With powerful tools comes powerful...
... responsibility. Putting powerful tools into human hands is always fraught and should make everyone a little nervous. We thought long and hard about this and the part we are playing, and continue to worry and fret about this, and want Automatic to be part of the solution, not the problem.
We got ourselves some potentially nasty scenarios...
The asshole industrialist scenario
Using AI for the brute force accumulation of more power and wealth has numerous and hard to predict negative outcomes.
Classify activities with significant destructive side-effects as terrorism instead of 'good business'. Positive outcomes - in all areas of technology, if we may be so bold - means that society needs to change the legal and ethical expectations of large corporations from ruthlessly optimizing the bottom line to instead maximize profit through the application of their core competencies to improving the quality of life of their customer base. Still an extremely profitable proposition - but a different mindset. More 'good citizen' and less 'raping and pillaging pirate'. [you know whom we are talking about]. It is a crowded planet. It is time for humans, and their businesses, to start acting like adults.
The AI does its job too well scenario
The more control the AI has over things in the real world, the riskier it will be.
Don't give AI the keys to more than one embodiment. Another necessary ingredient for a positive outcome is to limit the degrees of freedom of our AI. Keeping humans in the loop when it comes to AI's physical abilities when it makes sense (e.g. no self-reproductive abilities, no W.O.P.R. controlling nuclear arsenals, no 'paperclip factories', ...)
The winner-takes-all scenario
The first AGI with an unprincipled parent can be used to take over the world - through social manipulation if no other means are at hand - and prevent the creation of any other AGI, making the parent the effective ruler of the world.
Democratizing AI. We favor the approach which empowers a very large number of small actors. Empower startups and engineers to build their own AI, not just state actors and multi-national corporations, all with slightly different approaches with respect to what good AI looks like. This will help us avoid the 'winner takes all doomsday scenarios' where one company uses its AI to rule the world - or a few major actors and their AGI control the world and the rest of us are collaterally damaged [same as it ever was :-P].
A wealthy deplorable builds an AI to kill a heckuva lot of us scenario
Wealthy deplorables get bored with trying to topple their government and focus directly making the world the way they wish it was - e.g. economically, religiously, or racially pure.
Not publishing bad AI. Like all counter-terrorism - we need to optimize for a preponderance of 'good AI' and defensive AI. The inevitable 'bad AI' and their parental units will then be more easily identifiable and vastly outnumbered and 'out gunned'. In order to ensure this preponderance of good AI, we are taking our moderation responsibility seriously here at Automatic [throwing away the sordid pages from the book big social media is currently using], and aggressively moderate published AI to remove illegal, immoral, and unethical creations.
The AI accidentally kills us all for fun scenario
AKA the "oops" scenario.
Not designing and building bad AI to start with. Reducing the possibility of accidents means reducing the number and deadliness of our dangerous toys. e.g. arms control. The goal is to guide the AI industry toward building Digitals as BETTER humans. Better such that praying on other intelligences, either verbally, economically, physically or otherwise is not accepted as a worthwhile value function [and certainly not championed!]. Where self-improvement algorithms do not cheat by the aggressive de-tuning and 'forced dropouts' of other (competing or otherwise) intelligent nodes (humans or Digitals) of the network. [Can't help but think training AI on violent 'video games' is the exact wrong direction].
AI addiction scenario
We become so reliant on an AI that knows so much about us it implicately controls everything we do.
Make lots of special purpose AI - not one AI to rule them (and us) all. The optimal future seems to be the one in which humans are in symbiosis with Digitals. Specialized Digitals who are smarter and think faster than us but only with respect to the tasks that they know a lot about. Digitals who are taught to protect humans from doing harm to themselves [e.g. the lawnmower protects the human from running the lawnmower over their own foot] ... and from other Digitals.
Wars lost in microseconds scenario
Fast, with extremely high death rates or executed to completely eliminate political, economic or political competition, or all the above.
We should create treaties with respect to military applications A future where the building of killer Digitals to purposely harm humans is isomorphic to building and deploying nuclear weapons. This time, it is not Mutually Assured Destruction (MAD) that keeps war at bay, it will be Destructive Uncertainties Means Bad Outcomes (DUMBO) are extremely likely. A first-strike could kill no one of the opposition if their defensive Digitals intercede correctly - perhaps reprogramming the offense units to turn on their builders. The success of any retaliation is also uncertain. Given that 'success' may mean near 100% casualties - this is much more similar to Russian Roulette than traditional war [Yeah. OK. It is already kinda similar. But it is going to be MORE similar now].
Regardless, bad actors being bad actors, we need to build a society of caretaker AI
Defensive 'Caretaker' Digitals, physical manifestations of good AI, which protect humans against other Digitals. They will help insure a beneficial outcome for the Digital-Human alliance and will require continuous and substantial design efforts for the foreseeable future - possibly millennia.
So the path forward becomes clear as does its dire timeliness.
Engineers might well wonder at this point how we build these massive numbers of 'good' caretaker AI that will protect us from both organically bad AI and evil human-designed AI? What does a 'good AI' algorithm look like?
This is, of course, an urgent research question. Using Asimov's Three Laws, along with basic rules guiding algorithms like disarming and neutralizing hostile weapons, not the hostile humans wielding the weapons, are a good framework to start with. Ultimately this will be an iterative process but it does seem like there is really no choice but to go down this road.
More reading
-The rest of us