jdmcnair t1_j0ihvso wrote
Reply to comment by JVM_ in Is anyone else concerned that AI will eventually figure out how to build itself in three-dimensional space? by HeavierMetal89
>When the singularity does happen, powerful, but stupid AI will already be commonplace.
My personal "worst case scenario" imaginings of how things could go drastically wrong with AI is that there could be an AI takeover before it has actually been imbued with any real sentience or self-awareness.
It would be tragic if an ASI eventually decided to wipe out humanity for some reason, but it would be many multiples the tragedy if AI with a great capacity for merely simulating intelligence or self-awareness followed some misguided optimization function to drive humanity out of existence. In the former scenario at least we could have the comfort of knowing that we we're being replaced by something arguably better, but still in the spirit of our humanity. In the latter we're just snuffed out, and who knows if conditions would ever be right for conscious self-awareness to rise again.
JVM_ t1_j0im5kp wrote
...followed some misguided optimization function to drive humanity out of existence.
There's a thought experiment from years ago that went through a scenario, where a dumb-AI was told to corner the market on flowers or wheat or something innocuous, and the logical progression of what it felt it needed to control and takeover lead to the end of humanity. Google is clogged with AI now so I can't find it.
I agree with your sentiment, were worried about intelligent nuclear bombs, when a misplaced dumb one will do the same job. At least a smart bomb you can face and fight, an accidental detonation one you can't.
"The real troubles in your life
Are apt to be things that never crossed your worried mind
The kind that blindsides you at 4 p.m. on some idle Tuesday"
Remember to wear your sunscreen.
Viewing a single comment thread. View all comments