drekmonger t1_j7lgpnf wrote on February 7, 2023 at 5:37 PM

Reply to comment by EddgeLord666 in 200k!!!!!! by Key_Asparagus_919

I imagine the notion of self will be eliminated. In the bad outcome, the robot overlords have no use for us. In the better outcome, your circumstances will be so grossly changed that whatever there is of "you" that's left over will be unrecognizable as such. I don't imagine a true continuity as plausible.

In the more neutral outcome, we become pets in a zoo, not ascended transhumanistic beings.

EddgeLord666 t1_j7lh7nj wrote on February 7, 2023 at 5:40 PM

Well unlike most people on this sub, I think transhumanism should be prioritized over the creation of AGI. I’m more interested in AI serving us as tools or augmenting our capabilities than ruling over us. Furthermore, you absolutely could have continuity of consciousness as long as augmentation happened in a ship of Theseus way, say by gradually boosting your IQ by 20 points every year instead of all at once.

drekmonger t1_j7lia04 wrote on February 7, 2023 at 5:47 PM

The Singularity, as it was originally imagined, included potential scenarios for transhumanism over a technological singularity. The original essay is still well worth the read, even 30 years later.

But the doomsday scenario the essay was ultimately warning against was that the Singularity would occur rapidly as a shocking cascade of events.

Perhaps in the "pet human" scenario, a benevolent ASI might slowly augment people as individuals.

Regardless, the problem is one of alignment, and I don't think you or I have much say in that. Even if a relatively benevolent organization like OpenAI develops the first AGI, their competitors (like, say, China's AI research efforts) won't be so benevolent.

As in capitalism, the most unethical strategy will tend to dominate ethical strategies. The "bad" AIs will win any race.

EddgeLord666 t1_j7livt6 wrote on February 7, 2023 at 5:51 PM

So far we are not at the stage where the Singularity is likely to be imminent, contrary to what some people here say. That means we probably have anywhere from 1 to 3 decades for the “good” people to coordinate and plan ways for it to happen in a more beneficial way or stop it from happening at all if that is deemed more desirable. That is really what people should be using this sub for, not just idle speculation.