singulthrowaway

singulthrowaway t1_jea8vln wrote

Signed.

It's definitely a step in the right direction, but if you ask me you'd also have to shut down existing labs (including in China, so you'd have to make international agreements) and tightly control, again internationally, who is allowed to buy state of the art GPUs. Failing that, I'm not sure if open sourcing it is the correct move. I'd be fine with it being closed-source for now to avoid national efforts with more nefarious goals benefiting from its results so long as the people involved in the international project are legally bound to use it for the good of humanity as a whole, with mechanisms in place to ensure this.

0

singulthrowaway t1_jcja76s wrote

  1. ASI is given the goal to do exactly what humans collectively want right now, not something like CEV which is what they would want if they were smarter and had thought things through properly. As most people on Earth are members of religions which hold that the absolute most important thing in life is to believe in that religion and praise the respective god(s), ASI converts the whole universe into praisonium that sings hymns to Jahwe/Allah/Vishnu until the stars fizzle out. The end.
  2. Variation on this: ASI creates heaven and hell in accordance with the eschatology desired by the Christian and Muslim majority of humanity (55%). All nonbelievers go to hell immediately of course, but as Muslims go to hell in Christianity and Christians go to hell in Islam, ASI makes copies of them to have 1 instance to send to heaven and 1 to hell for each (coin flip decides whether the original goes to heaven and copy to hell or vice versa, good luck).
  3. The one I think is actually most likely to happen and has been discussed to death, so not really original, but it's going to happen, just watch: ASI aligned to AI company founder who happens to be fairly high on the narcissism spectrum. Remakes universe in his image, him and his family and buddies have orgasmically great lives forever until the end of time, we're all either dead if we're lucky or kept around as subjects for entertainment and for contrast to make already orgasmically great lives seem even greater by comparison to our miserable existence. (Speaking of hell, this has actually been used as a justification for it by certain Christian theologians: it's there so that the people in heaven can gawk at it and delight in the suffering of the wicked and nonbelievers. So it's not that farfetched that someone would actually want this).
5