Viewing a single comment thread. View all comments

ihateshadylandlords t1_iv3co1t wrote

Reply to comment by ReadSeparate in How do you think an ASI might manifest? by SirDidymus

>The moment ASI comes online is the moment money loses all of its value

That’s assuming whoever creates it will let it run on its own. There’s a whole subreddit dedicated towards why that’s a problem (/r/controlproblem). I really doubt the founders and employees will let their ASI run wild. For anyone to not recoup their investment and let their product run wild is silly imo.

1

ReadSeparate t1_iv3hlhu wrote

I don’t think that there’s a difference in regard to the control problem by asking the ASI to do any task. Whether they ask it to make money or they ask it to upload all of our minds to the hive mind and build a dyson sphere around the Sun, I don’t see it making any difference if it’s misaligned. If it’s misaligned, it’s misaligned. You could ask it simply to say hello and it could still cause issues.

Why would they want to recoup their investment? Money doesn’t mean anything in this scenario. ASI is the absolute pinnacle of the universe and money is just a social construct invented by some upright walking apes. It’s like chimps worrying about bananas when they’ve stumbled upon modern food supply chains.

1

ihateshadylandlords t1_iv3rnlh wrote

Who knows if they’ll even let their ASI do the tasks. They might ask how to do it on their own to ensure the ASI stays as an Oracle like entity and not some runaway genie.

>Why would they want to recoup their investment?

Unless the ASI is a genie that can turn everything around in a split second, they’re most likely going to want to take care of themselves first and everyone else right after that.

2

ReadSeparate t1_iv3xw7n wrote

Even if an ASI is an oracle alignment is still just as much of an issue. It can tell them to do something that sounds completely harmless to even the smartest of humans and even non-ASI AGIs, but in reality lets it out of the box.

> Unless the ASI is a genie that can turn everything around in a split second, they’re most likely going to want to take care of themselves first and everyone else right after that.

What do you mean? That's exactly what ASI is. We're talking about something orders of magnitudes more intelligent than Albert Einstein here. A machine like that will be capable of recursively improving its own intelligence at an insane rate and will eventually know how to achieve any goal compatible with the laws of physics in the most efficient way possible for any possible set of constraints. That is basically by definition a magical genie that can do anything in a split second.

Every point you're making makes sense IF you're talking about just human-level AGI, but it makes no sense for ASI.

1

ihateshadylandlords t1_iv3z7of wrote

> Even if an ASI is an oracle alignment is still just as much of an issue. It can tell them to do something that sounds completely harmless to even the smartest of humans and even non-ASI AGIs, but in reality lets it out of the box.

You’re assuming the ASI will be sentient. Teams are doing everything to ensure it’s not sentient.

> What do you mean? That's exactly what ASI is. We're talking about something orders of magnitudes more intelligent than Albert Einstein here. A machine like that will be capable of recursively improving its own intelligence at an insane rate and will eventually know how to achieve any goal compatible with the laws of physics in the most efficient way possible for any possible set of constraints. That is basically by definition a magical genie that can do anything in a split second.

Okay. Then the owners will probably use this non-sentient tech to take care of themselves and the rest of us next.

1

ReadSeparate t1_iv3zj0j wrote

I’m not assuming it’ll be sentient, I’m just saying an Oracle ASI equally as dangerous as one with agency. It MIGHT be sentient. Or it might NOT be sentient, but still dangerous, I.e. the paper clip maximizer scenario.

> Okay then the owners will probably use this Non-sentient tech to take care of themselves

Like just AGI you mean? Yeah I agree with that of course. But ASI, again, seems short sighted. If Google makes human level AGI, but it’s just as smart as say Einstein, yeah of course they’ll use it to get richer. But if they create something that makes Einstein look like an ant, they’d be foolish to use it in such a way.

1

ihateshadylandlords t1_iv40poj wrote

> I’m not assuming it’ll be sentient, I’m just saying an Oracle ASI equally as dangerous as one with agency. It MIGHT be sentient. Or it might NOT be sentient, but still dangerous, I.e. the paper clip maximizer scenario.

Meh, the dangers of an ASI can be discussed in another thread. We were initially talking about how an ASI might manifest, so it’s getting off course.

>Like just AGI you mean? Yeah I agree with that of course. But ASI, again, seems short sighted. If Google makes human level AGI, but it’s just as smart as say Einstein, yeah of course they’ll use it to get richer. But if they create something that makes Einstein look like an ant, they’d be foolish to use it in such a way.

Okay. Just don’t be surprised if companies keep doing what they’ve been doing for literally thousands of years and use their products to make a profit.

1