Viewing a single comment thread. View all comments

ihateshadylandlords t1_iv36fsn wrote

Reply to comment by ReadSeparate in How do you think an ASI might manifest? by SirDidymus

What if I told you executives might use it to make money AND solve the world’s problems?

0

ReadSeparate t1_iv38nua wrote

I still think that's absurd. We're not talking about human level AGI here, we're talking about ASI. The moment ASI comes online is the moment money loses all of its value. If they do anything except use it to transition humanity into the next thing we're going to evolve into, I'll think they're short-sighted.

2

World_May_Wobble t1_iv4guq0 wrote

Why wouldn't money have value in a post-ASI world? I assume even super-intelligent, digital minds will need to find maximally efficient ways to distribute resources over very large, technically complex networks. Money's one way of doing that.

2

ReadSeparate t1_iv6blo0 wrote

Why would it need symbols to do that though? It would just do it directly. The reason why humans use money is because we don’t know the direct comparison from iPhones to chickens.

Additionally, there would not be market forces in such a system, so nothing would have a price, just an inherent value based on scarcity/utility. That wouldn’t change, they’d just be fundamental constants, more of less.

1

World_May_Wobble t1_iv6k0dr wrote

>Why would it need symbols to do that though?

I think bartering has problems besides converting between iPhones and chickens. Even if you know how many chickens an iPhone is worth, what if one ASI doesn't *want* iPhones? Then you can't "just do it directly," you have to find an intermediary agent who wants your iPhone who has something chicken-ASI wants.

Then symbols have other benefits. For example, you can't pay in fractions of an iPhone, but symbols are infinitely divisible, and symbols store value longer than chickens, which die and rot.

>there would not be market forces in such a system

Why not? Agents are (I presume) exchanging things based on their supply and demand. That's a market.

1

ReadSeparate t1_iv6p95l wrote

Are we talking about a world in which there are multiple ASIs existing at the same time? In that case you could be right, I have no idea how to model such a world though. I have no idea what their systems would look like. Would they compete? Would they cooperate? Would they merge? Would game theory still apply to them in the same way? I have no answers for any of those.

I was under the assumption that we were talking about a singular ASI with complete control over everything. I don’t know why the ASI, or whoever is controlling it, would allow any other ASIs to come into existence.

1

World_May_Wobble t1_iv6zi3l wrote

We have to make a lot of assumptions, and there's very little to anchor those assumptions to. So all we can say is given set of assumptions x, you tend toward world y.

One of my assumptions is that, depending on its capabilities, constraints, and speed of takeoff, an ASI may not be in a position to establish a singleton. Even an uploaded human mind is technically superintelligent, and it's easy to imagine a vast ecosystem of those forming.

Even if you imagine a singleton arising, you have to make some assumptions about its activities and constraints. If it's going to be doing things in places that are physically separated, latency may be an issue for it, especially if it's running at very high speeds. It may want to delegate activities to physically distributed agents. Those may be subroutines, or whole copies of the ASI. In either case, you again have a need for agents to exchange resources.

1

ihateshadylandlords t1_iv3co1t wrote

>The moment ASI comes online is the moment money loses all of its value

That’s assuming whoever creates it will let it run on its own. There’s a whole subreddit dedicated towards why that’s a problem (/r/controlproblem). I really doubt the founders and employees will let their ASI run wild. For anyone to not recoup their investment and let their product run wild is silly imo.

1

ReadSeparate t1_iv3hlhu wrote

I don’t think that there’s a difference in regard to the control problem by asking the ASI to do any task. Whether they ask it to make money or they ask it to upload all of our minds to the hive mind and build a dyson sphere around the Sun, I don’t see it making any difference if it’s misaligned. If it’s misaligned, it’s misaligned. You could ask it simply to say hello and it could still cause issues.

Why would they want to recoup their investment? Money doesn’t mean anything in this scenario. ASI is the absolute pinnacle of the universe and money is just a social construct invented by some upright walking apes. It’s like chimps worrying about bananas when they’ve stumbled upon modern food supply chains.

1

ihateshadylandlords t1_iv3rnlh wrote

Who knows if they’ll even let their ASI do the tasks. They might ask how to do it on their own to ensure the ASI stays as an Oracle like entity and not some runaway genie.

>Why would they want to recoup their investment?

Unless the ASI is a genie that can turn everything around in a split second, they’re most likely going to want to take care of themselves first and everyone else right after that.

2

ReadSeparate t1_iv3xw7n wrote

Even if an ASI is an oracle alignment is still just as much of an issue. It can tell them to do something that sounds completely harmless to even the smartest of humans and even non-ASI AGIs, but in reality lets it out of the box.

> Unless the ASI is a genie that can turn everything around in a split second, they’re most likely going to want to take care of themselves first and everyone else right after that.

What do you mean? That's exactly what ASI is. We're talking about something orders of magnitudes more intelligent than Albert Einstein here. A machine like that will be capable of recursively improving its own intelligence at an insane rate and will eventually know how to achieve any goal compatible with the laws of physics in the most efficient way possible for any possible set of constraints. That is basically by definition a magical genie that can do anything in a split second.

Every point you're making makes sense IF you're talking about just human-level AGI, but it makes no sense for ASI.

1

ihateshadylandlords t1_iv3z7of wrote

> Even if an ASI is an oracle alignment is still just as much of an issue. It can tell them to do something that sounds completely harmless to even the smartest of humans and even non-ASI AGIs, but in reality lets it out of the box.

You’re assuming the ASI will be sentient. Teams are doing everything to ensure it’s not sentient.

> What do you mean? That's exactly what ASI is. We're talking about something orders of magnitudes more intelligent than Albert Einstein here. A machine like that will be capable of recursively improving its own intelligence at an insane rate and will eventually know how to achieve any goal compatible with the laws of physics in the most efficient way possible for any possible set of constraints. That is basically by definition a magical genie that can do anything in a split second.

Okay. Then the owners will probably use this non-sentient tech to take care of themselves and the rest of us next.

1

ReadSeparate t1_iv3zj0j wrote

I’m not assuming it’ll be sentient, I’m just saying an Oracle ASI equally as dangerous as one with agency. It MIGHT be sentient. Or it might NOT be sentient, but still dangerous, I.e. the paper clip maximizer scenario.

> Okay then the owners will probably use this Non-sentient tech to take care of themselves

Like just AGI you mean? Yeah I agree with that of course. But ASI, again, seems short sighted. If Google makes human level AGI, but it’s just as smart as say Einstein, yeah of course they’ll use it to get richer. But if they create something that makes Einstein look like an ant, they’d be foolish to use it in such a way.

1

ihateshadylandlords t1_iv40poj wrote

> I’m not assuming it’ll be sentient, I’m just saying an Oracle ASI equally as dangerous as one with agency. It MIGHT be sentient. Or it might NOT be sentient, but still dangerous, I.e. the paper clip maximizer scenario.

Meh, the dangers of an ASI can be discussed in another thread. We were initially talking about how an ASI might manifest, so it’s getting off course.

>Like just AGI you mean? Yeah I agree with that of course. But ASI, again, seems short sighted. If Google makes human level AGI, but it’s just as smart as say Einstein, yeah of course they’ll use it to get richer. But if they create something that makes Einstein look like an ant, they’d be foolish to use it in such a way.

Okay. Just don’t be surprised if companies keep doing what they’ve been doing for literally thousands of years and use their products to make a profit.

1

TheSingulatarian t1_iv3vx3j wrote

There will be a lag time of several to 10 years before the implications of ASI start to show themselves.

0