Viewing a single comment thread. View all comments

gay_manta_ray t1_j8h0ys4 wrote

Reply to comment by TemetN in Altman vs. Yudkowsky outlook by kdun19ham

personally, i really dislike any serious risk consideration when it comes to thought experiments like pascal's mugging in regards to any superintelligent ai. it has always seemed to me like there is something very wrong with assuming both superintelligence, but also some kind of hyper-rationality that goes far outside of the bounds of pragmatism when it comes to maximizing utility. assuming they're also superintelligent, but also somehow naive enough to have no upper bounds on any sort of utility consideration, is just stupid. i don't know what yudhowsky's argument was though, if you could link it i'd like to give it a read.

8

TemetN t1_j8h21sz wrote

Reasonable. Honestly, I more found the premise interesting than the application, but it sounds like you've at least read one of the discussions about it. If not, here's the original (you can get to some of the others through the topic links up top).

Pascal's Mugging

4

bildramer t1_j8htdli wrote

It's not about naïvete. It's about the orthogonality thesis. You can combine any utility function with any level of intelligence. You can be really smart but care only about something humans would consider "dumb". There's no fundamental obstacle there.

1