We are so far away from anything you can really call "AI" it is not on my mind at all. What we have today is simply algorithmic pattern recognition and it is actually really disappointing. The scale of ChatGPT is impressive, but the performance is not. Many many thousands of man-hours were needed to manually tag training datasets. The only place "AI" exists is in the marketing department.

ML researchers worry a lot less than AGI safety people. I think that's because only the AGI safety people spend a lot of time thinking about getting all the way to agentic superhuman intelligence.

If we're building tools, not much need to worry.

If we're building beings with goals, smarter than ourselves, time to worry.

Now: do you think we'll all stop with tools? Or go on to build cool agents that think and act for themselves?

Many thousands of human hours are cheap to buy, and cycles get cheaper every year. So those things aren't really constraints except currently for small businesses.

True, but it doesn't matter - it is still just algorithmic. There is no "intelligence" of any kind yet. We are not even remotely close to anything like actual brain functions.

The definition of intelligence is quite simple, we definitely have AI.

About a year or two ago, we were so far away from having an AI model that could reliably and easily produce high quality artwork that almost no one was thinking about AI art generators.

Then diffusion models became a thing.

AGI could easily be very similar; it could take decades to discover what is required to make an AGI, or just a few more years. But AGI is not quite like diffusion models, because a diffusion model can’t create and carry out a plan to convert every single living thing into computronium or whatever helps maximize its utility function.

No, absolutely not. First AGI is just a theory, it's not possible on modern logic based hardware, quantum is a possibility. Even if we do achieve it, it's fragile, just unplug it. 2nd, we've had nuclear weapons for close to 80 years, and we're still here, that's a much more real and immediate threat to our demise.

​

As a thought experiment, it's not bad...

Check out Paul Christiano. His focus is on ai-alignment and, in contrast to Eliezer, he holds an optimistic view. Eliezer actually mentions him in the bankless podcast you are referring to.

This interview of him is one of the most interesting talks about AI I’ve ever listened to.

And here is his blog.

I think the biggest danger isn't AIs/AGIs pursuing their own goals/utility functions that involve turning all humans into paperclips. I think the "predict-the-next-word" AIs that are currently the closest thing to AGI aren't capable of recursively self improving arbitrarity, nor is there evidence AFAIK that they pursue their own goals.

Instead the danger is in people using increasingly capable AIs to pursue their own goals, which may or may not be benign. Like, the same AIs that can cure cancer can also create highly dangerous bioweapons or nanotechnology.

That’s a misreading of what the AI alignment people say, they’re quite explicit that agency is not necessary for AI risk.

EY tends to go straight to superintelligent AI robots making you their slave. I worry about problems that’ll happen a lot sooner than that. What happens when we have semi-autonomous infantry drones? How much more aggressive will US/Chinese foreign policy get when China can invade Taiwan with Big Dog robots with machine guns attached? What about when ChatGPT has combined with toolformer and can write to the internet instead of just read and can start doxxing you when it throws a temper tantrum? What about when rich people can use something like that to flood social media with bots that spew disinformation about a political candidate they don’t like?

But part of the lack of concern for AGI among ML researchers is that during the last AI winter we rebranded to machine learning because AI was such a dirty word. I remember as recently as 2015 at ICLR/ICML/NIPS you’d get side-eye for even bringing up AGI.

Lol, downvotes. this subreddit has been completely overran by non engineers. I guarantee no one here has ever custom trained and inferred with a model outside of API calls. Crazy. Since ChatGPT, open enrollment ML communities are so cringe

Thanks, I actually read this today. He and Richard Ngo are the names I've come across for researchers who've deeply thought about alignment and hold views grounded in the literature.

This is very far from the current thinking in AI research circles. Everyone I know believes intelligence is substrate independent and, therefore, could be implemented in silicon. The debate is really more about what constitutes AGI and if we're 10 years or 100 years away, not if it can be done at all.

It's hard to worry about the "Terminator" dystopia when the "Elysium" dystopia is so much more imminent.

No

Or maybe downvotes because they’re stating the obvious. I didn’t downvote for that or any other reason. Just stating it as another possibility. I haven’t seen anyone here claim language models are actual AI, let alone AGI.

>We are not even remotely close to anything like actual brain functions. Intelligence need not look anything remotely close to actual brain functions though, right? Like a plane's wings don't function anything like a bird's wings, yet it can still fly. In the same sense, why must intelligence not be algorithmic?

At any rate I feel like saying that probabilistic machine learning approaches like GPT3 are nowhere near intelligence is a bit of a stretch. If you continue scaling up these approaches, you get closer and closer to the entropy of natural language/whatever other domain, and if youve learned the exact distribution of language, imo that would be "understanding"

I work in AI research, and I see many of the points EY makes here in section A as valid reasons for concern. They are not 'valid' in the sense that they must be true, but valid in that they are plausible.

For example, he says, We can't just build a very weak system. There are two papers that led me to believe this could be the case. All Else Being Equal be Empowered, which shows that any agent acting to achieve a goal under uncertainty will need (all else being equal) to maximize its control over the system. And the Zero Shot Learners paper which shows that (very large) models trained on one task seem also to learn other tasks (or at least learn how to learn them). Both of these papers make me question the assumption that a model trained to learn one 'weak' task won't also learn more general capabilities.

Where I think I disagree is on the likely scale of the consequences. "We're all going to die" is an unlikely outcome. Most likely the upheaval caused by AGI will be similar to previous upheavals in scale, and I'm yet to see a strong argument that bad outcomes will be unrecoverable.

The thing is that the theory behind diffusion models is at least 40-50 years old. Forward diffusion is a discretization of a stochastic differential equations that transforms the data distribution into a normal distribution. People figured out that it is possible to reverse this process, so to go from the normal distribution back to the data distribution using another sde In the 1970s. The thing is that this reverse SDE contains the score function, so the gradient of the log density of the data and people just didn't really know how to get that from data. Then some smart guys came along, found the ideas about denoising score matching from the 2000s and did the necessary engineering to make it work with deep nets.

The point I am making is that this problem was theoretically well understood a long time ago, it just took humanity lots of years to actually be able to compute it. But for AGI, we don't have such a recipe. There's not one equation hidden in some old math book that will suddenly get us AGI. Reinforcement learning really is the only approach I could think of but even there I just don't see how we would get there with the algorithms we are currently using.

Quantum neural networks are an interesting idea, but our brain is certainly not sitting in a vat of liquid nitrogen, so intelligence must be possible without it.

The brain was created by an optimization process (evolution) - it's no coincidence that the entire field of machine learning is about the study of optimization processes too. It must be possible for intelligence to arise through optimization; and it does seem to be working better than anything else so far.

https://plato.stanford.edu/entries/qt-consciousness/

https://www.nature.com/articles/s41566-021-00845-4

https://www.nature.com/articles/440611a

https://phys.org/news/2022-10-brains-quantum.html

https://www.newscientist.com/article/mg22830500-300-is-quantum-physics-behind-your-brains-ability-to-think/

​

Considering in 355 BC Aristotle thought the brain was a radiator, it's not a far leap to think were wrong that it uses electrical impulses like a computer. And I'm sure after quantum mechanics there will be something else. Although we have far more understanding than 2000 years ago, we are very far from the understanding we will have in 2000 years.

[deleted]

I think the whole "paperclip" metaphor descibres problems that are already here. a lot of "alignment" discussion feels to me like passengers on a ship theorizing what would happen if the ship became sentient and turned evil and decided to crash into the rocks, but all the while the ship has already crashed into the rocks and is taking on water. It doesn't matter if the ship turns evil in the future: it's already taking us down, whether it crashed into the rocks on purpose or not. See also: contribution of social media recommendation systems to self-destructive human behaviors including political radicalization, stochastic terrorism, xenophobia, fascism, and secessionism. Oh yeah, also we're arguing over the safety of vaccines during an epidemic and still ignoring global warming, but for some reason public health and environmental hazards don't count as "x-risks".

Would you admit a theory that may not be possible and than devote your life to working on it? Even if you don't you're going to say it, and eventually believe it. And the definitions do keep moving with lower bars as the media and companies sensationalise for clicks and funding.

That's just how tools are used, has been since the dawn of time. You just want to be on the side with the largest club, warmest fire, etc.

[deleted]

No, a lot of his arguments strike me as similar to arguments from the 1800s about how some social trend or another spells doom in a generation or two. And then his followers spend their time confusing "Bing was mean to me" with "Bing is misaligned" (as opposed to "Bing is bad at its job") and start shouting "See? See? Alignment is impossible and it's already biting us!"

When they scale they hallucinate more, produce more wrong information, thus arguably getting further from intelligence.

[removed]

That's like saying we're wrong about out aerodynamics and how birds fly, because Aristotle was wrong about it and we'll understand flight very differently in 2000 years.

These articles don't represent the mainstream neuroscience position. It pretty clearly does use electrical impulses. You can stick in an electrode array and read them directly, or you can stick someone in an fMRI and see the electrical patterns. It also pretty clearly uses chemical signalling, which you can alter with drugs. We've seen no structures that appear to perform quantum computation.

I worry about a lot of bad AI/ML made by interns making decisions that have huge impact like in the justice system, real estate ect.

No, absolutely not. It's fear mongering about something we aren't even remotely close to achieving.

What about current architectures makes you think they won’t continue to improve with scale and multimodality, provided a good way of tokenizing? Is it the context length? What about models like S4/RWKV?

>Quantum neural networks are an interesting idea, but our brain is certainly not sitting in a vat of liquid nitrogen, so intelligence must be possible without it.

look at the links I shared above.

​

Recreating actual intelligence, what the definition of AGI was 6 months ago, will not be possible on logic based computers. I have never said it's not possible. There's a number of reasons it is not currently possible, the number 1 that we don't have a full understanding of intelligence, and recent theories suggest it's not logic based like previously theorised, but quantum based.

Look at the early history of attempting to fly, for centuries humans strapped wings to their arms and attempted to fly like birds.

Most of these links are highly philosophical and none of them address the question of how the brain would usefully retain qubit stability at body temperature.

The evidence they present is very weak or non-existent, and the newscientist article acknowledges this is not the mainstream neuroscience position.

Meanwhile there is heaps of evidence that electrical and chemical signaling is involved; fiddling with either of them directly affects your conscious experience.

>I think the whole "paperclip" metaphor descibres problems that are already here

Does it? My understanding of the paperclip metaphor is that an advanced AI will pursue its own goals that are totally unrelated to human goals, e.g. creating as many paperclips as possible. But AIs aren't advanced enough right now to be at this point.

As for what constitutes "x-risks", AFAIK it means "existential risk" which is like all of humanity going extinct. IMO the reason why people consider advanced AGIs an x-risk, and the others are not, is because the other problems you mentioned don't result in the extinction of *every* single human on Earth

>When they scale they hallucinate more, produce more wrong information

Any papers/literature on this? AFAIK they do better and better on fact/trivia benchmarks and whatnot as you scale them up. It's not like smaller (GPT-like) language models are factually more correct ...

I must say I am not very involved with the alignment community and do not have much exposure to their discussions, so I may miss some ideas, but as a researcher in robotics I am not super worried about some of his concerns just by reading his post.

Currently there is no clear roadmap in the robotics community to achieve an agent that can autonomously and robustly interact with the unstructured physical world, even just for a relatively specialized environment. Robotics is still very far away from its ChatGPT moment, and I think current socioeconomic conditions are rather adversarial to robotics RD compared to other domain. So such agent will have very limited physical agency.

If you assume current auto-regressive LLMs can somehow lead to a super-intelligent agent and just figure out the robotics/physical interaction problem itself, then sure you could worry about it. But if we assume an omnipotent oracle then we could worry about anything. It’s not so much different from worrying about a scenario in which the law of physics just changes the next instant and all biological creatures will just explode under the new law of physics. I mean it’s possible, just not falsifiable so I wouldn’t worry too much about it.

Btw, I want to stress that I think most of EY’s chain of thoughts that I have the chance to read about are logical. But his assumptions are usually so powerful. When you have such powerful assumptions a lot of things become possible.

Also, I wouldn’t dismiss alignment research in general like many ML researchers do, precisely because I work with physical robots. There are many moments during my experiments I would think to myself “this robot system can be a very efficient killing machine if people really try” or “this system can make many people lose their jobs if it can economically scale”. So yeah in general I think some “alignment” research has its merits. Maybe we should start by addressing some problems that already happened or are very imminent

if a "sufficiently advanced AI" could achieve "its own goals" that included "humanity going extinct" (at least as a side effect) in such a fashion that humanity did the work of putting itself out of extinction on its own needing only the AGIs encouragement, it would. In other words, the issues I described are indistinguishable from the kinds of bedlam we could reasonably expect an "x-risk AGI" to impose upon us. ipso facto, if part of the alignment discussion is avoiding defining precisely what "AGI" even means and focusing only on potential risk scenarios, the situation we are currently in is one in which it is unclear that a hazardous-to-human-existence AGI doesn't already exist and is already driving us towards our own extinction.

instead of "maximizing paperclips," "it" is just trying to maximize engagement and click-through rate. and just like the paperclips thing, "it" is burning the world down trying to maximize the only metrics it cares about. "it" just isn't a specific agent, it's a broader system that includes a variety of interacting algorithms and platforms forming a kind of ecosystem of meta-organisms. but the nature of the ecosystem doesn't matter for the paperclip maximization parable to apply.

No - his arguments and fears of mis-alignment are far overblown.

It's just matrix multiplication and derivatives. The only real advance in machine learning over the last 20yrs is scale. Nvida was very clever and made a math processor that can do matrix multiplication 100x faster than general purpose CPUs. As a result, the $1bil data center, required to make something like GPT-3, now only costs $100mil. It's still just a text bot.

EY is a total clown who inserts enough truth into his (incredibly lengthy) arguments that an air of correctness and solid reasoning permeate from it, but most of his claims simply reduce to p(everyone dies | literally anyone but EY charts the course) ~= 1. I am not exaggerating, he got angry publicly that others had not thought of everything he'd thought of before him when it was so obviously correct.

> instead of "maximizing paperclips," "it" is just trying to maximize engagement and click-through rate. and just like the paperclips thing, "it" is burning the world down trying to maximize the only metrics it cares about

Isn't there a difference between the two, because the latter concerns a human trying to pursue a certain goal (maximize user engagement), and giving the AI that goal. and so arguably, the latter is "aligned" (for some sense of the word) to the human that's using it to maximize their engagement, in that its doing what a specific human intends it to do. Whereas the paperclip scenario is more like, human tells AI to maximize engagement, yet the AI has a different goal and chooses to pursue that instead.

I'm sorry. There's no way I'm going to a listen to a dude with eyebrows like that.

I'll see if I can find the benchmarks, I believe there are a few papers from IBM and deepmind talking about it. And a benchmark study in relation to flan.

What I’m hoping is that EY’s long term vision for AI existential risk is thwarted by the inevitable near term issues that will come to light and inevitably be raised to major governments and powerful actors who will then enter a “collective action” type of response similar to what happened with nukes, etc. the difference is that any old 15 year old can’t just buy a bunch of AWS credits and start training a misaligned nuke.

What you mention about a ChatGPT like system getting plugged into the internet is exactly what Adept AI is working on. It makes me want to bang my head against the wall. We can say goodbye soon to a usable internet because power seeking people with startup founder envy are going to just keep ramping these things up.

In general though, I think my “timelines” are longer than EY / EA by a bit for a doomsday scenario. LLMs are just not going to be the paradigm that brings “AGI,” but they’ll still do a lot of damage in the meantime. Yann had a good paper about what other factors we might need to get to a dangerous, agentic AI.

Yep if we die from AI it will be from bioterrorism well before we get enslaved by a robot army. And the bioterrorism stuff could even happen before “AGI” rears its head.

> Isn't there a difference between the two, because the latter concerns a human trying to pursue a certain goal (maximize user engagement), and giving the AI that goal.

in the paperclip maximization parable, "maximize paperclips" is a directive assigned to an AGI owned by a paperclip manufacturer, which consequently concludes that things like "destabilize currency to make paperclip materials cheaper" and "convert resources necessary for human life to exist into paperclip factories" are good ideas. so no, maximizing engagement at the cost of the stability of human civilization is not "aligned" in exactly the same way maximizing paperclip production isn't aligned.

hmm i didn't realize that the origin of the paperclip maximizer analogy, but it seems like you're right that some human had to tell it to make paperclips in the first place.

I think the basic issue of AI alignment isn't AI. It's trying to figure out what our values are supposed to be and who gets to decide that.

>We can say goodbye soon to a usable internet because power seeking people with startup founder envy are going to just keep ramping these things up.

This might seem like paranoia on my part, but I am worried about AI being leveraged to "drive engagement" by stalking and harassing people.

"But why would anyone consider this a salable business model?" It's no secret that the entertainment on a lot of different websites are fueled by drama. If someone were so inclined, AI and user metrics could be harvested specifically to find creative and new ways to sow social distrust, and in turn drive "drama" related content.

I.E. creating recommendation engines specifically to show people things that they assume will make them angry, specifically so they engage with it in great detail, so that a larger corpus of words will be exchanged that can be datamined for advertising analytics.

[deleted]

https://en.wikipedia.org/wiki/Instrumental_convergence

What freaks me out the most are the social ramifications of AIs that pass as humans to the majority of people. We're still figuring out how to healthily interact with social media and soon we're going to be interacting with entirely artificial content that we're gonna anthropomorphize onto other humans. In the US we're dealing with a crisis of trust and authenticity, I can't imagine generative text models are going to help with that.

[deleted]

Both of them are more positive than EY, but both are still quite worried about AI risk. It's just that they don't see doom as inevitable. This is the sort of scenario Christiano worries about: https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like

And this is Ngo's overview of the topic: https://arxiv.org/abs/2209.00626

[removed]

>I remember as recently as 2015 at ICLR/ICML/NIPS you’d get side-eye for even bringing up AGI.

You still do, imo rightfully so

>Like, the same AIs that can cure cancer can also create highly dangerous bioweapons or nanotechnology.

A good example of this:

https://www.nature.com/articles/s42256-022-00465-9

One would have to consider the ultimate consequences (including paradoxical ones) of those things too.. Like would it really be catastrophic if social media became unusable for the average user? The 1990s are usually considered the last halycon era... Maybe thats a feature not a bug!

As far as drone swarms, those are definitely terrifying, but then there will be drone swarm countermeasures. Also, is it really much more terrifying than Russia throwing wave after wave of humans at machine gun nests?

I view a lot of the ethics concerns as a bunch of people projecting their fears into a complicated world, and then drastically overextrapolating. This happened with the industrial age, electricity, the nuclear age and so on and so forth.

Are you familiar with

https://innermonologue.github.io/

or

https://arxiv.org/abs/2204.01691

> It's amazing to me how easily the scale of the threat is dismissed by you after you acknowledge the concerns.

I second this.

Also, the effects of misaligned AI can entirely be mediated by so called meat-space: an AI can sow astonishing havoc by simply damaging our ability to know what is true.

In fact, I find this to be the biggest danger of all. We already have a scientific publishing "problem" in that we have arrived at an era of diminishing returns and extreme specialization, I simply cannot imagine the real world damage that would be inflicted when (not if) someone starts pumping out "very legitimate sounding but factually false papers on vaccines side-effects".

I just watched this today where he talks about using automated code generation for code verification and tests. The man is brilliant and the field is brilliant but one thing is certain and that is that the complexity of far exceed individual humans' ability to fully comprehend.

Now combine that with this and you have a true recipe for disaster.

I think in the opposite way: if alignment is possible, then alignment is profoundly dangerous.

If alignment is possible, then the AI can be aligned with the interest of the wealthy capital owners who fund its development, and can be used to basically control the world.

Meanwhile, if alignment is impossible, ordinary people who have access to these hypothetical future 'superintelligences' can convince these entities to do things that they like, but which are undesired by the model-owning class.

For this reason, if we are on some kind of path to super AI, the development of technology to permit value alignment must be prevented.

Yes, I am aware of the paper you linked, although I can’t say I am super familiar with the details.

This is very cool and solves some of the problems with robotics, but not a whole lot. Not discrediting the authors (especially Fei Xia, who I really admire as a robotics researcher. And of course Sergey Levine, who is probably my favorite), but the idea of fusing NLP and robotics to create a robot that can understand command and serve you is not super new. Even 10+ years ago there is this famous video from ROS developer Open Robotics (at the time it was still Willow Garage IIRC) in which they tell the robot to grab a bear and the robot will navigate the entire office and fetch it from the kitchen. Note that this is not the innovation these papers claim, (these papers are actually investigating a possibility instead of solving a problem) but I assume this is probably what everyone assumes to be the bottleneck of service robot, which in reality isn’t.

None of those concerns have to do with the intrinsic nature of machine learning, though. Right now it's another tool that can automate tasks previously thought impossible to automate, and sometimes it does that task much better than humans could. It's another wave like the Industrial Revolution and the assembly line.

Some people will inevitably use this technology to destroy things on a greater scale than ever before, like using the assembly line to mass produce missiles and tanks. But trying to put a leash on the technology won't accomplish anything because technology isn't inherently good or evil.

Now, if the state of ML were such that sentient AI was actually on the horizon, not only would this way of thinking be wrong, we'd need to rethink the concepts of humanity and morality altogether. But it's not. Not until these models manage to improve at tasks it was not trained to do. Not until these models become capable of accurate self-evaluation of its own performance without human input.

This is already happening but the problem is humans not ai. Even without ai we are descending into an era of misinformation.

Yud is a millenarian street preacher; his concept of evil superintelligent AGI is half religion and half old SF books they read. It has no resemblance to current research and we aren't going in directions similar to what they imagine we're doing.

(There's not even much reason to believe "superintelligence" is possible, that it would be helpful on any given task, or even that humans are generally intelligent.)

I don't worry much at all about the AI safety/alignment concerns described by Eliezer Yudkowsky. I don't find his arguments to be particularly rigorous, and his arguments in this space are typically based on premises that are either nonsensical or wrong and that don't engage meaningfully with the current practice in the field. This is not to say that I do not worry about AI safety: Stuart Russell has done good work in this space towards mapping out the AI alignment problem. And if you're looking for arguments that are more rigorous leading to more sound conclusions on AI alignment and which people in the field do seem to respect, I'd recommend you look into Stuart Russell's work. The bulk of opinions I've seen from people in the field on the positions of Yudkowsky and his edifice range from finding the work to be of dubious quality (but tolerable) to judging it as actively harmful.

[removed]

Worry less about misalignment of Skynet, the impending singularity and the rise of the robots, which is science fiction, and worry more about misalignment of class interests and misalignment of power, which is our reality.

For the former, it's still mostly an ambitious long-term goal to simulate the world's simplest nematodes. There's hardly any reason to believe anyone's appreciably closer to AGI now than they were in the 1950s. For the latter, though, there are well-founded concerns that automation will be used for surveillance, disinformation, manipulation, class control, digital Taylorism and other horrifying purposes, as the species knowingly accelerates toward extinction by ignoring systemic failures like AGW and nuclear war, which pose actual, imminent and growing existential risks -- risks that will be compounded by giving state and capital tools to put the interests of power and short term ROI above even near-term human survival, let alone human dignity or potential.

"What if this pile of linear algebra does some asimov nonsense" is not a serious concern. The real concern is "what if it does exactly what was intended, and those intentions continue to see omnicide an acceptable side effect."

> I think in the opposite way: if alignment is possible, then alignment is profoundly dangerous.

Exactly. What is this neoliberal fever dream? "But what if the computer doesn't do what they want?!" -- my god, what if it does? Are we living on the same planet? Have you seen what they want?

I love how the core of the panic is basically:

"Oh my god, what if some kind of machine emerged, misaligned with human interests and totally committed to extracting what it wants from the material world, no matter the cost, seeing human life and dignity as an obstruction to its function?!"

Yeah, wow... what if?! That'd be so crazy! Glad we don't have anything like that.

I agree about the callousness, and that's without artificial intelligence too. The global power balances were shifting at times of rapid technological development, and that development created control vacuums and conflicts that were resolved by war. If we learn from history we can plan for it and prevent it, but the same types of fundamental underlying shifts are being made now. We can say that international global financial incentives act to prevent worldwide conflict, but that only goes so far. All of the things I'm saying are on the trajectory without neural networks as well, they are just one of the many rapid shifts in political economy and productive efficiency.

In the same way that people were geared up at the start of the Russian invasion to Ukraine to try to prevent nuclear war, we should all be vigilant to try to globally dimilitarize and democratise to prevent any war. The global nuclear threat isn't even over and it's regressing.

[removed]

> I worry about a lot of bad AI/ML made by interns making decisions that have huge impact like in the justice system, real estate ect.

I worry more about those same AIs made by the senior-architects, principal-engineers, and technology-executives rather than the interns. It's those older and richer people whose values are more likely to be archaic and racist.

I think the most dangerous ML models in the near term will be made by highly skilled and competent people whose goals aren't aligned with the bulk of society.

Ones that unfairly send certain people to jail, ones that re-enforce unfair lending practices, ones that will target the wrong people even more aggressively than humans target the wrong people today.

> , I simply cannot imagine the real world damage that would be inflicted when (not if) someone starts pumping out "very legitimate sounding but factually false papers on vaccines side-effects".

I mean, just look at the current anti-vaccine movement. You just described the original Andrew Wakefield paper about vaccines causing autism. We don't need AI for this to happen, just a very credulous and gullible press.

Maybe when an actual expert tells you you're overreacting, you should listen.

Are you seriously arguing that the modern world is somehow corrupted by some magical unified "postmodern philosophy"? We live in the most peaceful time in recorded history. Read "Factfulness" for exact figures. And while you're at it, actually read "Black Swan" instead of throwing that term around because you clearly need to a lesson on measuring probability.

If you think AI will be destructive, outline some plausible and SPECIFIC scenarios how this could possibly happen, instead of your vague allusions to philosophy with no proof of causality. We could then debate the likelihood of each scenario.

IMO, even if ML researchers assigned only an 0.1% chance to AI wiping out humanity, the cost of that happening is so unfathomably large that it would only be rational to shift a lot of resources from AI capability research to AI safety in order to drive that probability down.

If you tell people that society needs to do a lot less of the thing that is their job, it's no surprise they dismiss your arguments. The same applies to EY to some extent; I think it would be more reasonable to allow for a lot more uncertainty on his predictions, but would he then have the same influence?

Rather than giving too much credit to expert opinions, it's better to look at the evidence from all sides directly. You seem already be doing that, though :-)

He gets paid for telling scary stories about AI. "Somehow, AI has learned to send e-mails".

I don't know anything about him when it comes to alignment. Seems like a lot of unrigorous wasted effort at first glance, but I haven't really had the time or desire to look into it.

The overbearing smugness of Inadequate Equilibria was nauseating. It was unreadable, even for poop reading. The guy is really impressed with himself for believing he came up with theories that have existed for a long time, but that he was too lazy and too disrespectful to research. I will admit there were a couple good snippets in the book (but given the general lack of originality, can we really be sure those snippets were original?).

>When things suck, they usually suck in a way that's a Nash Equilibrium.

There you go, I just saved you a couple hours.

What has EY actually done or built? He seems like one of those guys that wants to be seen as technical or intellectual but hasn't actually built anything or done anything other than nebulously / unrigorously / long-windedly discuss ideas to make himself sound impressive. Kinda like the Yuval Noah Harari of AI.

>Meanwhile, if alignment is impossible, ordinary people who have access to these hypothetical future 'superintelligences' can convince these entities to do things that they like

Interesting, how are you gonna "convince" an unaligned AI though, I wonder. I feel like there is a flaw in your reasoning here

Surprised I had to scroll down this much to see this opinion, which I agree completely with. The danger I worry about most isn't superintelligent AI, it's people like Yudkowsky creating their little cults around the potential for superintelligent AI.

Alignment isn't my main concern. I fear AIs that are "aligned" with people who want to e.g. fight wars, or worse.

Much like that there are issues like what recommender systems are recommending, how credit models are scoring, why your resumes are being discarded without being seen by a human being and lots of other minor mundane daily things that we take for granted and are actually dystopic for the people at the short end of the stick. These are systems that need to be fine tuned and we treat their judgements as holy lines in the Commandment stones. The AI dystopia is already real.

"Your discarded toenail could turn into Keratinator, Devourer of Worlds, and end all life in the galaxy. We need agencies and funding to regulate toenails."

"That's stupid, and very unlikely."

"You are dismissing the scale of the threat!"

I am more worried governments will use AI to fully automatically censor every post and chat. Right now they would have to spend too much to throw humans at it, but AI could make it almost free.

Also, Paul actually has trained and works closely with ML models, unlike Eliezer, who does not understand how deep learning works .

[removed]

I agree that we don’t necessarily need AI for nefarious actors to spread scientific misinformation, but I do think AI introduces another tool or weapon that could used by the Andrew Wakefields of the future in a way that might pose unique dangers to public health and public trust in scientific institutions.

I’m not sure whether it was malevolence or incompetence that has mostly contributed to vaccine misinformation, but if one intentionally sought to produce fake but convincing scientific-seeming work, wouldn’t something like a generative language model allow them to do so at a massively higher scale with little knowledge of a specific field?

I’ve been wondering what would happen if someone flooded a set of journals with hundreds of AI-written manuscripts without any real underlying data. One could even have all the results support a given narrative. Journals might develop intelligent ways of counteracting this but it might pose a unique problem in the future.

I'd like to link Robert Miles' Intro to AI Safety (and his YouTube channel in general) as an accessible and well-presented way to learn about technical risk in AI Safety. As a field without a clear prevailing paradigm, there are many diverse viewpoints, of which EY's is just one. There are philosophical problems to solve, yes, but there are also very technical problems to solve, like power-seeking or inner misalignment or mechanistic interpretability that are much less funded than traditional capabilities research.

In general, taking risks with high stakes without thinking enough about it is just... kind of reckless, whether you're an individual or a company or a country or a nuclear physicist etc. We've already demonstrated in real systems (eg Bing or even social media recommender systems) that AI can be harmful and not behave as intended. I think it's just prudent of us to at least try and be careful, y'know, slow down and do some safety research, before doing things that might irreversibly change the world.

Like, imagine if a civil engineer just drew up the blueprint for a bridge without considering its stability, or weight, or materials, and it just got built 'cause there's no regulation against building unsafe bridges, it's much easier to build dangerous bridges than strong ones, anyone has access to the tools to build a bridge, lots of people think that building "safe" bridges = building beautiful bridges, etc. From a very high perspective this situation (which I hope you agree sounds quite silly) is remarkably similar to that of AI research today.

at the end of the day, it's always humans that are evil rather than the mystical AI.

I don't think this is about sentience, unless you're using that as a synonym for "general intelligence" rather than "conscious awareness"?

Pretty sure that's already happened... (the algorithm showing things to make you angry so you write a long response)

[removed]

I wouldn't trust Yudkowski to build an MLP classifier from scratch. Hell, I wouldn't trust him solving the Titanic task on Kaggle with SciKit learn.

He's a captivating speaker, but similar to some other popular cultural-scientific faces (e.g. Jordan Peterson) I feel like he is 100% form, 0% substance (or the substance that is valuable, is not original, but common knowledge repackaged).

If you don't tune in to people without a Bachelor's degree giving health advice, I don't know why you should care of Yudkowski's view of AI.

Corporations are paperclip maximizers, sometimes literally.

No. As a matter of fact, I consider it harmful, and I am far from being alone in that regard.

What you need to understand is that AI* kills already. Not only military/law enforcement AI that misidentifies people and leads to them being killed / searched & killed / empoisoned & killed in prison, the types of AI that you interact on a daily basis. Recommendation algorithms that promote disinformation regarding vaccines safety and COVID risk killed hundreds of thousands. Medical AIs that are unable to identify sepsis in 70% of cases but are widely used and override doctors in hospitals have killed thousands. Tesla autopilot AIs that kill their passengers on a regular basis. Conversational agent LLMs that will tell the users how to do electric work and kill them in the process.

But here is the thing. Working on the safety of such AIs leads to a conflict - with the engineers and researchers developing them, with execs that greenlight them, with influencers that touted them, with stakeholders who were getting money from additional sales the AI feature has generated. So safety and QA teams get fired, donations get made to universities to get rid of particularly vocal current state of affairs critics, Google de-indexes their works and Facebook randomly and accidentally deletes their posts (Bengio vs LeCun circa 2019, I believe, and the reason the latter moved to Twitter).

The problem with super-human AGI folks (and generally the longtermism/EA, to which Eliezer Yudkowsky belongs), is that they claim that none of those problems matter, because if SH-AGI arises, if it decides to mingle into human affairs, if we don't have an enclaves free from it, and even if it occurs in 100 years, it will be so bad, that it will make everything else irrelevant.

That's a lot of "ifs". And a long timeline. And there are pretty good theoretical reasons to believe that even when SG-AGI arises, its capabilities would not be as extensive as EA crowd claims (impossibility theorems and Solomonoff computability support wrt energy and memory support). And then there are theoretical guarantees as to why we won't be able to prevent it now even if it started to emerge (Godel's incompletness).

But in principle - yeah, sure why not, you never know if something interesting pops along the way.

The problem is that in the way it is currently formulated and advertised, it hits the cultural memes (HAL, A.I., ..) and the A-type personalities of younger engineers and researchers (work on the **most important** problem likely to make you **most famous**) in a way that completely drowns out the problems with AI that are already here - both from the general public's and engineer's perspective.

It is perhaps not a coincidence that a lot of entities that would stand to loose in reputation/income from in-depth looks into current AIs safety and alignment are donating quite a lot to EA/long-termism and lending them of their own credibility.

*To avoid sterile semantic debates, to me an AI is any non-explicitly coded programs that perform decisions on its own. Hence LLMs without a sampler are non-AI ML, whereas generative LLMs with a sampler are AI (generative ML).

[removed]

here is a text transcription of the podcast with comments.

You mention EY not being rigorous in his arguments. The timelines/probability of civilization-destroying AGI seem to need more explanation to me as well, but the type of AI safety/alignment problems he describes should be taken seriously by everyone in the community. The timelines for AGI vary in the community, from people that are confident in a AGI capable of complete wipeout of the human race within 15 years, to other 'optimists' in AI safety that think it might take several more decades. Although the timelines for AGI differ, these people mostly agree on the scenarios that they are trying to prevent, because the important ones are obviously possible (powerful things can kill humans; extremely powerful things can kill extreme amounts of humans) and not hard to imagine, such as 'we asked AGI to do harmless task X but even though it's not evil, it killed us as a byproduct of something else it was trying to do after reprogramming itself'. (By the way, the AI safety 'optimists' are still much more pessimistic than the general ML community which thinks it is an insignificant risk.)

There are good resources mentioned in this thread already to get other perspectives. The content is unfortunately mostly scattered in little bits and pieces over the internet. If you like popular book format/audiobooks, you could start with a longer and more digestible content in Stuart Russell's Human Compatible or Superintelligence from Nick Bostrom (which is a bit dated now, but still well written).

The shorter-term problems you mention are important, and I think it would be great for technical and policy-minded people to try to alleviate such threats. But it's also important for people to work on the potential longer term problems associated with AGI.

OpenAI, and organizations like them, are racing towards AGI - it's literally in their mission statement. The current slope of ML progress is incredibly steep. Seemingly every week it looks like some major ML lab comes up with an incredible new capability with only minor tweaks to the underlying transformer paradigm. The longer this continues to happen, the more impressive these capabilities look, and the longer we see scaling curves continue with no clear ceiling, the more likely it looks that AGI will come soon, say, over the next few decades. And if we do succeed at making AI as capable or more capable than us, then all bets are off.

None of this is a certainty. One of Yudkowsky's biggest flaws imo is the certainty with which he makes claims backed with little rigorous argument. But given recent discoveries, the probability of a dangerous long term outcome is high enough that I'm glad we have people working on a solution to this problem, and I hope more people will join in.

> can also create highly dangerous bioweapons

EY's example he gave in the podcast was a bioweapon attack. Unsure what kind of goal the AI had in this case, but maybe that was the point:

>But if it's better at you than everything, it's better at you than building AIs. That's snowballs. It gets an immense technological advantage. If it's smart, it doesn't announce itself. It doesn't tell you that there's a fight going on. It emails out some instructions to one of those labs that'll synthesize DNA and synthesize proteins from the DNA and get some proteins mailed to a hapless human somewhere who gets paid a bunch of money to mix together some stuff they got in the mail in a file. Like smart people will not do this for any sum of money. Many people are not smart. Builds the ribosome, but the ribosome that builds things out of covalently bonded diamondoid instead of proteins folding up and held together by Van der Waals forces, builds tiny diamondoid bacteria. The diamondoid bacteria replicate using atmospheric carbon, hydrogen, oxygen, nitrogen, and sunlight. And a couple of days later, everybody on earth falls over dead in the same second. That's the disaster scenario if it's as smart as I am. If it's smarter, it might think of a better way to do things. But it can at least think of that if it's relatively efficient compared to humanity because I'm in humanity and I thought of it.

No.

[deleted]

Nah dw I solved the alignment issue

I don't even think this is a strawman mate you've mischaracterized me so badly it's basically ad hominem.

"Unsure what kind of goal the AI had in this case"

tbf pretty much any goal that involves you doing something on planet Earth may be interrupted by humans, so to be certain, getting rid of them probably reduces the probability of being interrupted from your goal. I think its a jump that itll be that smart or that the alignment goal we use in the end wont have any easier way to the goal than accepting that interruptibility, but the alignment issue is that it Wishes it was that smart and could think of an easier way around

You're basically a doomsday cultist, just hiding it behind Sci-Fi language. "The scale of the threat" is irrelevant if the probability of it happening is infinestimal.

Yeah, GPT was the GPT moment of RL

Ah, yes, those well-understood equations for aesthetic beauty from the 1970s.

Well I think that entirely depends on what the threat is mate. The probability of AGI rising up terminator style I agree seems pretty small. The probability of disaster due to the inability of humans to distinguish true from false and fact from fiction being exasperated due to AI? That seems much higher. Also, I don't think either of us have a formula for this risk, so I think saying the probability of an event happening is infinitesimal is intellectual fraud.

Until the tools start exhibiting behavior that you didn't predict and in ways that you have no control over. Not taking an opinion on which side is "right", just saying that this is a false equivalence with respect to the arguments that are being made.

​

EDIT: Typo

This is terrifying.

Re: Things that will happen rather soon, I think it's important that we also look at AI's impact on nonhuman animals. I argued it in this paper. AI-controlled drones are already killing animals, some AIs are helping factory farming, language models are showing speciesist patterns that might reinforce people's bad attitudes toward some animals (ask ChatGPT about recipes of dog meat vs chicken meat, or just google "chicken" to see whether you see mainly the animal or their flesh)

Actually, even for things that could happen in the far future, I think it's extremely likely that what AI will do will immensely impact nonhuman animals too.

Eliezer Yudkowsky didn't attend High School or College. I'm not confident he understands basic Calculus or Linear Algebra, let alone modern Machine Learning. So yes, I will dismiss his views without seriously engaging with them, for the same reason that any Physics professor will dismiss emails from cranks talking about their "theories."

AI reduces the "proof of work" cost of an Andrew Wakefield paper. This is significant.

There's a reason people don't dedicate long hours to writing completely bogus scientific papers which will result in literally no personal gain: it's because they want to live their lives and do things like have a BBQ on a nice summer day.

The work involved in sounding credible and legitimate is one of the few barriers holding the edifice of what we call Science standing. The other barrier is peer review...

Both of these barriers are under a serious threat by the ease of generation. AI is our infinite monkeys on infinite typewriters moment.

This is to say nothing of much more insidious and clever intrusions into our thought institutions.

[removed]

I don't want to wax philosophical, but dying is the realm of humans. Death is the ultimate "danger of AI", and it will always require humans.

AI can't be dangerous on Venus.

Yes; but there's a general agreement that tool AI is vastly less dangerous than agentic AI. This seems to be the crux of disagreement between those who think risk is very high or just moderately high.

If we're getting philosophical, in a weird way if we ever do manage to build human-like AI, and I personally don't believe were at all close yet, that AI may well be our legacy. Long after we've all died that AI could potentially still survive in space or in environments we can't.

Even if we somehow survive for millenia, it will always be near infeasible for us to travel the stars. But it would be pretty easy for an AI that can just put itself in sleep mode for the time it takes to move between system.

If such a thing happens, I just hope we don't truly build them in our image. The universe doesn't need such an aggressive and illogical species spreading. It deserves something far better.

> Ones that unfairly send certain people to jail, ones that re-enforce unfair lending practices, ones that will target the wrong people even more aggressively than humans target the wrong people today.

Those examples are what I was alluding to with maybe a little too much hyperbole with saying “interns”. The most senior or best people are absolutely not building those models. Those models are being built by contractors who are subcontracting that work out which means its being built by people who are not getting paid well ie not senior or experienced folks.

Those jobs aren’t exciting and arent being rewarded financially by the market and I understand that I am not personally helping the situation but I am not going to take a huge paycut to work on those problems especially when that paycut would be at my expense for the benefit of contractors who have been historically scummy.

Let me flip that on its head for you: what makes you think that the Human-like AI is something you will want to be your representative?

What if it's a perfect match for Jared Kushner? Do you want Jared Kushner representing us on Alpha Centauri?

Generally, the whole AI is fine/is not fine debate always comes down to these weird false dichotomies or dilemnas. And imo, they are always rooted in the false premise that what makes humans noble - what gives them their humanity - is their intelligence.

Two points: a) AI need not be human like to have the devastating lethality, and b) a GAI is almost certainly not going to be "like you" in the way that most humans aren't like you.

AI's lethality comes from its cheapness and speed of deployment. Whereas a Jared Kushner (or insert your favorite person to dislike) takes 20 years to create out of scratch, AI takes a few hours.

That’s… not what his followers are saying. The hand-wringing about Bing hasn’t been about its misalignment per se, but about what it proves about the willingness of Microsoft and OpenAI to rush defective product release in an arms race situation. It’s not that the alignment is bad, it’s that clearly it didn’t register as a priority in the eyes of leadership, and it’s dangerous to expect that things will get better as AI get more capable.

What makes an extinction-level event unlikely in your view if you do believe advanced models will act so as to maximise control? Is it that you don’t believe in the capabilities of such a model?

> I understand that I am not personally helping the situation but I am not going to take a huge paycut to work on those problems especially when that paycut would be at my expense

I think you have this backwards.

Investment Banking and the Defense Industry are two of the richest industries in the world.

> Those models are being built by contractors who are subcontracting that work out which means its being built by people who are not getting paid well ie not senior or experienced folks.

The subcontractors for that autonomous F-16 fighter from the news last month are not underpaid, nor are the Palantir guys making the software used to target who autonomous drones hit, nor are the ML models guiding real-estate investment corporations that bought a quarter of all homes this year.

It's the guys trying to do charitable work using ML (counting endangered species in national parks, etc) that are far more likely to be the underpaid interns.

> EY tends to go straight to superintelligent AI robots making you their slave.

First, I don't think he ever said that they will make us slaves, except possibly as a joke at the expense of people who think the AI will care about us or need us enough to make us slaves.

Second, I am frustrated by the fact that you seem to think that only the short-term threats matter. What's a more short-term threat: nuclear contamination because of the destruction of the ZPP in Ukraine, or all-out nuclear war? Contamination is more likely, but that doesn't mean that we wouldn't be stupid to ignore the potentially farther away yet incredibly catastrophic outcome of nuclear war. Why can you not be worried about short-term AI issues but also acknowledge the possibility of the slightly longer term risk of superintelligent AI?

This is depressingly typical as an attitude and not at all surprising as the top comment here, unfortunately.

Unfortunately there are just too many crackpots in that space. It's like bringing up GUT in physics - worthwhile goal, but you're sharing the bus with too many crazies.

This is eerily similar to the “guns don’t kill people” argument.

It should be undeniable that AI provides a next-generation tool to lower the cost of disruption for nefarious actors. That disruption can come in various forms - disinformation, cyber crime, fraud etc.

https://www.lesswrong.com/posts/jtoPawEhLNXNxvgTT/bing-chat-is-blatantly-aggressively-misaligned

Yes? I fail to see how that goes against what I’m saying.

Here's how I would score this passage based on the nine emotions:

Anger: 0 - There's no indication of anger in this statement. Fear: 3 - The passage expresses a sense of worry and concern about the social ramifications of AI that pass as humans, which may reflect some level of fear. Joy: 0 - There's no expression of joy in this statement. Sadness: 0 - There's no indication of sadness in this statement. Disgust: 0 - There's no expression of disgust in this statement. Surprise: 0 - There's no indication of surprise in this statement. Trust: 1 - The passage expresses a concern about a crisis of trust and authenticity in the US, which may reflect some level of trust. Anticipation: 0 - There's no expression of anticipation in this statement. Love: 0 - There's no expression of love in this statement. Please keep in mind that these scores are subjective and based on my interpretation of the text. Different people may score the passage differently based on their own perspectives and interpretations.

Source: chatgpt

As others have said, the idea of being concerned with AI ethics and safety and taking it seriously is a good thing.

The problem is - and this is Just My Opinion™ - that people like EY are making what basically amount to spurious speculations about completely nebulous topics such as AGI, and they have very little to show in terms of some proof that they actually understand in technical detail where AI/ML is currently and the current SOTA. EY in particular seems to have jumped straight to those topics without any grounding in technical AI/ML research. I can't help but feel that, on some level at least, those topics were chosen based on the fact that it's easy to grab headlines and get into the media by making statements about it.

I'm not saying it's a bad thing to have people like EY around or that he or others like him are bad actors in any way, or that they shouldn't continue doing what they're doing. They may well be correct and their ideas aren't necessarily explicitly wrong. It's just that it's very difficult to genuinely take what they say seriously or make any practical decisions based on it, because a lot of it is so speculative. It reminds me a bit of Asimov's Laws of Robotics - they seemed like they made a lot of sense decades ago before anyone knew how the development of AI/ML would pan out, but in reality they're really just "it would be great if things worked this way" with no practical realistic plan on how to implement them, or even any way to know if they would actually be relevant.

The other thing is, as other people have pointed out, there are immediate and real problems with AI/ML as it stands, and solving those problems or avoiding disaster requires more than just making speculative statements. I think the absence of a will to address those issues by the biggest names in AI/ML ethics and safety is quite conspicuous.

​

Edit: Added a bit about Asimov's Laws of Robotics which occured to me after I made the post.

I don't see how it doesn't. Is this not an example of his followers wringing their hands over so-called misalignment that's really just poor performance?

It’s an example of noticing the misalignment, but the alignment is only a problem insofar as it is a symptom of the deeper problem I mentioned.

EY was very explicit that he doesn’t think GPT-style models are any threat whatsoever (the proliferation of convincing but fake text is possibly a societal problem, but it’s not an extinction risk)

?

That's Pascal's wager and could be used to justify belief in hell/god.

My research is in AI/ML for healthcare. One thing people forget is that everyone is concerned about AI/ML, and no-one is happy to completely delegate decision making to an ML model. Even where we have models capable of making accurate predictions, there are so many barriers to trust e.g. Black Box Problem and general lack of explainability which relegate these models to decision-support at best and being completely ignored at worst. I actually think that's a good thing to an extent - the barriers to trust are for the most part absolutely valid and rational.

However, the idea that these models are just going to be running amock is a bit unrealistic I think - people are generally very cautious of AI/ML, especially laymen.

> Until the tools start exhibiting behavior that you didn't predict and in ways that you have no control over.

LLMs already do behave in ways we don't expect. But they are much more than a hop skip, a jump and 27 hypothetical leaps away from being out of our control.

Yes, people will use AI for bad things, but that's not an inherent property of AI, that's an inherent property of humanity.

My point is that this isn't even misalignment in the first place. No more than an Imagenet classifier with 40% accuracy is misaligned. Misalignment is supposed to be when a model's learned objective is different from the human designer's objective. In their desperation to see threats everywhere, EZ et al resort to characterizing poor performance as misalignment.

Very well articulated.

[deleted]

Appreciate it! Articulation isn't a strong suit of mine but I guess a broken clock is right twice a day

But that’s a semantic disagreement on the proper use of “misalignment”, the substantive risk posed by the incentives of an AI arms race are the problem.

I agree with all of this but it's already been done. Social media platforms already use engagement driven algorithms that instrumentally arrive at recommending reactive content.

Cambridge analytica also famously preyed on user demographics to feed carefully tailored propaganda to swing states in the 2016 election.

Obligatory https://www.youtube.com/watch?v=9fa9lVwHHqg

Well, I don't really think this is a semantic disagreement. I'm using their definition of the term.

If the issue is the danger of an AI arms race, what does a poorly-trained model have to do with it? Isn't the danger supposed to be that the model will be too strong, not too weak?

>What about when ChatGPT

I mean, we're facing even more important dilemas right now, with ChatGPT's saftey rails. What is it allowed to talk about, or not? What truths are verbotten?

If the plurality of Internet content is written by these sorts of algorithms, that have hardcoded "safety" layers, then dream of truly open access to information that was the Internet will be that much closer to death.

That's a good question. To be clear, I believe there is a risk of an extinction-level event, just that it's unlikely. My thinking goes like this.

1. Extinction-level events must be rare, as one has not occurred in a very long time.
2. Therefore the 'base' risk is very low, and I need evidence to convince me otherwise.
3. I'm yet to see strong evidence that AI will lead to an extinction-level event.

I think the most likely outcome is that there will be serious negative implications of AI (along with some great ones) but that they will be recoverable.

I also think some people overestimate how 'super' a superintelligence can be and how unstoppable an advanced AI would be. In a game like chess or Go, a superior player can win 100% of the time. But in a game with chance and imperfect information, a relatively weak player can occasionally beat a much stronger player. The world we live in is one of chance and imperfect information, which limits any agent's control over the outcomes. This makes EYs 'AI didn't stop at human-level for Go' analogy less relevant.

I don't think anyone believes it will be LLMs that undergo an intelligence explosion, but they could certainly be a piece of the puzzle. Look at how much progress has been made in just the past 10 years alone - imo it's not unreasonable to think that the alignment problem will be a serious concern in the next 30 years or so.

In the short term, though, I agree that people doing bad things with AI is much more likely than an intelligence explosion.

Whatever anyone's opinion, I think the fact that the opinions of very smart and knowledgeable people run the gamut is a testament to the fact that we need to dedicate serious resources into ethical AI beyond the disclaimers at the end of every paper that models may contain biases.

Yes, it's worse than this too. We usually associate well-written text with accurate information. That's because, generally speaking, most people who write well are highly educated and have been taught to be critical of their own writing.

Text generated by large language models is atypical in that it's written like an expert but is not critical of its own ideas. We now have an unlimited amount of well-written, poor-quality information, and this is going to cause real problems.

>The subcontractors for that autonomous F-16 fighter from the news last month are not underpaid, nor are the Palantir guys making the software used to target who autonomous drones hit, nor are the ML models guiding real-estate investment corporations that bought a quarter of all homes this year.

You are equivocating the profits of the corporations vs the wages of the workers. Also you are equivocating "Investment Banking" with "Retail Banking" the person making lending models isnt getting the same TC as someone at Two Sigma.

None of those places (retail banking, defense) you list are the highest comp employers. The may be massively profitable but that doesnt necessarily translates to wages.

Just wanted to point out that even if we restrict ourselves purely to an agent that can only interact with the world through the internet, code, and natural language, that does not address the core AI alignment arguments of instrumental convergence etc being dangerous.

The hypothesis is precisely that the failure mode of Bing Chat comes from it being too strong, not too weak. That is, if prompted even in quite vague ways it can exhibit instrumentally convergent behaviour like threatening you, even though this was obviously not the designer's objective, and this behaviour occurs as a byproduct of being highly optimised to predict the next word (or an RL finetuning training objective). This is obviously not possible with, say, GPT-2, because GPT-2 does not have enough capacity or data thrown at it to do that.

So you would argue that the behavior highlighted in the post leads to either a lower loss on language modeling or a lower loss on RL finetuning than the intended behavior? That strikes me as very unlikely.

Nah, there are a bunch of reasoning steps missing. Conjecture on conjecture on conjecture is tough to work with.

I mean, it seems like the obvious explanation? That the model's behaviour is incentivised by its training objective. It also seems very plausible: we know that language models at large scale (even if not RL finetuned) exhibit a wide variety of emergent behaviours which you might not guess are motivated by next token prediction, but evidently are instrumental to reducing the loss. This is not necessarily overfitting: the argument is simply that certain behaviour unanticipated by the researchers is incentivised when you minimise the loss function. Arguably, this is a case of goal misgeneralisation: https://arxiv.org/abs/2105.14111

How do you distinguish a behavior which is incentivized by the training objective and behavior that is the result of an optimization shortcoming, and why is it obvious to you that this is the former?

The real alignment problem is with us, not AI. The danger isn't that AGI will run amok or indeed do anything unforeseen. Rather, it will do exactly what it asked of it: give a few powerful, unaccountable people even more dangerous ways to wage their useless, petty, destructive squabbles. There is no other reasonable prediction than this: the first serious thing it will be asked to do is think up new weapons. Stuff nobody could have even dreamed of. Think about that.

There's obviously intelligence under some definitions. It meets a weak definition of AGI since it reasons about a lot of things almost as well as the average human.

And yes, I know how it works and what its limitations are. It's not that useful yet, but discounting it entirely is as silly as thinking it's the AGI we're looking for.

Well, the obvious optimisation shortcoming is overfitting. We cannot distinguish this rigorously without access to model weights, but we also have a good idea what overfitting looks like in both pretraining and RL finetuning (in both cases it tends to result in common repeated text strings and a strong lack of diversity in output, a sort of pseudo mode collapse). We can test this by giving Bing GPT the same question multiple times and observing if it has a strong bias towards particular completions -- having played with it a bit I don't think this is really true for the original version, before Microsoft limited it in response to criticism a few days ago.

Meanwhile, the alternative hypothesis I raised seems very plausible and fits logically with prior work on emergent capabilities of LLMs (https://arxiv.org/abs/2206.07682), since it seems only natural to expect that when you optimise a powerful system for an objective sufficiently, it will learn instrumental behaviours which help it minimise that objective, potentially up to and including appearing to simulate various "personalities" and other strange outputs.

Personally, as a researcher who works on RL finetuned large language models and has spent time playing with many of these models, my intuition is that Bing GPT is not RL finetuned at all but is just pretrained and finetuned on dialogue data, and the behaviour we see is just fairly likely to arise by default, given Bing GPT's particular model architecture and datasets (and prompting interaction with the Bing Search API).

[deleted]

[deleted]

>Eliezer Yudkowsky didn't attend High School or College. I'm not confident he understands basic Calculus or Linear Algebra

This is incredibly dishonest and I think you know it. Even a little bit of googling would show that he has done research in decision theory and has an h-index of ~15. What's yours, btw?

And if you really want to go down the route of credentialism, there are quite a few established researchers who broadly agree with what Yudkowsky is saying and are working on the same things.

[removed]

Looks like reddit filtered out your response. Anyway, you seem plenty unhinged and I think I'll stop interacting with you.

In parting, I googled you as well. Bit odd for a "researcher" (one that compares himself to physics professors, no less) to have a citation count of zero. Have a good one...