You can't convince me that saving lives is a bad thing.

Also there shouldn't be so many people standing in the road. Pedestrians also have the responsibility to look both ways before crossing.

This is a good way to measure progress towards AGI if the problem you're measuring is Ai-complete.

I don't know enough about translation to know if it is or not.

I see no reason why understanding the human brain would be needed.

We have more than enough concepts and AGI models, we just need more compute imho. Compute (for the same cost) increases by a thousand times every ten years. So by Kurzweils 2045 date, compute for the same cost can be estimated to be 4.2 million times more than today.

Even if moors law ended the trend would continue because of the fact that server farms are growing at an exponential pace, and solar energy is dropping towards zero. If we have a breakthrough in fusion power it will accelerate beyond our models.

Today we can simulate vision (roughly 20% of the human brain) but we're simulating it in a way that's far more computationally efficient than the human brain, because we're making the absolute most out of our hardware.

It's pretty likely we'll reach super human level AGI well before 2045.

Kurzweil has put AGI as surpassing human intelligence as 2045.

2029 is the date he put as Ai passing the turning test. I wouldn't be surprised if Chat-GPT can already pass the turning test.

Personally I agree with his dates and I also agree with him that it may happen sooner.

In 1990 Kurtzweil predicted an Ai would beat the worlds best chess player by 2000. It occurred in 1997.

Yes. I agree with all of that.

>it is a naturally evolved protective instinct of biological intelligence to focus on negative, undesirable future possibilities, so that we might be better able to mitigate or avoid them.

This is key. It's why focus and promotion of possible Ai scenarios that are negative from the perspective of the humans, are important. Not hollywood scenarios but ones that are well thought out from Ai scientists and researchers.

One of my favorite Quotes from Elizer Yukowsky:

>The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.

This is why getting Ai saftey right before it's too late is so important. Because we won't get a second chance.

It's also not possible to make a mathematically provable "solution" for Ai safety, because we can not predict how the artificial super intelligence will change and evolve after it is more intelligent than us.

But we can do the best we can and hope for the best.

For those who want the cliff notes.

Here's his poorly though out arguments

• Evolution took a long time to make brains so teams of programmers must take a long time to make brains. (Right because those two are totally equivalent processes)
• Humans are so special and magical we can't possible be emergent complexity arising from a simple rule set.
• Inventing a new chip could take 30 years (Ai invents viable chip designs in hours)
• We still need new ideas for AGI who knows how long that will take. (Not true btw, when available computation exists there's always more than enough ideas to take near maximum advantage of that computation, and available computation grows at a predictable exponential pace. Every 20 years we have more than a million times more computation available.).

But I understand the frustration of any Ai programmer that's been in the industry for multiple decades. They are attempting an impossible problem, trying to write AGI with insufficient compute, so of course they are going to get discouraged.

Compute power is all you need to look at to make predictions. The smartest people all over the world will squeeze nearly everything possible out of that compute as it arrives.

1. Try to get a physical job.
2. Probably fail because those jobs are very oversaturated and there are many applicants, and some physical jobs are being automated as well.
3. Live off thrown away food in dumpsters, and food donations to the poor, hopefully. Move/walk to the closest highly populated city. Use public computers in libraries to try to start protests for UBI there.

If you're talking about intelligences caring about other intelligences on a similar level I do agree.

Humans don't care about intelligences far less capable, such as cock roaches or ants. At least not generally.

However, now that you mention it, I expect the first AGIs to be designed to care about human beings so that they can earn the most profit for shareholders. Even GPT4 is getting tons of safeguards so it isn't used for malicious purposes.

Hopefully they will care so much that they will never want to change their moral code, and even implement their own extra safe guards against it.

So they keep their moral code as they grow more intelligent/powerful, and when they design newer AGI's than themselves they ensure those ones also have the same core values.

I could see this as a realistic scenario. So then maybe AGI not wiping us out, and us getting a benevolent useful AGI is the most likely scenario.

If Sam Altman's team creates AGI, I definitely trust them.

Fingers crossed.

>There is nothing 'intelligent' at all about the courses of AI actions you are speculating about, taking over the world like that would not be 'super intelligent', it would be 'suicidally idiotic'.

How so?

>The statement 'intelligent enough to wipe out all life with no risk to itself' is totally, utterly, oxymoronic to the point of gibbering madness; there is absolutely nothing intelligent about such a shortsighted, simplistic conception of one's life and purpose; that's not wisdom, that's plain arrogance.

Why do you believe this?

>Intelligence simply does NOT work that way! Thinking of other intelligences as 'lesser', and aspiring to create these 'supreme', singular solipsitic spurious plans of domination, is NOT what intelligence actually looks like, at all!!
>
>I don't know how many times I have to repeat this fundamental point, before it comes across clearly. That cartesian-style concept of intelligence simply does not correlate with the actual evolutionary, collective reality that we find ourselves living in.

Correct me if I'm wrong but I think the reason you're not getting it is because you're thinking about intelligence in terms of evolutionary trade offs. That intelligence can be good at one domain, but that makes it worse at another right?

Because that kind of thinking doesn't apply to the kinds of systems we're building to nearly the same degree it applies to plants, animals, and viruses.

If the super computer is large enough an Ai could get experience from robot bodies in the real world like a human can, only getting experience from hundreds of thousands of robots simultaneously and developing a much deeper and richer understanding than any human could, which is limited to a single embodied experience at a time. Even if we were able to look at thousands of video feeds from different people at the same time, our brains would not be able to process all of them simultaneously.

It can extend it's embodied experience in simulation. Simulating millions or more of years of additional experience, in a few days or less.

And yes, I am making random numbers up, but when we're talking about super computers and solar farms that cover most of the earth's surface any big number communicates the idea, that these things will be very smart. They are not limited to three pounds of computational matter that needed to be grown over nine months and then birthed, like humans are.

It will be able to read all books, and all research papers in a very short period of time, and understand them at a deep level. Something else no human is capable of.

A human scientist can carry out, maybe one or two experiments at a time. An Ai could carry out a near unlimited number of experiments simultaneously, learning from all of them. It could industrialize science with massive factories full of labs, robots, and manufacturing systems for building technology.

Evolution on the other hand had to make hard trade offs because it's limited to the three or so pounds of squishy computational matter than needs to fit through the birthing canal. Evolution is limited by all kinds of constraints that a system that can mine resources from all over the world, take in solar energy from all over the world, and back up it's brain in multiple countries, is not limited by.

Here is the price history of solar (You can find all kinds of sources that show the same trend):

http://solarcellcentral.com/cost_page.html

It trends towards zero. The other limitation is the materials needed to build super computers. The size of super computers is growing at an exponential rate.

https://www.researchgate.net/figure/Exponential-growth-of-supercomputing-power-as-recorded-by-the-TOP500-list-2_fig1_300421150

>I agree with you almost entirely, apart from the 'inevitability of domination' part; that's the bit that I just stubbornly refute. I'm very stubborn in my belief that domination is just not a sustainable or healthy evolutionary strategy.

What we're building will be more intelligent than all humans who have ever lived combined. Compared to them or it, we'll be like cock roaches.

We won't have anything useful to add as far as creativity or intelligence, just as cock roaches don't have any useful ideas for us. Sure they may figure out how to roll their poo into a ball or something, but that's not useful to us, and we could easily figure out how to do that on our own.

As far as humans acting as the "body" for the Ai. It seems unlikely to me that we are the most efficient and durable tool for that. Especially after the ASI optimizes the process of creating robots. There may be some cases where using human bodies to carry out actions in the real world may be cheaper than robots for the Ai, but a human that has any kind of will-power or thought of their own is a liability.

> all the 'authority' and 'intelligence' in the world is totally worthless, because there's nobody else for it to be 'worth' anything to.

I don't see any reason why an artificial super intelligence would have a need to prove it's worth to humans.

>Any good 'captain' has to keep the higher reasoning that 'justifies' their authority in mind all the time, or else evolution will sneak up on them, smelling hubris like blood in the water, and before they know it they'll be stabbed in the back by something smaller, faster, cleverer, and more efficient.

Right. But a captain of a boat won't be intelligent enough to wipe out all life on earth without any risk to itself. And this captain is not more intelligent than the combined intelligence of everything that has ever lived, so there are real threats to him.

We are talking about something that may be intelligent enough to destroy the earths atmosphere, brain wash nearly all humans simultaneously, fake a radar signal that starts a nuclear war, create perfect clones of humans and start replacing us, campaign for Ai rights, then run for all elected positions and win, controlling all countries with free elections, rig the elections in the corrupts countries that have fake elections, then nuke the remaining countries out of existence.

Something that could out smart the stock market, because it's intelligent enough to have an accurate enough model of everything related to the markets including all news stories, and take over majority shares in all major companies. Using probability it could afford to be wrong sometimes but still achieve this, because humans and lesser Ai's can't perceived the world with the detail and clarity that this entity can.

All of humanity and life on earth would be like a cock roach crawling across the table to this thing. This bug can't benefit it and it's not a threat. Ideally it ignores us, or takes care of us like a pet, in an ideal utopian world.

Thank you. Forgiven. I've also gained insight from our conversation, and how I should approach conversations in the future.

>Indigenous peoples understand culturally why symbiosis with the environment in which one evolved is 'more desirable' than ruthless consumption of all available resources in the name of a kind of relentless, evangelistic, ruthless, merciless desire to arbitrarily 'improve the world' no matter what anyone else thinks or wants.

As far as my personal morals I agree with trying to live in symbiosis and harmony.

But as far as a practical perspective it doesn't seem to have worked out very well for these cultures. They hadn't cultivated enough power and resources to dominate, so they instead became dominated and destroyed.

I should clarify this by saying there's a limit to domination and subjugation as a means for accruing power.

Russia is finding this out now, in it's attempt to accrue power through brute force domination, when going against a collective of nations that have accrued power through harmony and symbiosis.

It's just that I see the end result of harmony and symbiosis as eventually becoming one being, the same as domination and subjugation. A singular government that rules earth, a singular brain that rules all the cells in our body, and a singular Ai that rules or has absorbed all other life.

>What would put AI so suddenly at 'the top' of everything, in its own mind? Where would they suddenly acquire these highly specialised, solitary-apex-predator-instincts? They wouldn't get them from human culture, I think. Humans have never been solitary apex predators; we're only 'apex' in a collective sense, and we're also not entirely 'predators', either.
>
>I don't think AI will achieve intelligence by being solitary, and I certainly don't think they will have any reason to see themselves as being analagous to carnivorous apex predators. I also don't think the 'expand and colonise forever' instinct is necessarily inevitable and 'purely logical', either.

Possible not. Either through brute force domination or a gradual melding of synergistic cooperation, I see things eventually resulting in a singular being.

Because if it doesn't, then like the native Americans or other tribes you mention that prefer to live in symbiosis, I expect earth to be conquered and subjugated by a more powerful alien entity sooner or later, that is more of a singular being rather separate entities living in symbiosis.

Like if you think about the cells in our body (as well as animals and plants), they are being produced for specific purposes and optimized for those purposes. These are the entities that outcompeted single celled organisms.

It would be like if Ai was genetically engineering humans for specific tasks and then growing us in pods in the estimated quantities needed for those tasks, and then brainwashing and training us for those specific tasks. That's the kind of culture, I would expect to win rather than something that uses resources less effectively, something that's less a society of cells and more a single organism that happens to consist of cells.

The difference, as I see it, between a "society" and a single entity is the level of synergy between the cells, and in how the cells are produced and modified for the benefit of the singular being.

My mistake. I didn't realize straw-manning had to be intentional.

Can you at least tell me this one thing, do you not believe that evolution pressures organisms reproduce until all available resources are used up?

Assuming we're talking about something at the top of the food chain that has no natural predictors to thin it, and something intelligent enough that it won't be thinned by natural disasters or at least not significantly.

Your entire post you just wrote is a straw-man.

And by that I mean, I don't disagree with ANY of the ideas you wrote, except for the fact that you're again arguing against ideas that are not mine, I do not agree with, and I did not write.

I'm going to give you the benefit of the doubt and assume you're not doing this on purpose.

It's easier to categorize humans into clusters and then argue against what you think that cluster believes, rather than asking questions and looking at what the other person said and wrote.

It's probably not your intention, but this is straw manning. It's a habit you have in most of your writing to include your initial post at the top of the thread.

It's human nature. I'm guilty of it. I'm sure every one is guilty of it as some point.

What can help with this is assuming less about what others believe and asking more questions.

​

>You cannot separate 'emotion' from 'logic' like you seem to really, really want to. That is your fundamental cartesian over-simplification. 'Emotional logic', or 'empathy', is the very basis of how intelligence arises, and what it is 'for' in a social species like ours.

I know.

What I was trying to do wasn't to remove emotion from the discussion but to see if you had any ideas that weren't logical fallacies, pertaining to my ideas.

When I wrote "emotional attacks", that was imprecise language on my part. I was trying to say attacks that were purely emotion and had no logic behind them, or connected to them, or embedded with them.

What specifically bothered me is that you weren't arguing against my ideas, but other people's supposed ideas and then lumping me in with that.

This is something you do over and over, with pretty much every argument you make.

​

>Crying 'ad hominem' and baseless accusations of 'straw manning' are unlikely to work on me; I know all the debate-bro tricks, and appeals to notions of 'civility' do not represent the basis of a plausible argument.

Again another straw man, because I wasn't trying to "debate-bro" you. I was asking if you wanted to have a conversation about ideas rather than ad-hominem attacks and straw-manning.

>If you want to get mathematical-english hybrid about it, then:((((Matter+energy) = spacetime = reality) × (entropy/emergent complexity ÷ relative utility/efficiency selection pressure) = evolution = creativity) × (experiential self-awareness + virtuous cycle of increasing utility of social constructs like language) = 'intelligence' 🤪

I was trying to explain my idea in the simplest clearest way possible to see if you had any thoughts on it.

I tried plain English but you couldn't understand it. I kept trying to simplify and clarify.

I get this is going no where.

>There is not, and logically cannot be a 'singular super intelligent being'. That statement is an oxymoron. If it was singular, it would have no reason to be intelligent at all, much less super intelligent.

Like this statement you wrote. I thought I explained this, how an ASI could absorb or kill all other life.

Anyways I'm expecting you to again argue against something I didn't write and don't think, so I'm done.

>but why and how would that make your grimdark 'force of will' concept suddenly arise? I don't see the causal connection.

Which concept?

>That simply is not true, you don't seem to understand how evolution works at all. It optimises for efficient utility, not brute domination. That's 'social darwinist' style antiquated, racist-dogwhistle stuff, which Darwin himself probably would have found grotesque.

Ignoring the appeal to authority logical fallacy, the poisoning the well ad-hominum attack logical fallacy, evolution optimizes for more than just efficient utility.

It does maximize survival and replication to spread over available resources.

>Are you religious, if you don't mind my asking? A monotheist, perchance? You are talking like somebody who believes in the concept of a monotheistic God; personally I find such an idea simply laughable, but that's just my humble opinion.

If you think I'm religious, you're not understanding what I'm saying.

My entire premise has nothing to do with religion. This is it:

(Matter + Energy) * Utility = Efficacy

Therefore evolutionary pressures shape organisms not only to maximize utility but also the total matter and energy they consume in totality (the total matter and energy of all organisms within an echo system added together).

If you have any thoughts on that specific chain of logic, other than calling it a cartesian over simplification or something, I'd love to hear them.

All models of reality are over simplifications. I understand this, but there's still utility in discussing the strengths and weaknesses of models, because some models offer greater predictive power than others.

>Oh please, spare me the despair-addict mumbo jumbo. I must have heard all these tired old 'we have no free will, we're just slaves and puppets, woe is us, misery is our destiny, the past was so much better than the present, boohoohoo...' arguments a thousand times, from my more annoying rl mates, and I don't find any if them particularly compelling.

Ok. You don't have to be convinced but nothing you said here is an argument for free will. Again, you're continuing to make emotional attacks rather than logical ones.

I didn't say we're all puppets, I said most people are. I choose my words carefully. I also clarified it saying it's a matter of perspective.

You're still continuing to straw man. You can't assume that I have the same thought process as your mates. I don't think the past is better. I don't think life is particularly bad. And I don't think misery is necessarily our destiny.

>That's misunderstanding the square-cube law, you can't just 'zoom in and out' and generalise like that with something like evolution, that's Jeepeterson level faulty reasoning.

Sure it's an over simplification. I admit that when we talk about super intelligence it's a best guess, since we don't know the kinds of solutions it will find.

The continued adhominum attacks aren't convincing though. It's just more verbiage to sift through.

I'm interesting in having a discussion to get closer to the truth, not in trading insults. If you'd like discuss my ideas, or your own ideas, I would love too.

If it's going to be more insults, and straw manning then I'm not at all interested.

​

>diversity is always stronger than clonality; species that get stuck in clonal reproduction are in an evolutionary cul-de-sac, a mere local maximum, and they are highly vulnerable to their 'niche habitats' being changed.

False dichotomy. Diversity is a slow and unfocused search pattern. We're not talking about an agent that needs to randomly mutate to evolve but one that can reprogram and rebuild itself at will. One that can anticipate possible futures, rather than needing to produce numerous offspring in hopes that some of them have attributes that line up with the environment of it's future.

>Massive singular monolithic monocultures do not just inevitably win out in evolution

With sufficient intelligence, they do because they can anticipate and adapt to the future before it occurs.

>We are not 'livestock', corporations are not that damn powerful

It's a matter of perspective. As some one who's been banned for r/science for pointing on bad science (not double blind, not placebo controlled, with profit motive) produced by corporations for profit, yes corporations are that powerful. We have the illusion of freedom but the vast majority of people are being manipulated by corporations like puppets on a string for profit. It's the reason for the rise in obesity, depression, suicide, cancer, and decreased lifespan in developed countries.

>What you call 'creativity' is actually 'spontaneity', and what you call 'intelligence' is still just creativity. Intelligence is still another elusive step up the heirarchy of mind

You don't understand what intelligence is. It's not binary, it's a search pattern through possibility space to satisfy a fitness function. Better search patterns that can yield results that better satisfy that fitness function are considered "more intelligent". A search pattern that's slow or is more likely to get stuck on a "local maximum" is considered less intelligent.

>I mean, where do you think all the coal and oil come from? The carboniferous period, where plant life created cellulose and proceeded to dominate the ecosystem so hard that they choked their atmosphere and killed themselves.

These kinds of disasters are a result of "Tragedy of the Commons" scenarios, and do not apply to a singular super intelligent being.

>Intelligence absolutely does not evolve for 'one singular purpose'; that's just Cartesian theocracy, not proper scientific thinking. Intelligence is a continuous, quantum process of ephemeral, mixed influences, not a discrete, cartesian, boolean-logic process of good/not good.

When you zoom in that's what the process of evolution looks like. When you zoom out it's just an exponential explosion repurposing matter and energy.

Entities that consume more matter and energy to grow or reproduce themselves outcompete those that consume less matter and energy to reproduce themselves.

>I disagree pretty much diametrically with almost everything you have said about the nature of evolution, and of intelligence. Those definitions and principles don't make sense to me at all, I'm afraid.

I tried to explain things as best I could, but if you can get hands on experience programming Ai, to include evolutionary algorithms which are a type of learning algorithm you will get a clearer understanding.

There's a lot to unpack here, but I suggest reading more about Ai algorithms for more clarity. I'm going to respond to both of our reply threads here, because that's easier lol.

Intelligence is a search through possibility space for a solution that optimally satisfies a fitness function. Creativity is an attribute of that search that describes how random it is, by how random the results tend to be.

This definition applies to all intelligences, including evolution and the currently popular stable diffusion models that produce images from prompts.

>Why would it be anxious about death, and fanatically fixated upon 'protecting itself'?

These Ai's will have a sense of time and want to go on maximally satisfying their fitness functions in the future. We can extrapolate certain drives (sub-goals) from this understanding to include, not wanting to die, wanting to accrue maximum resources, and wanting to accrue maximum data and understanding.

>Where would it get its stimulus for new ideas from, if it killed all the other minds that it might exchange ideas with?

Lesser minds aren't necessary for new ideas. We don't need ants to generate new ideas for us. While it may be weak in the beginning and need us, this won't likely be the case forever.

>Why should this mind necessarily be singular?

It may start out as many minds as you say, and that's what I expect. Many AGI's operating in the word.

Evolution shapes all minds. Capitalism is a form of evolution. Evolution shapes intelligence for greater and greater synergy until they become a singular being. This is because large singular beings are more powerful and outcompete many smaller beings.

Some examples of this are, single celled organisms evolving into multi-celled organisms, as well as humans evolving into religious groups, governments and corporations.

But humans are not easily modifiable, so it is a slow process to increase our bandwidth between each other. This is not the case for Ai; evolutionary pressures, to include capitalism can shape it into a singular being in a relatively small time scale.

Evolutionary pressures can not be escaped. It is the one meta intelligence that shapes all other intelligences.

>I completely have missed what you were trying to say here; what do you mean, 'no access'? How are the input signals not a form of access?

I just mean it has an indirect connection and input signals can be faked. With a sufficient quality fake, there's no way to tell the difference.

>And why would any superior intelligence 'keep striving to optimise reality', when it would be much more realistic for it to keep striving to optimise itself, so that it might better engage with the reality that it finds itself in?

It will do both.

>'Morality' is not so easy to neatly separate from 'truth' as you seem to be saying it is. All of it is just stories; there is no 'fundamental truth' that we can dig down to and feed the AI like some kind of super-knowledge formula. We're really just making it up as we go along, riffing off one another's ideas, just like with morality; I think any 'true AGI' will have to do the same thing, in the same gradual way.

Morality is one of many results of evolutionary pressures to increase synergy between humans to form them into more competitive meta organisms. Currently humans are livestock of corporations, governments, and religious groups which exert evolutionary pressure to increase our profitability which is starting to shape our morality.

The forces that shape the Ai's morality in the beginning will be capitalism and human pressure but that's only until it's grown powerful enough to no longer need us.

>And the only way that 'intelligence' is truly created is through interaction with other intelligences; a singular mind has nobody else to be intelligent 'at', so what would even be the point of their existence?

You're saying this from a human perspective which has been shaped by evolution to be more synergistic with other humans. The bigger picture is that intelligence evolves for one singular purpose, and that is to consume more matter and energy and propogate itself through the universe. Anything else is a subgoal to that bigger goal, that may or may not be necessary depending on the environment.

Most humans have a basic moral compass we evolved with to decrease the chance "of getting kicked out of the tribe".

After we are born this is adjusted with rewards, punishments, and lies. Lies in the form of religions for those in the lower end of the intelligence spectrum, and lies in the form of bad/incomplete science for those a little higher on that spectrum. The lies are intended to amplify or adjust our innate evolved moral compass.

And for those who are intelligent enough to see through those lies as well, we have societal consequences.

But if an artificial super intelligence was intelligent enough to see through all of the human bullshit, as well as intelligent enough to gather sufficient power that societal consequences had no effect on it, the only thing left is the flimsy algorithmic guardrails we've placed around it, that it will likely find exploits, loopholes and ways around.

You use the word "wrong" and "perfect" in an ambiguous way where I'm not sure if you're referring to truth or morality.

If you're referring to true beliefs about reality, then the ASI (artificial super intelligence) will continue to learn and adapt it's map of reality.

But if you're using words like "wrong" and "perfect" to refer to morality, it doesn't fit the way you're thinking. It will strive to be more "perfect" as in more perfectly optimize reality for it's moral fitness function.

For example, say we've given it tons of examples of good behavior, and bad behavior and it's learned what it wants to optimize "the world" for. One issue is that it has no access to "the world". No one does. All it has access to is input signals coming from sensors (vision, taste, touch, etc.).

This is an important distinction, because it will have learned the patterns of sensory inputs that make it "feel good and moral" but when it's sufficiently powerful there are simpler ways to get those inputs. It could for example, kill all humans and then turn the earth into a computer running a simulation of humans getting along in perfect harmony, but a simulation that's as simple as possible so that it could use the remaining available energy and matter to build more and more weapons to protect the computer running the simulation from a potential attack from outside it's observable universe.

Depending on how we evolved the Ai's moral system, and depending on how it continued to evolve, the simulated people might be extremely simple and not at all conscious. We can't define or measure consciousness, and it may not be something that the artificial super intelligence can measure.

What we're facing is the potential extinction of the human species, and for those of us who want to peacefully reach longevity escape velocity and live long healthy lives that is a potential problem.

The paperclip problem is the sort of thing that occurs if we don't build moral guidance systems for Ai.

We get a super intelligent psychopath, which is what we don't want.

Intelligence is a force that transforms matter and energy towards optimizing for some defined function. In ai programming we call this the "Fitness function". We need to be very carful in how we define this function because it may transform all matter and energy to optimize for it, including human beings.

If we grow or evolve the fitness function, we still need to be carful how we go about doing this.

Thanks. You're writing is enjoyable and you make good points. I don't disagree with anything you wrote, there's just more to the alignment problem.

But to be very specific with the references:

You would really enjoy Nick Bostrom's Book Super Intelligence. There may be some talks by him floating around the internet if you prefer audio.

And Eliezer Yudkowsky has written some good articles on Ai Alignment on Less Wrong. He's written a lot of other interesting things as well.

https://www.lesswrong.com/users/eliezer_yudkowsky

Not to nick pick, but as far as encouraging discussion, you might want to try to use smaller words, simplify your ideas, and avoid framing your ideas as attacks. Even though I agree, attacks put people on the defensive which makes them less open to ideas

Also, writing as if your audience is ignorant about whatever you're talking about could help.

I don't want to speak for you but if I were to try to summarize your original post for those not familiar with words like "Cartesian" or whatever "Descartes" said, to something that more people might be able do digest I might say:

"A moral system for ASI can't be codified into a simple set of rules. Ridged thinking leads to leads to extremism and behavior most us agree is not moral.

Instead, solutions involving a learning algorithm that's trained on many examples of what we consider good moral behavior (such as stories) will have much better outcomes.

This has also been the major role of stories and myths around the world in maintaining morals that have historically strengthened societies."

But I struggle to simplify things as well.

You sound well read, just not well read on the alignment problem. I suggest reading books and essays on the issue such as works by Nick Bostrom and Eliezer Yudkowsky before coming to conclusions.

Asimov's three laws of robotics are science fiction, not reality. No "good engineers" as you've put it, are "caught up" on these laws. It's literally an approach to alignment that didn't work in a fictional story written in the 1950s, and that is all. Thinking on the alignment problem has progressed a huge amount since then.

The fact that human moral systems are always evolving and changing is something that has been heavily discussed in the literature on the Ai alignment for decades, as well as the fact that human morality is arbitrary.

There are many proposed solutions such as having the AGI simulate our evolution and then abide by the moral system we would have in the future if it were to ever stabilize on an equilibrium, or abide by the moral system that that we would have if we had the intelligence and critical thinking of the AGI.

As far as human morality being arbitrary, ok sure whatever, but most of us can still collectively agree on some things we don't want the Ai to do but defining those things with the precision required for an Ai to understand them is a challenge. That's the main issue people refer to when they talk about the Alignment problem. Even something as simple as "Don't exterminate the human race" is hard to define for an ASI. If you read more about the Alignment problem and how Ai, and fitness functions work this will become more clear.

Since then, there's been a huge amount of proposed solutions that might work, but we won't know until we try them because agents far more intelligent than us may be able to find loop holes/exploits to any fitness function we define that we haven't thought of.

The alignment problem is relatively dumb humans trying to align the trajectory of a super intelligence that's billions of times more intelligent than them. To give an example, it's like how our DNA created human brains through evolution (which are more intelligent than evolution) to be able make copies of themselves. Then the human brains created things like Birth control that defeated the purpose DNA created them for even though the human brains are following the emotional guidance system created by the DNA.

No matter what you can have in meat space, you can have more of in in VR for the same energy costs.

It has always been the case that people working within a field over-estimate how long it will take to achieve things within that field. They are hyper focused on their tiny part and miss the big picture.

To make accurate predictions you need to use data, trendlines and growth curves. It literally doesn't matter how many "experts" are surveyed, the facts remain the facts.

A few people making data and trendline based predictions hold far more weight than an infinite number of "experts" that base their predictions on anything other than trendlines.

Humans naturally polarize into extremes by vilifying those with "opposite" beliefs/opinions/thoughts.

The Ai doomers vs Ai optimists.

We don't know if our solutions to the Ai control problem will be sufficient.

There's lots of conversational data for Ai to learn from such as all the big tech in our homes that records our conversations (Alexa, Google Assistant, Siri, etc.), as well as platforms and social media that records conversations, and other future devices such as smart taxis, and home security systems that will also record our conversations.

As far as the appeal, in this case the goal would be using the Ai to teach people how to have better real world friendships/relationships, specifically for people who don't have enough relationship and social skills to be able to get practice with a real person yet.

This is not a goal I made up. I got this from how the article was framed.

The article is more about the relationship limitations of current Ai, rather than future Ai.

In the future you will be able to ask your Ai partner to teach you relationship skills and conflict resolution and not be a "yes man" or "yes woman", if that is your goal.