Yep. Convenient how the open letter does nothing at all to slow anyone down who isn't at gpt-5 researching stage. And it's only 6 months, maybe renewed a couple times - about the length of the gap between OAI and the second place group.

Like getting the refs in an auto race to slow down only the car multiple laps ahead.

I don't care who he is, it doesn't fit measurements he is aware exist.

Fine let's spend a little effort debunking this:

From:

https://medium.com/@francois.chollet/the-impossibility-of-intelligence-explosion-5be4a9eda6ec

Intelligence is situational — there is no such thing as general intelligence.

This is empirically false and not worth debating. Current sota AI use very very very simplistic algorithms and are general, and slight changes to the algorithm result in large intelligence increases.

This is so wrong I will not bother with the rest of the claims, this author is unqualified

​

From:

https://globalprioritiesinstitute.org/wp-content/uploads/David-Thorstad-Against-the-singularity-hypothesis.pdf

Extraordinary claims require extraordinary evidence

- you could have "debunked" nuclear fission in 1943 with this argument and sat comfortably in the nice japanese city of hiroshima unworried. Sometimes you're just wrong.

Good ideas become harder to find

This is true but misleading. We have many good ideas, like fusion rocket engines, flying cars, genetic treatments to disable aging, nanotechnology. As it turns out the implementation is insanely complicated and hard. Sometime AI can do much better than us.

Bottlenecks

True but misleading. Each bottleneck can be reduced at an exponential rate. For example if we actually have AGI right now, we'd be building as many robots and AI accelerator chips as physically can, and also increasing the rate of production exponentially.

Physical constraints

True but misleading, the solar system has a lot of resources. Growth will stop when we have exhausted the entire solar system of accessible solid matter.

​

Sublinearity of intelligence growth from accessible improvements

True but again misleading, even if intelligence is sublinear we can make enormous brains, and there are many tasks, mentioned above, we as individual humans are too stupid to make short term progress on, so investors won't pay to develop them.

So even if the AGI system has 1 million times the computational power of a human being, but is "only 100" times as smarter and works 24 hours a day, it can still make possible to make working examples of many technologies in short timelines. Figure out biology and aging in 6 months of frenetic round the clock experiments using millions of separate robots. Figure out a fusion rocket engine by building 300,000 prototypes of fusion devices of various scales. And so on.

Human beings are not capable of doing this, no human alive can even hold in their head the empirical results of 300k engine builds and field geometries. So various humans have to "summarize" all the information and they will get it wrong.

It is not learning from your chats. Apparently OAI does farm for information from CHATGPT queries specifically for RL runs. And I was mentioning that in order for "plugin" support to work even sorta ok the machine absolutely has to learn from it's mistakes.

Remember all it knows is a plugin claims to do something by a description. The machine needs to accurately estimate if a particular user request is going to actually be satisfied by a particular plugin and also how to format the query correctly the first time.

Without this feature it would probably just use a single plugin, ignoring all the others, or get stuck emitting malformed requests a lot and just guess the answer like it does now.

So this is an Open domain hallucination:

​

Closed domain hallucinations refer to instances in which the model is instructed to use only information provided

in a given context, but then makes up extra information that was not in that context. For example, if you ask the

model to summarize an article and its summary includes information that was not in the article, then that would be a

closed-domain hallucination.

Open domain hallucinations, in contrast, are when the model confidently provides false

information about the world without reference to any particular input context.

​

​

They handled this via : For tackling open-domain hallucinations, we
collect real-world ChatGPT data that has been flagged by users as being not factual, and collect
additional labeled comparison data that we use to train our reward models.

​

​

Not very productive. The best way to check references would be using a plugin and instructions to the model to "check references". The machine also needs to have RL training so that it will use the plugin and use it correctly the first time.

I mean you could simply grab a heap of exams the school district already gave and the standardized tests and just use that. Not saying this is an optimal standard but it's what we already use.

Yep. Plus in that 5 percent category the AI could be designed to inform the supervising doctor when it has low confidence so that when possible humans augment it.

Doctors commit grave errors now. They won't switch to AI until it is substantially better than humans .

Sue the doctor using AI. For essentially as far as we can imagine, a human doctor will still be on paper the one practicing medicine. They are just using AI as a tool to be more productive.

As the AI gets stronger and stronger the human does less and less, as the AI itself has a confidence level that with the right software algorithms can be extremely accurate. So the human doctor let's AI do it all for the cases where the AI is very confident.

Because many AIs will check anything done for you, this accidental amputation is unlikely, and most suits are going to fail because exact records of their AI reasoning and wha rit new are kept. So you can just see in the logs the AI did everything possible and picked the highest success probability treatment.

And rigged it where new doctors aren't trained in sufficient numbers, meaning that even bad doctors so long as they meet some low bar can continue practicing until age 70 or 80 or higher.

Same for humans. Tv shows like ER where they ask for a CBC and chem7 every time...

A personalized AI tutor and a curriculum with objective measurements, where once a student scores high enough they finish, would probably make teachers fairly unnecessary other than as a "hall monitor" to oversee groups of kids on their devices being taught by AI.

Or another way to put it, the mistake RATE is probably significantly lower for even relatively crude AI. Human doctors make a very large error rate, it may be 30 percent plus. Wrong diagnosis, suboptimal prescription, failure to consider relevant life threatening issues by overly focusing on the chief complaint.

(I define "error" by any medical treatment that is measurably less effective than the current gold standard, for all issues the patient has)

If known anti aging drugs work, human doctors commit essentially a 100 percent error rate by failing to prescribe them.

Current AI can fail in certain situations, so I think human doctors and other AI should be checking their work, but yeah, if you want to live get an AI doctor.

Right now, you can go get human surgeries that attempt to transition one gender to another.

Obviously the surgeries are not able to fix many things, and leave scars and all kinds of damage.

Demos of medical labs 3d printing human organs have existed for 15+ years, but a combination of bureaucratic inertia and just flat problems with the printed organs have prevented their use.

Presumably if AI is in charge of the organ production, and it's had an enormous amount of practice doing it and many scientific experiments to understand it fully, much better organs could be created, new skin, new structures, whole limbs, and so on.

This would probably be initially be used to help the elderly - since you can basically replace their bodies except the brain this way - but eventually there would be perfect gender reassignment surgery.

Presumably eventually with nanotechnology, surgical incisions might be far tighter and cleaner - right along the line of cells, cutting structures without damage, and more importantly, suturing might be exact, where all the nerves and individual fibers in each muscle are actually reattached correctly, using protein based glue similar to how the cells bond now, and a lot less pain and inflammation after the patient wakes up - maybe none.

All that pain and scarring and swelling is basically because current surgeons don't have any better tools, this is the best they can do. (medical science does know the reason for a lot of it but has failed to develop tools to prevent it)

They could fine tune it, use prompting or multiple pass reasoning, give it an internal python interpreter. Lots of options that would more fairly produce results closer to what this generation of compute plus model architecture is capable of.

I don't know how well that will do but i expect better than median human as these are the result google got who were using a weaker model than gpt-4.

Well there is a solution to this. Instead of assuming the AI can figure out how to repair any arbitrary house (though it might actually be doable), if houses were factory built, the robots/AI can be much more limited.

Basically, make the whole house/office building out of flat panels and other objects designed to fit into the dimensions of a max size load truck and easy to assemble on site. Robot trucks haul the pieces to the job site, robot cranes lift them into place, robots probably ride the piece up (no OSHA standards for them!) and use their arms to pull it into alignment and bolt/weld into place.

So the solution would be to basically take a 100 year old house and replace it with a brand new house where it's been made to look externally like the same house.

The new stuff would be robot repairable, with everything behind panels that robots can easily remove and subdivided into modules that can be easily removed and replaced.

My bad, I checked the dates in a hurry.

By "some point" you meant 2 weeks ago right? https://arxiv.org/abs/2302.04761

>Why is python so widely used in AI when it’s a really inefficient language under the hood? Wouldn’t Rust be better to optimize models? Or do you just need that optimization at the infrastructure level while the models are so high level it doesn’t matter?

You make calls to a high level framework, usually pytorch, that have the effect of creating a pipeline. "Take this shape of input, inference it through this architecture using this activation function, calculate the error, backprop using this optimizer".

The python calls can be translated to a graph. I usually see these in *.onnx files though there are several other representations. These describe how the data will flow.

In the python code, you form the object, then call a function to actually inference it a step.

So internally it's taking that graph, creating a GPU kernel that is modified for the shapes of your data, compiling it, and then running it on the target GPU. (or on the project i work on, it compiles it for what is a TPU).

The compile step is slow, using a compiler that is likely C++. The loading step is slow. But once it's all up and running, you get essentially the same performance as if all the code were in C/C++, but all the code you need to touch to do AI work is in Python.

Mistakes: Depends on the outcome of efforts to try to reduce answering errors. If self introspection works, months.

More context memory: Weeks to months. There already are papers that set up the groundwork: https://arxiv.org/abs/2302.04761 . Searching the past log for this same session (past our token window) is easily integratable with the toolformer architecture.

There are also alternate architectures that may also enormously increase the window.

AGI : it is possible within a few years. Whether it happens depends on the trajectory of outside investment. If Google and Microsoft go into an all out AI war where each are spending 100B plus annually? A few years. If current approaches "cap out" and the hyper diminishes? Could take decades.

Singularity: shortly after AGI is good enough to control robotics for most tasks. So shortly after AGI probably. (shortly meaning a matter of months to a few years)

Several friends. Others at AI startups. Somehow they are self taught. Good at Python, has a framework that uses some cool hacks included automated function memoization.

Note that until very recently, like 2 months now, OpenAI was kind of not the best option for elite programmers. It was all people on a passion project. The lottery winners were at Deepmind or Meta.

Have several friends there also. The Meta friends are all the usual background, with the graduate degree and 15+ yoe in high performance GPU work.

I thought it would require a lot of things. But here we are.

Open source devs have re-created the core of an LLM like GPT-3 (it's what powers chatGPT and BingChat) in a few thousand lines of code.

It's really not that complicated. https://github.com/EleutherAI/gpt-neox

And yet this one repeated algorithm and a few tricks in training and we can get like 50% of human intelligence right there.

Note: I work in AI , and have friends who work at OpenAI.

Computer science.

The reason why the other 2 subjects don't matter is they essentially are not used now. Neither neuroscience or cognitive science is relevant for current AI research. Current methods have long since left needing to borrow from nature. The transformer or current activation functions for ANNs do not borrow anything but the vaguest ideas from looking at old neuroscience data.

Current AI research is empirical. We have tasks we want the AI to do, or output we want it to produce, and we will use whatever actually works.

The road to AGI - which may happen before you graduate, it's happening rapidly - will be likely from recursion. Task an existing AI with designing a better AI. By this route, less and less human ideas or prior human knowledge will be used as the AI architectures are evolved in whatever direction maximizes performance.

For an analogy: only for a brief early period in aviation history did anyone study birds. Later aerofoil advancements were made by building fixed shapes and methodically studying variations on those shapes in a wind tunnel. Eventually control surfaces like flaps and other active wing surfaces were developed, still nothing from birds - the shapes all came from empirical data, and later CFD data.

Similarly, none of the other key element of aviation: engines: came from studying nature either. The krebs cycle was never, ever used in the process of making ever more powerful combustion engines. They are so different there is nothing useful to be learned.

From you and every mortal perspective that has no cost.

Also how useful is quantum chemistry.

You can probably just "memorize the rules" with a neural network, the way protein folding was solved, and not actually simulate the quantum chemistry. This is drastically faster and almost as accurate.

This means you just do a bunch of chemistry experiments, or load in the data from already performed experiments, and figure out the rules so you can predict the experiments you didn't perform. Neural networks can already learn most possible functions so they can approximate what a quantum chemistry sim would theoretically be exact for.

And the approximations can be potentially just as accurate : remember your input data has finite resolution. (significant figures)