GuyWithLag

GuyWithLag t1_j1q43hg wrote

There's good indications that one can trade-off training time and corpus size against model size, making the post-training per-execution cost smaller.

Note that ChatGPT is already useful to very many people; but training a new version takes time, and I'm guessing that OpenAI is currently still in the iterative development phase, and each iteration needs to be short as it's still very early in the AI game.

1

GuyWithLag t1_j1hgj0h wrote

Reply to comment by Ortus12 in Hype bubble by fortunum

Dude, no. Listen to the PhDs - the rapture isn't near, not yet at least.

On a more serious note: This is what the OP refers to when talking about a "hype bubble". The professionals working in the field actually know that the current crop of AI models are definitely not suitable for the architecture of AGI, except maybe as components thereof. Overtraining is a thing, and it's also shown that overscaling is also a thing. Dataset size is king, and the folks that create the headline-grabbing models already fed the public internet to the dataset.

From a marketing standpoint, there's the second-mover advantage: see what other did, fix issues and choose a different promotion vector. You're looking at many AI announcements in a short span due to the bandwagon effect, caused by a small number of teams showing multiple years' worth of work.

6

GuyWithLag t1_izg8rz0 wrote

Was this written by an AI? because it's veering hard to a similar topic after the first paragraph.

>penmanship in the age of keyboards

Bad example, in both cases you need to know what you want to write, and how to express it. I'm of the position that the approach used by OP will lead to a shallower understanding of the topics he delegates to the AI for research, and that he himself will not have the necessary foundations to generate advances (novel things, sure, he'll get from the AI recombining the current state of the art).

2

GuyWithLag t1_ize6696 wrote

>4.0 GPA stem club member robotics club member internship at a biology lab

But your thinking _as expressed by the post you wrote_ is of a teenager; you writing is sub-par and somewhat rambling. Have you actually integrated and internalized anything of what you were taught in that year, besides by osmosis from GPT-3? And no, I'm not talking about bloody facts, mate.

You probably should pass this piece through a GPT-3-equivalent cleanup process. But here lies the rub: you will need to do this for everything that you do going forward, and the facade will need to never fall.

11

GuyWithLag t1_ix8lmg8 wrote

>If you have a follow up to Gato that's 10x or 100x larger, the ability to cross/interpolate its knowledge across learned skills, and has a context window larger than 8,000 tokens, then you're approaching something like a proto-AGI.

And exactly this is why I think we're missing some structural / architectural component / breakthrough - the current models have the feel of unrolled loops.

2

GuyWithLag t1_irm805h wrote

>This is all correct, except the experts were extremely reluctant and slow to accept that they were wrong about aerosol droplet size.

This is an adaptation to science and public health reporting. The general public has been trained to not understand the scientific process, and "hey folks, we were wrong in our research and now you need to do X and stop doing Y" makes Joe "Moron" Public go "Hah, see? Science can be wrong, what else have they gotten wrong too?".

This isn't helped by there being monied interests in play.

29