Submitted by xutw21 t3_ya21gp in singularity
FirstOrderCat t1_it9oqhg wrote
Reply to comment by CommentBot01 in U-PaLM 540B by xutw21
> Currently deep learning and LLM are very successful and not even close to its limit.
to me it is opposite, companies already invested enormous resources, but LLM can solve some simplistic limited scope tasks, and no much AGI-like real applications have been demonstrated.
AsthmaBeyondBorders t1_ita7j8l wrote
LLMs are best when coupled with other AIs for natural language commanding. Instructing a robot on what to do using natural language and chain of thought instead of pre determined scripts. Instructing an image generator like stable diffusion and Dall-E on what to draw based on language instead of complicated manual adjustment of parameters and code. I'd say those are very necessary applications.
You may be looking at LLM models on their standalone form but don't forget LLMs are behind stable diffusion, dreamfusion, dreambooth, etc.
visarga t1_itapbde wrote
The same CLIP architecture that guides SD to draw pretty images can also guide an industrial robot to accomplish tasks.
FirstOrderCat t1_ita7yel wrote
> form but don't forget LLMs are behind stable diffusion, dreamfusion, dreambooth, etc.
But its not discussed AGI, it is more stochastic parroting, or style transferring.
AsthmaBeyondBorders t1_ita8bqk wrote
Yeah but we can't jump from nothing to AGI, LLMs have been very useful so it makes sense to continue pushing their limits until we hit a wall (and we haven't hit that wall yet).
FirstOrderCat t1_itaaaxe wrote
> and we haven't hit that wall yet
why do you think so?
AsthmaBeyondBorders t1_itabm60 wrote
Look at the post you are replying to.
A wall is when we can't improve the results of the last LLMs.
New LLMs, both with different models and bigger scale, not only improve the performance of the last LLMs on tasks we already know they can do, but we also know there are emergent skills that we may still find scaling up. The models become capable of doing something completely new just because of scale, when we scale up and stop finding emergent skills then that's a wall.
FirstOrderCat t1_itacdp4 wrote
>A wall is when we can't improve the results of the last LLMs.
The wall is a lack of break through innovations.
Latest "advances" are:
- build Nx larger model
- tweak prompt with some extra variation
- fine-tune on another dataset, potentially leaking benchmark data to training data
- receive marginal improvement in benchmarks irrelevant to any practical task
- call your new model with some epic-cringe name: path-to-mind, surface-of-intelligence, eye-of-wisdom
But none of these "advances" somehow can replace humans on real tasks, with exception to style-transfer of images and translation.
AsthmaBeyondBorders t1_itaclhq wrote
The problem is you don't know what emergent skills are yet to be found because we didn't scale enough. And "breakthrough" may well be one of the emergent skills we haven't reached yet
FirstOrderCat t1_itad0zs wrote
>The problem is you don't know what emergent skills are yet to be found because we didn't scale enough.
Yes, and you don't know if such skills will be found and we hit the wall or not yet.
AsthmaBeyondBorders t1_itad4q2 wrote
There is a very old solution to finding that out. It is to scale and check instead of guessing
FirstOrderCat t1_itaeypy wrote
this race maybe over.
On the graph guy is proud of getting 2 points in some synthetic benchmark, while spending 4 millions TPUv4 hours = $12M.
At the same time we hear that Google cuts expenses and considering layoffs, and LLM part of Google Research will be the first in the line, because they don't provide much value in Ads/Search business.
AsthmaBeyondBorders t1_itagbfz wrote
This model had up to 21% gains in some benchmarks, as you can see there are many benchmarks. You may notice this model is still 540B just like the older one, so this isn't about scale it is about a different model which can be as good and better than the previous ones while cheaper to train.
You seem to know a lot about Google's internal decisions and strategies as of today, good for you, I can't discuss stuff I have absolutely no idea about and clearly you have insider information about where google is going and what they are doing, that's real nice.
FirstOrderCat t1_itahazn wrote
>This model had up to 21% gains in some benchmarks, as you can see there are many benchmarks
Meaning they received less than 2 points in many others..
> it is about a different model which can be as good and better than the previous ones while cheaper to train.
Model is the same, they changed training procedure.
> You seem to know a lot about Google's internal decisions and strategies as of today
This is public information.
justowen4 t1_itau79p wrote
Epic commenting you two. The winner is….. AsthmaBeyondBorders !
Viewing a single comment thread. View all comments