Viewing a single comment thread. View all comments

RandomCandor t1_j47bx4j wrote

Reply to comment by currentscurrents in [D] Bitter lesson 2.0? by Tea_Pearce

To me, all that means is that the lay people will always be a generation behind from what the rich can afford to run

8

currentscurrents t1_j48csbo wrote

If it is true that performance scales infinitely with compute power - and I kinda hope it is, since that would make superhuman AI achievable - datacenters will always be smarter than PCs.

That said, I'm not sure that it does scale infinitely. You need not just more compute but also more data, and there's only so much data out there. GPT-4 reportedly won't be any bigger than GPT-3 because even terabytes of scraped internet data isn't enough to train a larger model.

6

BarockMoebelSecond t1_j48mepq wrote

Which is and has been the Status Quo for the entire history of computing, I don't see how that's a new development?

3

currentscurrents t1_j490rvn wrote

It's meaningful right now because there's a threshold where LLMs become awesome, but getting there requires expensive specialized GPUs.

I'm hoping in a few years consumer GPUs will have 80GB of VRAM or whatever and we'll be able to run them locally. While datacenters will still have more compute, it won't matter as much since there's a limit where larger models would require more training data than exists.

3