justowen4 t1_it645n5 wrote on October 21, 2022 at 5:14 AM

Reply to comment by ftc1234 in If you believe you can think exponentially, you might be wrong. Transformative AI is here, and it is going to radically change the world before the Singularity, and before AGI. by AdditionalPizza

In case you missed it, LLMs surprised us by being able to scale beyond expectations. The underestimation was because llms came from the nlp world with simple word2vec style word associations. In 2017 the groundbreaking “attention is all you need” paper showed the simple transformer architecture alone with lots of gpu time can outperform other model types. Why? Because it’s not an nlp word association network anymore, it’s a layered context calculator that uses words as ingredients. Barely worth calling them llms unless you redefine language to be integral to human intelligence

ftc1234 t1_it64mud wrote on October 21, 2022 at 5:19 AM

I know what LLMs are. They are a surprising development but the history of technology is littered with surprising discoveries and inventions. But there are very few inventions of the earth shattering variety. And I don’t believe that LLMs are of that variety for the reasons I stated. CNNs were all the rage before LLMs. And respected giants in the field such as Yann LeCun have also stated that LLMs are important but they aren’t everything.