The biggest limitation of GPT-3 wasn’t the size but the data. It was trained almost on the whole internet and still underfit. At the end of the day the goal of the model is to predict the next word I don’t think it will necessarily lead to AGI but definitely it will be great to see interesting properties emerging from such a simple objective function.