If I understand correctly, the model is optimised to effectively predict the next word. That says nothing of its internal representations or lack thereof. It could well be forming internal representations as an efficient strategy to predict the next word. As Sam Altman pointed out, we’re optimised to reproduce and nothing else, yet look at the complexity of living organisms.

EDIT: Just to add, it’s not quite the same thing, but another way of thinking of “most probable next word” is “word that a person would be most likely to write next” (assuming the training data is based on human writings). One way to get really good at approximating what a human would likely write given certain information would be to actually approximate human cognitive structures internally.