> So long as it’s a transformer model, GPT-4 will also be a query engine on a giant corpus of text, just with more of the holes patched up, so it’d be harder to see the demonstrative examples of it being that.

This claim has a strong scent of sophistry about it - any and all signs of intelligence can be handwaved away as interpolating to plausible text.

The explanations of failures are convincing, but the theory needs to go further and explain why larger models like GPT4 (and in some cases 3.5) are so much more effective at answering out-of-domain queries with explicit reasoning proceding from information that it does have. E.g 4 correctly answers the weights question and gives a clear explanation of its reasoning. And that isn't an isolated example.

It's not just an incremental improvement, there is a clear difference in kind.