Viewing a single comment thread. View all comments

visarga t1_jdtypz6 wrote

Reply to comment by Haycart in [D] GPT4 and coding problems by enryu42

Doesn't autoregressive decoding cache the states for the previous tokens when decoding a new token?

2

Haycart t1_jdu7hlp wrote

Oh, you are probably correct. So it'd be O(N^2) overall for autoregressive decoding. Which still exceeds the O(n log n) that the linked post says is required for multiplication, though.

1