Viewing a single comment thread. View all comments

suntehnik t1_j5ykwei wrote on January 26, 2023 at 1:44 PM

Reply to comment by besabestin in Few questions about scalability of chatGPT [D] by besabestin

Just speculation here: maybe they store generated text in a buffer and when they run out of memory buffer can be flushed to get allocation back for other tasks.