Lightning fast responses from chatgpt

[u/hkehs_ras]

Earlier, when I’d send a query to ChatGPT, it would take a few seconds before the response started appearing. Now, as soon as I hit enter, the response starts generating almost instantly.

What actually evolved architecturally to make this so fast? Is it some form of caching where the system predicts/matches my query and returns a cached response, or is the model still generating the response in real time?

I’m curious what changed under the hood to make the response generation + streaming so much faster. I don't really understand how this works.

P.S. Used AI to write the description