The KV cache: why LLM inference runs out of memory, not compute
Generating text one token at a time would be hopelessly slow without the key/value cache. It is also the thing that eats your GPU memory. A quick tour with the numbers.
Archive
All posts
Why your Postgres table is bigger than its data
MVCC never updates a row in place. Every UPDATE leaves a dead tuple behind, and VACUUM is the only thing that cleans it up. Here is how to see the bloat and keep it in check.