FRESH

Hacker News

From 300KB to 69KB per Token: How LLM Architectures Solve the KV Cache Problem

39 points by future-shock-ai

by coppsilgold

0 subcomment

There are also interesting approaches to more directly compress a large document or an entire codebase into a smaller set of tokens without getting the LLM to wing it. For example, Cartridges: <https://hazyresearch.stanford.edu/blog/2025-06-08-cartridges>
They basically get gradient descent to optimize the KV cache while freezing the network.

by az09mugen

1 subcomments

by refulgentis

0 subcomment

by LuxBennu

1 subcomments

good overview of the architecture side but worth mentioning there's another axis that stacks on top of all of this: you can quantize the kv cache itself at inference time. in llama.cpp you can run q8 for keys and q4 for values and it cuts cache memory roughly in half again on top of whatever gqa or mla already saves you. i run qwen 70b 4-bit on m2 max 96gb and the kv quant is what actually made longer contexts fit without running out of unified memory. keys need more precision because they drive attention scores but values are way more tolerant of lossy compression, so the asymmetry works out.