Skip to content
All tags

#kv-cache

2 posts
ai deep-dive The Agent Production Line

Context and Memory: Where Agents Actually Fail

Chroma tested 18 frontier models and all of them degrade as input grows — as a cliff, not a slope. Memory failures are usually retrieval failures in disguise. And the real cost of KV cache is bandwidth, not storage: every generated token reads the whole cache.

#context-engineering#memory#ai-agent#llm#kv-cache
ai guide

TurboQuant+ — Two-Stage Quantization to Compress KV Cache to 2-bit, Running 100B Models on a MacBook

TurboQuant+ is an open-source implementation of a Google Research ICLR 2026 paper that uses PolarQuant + QJL two-stage quantization to compress the KV cache by 3.8-6.4x, enabling consumer hardware to run larger models with longer contexts.

#turboquant#kv-cache#quantization#llm-inference#llama-cpp#apple-silicon