RaymondHuang210129/llama.cpp-adaptive-kv-streaming

LLM inference in C/C++

Summary

This is an experimental fork of llama.cpp that adds "Adaptive KV Streaming V2," a memory scheme for running very long context LLMs (notably Qwen3.8-27B at 256K tokens) on a single CUDA GPU. The full KV history stays in pinned system RAM while only a working subset of resident pages plus one shared ring buffer occupy device memory, and a phase arena lends bounded views of one GPU allocation to either prefill or decode rather than holding both simultaneously. Speculative multi-token prediction (MTP) is attached as a 17th logical attention layer that borrows the same physical pool instead of allocating a second full GPU KV cache.

What it is useful for

The repository is useful when you need to serve long-context LLMs on a single consumer GPU without exhausting VRAM. The README documents qualification for one serial Qwen3.8-27B-style target on one CUDA GPU with Flash Attention and Q8_0 K / Q4_0 V caching. It also supports image requests via the matching F16 M-RoPE projector, with speculation disabled in that mode, and supports speculative multi-token decoding by giving MTP its own authoritative host history while sharing the same 17-layer physical pool.

Concrete numbers from a single run on an RTX 5070 Ti with Unsloth UD-IQ4_XS, 256/256 batches, UVM off, and a 2,240 MiB arena: a 256-token continuation at 256 Ki context measured 8.73 tok/s without MTP and 19.12 tok/s with draft length 3; at 32 Ki context the same settings measured 41.04 and 101.05 tok/s. The README also reports the effective decode KV grant as about 2,228 MiB without MTP and 2,217-2,219 MiB with draft length 3 in that arena, and a vision qualification run that settled at 15,466 MiB on the 16,303 MiB card. These are single-prompt, single-machine qualification measurements, not general speed guarantees.

How engineers can use it

Build from the repository root as documented:

cmake -S . -B build-v2 -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-v2 --target llama-server -j

A documented text-with-MTP launch sets --kv-stream-arena-mib 2240, --flash-attn on, --cache-type-k q8_0, --cache-type-v q4_0, --ctx-size 262144, plus --spec-type draft-mtp --spec-draft-n-max 3. The server derives the auxiliary KV layer count from the loaded model's NextN/MTP metadata; the legacy --kv-stream-auxiliary-layers 1 flag is optional and must match. The fixed-arena sweep can be reproduced with python3 benchmarks/run-fixed-span-sweep.py --mtp-lengths 0,1,2,3 --arena-mib 2240, and an offline harness at tools/server/tests/test_adaptive_vision.py compares arena and eager servers on deterministic local PNGs.

The README states several explicit limitations: vision plus MTP, audio/video arena execution, other accelerator streaming paths, parallel slots, separate draft models, CPU projector offload, LoRA, embeddings, and unqualified KV pairs are rejected. Multi-layer MTP metadata is detected but its streaming execution is not yet supported and returns an explicit capability error. The backend-neutral ownership contracts are documented in ggml-backend.h and ggml-backend-memory.h, but the pinned-host registration and complete streamed-attention adapter are CUDA-specific, and other accelerators need porting and their own qualification.