Sources

1. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. https://arxiv.org/abs/2503.01840
2. Accelerating Large Language Model Decoding with Speculative Sampling. https://arxiv.org/abs/2302.01318
3. Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation. https://arxiv.org/abs/2203.16487
4. Online Speculative Decoding. https://arxiv.org/abs/2310.07177
5. Accelerating LLM Inference with Staged Speculative Decoding. https://arxiv.org/abs/2308.04623
6. ParallelSpec: Parallel Drafter for Efficient Speculative Decoding. https://arxiv.org/abs/2410.05589
7. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. https://arxiv.org/abs/2401.10774
8. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. https://arxiv.org/abs/2205.14135
9. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. https://arxiv.org/abs/2307.08691
10. Efficient Memory Management for Large Language Model Serving with PagedAttention. https://arxiv.org/abs/2309.06180
11. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. https://arxiv.org/abs/2210.17323
12. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. https://arxiv.org/abs/2211.10438
13. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. https://arxiv.org/abs/2305.13245
14. A Survey on Efficient Inference for Large Language Models. https://arxiv.org/abs/2404.14294
15. Mastering LLM Techniques: Inference Optimization (NVIDIA). https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
16. LLM Inference Series: KV Caching, A Deeper Look (Pierre Lienhart). https://medium.com/@plienhar/llm-inference-series-4-kv-caching-a-deeper-look-4ba9a77746c8
17. Fast Transformer Decoding: One Write-Head is All You Need (Multi-Query Attention). https://arxiv.org/abs/1911.02150
18. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. https://arxiv.org/abs/2405.04434
19. DeepSeek-V3 Technical Report. https://arxiv.org/abs/2412.19437
20. Tensor Product Attention Is All You Need. https://arxiv.org/abs/2501.06425
21. Mistral 7B. https://arxiv.org/abs/2310.06825
22. Layer-Condensed KV Cache for Efficient Inference of Large Language Models (LCKV). https://arxiv.org/abs/2405.10637
23. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (CLA). https://arxiv.org/abs/2405.12981
24. You Only Cache Once: Decoder-Decoder Architectures for Language Models (YOCO). https://arxiv.org/abs/2405.05254
25. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. https://arxiv.org/abs/2306.14048
26. SnapKV: LLM Knows What You are Looking for Before Generation. https://arxiv.org/abs/2404.14469
27. HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing. https://arxiv.org/abs/2412.16187
28. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. https://arxiv.org/abs/2406.02069
29. Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference. https://arxiv.org/abs/2407.11550
30. CaM: Cache Merging for Memory-Efficient LLMs Inference. https://openreview.net/forum?id=LCTmppB165
31. D2O: Dynamic Discriminative Operations for Efficient Generative Inference of Large Language Models. https://arxiv.org/abs/2406.13035
32. KVMerger: Efficiently Serving Long-Context Large Language Model by KV Cache Merging. https://arxiv.org/abs/2407.08454
33. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. https://arxiv.org/abs/2402.02750
34. Optimizing Inference for Long Context and Large Batch Sizes with NVFP4 KV Cache (NVIDIA). https://developer.nvidia.com/blog/optimizing-inference-for-long-context-and-large-batch-sizes-with-nvfp4-kv-cache
35. PALU: Compressing KV-Cache with Low-Rank Projection. https://arxiv.org/abs/2407.21118
36. Efficient Streaming Language Models with Attention Sinks (StreamingLLM). https://arxiv.org/abs/2309.17453
37. Orca: A Distributed Serving System for Transformer-Based Generative Models. https://www.usenix.org/conference/osdi22/presentation/yu
38. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. https://arxiv.org/abs/2401.09670
39. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. https://arxiv.org/abs/2311.18677
40. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. https://arxiv.org/abs/2407.00079
41. Disaggregated Inference: 18 Months Later (Hao AI Lab). https://hao-ai-lab.github.io/blogs/distserve-retro/
42. SGLang: Efficient Execution of Structured Language Model Programs. https://arxiv.org/abs/2312.07104
43. LMCache: Turboboosting vLLM with 10x Faster Access to 100x More KV Caches. https://lmcache.ai/tech_report.pdf
44. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving. https://arxiv.org/abs/2310.07240
45. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. https://arxiv.org/abs/2405.16444
46. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. https://arxiv.org/abs/2303.06865
47. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management. https://www.usenix.org/conference/osdi24/presentation/lee
48. Introducing NVIDIA Dynamo: A Low-Latency Distributed Inference Framework. https://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/
49. MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding. https://arxiv.org/abs/2408.11049
50. KV Cache Compression for Long-Context Reasoning Models. https://arxiv.org/abs/2512.12008
51. Prompt Leakage Effect and Defense for KV-Cache Sharing in Multi-Turn Conversations (NDSS 2025). https://www.ndss-symposium.org/ndss-paper/prompt-leakage-effect-and-defense-for-kv-cache-sharing/