> For the complete documentation index, see [llms.txt](https://yeasy.gitbook.io/llm_internals/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/appendix/a4_references.md).

# A.4 推荐阅读与参考文献

## 核心论文

1. Vaswani, A., et al. (2017). [Attention Is All You Need](https://arxiv.org/abs/1706.03762). *NeurIPS 2017*.
2. Devlin, J., et al. (2019). [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805). *NAACL 2019*.
3. Radford, A., et al. (2018). [Improving Language Understanding by Generative Pre-Training](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf). *OpenAI*.
4. Brown, T., et al. (2020). [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165). *NeurIPS 2020*.
5. Raffel, C., et al. (2020). [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683). *JMLR 2020*.
6. Lewis, M., et al. (2020). [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://arxiv.org/abs/1910.13461). *ACL 2020*.
7. Liu, Y., et al. (2019). [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692).
8. Lan, Z., et al. (2019). [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942).
9. Clark, K., et al. (2020). [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://arxiv.org/abs/2003.10555). *ICLR 2020*.
10. Beltagy, I., et al. (2020). [Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150).
11. Zaheer, M., et al. (2020). [Big Bird: Transformers for Longer Sequences](https://arxiv.org/abs/2007.14062). *NeurIPS 2020*.
12. Kaplan, J., et al. (2020). [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361).
13. Hoffmann, J., et al. (2022). [Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556). *Chinchilla*.
14. Chowdhery, A., et al. (2022). [PaLM: Scaling Language Modeling with Pathways](https://arxiv.org/abs/2204.02311). *JMLR 2023*.
15. Warner, B., et al. (2024). [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (ModernBERT)](https://arxiv.org/abs/2412.13663).

## 架构改进

16. Su, J., et al. (2021). [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864).
17. Press, O., et al. (2021). [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://arxiv.org/abs/2108.12409).
18. Peng, B., et al. (2023). [YaRN: Efficient Context Window Extension of Large Language Models](https://arxiv.org/abs/2309.00071). *ICLR 2024*.
19. Geva, M., et al. (2021). [Transformer Feed-Forward Layers Are Key-Value Memories](https://arxiv.org/abs/2012.14913). *EMNLP 2021*.
20. Dao, T., et al. (2022). [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://arxiv.org/abs/2205.14135). *NeurIPS 2022*.
21. Shazeer, N. (2019). [Fast Transformer Decoding: One Write-Head is All You Need](https://arxiv.org/abs/1911.02150).
22. Ainslie, J., et al. (2023). [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://arxiv.org/abs/2305.13245). *EMNLP 2023*.
23. Jiang, A. Q., et al. (2023). [Mistral 7B](https://arxiv.org/abs/2310.06825).
24. Xiao, G., et al. (2023). [Efficient Streaming Language Models with Attention Sinks](https://arxiv.org/abs/2309.17453). *ICLR 2024*.

## 训练与对齐

25. Kingma, D. P. & Ba, J. (2014). [Adam: A Method for Stochastic Optimization](https://arxiv.org/abs/1412.6980). *ICLR 2015*.
26. Loshchilov, I. & Hutter, F. (2017). [Decoupled Weight Decay Regularization](https://arxiv.org/abs/1711.05101). *ICLR 2019*.
27. Ouyang, L., et al. (2022). [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155). *NeurIPS 2022*.
28. Rafailov, R., et al. (2023). [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/abs/2305.18290). *NeurIPS 2023*.
29. DeepSeek-AI. (2025). [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/abs/2501.12948).
30. Hu, E., et al. (2021). [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685). *ICLR 2022*.
31. Dettmers, T., et al. (2023). [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314). *NeurIPS 2023*.
32. Rajbhandari, S., et al. (2019). [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://arxiv.org/abs/1910.02054).
33. Microsoft DeepSpeed Team. (2021). [DeepSpeed: Accelerating large-scale model inference and training via system optimizations and compression](https://www.deepspeed.ai/2021/05/14/inference-release.html).
34. Shoeybi, M., et al. (2019). [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053).
35. Schulman, J., et al. (2017). [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347).
36. Bai, Y., et al. (2022). [Constitutional AI: Harmlessness from AI Feedback](https://arxiv.org/abs/2212.08073).
37. Zhou, C., et al. (2023). [LIMA: Less Is More for Alignment](https://arxiv.org/abs/2305.11206). *NeurIPS 2023*.
38. Shao, Z., et al. (2024). [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://arxiv.org/abs/2402.03300).
39. Ethayarajh, K., et al. (2024). [KTO: Model Alignment as Prospect Theoretic Optimization](https://arxiv.org/abs/2402.01306).
40. Hong, J., et al. (2024). [ORPO: Monolithic Preference Optimization without Reference Model](https://arxiv.org/abs/2403.07691).
41. Liu, S.-Y., et al. (2024). [DoRA: Weight-Decomposed Low-Rank Adaptation](https://arxiv.org/abs/2402.09353). *ICML 2024*.
42. Hayou, S., et al. (2024). [LoRA+: Efficient Low Rank Adaptation of Large Models](https://arxiv.org/abs/2402.12354).
43. Kalajdzievski, D. (2023). [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (rsLoRA)](https://arxiv.org/abs/2312.03732).
44. Goyal, P., et al. (2017). [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://arxiv.org/abs/1706.02677).
45. OpenAI. (2026). [IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs](https://arxiv.org/abs/2603.10521).
46. Kalajdzievski, D. (2024). [Scaling Laws for Forgetting When Fine-Tuning Large Language Models](https://arxiv.org/abs/2401.05605).
47. Biderman, D., et al. (2024). [LoRA Learns Less and Forgets Less](https://arxiv.org/abs/2405.09673). *TMLR 2024*.
48. Gudibande, A., et al. (2023). [The False Promise of Imitating Proprietary LLMs](https://arxiv.org/abs/2305.15717).
49. Yang, S., et al. (2023). [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://arxiv.org/abs/2311.04850).
50. Zhang, H., et al. (2024). [A Careful Examination of Large Language Model Performance on Grade School Arithmetic](https://arxiv.org/abs/2405.00332). *NeurIPS 2024*.
51. Springer, J. M., et al. (2025). [Overtrained Language Models Are Harder to Fine-Tune](https://arxiv.org/abs/2503.19206).

## 推理优化

52. Kwon, W., et al. (2023). [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180). *SOSP 2023*.
53. Leviathan, Y., et al. (2023). [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192). *ICML 2023*.
54. Cai, T., et al. (2024). [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://arxiv.org/abs/2401.10774).
55. Li, Y., et al. (2024). [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://arxiv.org/abs/2401.15077).
56. Nguyen, M., et al. (2024). [Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs](https://arxiv.org/abs/2407.01082).
57. Liu, H., et al. (2023). [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://arxiv.org/abs/2310.01889).
58. Milakov, M. & Gimelshein, N. (2018). [Online normalizer calculation for softmax](https://arxiv.org/abs/1805.02867). *NVIDIA*.
59. Zadouri, T., et al. (2026). [FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling](https://arxiv.org/abs/2603.05451).
60. Agrawal, A., et al. (2024). [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://arxiv.org/abs/2403.02310). *OSDI 2024*.
61. Keskar, N. S., et al. (2019). [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://arxiv.org/abs/1909.05858).
62. Wu, Y., et al. (2016). [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://arxiv.org/abs/1609.08144).
63. Xiao, G., et al. (2022). [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://arxiv.org/abs/2211.10438). *ICML 2023*.
64. Qin, R., et al. (2024). [Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving](https://arxiv.org/abs/2407.00079).

## 前沿架构

65. Brandon, W., et al. (2024). [Reducing Transformer Key-Value Cache Size with Cross-Layer Attention](https://arxiv.org/abs/2405.12981).
66. Gu, A. & Dao, T. (2023; revised 2024). [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752).
67. DeepSeek-AI. (2024). [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://arxiv.org/abs/2405.04434).
68. DeepSeek-AI. (2024). [DeepSeek-V3 Technical Report](https://arxiv.org/abs/2412.19437).
69. Fedus, W., et al. (2021). [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://arxiv.org/abs/2101.03961). *JMLR 2022*.
70. Wei, J., et al. (2022). [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903). *NeurIPS 2022*.
71. Yao, S., et al. (2022). [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629). *ICLR 2023*.
72. Yao, S., et al. (2023). [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://arxiv.org/abs/2305.10601).
73. Besta, M., et al. (2023). [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://arxiv.org/abs/2308.09687).
74. Shazeer, N., et al. (2017). [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://arxiv.org/abs/1701.06538). *ICLR 2017*.
75. Jiang, A. Q., et al. (2024). [Mixtral of Experts](https://arxiv.org/abs/2401.04088).
76. Kojima, T., et al. (2022). [Large Language Models are Zero-Shot Reasoners](https://arxiv.org/abs/2205.11916). *NeurIPS 2022*.
77. Lightman, H., et al. (2023). [Let's Verify Step by Step](https://arxiv.org/abs/2305.20050).
78. Hao, S., et al. (2024). [Training Large Language Models to Reason in a Continuous Latent Space (Coconut)](https://arxiv.org/abs/2412.06769).
79. Alayrac, J.-B., et al. (2022). [Flamingo: a Visual Language Model for Few-Shot Learning](https://arxiv.org/abs/2204.14198). *NeurIPS 2022*.
80. Li, J., et al. (2023). [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://arxiv.org/abs/2301.12597). *ICML 2023*.
81. Liu, H., et al. (2023). [Visual Instruction Tuning (LLaVA)](https://arxiv.org/abs/2304.08485). *NeurIPS 2023*.
82. Hsieh, C.-P., et al. (2024). [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://arxiv.org/abs/2404.06654).
83. Bai, Y., et al. (2023). [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://arxiv.org/abs/2308.14508). *ACL 2024*.
84. Zhang, X., et al. (2024). [InfiniteBench: Extending Long Context Evaluation Beyond 100K Tokens](https://arxiv.org/abs/2402.13718). *ACL 2024*.
85. Liu, N. F., et al. (2023). [Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172). *TACL 2024*.

## 教程与可视化

86. Jay Alammar. [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/).
87. Jay Alammar. [The Illustrated GPT-2 (Visualizing Transformer Language Models)](https://jalammar.github.io/illustrated-gpt2/).
88. Cho, A., et al. (2024). [Transformer Explainer: Learning LLM Transformers with Interactive Visual Explanation and Experimentation](https://arxiv.org/abs/2408.04619). 在线交互版见 [poloclub.github.io/transformer-explainer](https://poloclub.github.io/transformer-explainer/)。
89. Lilian Weng. [The Transformer Family](https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/).
90. HuggingFace. [Transformers Documentation](https://huggingface.co/docs/transformers/).
91. Sebastian Raschka. [LLMs-from-scratch](https://github.com/rasbt/LLMs-from-scratch)（*Build a Large Language Model (From Scratch)*, Manning, 2024 的官方配套代码，Apache-2.0）。本书讲原理，该仓库提供可逐步运行的 PyTorch 实现，二者互补；其 `ch04` 下的 GQA、MLA、SWA、跨层 KV 共享等对照实现，可与[第 10.2 节](/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/10.2_kv_cache.md)的四条压缩轴对读。

## 推荐书籍

92. Jurafsky, D. & Martin, J.H. *Speech and Language Processing* (3rd ed.). 第 10 章 Transformer 部分。
93. 邱锡鹏. 《神经网络与深度学习》. 第 15 章注意力机制与 Transformer。

## 官方模型、硬件与工程资料

94. OpenAI. [GPT-5 Model Documentation](https://developers.openai.com/api/docs/models/gpt-5).
95. OpenAI. [Retiring GPT-4o and other ChatGPT models](https://help.openai.com/en/articles/20001051).
96. Meta AI. [The Llama 4 herd](https://ai.meta.com/blog/llama-4-multimodal-intelligence/).
97. Google. [Introducing Gemini 1.5](https://blog.google/innovation-and-ai/products/google-gemini-next-generation-model-february-2024/).
98. Google Cloud. [Our eighth generation TPUs: TPU 8t and TPU 8i](https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/eighth-generation-tpu-agentic-era/).
99. NVIDIA. [H100 GPU Product Specifications](https://www.nvidia.com/en-us/data-center/h100/).
100. PyTorch. [DistributedDataParallel Documentation](https://docs.pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html).
101. DeepSpeed. [ZeRO Documentation](https://deepspeed.readthedocs.io/en/stable/zero3.html).
102. Qwen Team. [Qwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated Parameters](https://qwenlm.github.io/blog/qwen-moe/).
103. Qwen. [Qwen2-57B-A14B Model Card](https://huggingface.co/Qwen/Qwen2-57B-A14B).
104. Qwen Team. [Qwen2.5 Technical Report](https://arxiv.org/abs/2412.15115).
105. Chen, J., Liang, Y. & Liu, Z. (2026). [DFlash: Block Diffusion for Flash Speculative Decoding](https://arxiv.org/abs/2602.06036). *ICML 2026*.
106. Cheng, X., et al. (2026). [DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation](https://arxiv.org/abs/2607.05147). *DeepSeek*.
107. Austin, J., et al. (2021). [Structured Denoising Diffusion Models in Discrete State-Spaces](https://arxiv.org/abs/2107.03006). *NeurIPS 2021*.
108. Lou, A., Meng, C. & Ermon, S. (2024). [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://arxiv.org/abs/2310.16834). *ICML 2024*.
109. Sahoo, S., et al. (2024). [Simple and Effective Masked Diffusion Language Models](https://arxiv.org/abs/2406.07524).
110. Nie, S., et al. (2025). [Large Language Diffusion Models](https://arxiv.org/abs/2502.09992).
111. Arriola, M., et al. (2025). [Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models](https://arxiv.org/abs/2503.09573).
112. Chen, Y., et al. (2025). [Reasoning Models Don't Always Say What They Think](https://arxiv.org/abs/2505.05410). *Anthropic*.
113. Google DeepMind (2026). [DiffusionGemma Model Card](https://ai.google.dev/gemma/docs/diffusiongemma/model_card).
114. Lei, Y., et al. (2026). [The Energy Cost of Execution-Idle in GPU Clusters](https://arxiv.org/abs/2604.04745).
115. Zhong, T., et al. (2026). [Diagnosing Training Inference Mismatch in LLM Reinforcement Learning](https://arxiv.org/abs/2605.14220). *ByteDance & University of Virginia*.

## 本轮加深新增

以下条目在第 7、9、10、11 章加深时引入，题目与作者经 arXiv API 核对。

116. Santurkar, S., et al. (2018). [How Does Batch Normalization Help Optimization?](https://arxiv.org/abs/1805.11604). *NeurIPS*
117. Grattafiori, A., et al. (2024). [The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783).
118. Abbas, A., et al. (2023). [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://arxiv.org/abs/2303.09540).
119. Muennighoff, N., et al. (2023). [Scaling Data-Constrained Language Models](https://arxiv.org/abs/2305.16264).
120. Shumailov, I., et al. (2023). [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://arxiv.org/abs/2305.17493).
121. Shi, W., et al. (2023). [Detecting Pretraining Data from Large Language Models](https://arxiv.org/abs/2310.16789).
122. McCandlish, S., et al. (2018). [An Empirical Model of Large-Batch Training](https://arxiv.org/abs/1812.06162).
123. Zoph, B., et al. (2022). [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://arxiv.org/abs/2202.08906).
124. Dehghani, M., et al. (2023). [Scaling Vision Transformers to 22 Billion Parameters](https://arxiv.org/abs/2302.05442).
125. Team, C. (2024). [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://arxiv.org/abs/2405.09818).
126. Team, G., et al. (2024). [Gemma 2: Improving Open Language Models at a Practical Size](https://arxiv.org/abs/2408.00118).
127. Liu, J., et al. (2025). [Muon is Scalable for LLM Training](https://arxiv.org/abs/2502.16982).
128. Team, G., et al. (2025). [Gemma 3 Technical Report](https://arxiv.org/abs/2503.19786).
129. Team, K., et al. (2025). [Kimi K2: Open Agentic Intelligence](https://arxiv.org/abs/2507.20534).
130. Wang, S., et al. (2026). [Why Muon Outperforms Adam: A Curvature Perspective](https://arxiv.org/abs/2606.04662).
131. Team, K., et al. (2026). [Kimi K3: Open Frontier Intelligence](https://arxiv.org/abs/2607.24653).
132. Chen, T., et al. (2016). [Training Deep Nets with Sublinear Memory Cost](https://arxiv.org/abs/1604.06174).
133. Micikevicius, P., et al. (2017). [Mixed Precision Training](https://arxiv.org/abs/1710.03740). *ICLR 2018*
134. Harlap, A., et al. (2018). [PipeDream: Fast and Efficient Pipeline Parallel DNN Training](https://arxiv.org/abs/1806.03377).
135. Shazeer, N., et al. (2018). [Mesh-TensorFlow: Deep Learning for Supercomputers](https://arxiv.org/abs/1811.02084).
136. Huang, Y., et al. (2018). [GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism](https://arxiv.org/abs/1811.06965).
137. Narayanan, D., et al. (2020). [Memory-Efficient Pipeline-Parallel DNN Training](https://arxiv.org/abs/2006.09503). *ICML 2021*
138. Li, S., et al. (2020). [PyTorch Distributed: Experiences on Accelerating Data Parallel Training](https://arxiv.org/abs/2006.15704).
139. Ren, J., et al. (2021). [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://arxiv.org/abs/2101.06840).
140. Narayanan, D., et al. (2021). [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://arxiv.org/abs/2104.04473).
141. Rajbhandari, S., et al. (2021). [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://arxiv.org/abs/2104.07857).
142. Korthikanti, V., et al. (2022). [Reducing Activation Recomputation in Large Transformer Models](https://arxiv.org/abs/2205.05198).
143. Micikevicius, P., et al. (2022). [FP8 Formats for Deep Learning](https://arxiv.org/abs/2209.05433).
144. Zhao, Y., et al. (2023). [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://arxiv.org/abs/2304.11277).
145. Qi, P., et al. (2023). [Zero Bubble Pipeline Parallelism](https://arxiv.org/abs/2401.10241).
146. NVIDIA, N., et al. (2025). [Pretraining Large Language Models with NVFP4](https://arxiv.org/abs/2509.25149).
147. DeepSeek-AI, D., et al. (2026). [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://arxiv.org/abs/2606.19348).
148. Touvron, H., et al. (2023). [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/abs/2307.09288).
149. Sheng, G., et al. (2024). [HybridFlow: A Flexible and Efficient RLHF Framework](https://arxiv.org/abs/2409.19256).
150. Yu, Q., et al. (2025). [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://arxiv.org/abs/2503.14476).
151. Liu, Z., et al. (2025). [Understanding R1-Zero-Like Training: A Critical Perspective](https://arxiv.org/abs/2503.20783).
152. Vijayakumar, A., et al. (2016). [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://arxiv.org/abs/1610.02424). *AAAI 2018*
153. Koehn, P., et al. (2017). [Six Challenges for Neural Machine Translation](https://arxiv.org/abs/1706.03872).
154. Fan, A., et al. (2018). [Hierarchical Neural Story Generation](https://arxiv.org/abs/1805.04833).
155. Holtzman, A., et al. (2019). [The Curious Case of Neural Text Degeneration](https://arxiv.org/abs/1904.09751). *ICLR 2020*
156. Stahlberg, F., et al. (2019). [On NMT Search Errors and Model Errors: Cat Got Your Tongue?](https://arxiv.org/abs/1908.10090). *EMNLP 2019*
157. Basu, S., et al. (2020). [Mirostat: A Neural Text Decoding Algorithm that Directly Controls Perplexity](https://arxiv.org/abs/2007.14966).
158. Meister, C., et al. (2022). [Locally Typical Sampling](https://arxiv.org/abs/2202.00666). *TACL 2022*
159. Wang, X., et al. (2022). [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://arxiv.org/abs/2203.11171). *ICLR 2023*
160. Gao, L., et al. (2022). [Scaling Laws for Reward Model Overoptimization](https://arxiv.org/abs/2210.10760).
161. Hewitt, J., et al. (2022). [Truncation Sampling as Language Model Desmoothing](https://arxiv.org/abs/2210.15191). *EMNLP*
162. Kirchenbauer, J., et al. (2023). [A Watermark for Large Language Models](https://arxiv.org/abs/2301.10226). *ICML 2023*
163. Park, K., et al. (2024). [Grammar-Aligned Decoding](https://arxiv.org/abs/2405.21047). *NeurIPS 2024*
164. Brown, B., et al. (2024). [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://arxiv.org/abs/2407.21787).
165. Snell, C., et al. (2024). [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://arxiv.org/abs/2408.03314).
166. Muennighoff, N., et al. (2025). [s1: Simple test-time scaling](https://arxiv.org/abs/2501.19393).
167. Ma, X., et al. (2025). [dKV-Cache: The Cache for Diffusion Language Models](https://arxiv.org/abs/2505.15781).
168. Wu, C., et al. (2025). [Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding](https://arxiv.org/abs/2505.22618).
169. Hinton, G., et al. (2015). [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531).
170. Kim, Y., et al. (2016). [Sequence-Level Knowledge Distillation](https://arxiv.org/abs/1606.07947). *EMNLP 2016*
171. Mirzadeh, S., et al. (2019). [Improved Knowledge Distillation via Teacher Assistant](https://arxiv.org/abs/1902.03393). *AAAI 2020*
172. Mishra, A., et al. (2021). [Accelerating Sparse Deep Neural Networks](https://arxiv.org/abs/2104.08378).
173. Dettmers, T., et al. (2022). [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339). *NeurIPS 2022*
174. Frantar, E., et al. (2022). [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323). *ICLR 2023*
175. Frantar, E., et al. (2023). [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://arxiv.org/abs/2301.00774).
176. Chen, C., et al. (2023). [Accelerating Large Language Model Decoding with Speculative Sampling](https://arxiv.org/abs/2302.01318).
177. Lin, J., et al. (2023). [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://arxiv.org/abs/2306.00978). *MLSys 2024*
178. Gu, Y., et al. (2023). [MiniLLM: On-Policy Distillation of Large Language Models](https://arxiv.org/abs/2306.08543). *ICLR 2024*
179. Sun, M., et al. (2023). [A Simple and Effective Pruning Approach for Large Language Models](https://arxiv.org/abs/2306.11695). *ICLR 2024*
180. Agarwal, R., et al. (2023). [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://arxiv.org/abs/2306.13649). *ICLR 2024*
181. Dao, T. (2023). [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://arxiv.org/abs/2307.08691).
182. Jacobs, S., et al. (2023). [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://arxiv.org/abs/2309.14509).
183. Rouhani, B., et al. (2023). [Microscaling Data Formats for Deep Learning](https://arxiv.org/abs/2310.10537).
184. Brandon, W., et al. (2023). [Striped Attention: Faster Ring Attention for Causal Transformers](https://arxiv.org/abs/2311.09431).
185. Fu, Y., et al. (2024). [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://arxiv.org/abs/2402.02057).
186. Liu, Z., et al. (2024). [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://arxiv.org/abs/2402.02750).
187. Gromov, A., et al. (2024). [The Unreasonable Ineffectiveness of the Deeper Layers](https://arxiv.org/abs/2403.17887). *ICLR*
188. Shah, J., et al. (2024). [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://arxiv.org/abs/2407.08608).
189. Muralidharan, S., et al. (2024). [Compact Language Models via Pruning and Knowledge Distillation](https://arxiv.org/abs/2407.14679).
190. Zhao, C., et al. (2025). [Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures](https://arxiv.org/abs/2505.09343).
191. Willard, B., et al. (2023). [Efficient Guided Generation for Large Language Models](https://arxiv.org/abs/2307.09702).
192. Chen, L., et al. (2023). [Punica: Multi-Tenant LoRA Serving](https://arxiv.org/abs/2310.18547).
193. Sheng, Y., et al. (2023). [S-LoRA: Serving Thousands of Concurrent LoRA Adapters](https://arxiv.org/abs/2311.03285).
194. Zheng, L., et al. (2023). [SGLang: Efficient Execution of Structured Language Model Programs](https://arxiv.org/abs/2312.07104).
195. Sheng, Y., et al. (2023). [Fairness in Serving Large Language Models](https://arxiv.org/abs/2401.00588).
196. Holmes, C., et al. (2024). [DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference](https://arxiv.org/abs/2401.08671).
197. Beurer-Kellner, L., et al. (2024). [Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation](https://arxiv.org/abs/2403.06988).
198. Srivatsa, V., et al. (2024). [Preble: Efficient Distributed Prompt Scheduling for LLM Serving](https://arxiv.org/abs/2407.00023).
199. Tam, Z., et al. (2024). [Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models](https://arxiv.org/abs/2408.02442).
200. Fu, Y., et al. (2024). [Efficient LLM Scheduling by Learning to Rank](https://arxiv.org/abs/2408.15792).
201. Dong, Y., et al. (2024). [XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models](https://arxiv.org/abs/2411.15100). *MLSys*
202. Team, K., et al. (2025). [Kimi Linear: An Expressive, Efficient Attention Architecture](https://arxiv.org/abs/2510.26692).
203. Pakhomov, E., et al. (2025). [Convomem Benchmark: Why Your First 150 Conversations Don't Need RAG](https://arxiv.org/abs/2511.10523).
204. Lai, X., et al. (2026). [MiniMax Sparse Attention](https://arxiv.org/abs/2606.13392).
205. Jain, S., et al. (2019). [Attention is not Explanation](https://arxiv.org/abs/1902.10186). *NAACL 2019*
206. Dong, L., et al. (2019). [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://arxiv.org/abs/1905.03197). *NeurIPS*
207. Voita, E., et al. (2019). [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://arxiv.org/abs/1905.09418). *ACL 2019*
208. Michel, P., et al. (2019). [Are Sixteen Heads Really Better than One?](https://arxiv.org/abs/1905.10650). *NeurIPS 2019*
209. Kitaev, N., et al. (2020). [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451). *ICLR 2020*
210. Bhojanapalli, S., et al. (2020). [Low-Rank Bottleneck in Multi-head Attention Models](https://arxiv.org/abs/2002.07028).
211. Dong, Y., et al. (2021). [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://arxiv.org/abs/2103.03404).
212. Sennrich, R., et al. (2015). [Neural Machine Translation of Rare Words with Subword Units](https://arxiv.org/abs/1508.07909). *ACL 2016*
213. He, K., et al. (2015). [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385).
214. Veit, A., et al. (2016). [Residual Networks Behave Like Ensembles of Relatively Shallow Networks](https://arxiv.org/abs/1605.06431).
215. Ethayarajh, K. (2019). [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://arxiv.org/abs/1909.00512). *EMNLP 2019*
216. Zhang, B., et al. (2019). [Root Mean Square Layer Normalization](https://arxiv.org/abs/1910.07467). *NeurIPS 2019*
217. Xiong, R., et al. (2020). [On Layer Normalization in the Transformer Architecture](https://arxiv.org/abs/2002.04745).
218. Shazeer, N. (2020). [GLU Variants Improve Transformer](https://arxiv.org/abs/2002.05202).
219. Shen, S., et al. (2020). [PowerNorm: Rethinking Batch Normalization in Transformers](https://arxiv.org/abs/2003.07845).
220. Meng, K., et al. (2022). [Locating and Editing Factual Associations in GPT](https://arxiv.org/abs/2202.05262). *NeurIPS 2022*
221. Wang, T., et al. (2022). [What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?](https://arxiv.org/abs/2204.05832).
222. Hase, P., et al. (2023). [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://arxiv.org/abs/2301.04213). *NeurIPS 2023*
223. Land, S., et al. (2024). [Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models](https://arxiv.org/abs/2405.05417). *EMNLP 2024*
224. Chen, S., et al. (2023). [Extending Context Window of Large Language Models via Positional Interpolation](https://arxiv.org/abs/2306.15595).
225. Rozière, B., et al. (2023). [Code Llama: Open Foundation Models for Code](https://arxiv.org/abs/2308.12950).
226. Joshi, M., et al. (2019). [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://arxiv.org/abs/1907.10529). *TACL*
227. Lee, K., et al. (2021). [Deduplicating Training Data Makes Language Models Better](https://arxiv.org/abs/2107.06499). *ACL 2022*
228. Tay, Y., et al. (2022). [UL2: Unifying Language Learning Paradigms](https://arxiv.org/abs/2205.05131).
229. Xie, S., et al. (2023). [DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining](https://arxiv.org/abs/2305.10429). *NeurIPS 2023*
230. Berglund, L., et al. (2023). [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://arxiv.org/abs/2309.12288).
231. Sardana, N., et al. (2023). [Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://arxiv.org/abs/2401.00448).
232. Besiroglu, T., et al. (2024). [Chinchilla Scaling: A replication attempt](https://arxiv.org/abs/2404.10102).
233. Penedo, G., et al. (2024). [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://arxiv.org/abs/2406.17557).
234. Shazeer, N., et al. (2018). [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://arxiv.org/abs/1804.04235).
235. Zhang, J., et al. (2019). [Why are Adaptive Methods Good for Attention Models?](https://arxiv.org/abs/1912.03194).
236. Dettmers, T., et al. (2021). [8-bit Optimizers via Block-wise Quantization](https://arxiv.org/abs/2110.02861).
237. Yang, G., et al. (2022). [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://arxiv.org/abs/2203.03466). *NeurIPS 2021*
238. Hu, S., et al. (2024). [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://arxiv.org/abs/2404.06395).
239. Qi, X., et al. (2023). [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://arxiv.org/abs/2310.03693).
240. Azar, M., et al. (2023). [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://arxiv.org/abs/2310.12036).
241. Meng, Y., et al. (2024). [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://arxiv.org/abs/2405.14734). *NeurIPS 2024*
242. Razin, N., et al. (2024). [Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization](https://arxiv.org/abs/2410.08847). *ICLR 2025*
243. Lambert, N., et al. (2024). [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/abs/2411.15124).
244. Zheng, C., et al. (2025). [Group Sequence Policy Optimization](https://arxiv.org/abs/2507.18071).
245. Pascanu, R., et al. (2012). [On the difficulty of training Recurrent Neural Networks](https://arxiv.org/abs/1211.5063).
246. Mikolov, T., et al. (2013). [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/abs/1301.3781).
247. Bahdanau, D., et al. (2014). [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473). *ICLR 2015*
248. Cho, K., et al. (2014). [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://arxiv.org/abs/1409.1259).
249. Sutskever, I., et al. (2014). [Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215).
250. Luong, M., et al. (2015). [Effective Approaches to Attention-based Neural Machine Translation](https://arxiv.org/abs/1508.04025). *EMNLP 2015*
251. Gehring, J., et al. (2017). [Convolutional Sequence to Sequence Learning](https://arxiv.org/abs/1705.03122).
252. Tenney, I., et al. (2019). [BERT Rediscovers the Classical NLP Pipeline](https://arxiv.org/abs/1905.05950). *ACL 2019*
253. Dosovitskiy, A., et al. (2020). [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/abs/2010.11929).
254. Touvron, H., et al. (2023). [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971).
255. Yang, Z., et al. (2019). [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237).
256. Reimers, N., et al. (2019). [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/abs/1908.10084). *EMNLP 2019*
257. Karpukhin, V., et al. (2020). [Dense Passage Retrieval for Open-Domain Question Answering](https://arxiv.org/abs/2004.04906). *EMNLP 2020*
258. He, P., et al. (2020). [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://arxiv.org/abs/2006.03654).
259. Xiong, L., et al. (2020). [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://arxiv.org/abs/2007.00808).
260. He, P., et al. (2021). [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://arxiv.org/abs/2111.09543).
261. Wettig, A., et al. (2022). [Should You Mask 15% in Masked Language Modeling?](https://arxiv.org/abs/2202.08005).
262. OpenAI, O., et al. (2023). [GPT-4 Technical Report](https://arxiv.org/abs/2303.08774).
263. Schaeffer, R., et al. (2023). [Are Emergent Abilities of Large Language Models a Mirage?](https://arxiv.org/abs/2304.15004).
264. Team, G., et al. (2024). [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://arxiv.org/abs/2403.05530).
265. OpenAI, O., et al. (2025). [gpt-oss-120b & gpt-oss-20b Model Card](https://arxiv.org/abs/2508.10925).
266. Katharopoulos, A., et al. (2020). [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://arxiv.org/abs/2006.16236). *ICML 2020*
267. Merrill, W., et al. (2023). [The Expressive Power of Transformers with Chain of Thought](https://arxiv.org/abs/2310.07923). *ICLR*
268. Lieber, O., et al. (2024). [Jamba: A Hybrid Transformer-Mamba Language Model](https://arxiv.org/abs/2403.19887).
269. Allen-Zhu, Z., et al. (2024). [Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws](https://arxiv.org/abs/2404.05405).
270. Dao, T., et al. (2024). [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://arxiv.org/abs/2405.21060). *ICML 2024*
271. Gao, L., et al. (2024). [Scaling and evaluating sparse autoencoders](https://arxiv.org/abs/2406.04093).
272. Lieberum, T., et al. (2024). [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://arxiv.org/abs/2408.05147).
273. Kalai, A., et al. (2025). [Why Language Models Hallucinate](https://arxiv.org/abs/2509.04664).
274. Jiralerspong, T., et al. (2026). [Cross-Architecture Model Diffing with Crosscoders: Unsupervised Discovery of Differences Between LLMs](https://arxiv.org/abs/2602.11729).
275. Dauphin, Y. N., et al. (2016). [Language Modeling with Gated Convolutional Networks](https://arxiv.org/abs/1612.08083). *ICML 2017*
276. Ramachandran, P., et al. (2017). [Searching for Activation Functions](https://arxiv.org/abs/1710.05941).
277. Qiu, Z., et al. (2025). [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://arxiv.org/abs/2505.06708).
278. Ziegler, D. M., et al. (2019). [Fine-Tuning Language Models from Human Preferences](https://arxiv.org/abs/1909.08593).
279. Stiennon, N., et al. (2020). [Learning to Summarize from Human Feedback](https://arxiv.org/abs/2009.01325). *NeurIPS 2020*