# 大模型原理与架构 | LLM Internals

## Docs

- [大模型原理与架构](https://yeasy.gitbook.io/llm_internals/readme.md)
- [第一章：从序列建模到 Transformer](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction.md)
- [1.1 序列建模的根本挑战](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction/1.1_seq_challenge.md)
- [1.2 RNN 与 CNN：成就与瓶颈](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction/1.2_rnn_cnn_limits.md)
- [1.3 注意力的诞生：让模型学会“看哪里”](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction/1.3_attention_birth.md)
- [1.4 Transformer 的提出与核心思想](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction/1.4_transformer_idea.md)
- [1.5 里程碑时刻：从学术论文到产业变革](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction/1.5_milestones.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction/summary.md)
- [第二章：注意力机制：为什么它是核心](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention.md)
- [2.1 查询-键-值：一种信息检索的直觉](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention/2.1_qkv_intuition.md)
- [2.2 缩放点积注意力：为什么要除以根号 d](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention/2.2_scaled_dot_product.md)
- [2.3 多头注意力：为什么多个子空间更好](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention/2.3_multi_head.md)
- [2.4 自注意力、交叉注意力与因果掩码](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention/2.4_self_cross_causal.md)
- [2.5 注意力的代价：复杂度与局限](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention/2.5_complexity_limits.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention/summary.md)
- [第三章：Transformer 核心组件解析](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components.md)
- [3.1 分词：从文本到词元](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.1_tokenization.md)
- [3.2 词嵌入：从离散符号到连续向量](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.2_embedding.md)
- [3.3 位置编码：为什么顺序信息必须显式注入](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.3_position_encoding.md)
- [3.4 前馈网络：Transformer 的“记忆层”](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.4_feedforward.md)
- [3.5 残差连接：梯度为什么能流过百层网络](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.5_residual.md)
- [3.6 层归一化：为什么选择 LayerNorm 而非 BatchNorm](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.6_layer_norm.md)
- [3.7 编码器-解码器：完整架构如何协同工作](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.7_full_architecture.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/summary.md)
- [第四章：位置编码的设计哲学](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/04_position_encoding.md)
- [4.1 正弦位置编码：频率与外推的直觉](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/04_position_encoding/4.1_sinusoidal.md)
- [4.2 可学习位置编码：灵活性与局限](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/04_position_encoding/4.2_learnable.md)
- [4.3 旋转位置编码：为什么旋转能编码相对位置](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/04_position_encoding/4.3_rope.md)
- [4.4 ALiBi 与其他相对位置方案](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/04_position_encoding/4.4_alibi_others.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/04_position_encoding/summary.md)
- [第五章：预训练：为什么“预测下一个词”能学到知识](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/05_pretraining.md)
- [5.1 自回归语言模型：从左到右的世界观](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/05_pretraining/5.1_autoregressive.md)
- [5.2 掩码语言模型：完形填空的智慧](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/05_pretraining/5.2_masked_lm.md)
- [5.3 编码器-解码器预训练：两种范式的统一](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/05_pretraining/5.3_encoder_decoder.md)
- [5.4 预训练数据：规模定律与数据质量的博弈](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/05_pretraining/5.4_data_scaling.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/05_pretraining/summary.md)
- [第六章：训练技术的底层逻辑](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/06_training_techniques.md)
- [6.1 损失函数与优化器：为什么选择 Adam](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/06_training_techniques/6.1_loss_optimizer.md)
- [6.2 学习率调度：为什么需要先预热再衰减](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/06_training_techniques/6.2_lr_schedule.md)
- [6.3 正则化策略：防止过拟合的多重手段](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/06_training_techniques/6.3_regularization.md)
- [6.4 批次与序列长度：效率与质量的平衡](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/06_training_techniques/6.4_batch_sequence.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/06_training_techniques/summary.md)
- [第七章：大规模分布式训练](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training.md)
- [7.1 数据并行：为什么简单复制就能加速](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.1_data_parallel.md)
- [7.2 ZeRO 优化：如何突破单卡显存限制](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.2_zero.md)
- [7.3 模型并行与张量并行：拆分权重的艺术](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.3_model_tensor_parallel.md)
- [7.4 流水线并行与混合并行策略](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.4_pipeline_hybrid.md)
- [7.5 激活重计算：用时间换空间的艺术](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.5_activation_checkpointing.md)
- [7.6 混合精度训练：精度与速度的权衡](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.6_mixed_precision.md)
- [7.7 检查点管理与容错](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.7_checkpoint.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/summary.md)
- [第八章：从预训练到对齐：让模型有用且安全](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/08_alignment.md)
- [8.1 监督微调：教模型“怎么回答”](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/08_alignment/8.1_sft.md)
- [8.2 RLHF：为什么需要人类反馈参与训练](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/08_alignment/8.2_rlhf.md)
- [8.3 DPO 与新型对齐：从复杂到简洁的演化](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/08_alignment/8.3_dpo.md)
- [8.4 参数高效微调：为什么不必更新所有参数](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/08_alignment/8.4_peft.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/08_alignment/summary.md)
- [第九章：解码策略：模型如何生成文本](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding.md)
- [9.1 自回归解码：逐词生成的机制](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding/9.1_autoregressive_decode.md)
- [9.2 贪心搜索与束搜索：确定性与近似搜索](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding/9.2_greedy_beam.md)
- [9.3 采样策略：温度、Top-k 与 Top-p 的设计直觉](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding/9.3_sampling.md)
- [9.4 结构化输出与约束解码](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding/9.4_constrained.md)
- [9.5 解码侧的推理时扩展：生成、搜索与验证](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding/9.5_test_time_scaling.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding/summary.md)
- [第十章：推理优化：第一性原理的分析](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization.md)
- [10.1 推理瓶颈分析：计算密集还是访存密集](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/10.1_bottleneck.md)
- [10.2 KV 缓存：为什么能避免重复计算](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/10.2_kv_cache.md)
- [10.3 FlashAttention：IO 感知的算法设计](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/10.3_flash_attention.md)
- [10.4 模型量化：用更少的位数表示权重与激活值](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/10.4_quantization.md)
- [10.5 剪枝与知识蒸馏：模型瘦身的两条路](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/10.5_pruning_distillation.md)
- [10.6 投机解码：为什么“先猜后验”能加速](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/10.6_speculative_decoding.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/summary.md)
- [第十一章：推理引擎与生产部署](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving.md)
- [11.1 推理引擎架构概览](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving/11.1_engines_overview.md)
- [11.2 连续批处理与 PagedAttention](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving/11.2_continuous_batching.md)
- [11.3 分离式 Prefill-Decode 架构](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving/11.3_disaggregated_serving.md)
- [11.4 硬件选型：GPU、TPU 与专用加速器](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving/11.4_hardware.md)
- [11.5 生产部署最佳实践](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving/11.5_best_practices.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving/summary.md)
- [第十二章：编码器系列模型](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/12_encoder_models.md)
- [12.1 BERT：双向理解的突破](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/12_encoder_models/12.1_bert.md)
- [12.2 RoBERTa、ALBERT 与 ELECTRA：BERT 的改进之路](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/12_encoder_models/12.2_roberta_albert.md)
- [12.3 长文本编码器：Longformer 与 BigBird](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/12_encoder_models/12.3_longformer_bigbird.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/12_encoder_models/summary.md)
- [第十三章：解码器系列与主流 LLM](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/13_decoder_models.md)
- [13.1 GPT 系列：从语言模型到通用能力平台的扩展之路](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/13_decoder_models/13.1_gpt_series.md)
- [13.2 Llama 家族：开放权重如何改变 LLM 格局](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/13_decoder_models/13.2_llama.md)
- [13.3 DeepSeek、Gemini 与其他前沿模型](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/13_decoder_models/13.3_deepseek_gemini.md)
- [13.4 编码器-解码器模型：T5 与 BART 的设计选择](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/13_decoder_models/13.4_t5_bart.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/13_decoder_models/summary.md)
- [第十四章：架构创新与未来趋势](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends.md)
- [14.1 高效注意力：突破平方复杂度的瓶颈](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.1_efficient_attention.md)
- [14.2 混合专家模型：为什么不必激活所有参数](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.2_moe.md)
- [14.3 状态空间模型与混合架构：注意力的挑战者](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.3_ssm_hybrid.md)
- [14.4 多模态 Transformer：统一不同模态的表示](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.4_multimodal.md)
- [14.5 AI Agent 与工具调用：让模型从“说”到“做”](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.5_agent_tool_use.md)
- [14.6 推理时计算扩展：让模型学会深度思考](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.6_test_time_scaling.md)
- [14.7 长上下文技术：从理论到工程实践](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.7_long_context.md)
- [14.8 机制可解释性：打开黑箱](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.8_interpretability.md)
- [14.9 未来展望](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.9_outlook.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/summary.md)
- [附录](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/appendix.md)
- [A.1 数学基础速查](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/appendix/a1_math_basics.md)
- [A.2 PyTorch 实现示例](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/appendix/a2_pytorch_examples.md)
- [A.3 主流模型参数速查表](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/appendix/a3_model_reference.md)
- [A.4 推荐阅读与参考文献](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/appendix/a4_references.md)
- [A.5 快变事实核验表](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/appendix/a5_volatile_facts.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on a page URL with the `ask` query parameter:
```
GET https://yeasy.gitbook.io/llm_internals/readme.md?ask=<question>
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.