# 大模型原理与架构 | LLM Internals

## Docs

- [大模型原理与架构](https://yeasy.gitbook.io/llm_internals/readme.md)
- [第一章：从序列建模到 Transformer](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction.md)
- [1.1 序列建模的根本挑战](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction/1.1_seq_challenge.md)
- [1.2 RNN 与 CNN：成就与瓶颈](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction/1.2_rnn_cnn_limits.md)
- [1.3 注意力的诞生：让模型学会“看哪里”](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction/1.3_attention_birth.md)
- [1.4 Transformer 的提出与核心思想](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction/1.4_transformer_idea.md)
- [1.5 里程碑时刻：从学术论文到产业变革](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction/1.5_milestones.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/01_introduction/summary.md)
- [第二章：注意力机制：为什么它是核心](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention.md)
- [2.1 查询-键-值：一种信息检索的直觉](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention/2.1_qkv_intuition.md)
- [2.2 缩放点积注意力：为什么要除以根号 d](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention/2.2_scaled_dot_product.md)
- [2.3 多头注意力：为什么多个子空间更好](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention/2.3_multi_head.md)
- [2.4 自注意力、交叉注意力与因果掩码](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention/2.4_self_cross_causal.md)
- [2.5 注意力的代价：复杂度与局限](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention/2.5_complexity_limits.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/02_attention/summary.md)
- [第三章：Transformer 核心组件解析](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components.md)
- [3.1 分词：从文本到词元](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.1_tokenization.md)
- [3.2 词嵌入：从离散符号到连续向量](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.2_embedding.md)
- [3.3 位置编码：为什么顺序信息必须显式注入](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.3_position_encoding.md)
- [3.4 前馈网络：Transformer 的“记忆层”](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.4_feedforward.md)
- [3.5 残差连接：梯度为什么能流过百层网络](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.5_residual.md)
- [3.6 层归一化：为什么选择 LayerNorm 而非 BatchNorm](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.6_layer_norm.md)
- [3.7 编码器-解码器：完整架构如何协同工作](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/3.7_full_architecture.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/03_components/summary.md)
- [第四章：位置编码的设计哲学](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/04_position_encoding.md)
- [4.1 正弦位置编码：频率与外推的直觉](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/04_position_encoding/4.1_sinusoidal.md)
- [4.2 可学习位置编码：灵活性与局限](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/04_position_encoding/4.2_learnable.md)
- [4.3 旋转位置编码：为什么旋转能编码相对位置](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/04_position_encoding/4.3_rope.md)
- [4.4 ALiBi 与其他相对位置方案](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/04_position_encoding/4.4_alibi_others.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-yi-bu-fen-ji-chu-pian/04_position_encoding/summary.md)
- [第五章：预训练：为什么“预测下一个词”能学到知识](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/05_pretraining.md)
- [5.1 自回归语言模型：从左到右的世界观](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/05_pretraining/5.1_autoregressive.md)
- [5.2 掩码语言模型：完形填空的智慧](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/05_pretraining/5.2_masked_lm.md)
- [5.3 编码器-解码器预训练：两种范式的统一](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/05_pretraining/5.3_encoder_decoder.md)
- [5.4 预训练数据：规模定律与数据质量的博弈](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/05_pretraining/5.4_data_scaling.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/05_pretraining/summary.md)
- [第六章：训练技术的底层逻辑](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/06_training_techniques.md)
- [6.1 损失函数与优化器：为什么选择 Adam](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/06_training_techniques/6.1_loss_optimizer.md)
- [6.2 学习率调度：为什么需要先预热再衰减](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/06_training_techniques/6.2_lr_schedule.md)
- [6.3 正则化策略：防止过拟合的多重手段](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/06_training_techniques/6.3_regularization.md)
- [6.4 批次与序列长度：效率与质量的平衡](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/06_training_techniques/6.4_batch_sequence.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/06_training_techniques/summary.md)
- [第七章：大规模分布式训练](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training.md)
- [7.1 数据并行：为什么简单复制就能加速](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.1_data_parallel.md)
- [7.2 ZeRO 优化：如何突破单卡显存限制](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.2_zero.md)
- [7.3 模型并行与张量并行：拆分权重的艺术](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.3_model_tensor_parallel.md)
- [7.4 流水线并行与混合并行策略](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.4_pipeline_hybrid.md)
- [7.5 激活重计算：用时间换空间的艺术](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.5_activation_checkpointing.md)
- [7.6 混合精度训练：精度与速度的权衡](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.6_mixed_precision.md)
- [7.7 检查点管理与容错](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/7.7_checkpoint.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/07_distributed_training/summary.md)
- [第八章：从预训练到对齐：让模型有用且安全](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/08_alignment.md)
- [8.1 监督微调：教模型“怎么回答”](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/08_alignment/8.1_sft.md)
- [8.2 RLHF：为什么需要人类反馈参与训练](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/08_alignment/8.2_rlhf.md)
- [8.3 DPO 与新型对齐：从复杂到简洁的演化](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/08_alignment/8.3_dpo.md)
- [8.4 参数高效微调：为什么不必更新所有参数](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/08_alignment/8.4_peft.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-er-bu-fen-xun-lian-pian/08_alignment/summary.md)
- [第九章：解码策略：模型如何生成文本](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding.md)
- [9.1 自回归解码：逐词生成的机制](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding/9.1_autoregressive_decode.md)
- [9.2 贪心搜索与束搜索：确定性与近似搜索](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding/9.2_greedy_beam.md)
- [9.3 采样策略：温度、Top-k 与 Top-p 的设计直觉](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding/9.3_sampling.md)
- [9.4 结构化输出与约束解码](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding/9.4_constrained.md)
- [9.5 解码侧的推理时扩展：生成、搜索与验证](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding/9.5_test_time_scaling.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/09_decoding/summary.md)
- [第十章：推理优化：第一性原理的分析](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization.md)
- [10.1 推理瓶颈分析：计算密集还是访存密集](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/10.1_bottleneck.md)
- [10.2 KV 缓存：为什么能避免重复计算](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/10.2_kv_cache.md)
- [10.3 FlashAttention：IO 感知的算法设计](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/10.3_flash_attention.md)
- [10.4 模型量化：用更少的位数表示权重与激活值](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/10.4_quantization.md)
- [10.5 剪枝与知识蒸馏：模型瘦身的两条路](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/10.5_pruning_distillation.md)
- [10.6 投机解码：为什么“先猜后验”能加速](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/10.6_speculative_decoding.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/10_inference_optimization/summary.md)
- [第十一章：推理引擎与生产部署](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving.md)
- [11.1 推理引擎架构概览](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving/11.1_engines_overview.md)
- [11.2 连续批处理与 PagedAttention](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving/11.2_continuous_batching.md)
- [11.3 分离式 Prefill-Decode 架构](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving/11.3_disaggregated_serving.md)
- [11.4 硬件选型：GPU、TPU 与专用加速器](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving/11.4_hardware.md)
- [11.5 生产部署最佳实践](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving/11.5_best_practices.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-san-bu-fen-tui-li-yu-bu-shu-pian/11_serving/summary.md)
- [第十二章：编码器系列模型](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/12_encoder_models.md)
- [12.1 BERT：双向理解的突破](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/12_encoder_models/12.1_bert.md)
- [12.2 RoBERTa、ALBERT 与 ELECTRA：BERT 的改进之路](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/12_encoder_models/12.2_roberta_albert.md)
- [12.3 长文本编码器：Longformer 与 BigBird](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/12_encoder_models/12.3_longformer_bigbird.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/12_encoder_models/summary.md)
- [第十三章：解码器系列与主流 LLM](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/13_decoder_models.md)
- [13.1 GPT 系列：从语言模型到通用能力平台的扩展之路](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/13_decoder_models/13.1_gpt_series.md)
- [13.2 Llama 家族：开放权重如何改变 LLM 格局](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/13_decoder_models/13.2_llama.md)
- [13.3 DeepSeek、Gemini 与其他前沿模型](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/13_decoder_models/13.3_deepseek_gemini.md)
- [13.4 编码器-解码器模型：T5 与 BART 的设计选择](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/13_decoder_models/13.4_t5_bart.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/13_decoder_models/summary.md)
- [第十四章：架构创新与未来趋势](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends.md)
- [14.1 高效注意力：突破平方复杂度的瓶颈](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.1_efficient_attention.md)
- [14.2 混合专家模型：为什么不必激活所有参数](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.2_moe.md)
- [14.3 状态空间模型与混合架构：注意力的挑战者](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.3_ssm_hybrid.md)
- [14.4 多模态 Transformer：统一不同模态的表示](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.4_multimodal.md)
- [14.5 AI Agent 与工具调用：让模型从“说”到“做”](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.5_agent_tool_use.md)
- [14.6 推理时计算扩展：让模型学会深度思考](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.6_test_time_scaling.md)
- [14.7 长上下文技术：从理论到工程实践](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.7_long_context.md)
- [14.8 机制可解释性：打开黑箱](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.8_interpretability.md)
- [14.9 未来展望](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/14.9_outlook.md)
- [本章小结](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/14_future_trends/summary.md)
- [附录](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/appendix.md)
- [A.1 数学基础速查](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/appendix/a1_math_basics.md)
- [A.2 PyTorch 实现示例](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/appendix/a2_pytorch_examples.md)
- [A.3 主流模型参数速查表](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/appendix/a3_model_reference.md)
- [A.4 推荐阅读与参考文献](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/appendix/a4_references.md)
- [A.5 快变事实核验表](https://yeasy.gitbook.io/llm_internals/di-si-bu-fen-mo-xing-yu-qian-yan-pian/appendix/a5_volatile_facts.md)