https://iharee.github.io/tags/%E5%81%8F%E5%A5%BD%E4%BC%98%E5%8C%96/Recent content in 偏好优化 on 二三事Hugozh-CNMon, 17 Aug 2026 16:07:41 +0800https://iharee.github.io/algorithms/po/Mon, 17 Aug 2026 16:07:41 +0800https://iharee.github.io/algorithms/po/<p>本文的强化学习（Reinforcement Learning，RL）前置知识参考自 Sutton 的《<a href="http://incompleteideas.net/book/RLbook2020.pdf">Reinforcement Learning: An Introduction</a>》。</p> <h1 id="语言模型的训练过程">语言模型的训练过程</h1> <p>在前几篇文章 <a href="https://iharee.github.io/algorithms/mlp/" style="color:#7799bb; font-weight: bold; &:active{color:#0066cc;} &:focus{color:#0066cc;}">MLP 与 BP 算法的数学原理</a>、<a href="https://iharee.github.io/algorithms/attention_and_transformer" style="color:#7799bb; font-weight: bold; &:active{color:#0066cc;} &:focus{color:#0066cc;}">经典注意力与经典 Transformer 的数学原理</a> 与 <a href="https://iharee.github.io/algorithms/moe" style="color:#7799bb; font-weight: bold; &:active{color:#0066cc;} &:focus{color:#0066cc;}">Mixture of Experts 的数学原理（~2026 文献综述）</a> 中，我们分别在理论上讨论了 BP 算法、注意力与 Transformer 以及 MoE 架构。本文希望进一步讨论语言模型中偏好优化的数学模型。</p> <hr> <p>一个现代 LLM 的典型训练过程可以被概括为以下几点关键步骤：</p> <ol> <li> <p>Pretraining（预训练）：使用海量的文本语料作为训练样本，按自回归语言建模目标 </p> <span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>P</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">x</mi><mo separator="true">;</mo><mi mathvariant="bold-italic">θ</mi><mo stretchy="false">)</mo><mo>=</mo><munderover><mo>∏</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mi>T</mi></munderover><mi>P</mi><mo stretchy="false">(</mo><msub><mi>x</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><msub><mi mathvariant="bold-italic">x</mi><mrow><mo>&lt;</mo><mi>t</mi></mrow></msub><mo separator="true">;</mo><mi mathvariant="bold-italic">θ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex"> P(\boldsymbol{x};\boldsymbol{\theta})=\prod_{t=1}^{T}P(x_t|\boldsymbol{x}_{&lt;t};\boldsymbol{\theta}) </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal" style="margin-right:0.13889em;">P</span><span class="mopen">(</span><span class="mord"><span class="mord"><span class="mord boldsymbol">x</span></span></span><span class="mpunct">;</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord"><span class="mord"><span class="mord boldsymbol" style="margin-right:0.03194em;">θ</span></span></span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:3.0954em;vertical-align:-1.2671em;"></span><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.8283em;"><span style="top:-1.8829em;margin-left:0em;"><span class="pstrut" style="height:3.05em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight">t</span><span class="mrel mtight">=</span><span class="mord mtight">1</span></span></span></span><span style="top:-3.05em;"><span class="pstrut" style="height:3.05em;"></span><span><span class="mop op-symbol large-op">∏</span></span></span><span style="top:-4.3em;margin-left:0em;"><span class="pstrut" style="height:3.05em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.13889em;">T</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:1.2671em;"><span></span></span></span></span></span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord mathnormal" style="margin-right:0.13889em;">P</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal">x</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord">∣</span><span class="mord"><span class="mord"><span class="mord"><span class="mord boldsymbol">x</span></span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mrel mtight">&lt;</span><span class="mord mathnormal mtight">t</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.1774em;"><span></span></span></span></span></span></span><span class="mpunct">;</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord"><span class="mord"><span class="mord boldsymbol" style="margin-right:0.03194em;">θ</span></span></span><span class="mclose">)</span></span></span></span></span><p> 训练模型，使模型学会根据已有上文预测下一个 token，获得基本的语言表达和推理能力。在完成这一步训练后，模型虽然已经具有大量语言、知识及一定的推理能力，但尚不能稳定地遵循人类指令或作为「AI 助手」交互。</p>