← 笔记Notes2026.10.05 · 2 分钟min

偏差–方差权衡，不是你能选的那件事The bias–variance trade-off is not a trade-off you get to make

The bias–variance trade-off is not a trade-off you get to make偏差–方差权衡，不是你能选的那件事

统计Statistics笔记Note 01

第三周就会学到的那个分解，以及其中没人提的一点：三项里有一项不归你管。

The decomposition everyone learns in week three, and the part of it that nobody mentions: one of the three terms is not yours to control.

statistics
model-selection
estimation

目录Contents

分解
为什么这件事比听起来更要紧
那该怎么办
诚实的总结

每一门统计课都教同一张图：一条测试误差对模型复杂度的曲线、一个漂亮的 U 形，以及中间那个你该去找的点。这张图是对的。它通常漏掉的一点是，你并不拥有这条曲线的高度——你能选的只是自己站在它的哪个位置。

分解

设目标 $y$ 由 $y = f(x) + \epsilon$ 生成，且 $\operatorname{Var}(\epsilon) = \sigma^2$ ，那么估计量 $\hat f$ 在点 $x$ 处的期望平方误差分成三项：

E [(y - \hat{f} (x))^{2}] = σ_{noise}^{2} + Bias^{2} [\hat{f} (x)] + Var [\hat{f} (x)]

式 1 —— 期望平方预测误差的分解。

第一项与 $\hat f$ 完全无关。它属于这个世界，不属于你的模型。这带来一个不太舒服的推论：误差有一个下界，多少聪明都搬不动它。

为什么这件事比听起来更要紧

如果噪声下界相对于你在比较的那几个模型之间的差异很大，那么你的比较测的其实是噪声。具体一点：若 $\sigma^2 = 1$ ，而两个候选模型的期望误差相差 $0.02$ ，你需要一个足够大的样本，才能在一个 $1$ 的背景里分辨出 $0.02$ 。多数时候你并没有这么大的样本。

那该怎么办

先把下界估出来。 重新标注一个子集，或者把同一个模型分别拟合到互不相交的两半数据上，看两个答案之间差多远。那个距离就是你这次实验的分辨率。
在固定协议下比较。 一次只改一件事，并且把划分、预处理和评价指标保持不变。
报告分辨率，而不只是结果。 「模型 A 比模型 B 好 0.02，而标注噪声是 0.31」是一句能用的话。「模型 A 比模型 B 好」不是。

诚实的总结

偏差与方差之间的权衡是真实的，你可以在它上面移动。你做不到的是把整条曲线压到 $\sigma^2$ 以下。把这个数字——哪怕只是大致地——说清楚，会改变你此后读每一个比较的方式。

归档于统计 · 2026.10.05

目录Contents

The decomposition
Why this matters more than it sounds
What to do about it
The honest summary

Every statistics course teaches the same picture: a curve of test error against model complexity, a pleasant U-shape, and a point in the middle that you are supposed to find. The picture is correct. What it usually leaves out is that you do not get to choose the height of the curve — only your position on it.

The decomposition

For a target $y$ generated as $y = f(x) + \epsilon$ with $\operatorname{Var}(\epsilon) = \sigma^2$ , the expected squared error of an estimator $\hat f$ at a point $x$ splits into three terms:

E [(y - \hat{f} (x))^{2}] = σ_{noise}^{2} + Bias^{2} [\hat{f} (x)] + Var [\hat{f} (x)]

Equation 1 — expected squared prediction error, decomposed.

The first term does not depend on $\hat f$ at all. It is a property of the world, not of your model. This has an uncomfortable consequence: there is a floor on the error, and no amount of cleverness moves it.

Why this matters more than it sounds

If the noise floor is large relative to the differences between the models you are comparing, then your comparison is measuring noise. Concretely, if $\sigma^2 = 1$ and the two candidate models differ in expected error by $0.02$ , you need a sample large enough to resolve $0.02$ against a background of $1$ . Most of the time, you do not have it.

What to do about it

Estimate the floor first. Re-label a subset, or fit the same model to two disjoint halves, and see how far apart the answers land. That distance is the resolution of your experiment.
Compare under a fixed protocol. Change one thing at a time, and keep the split, the preprocessing and the metric identical.
Report the resolution, not just the result. "Model A beats model B by 0.02, and the annotation noise is 0.31" is a usable sentence. "Model A beats model B" is not.

The honest summary

The trade-off between bias and variance is real, and you can move along it. What you cannot do is move the curve down past $\sigma^2$ . Being explicit about that number — even roughly — changes how you read every comparison you make afterwards.

Filed under Statistics · 2026.10.05