LT Blog

Readnest:我的阅读记录,为什么要搬出 Notion

Sat, 10 Oct 2026 00:00:00 +0000

很长一段时间,我的读书记录没有一个固定的地方。试了一圈现成的都不满意,我就自己写了一个,叫 Readnest。MIT 开源。

一、起因:那张表

我读书的地方很杂。

最早在微信读书,后来用掌阅精选和京东读书的学校图书馆账号,此外还有自己下载的电子书,图书馆借回来的纸质书。

所以很长一段时间里,我有一张 Notion 表。书名、作者、状态、评分、起止日期,外加一句读完的感想。刚做好时挺好用:阅读记录、计划、笔记都在一个表上。

大半年之后越用感觉越麻烦。

想知道今年读完了几本,得搭一个 rollup,改一次列名就坏一次。想知道某个平台的进度到哪了,要在四个 App 之间来回切,再把数字抄回来。最烦的是年底想回顾总结一下——得对着表格一行一行数。

还有分类这件事。同一类书,我在不同时候写成过「经管」「经管励志」「商业」三种样子。这三种写法在表格里毫无区别,在统计里就是三个类别。

阅读这件事有很多的乐趣在回顾。但是 Notion 感觉太笨重了，在手机上显示效果也不好。

于是我自己写了一个,叫 Readnest。

二、一开始就定下的三件事

这个项目有三个原则。

离线也能用。 核心功能不依赖账号,不依赖网络。保证隐私。

所有来源进来之后都长一个样。 不管这本书从哪个平台来,先整理成同一份记录再放进去。这样多支持一个来源只是多一个转换,书架、统计、档案都不用改。反过来也是同理——换平台读书的时候,记录不会跟着一起被锁死。

钥匙自己拿着。 Readnest 有些功能要用大模型,比如阅读报告。但密钥得你自己去申请、自己填进设置里。不搭服务器,也不代持。想用就用,不用就纯离线,一个功能都不会缺。

三、Readnest 现在什么样

五个栏目,从左到右。

书架。 所有来源的书进来之后是同一种卡片:封面、作者、分类、状态、评分。可以按状态筛,也可以只看某一类、某一平台。

记录。 笔记和阅读计划放在一块儿。读到一半想记点什么,随手写下;想给一本书定个期限,就建一条计划。提醒走的是手机本地通知:每日计划固定在晚上九点,读完一本书的提醒设在到期前三天。

统计。 今年读了多少、每月趋势、分类构成、评分分布、各个来源平台的占比。

阅读档案。 我本人最喜欢的一块。它基于已经读完并且评过分的书,画出你的口味分布,生成一组阅读性格标签。每个标签都能点开,看到它是从哪几个数字推出来的。可以导出成图片。

这里重点做了一些优化，比如我读书单里收藏了 200 本哲学书而尚未开始阅读，这说明不了偏好,那多半只是某个下午的一次批量导入。

设置。 一些个性化的设置。

四、书是怎么进去的

这是我最花时间的部分,因为现实里根本没有统一入口。

有开放接口的就走接口。微信读书这次意外发现有官方的开发者网关,比抓 cookie 的方案稳得多。读 goodreads、Notion 导出的 CSV,也能直接读。

没接口的,就用看得见的办法——截图。掌阅精选、京东读书专业版、本地电子书,这三个都没有公开途径,Readnest 的做法是拍一张书架截图,让手机自己读上面的字。这一步在你手机里就完成了,照片不发到任何地方。

图书馆借的纸质书,拍封面或者书脊,再去公开书目库补全作者、出版社、页数。

重复的书会自动合并。缺失的信息会被补上,但你自己写过的评分、摘要、笔记,永远不会被覆盖。这是刻意的:宁可少补一个字段,也不能把你写过的东西刷掉。

至于分类,所有平台各自的叫法进来之后都要先过一遍统一的词表。手写的”经管励志”会并入”管理”。这大概是那张 Notion 表留给我的唯一遗产。

五、AI 那一块

用到 AI 的地方,我都按一个原则来:让它做人做不了的事,别让它做人一眼就能做完的事。

拿阅读报告举例。最早我的做法很偷懒:把数据丢给模型,让它写。写出来的是”你今年读了 38 本书,平均每月 3.2 本”——这种话读者打开应用一秒就能自己看到,读完等于没读。

现在的分工是:数字全部在本地算好,模型只管说这些数字意味着什么。同样是九本,”分散在九个月”和”挤在同一个月”是两种完全不同的读者,这种判断才值得交给模型。

报告也不用你手动整理,更不用记着去看。年报会在次年你第一次打开应用时自动生成,月报同理,生成好了会提醒一声。

六、什么数据会离开你的手机

这一段必须写清楚,因为它关系到你敢不敢用。

留在手机上的:书架、进度、笔记、统计、设置,全都在本地。截图识别在你手机上跑完,图片不上传。你填写的密钥只存在系统密钥库里。

只有两种情况会往外发数据,而且都能关掉:

一是让 AI 整理书架识别结果。 这时你自己选中的那张图片会发到你自己配置的服务模型。每次导入都要单独点一次同意,随时可以关掉,关掉就退回本地识别。

二是生成阅读报告。 发出去的是统计结果,加上这个周期里相关书籍的书名、作者、分类、状态和评分——这样报告才点得出具体的书。笔记正文和划线内容不发。

另外补一句:应用里没有统计 SDK,没有广告 SDK,也没有崩溃上报。唯一的例外是截图识别用的那个 SDK 会向 Google 发送与图片内容无关的运行指标,这是 SDK 自身的行为、应用关不掉,隐私政策里写明了。完全不使用截图导入,就不会产生这类请求。

七、现在的状态

项	情况
版本	v1.0.1,Android 安装包在 GitHub Releases
语言	English、中文、Deutsch、Français、Español
皮肤	12 套,6 套纯色加 6 套贴图背景
协议	MIT
iOS	签名配置已备好,尚未上架

回头看,做这东西的起因简单得不太值一提:我只是不想再手工维护那张表了。

但做完之后发现,真正的变化不在省下的那点时间。而在于当回望的成本降到”打开就能看到”的时候,你真的会去看。一份写给自己看的年度阅读回顾,就摆在档案页里。

Readnest 免费,源码和下载都在 GitHub,产品介绍页有更多实机截图。

如果你也在好几个平台之间搬记录,希望它能让你少操一点心。

Readnest: Why I Moved My Reading Log Out of Notion

Sat, 10 Oct 2026 00:00:00 +0000

For a long time my reading records didn’t have a home. I couldn’t find an app that did this, so I built one: Readnest. MIT licensed.

1. Where it started: the table

I read in too many places.

WeRead first, then the school library accounts on Zhangyue and JD Read, plus ebooks I’ve downloaded myself, and paperbacks borrowed from the library.

For a long time, what held it together was a Notion table. Title, author, status, rating, start and finish dates, plus one line about how it landed. It worked fine at first — reading log, plans and notes all on one table.

After six months or so it just felt like more and more trouble.

To know how many books I’d finished this year I needed a rollup that broke every time I renamed a column. To know my progress on any given platform I had to switch between four apps and copy numbers across. And the worst case was year-end, when I wanted to look back and sum it up — counting rows one by one.

Then there were categories. For the same kind of book I had written, at different times, three different labels. In the table those three were indistinguishable. In any statistic they became three separate categories.

There’s a lot of pleasure in reading that comes from looking back. But Notion felt too heavy, and it didn’t display well on a phone.

So I built my own. It’s called Readnest.

2. Three things I decided before writing any code

Three rules for this project.

It has to work offline. Nothing essential depends on an account or a network. That’s what keeps it private.

Every source looks the same once it’s in. Wherever a book comes from, it gets normalized into one record before it’s stored. Supporting a new source means writing one adapter; the shelf, the statistics and the archives never change. It works the other way too — when you switch platforms, your records aren’t locked in with them.

You hold your own keys. Some features use a large language model, reading reports for instance. But you apply for the key yourself and paste it into settings. No server on my side, and I hold nothing for you. Use it or don’t; everything else works fully offline.

3. What Readnest looks like now

Five tabs, left to right.

Shelf. Books from every source arrive as the same card: cover, author, category, status, rating. Filter by status, or narrow it to one category or one platform.

Journal. Notes and reading plans together. Jot something down mid-book, or set yourself a deadline for one. Reminders are plain local notifications: daily plans ping at nine in the evening, single-book deadlines warn three days ahead.

Statistics. Books this year, monthly trend, category mix, rating spread, share by source platform.

Archive. My favourite section. Based on books you’ve finished and rated, it draws your taste distribution and generates a set of reading-personality tags. Every tag can be tapped to see which numbers it came from. Export it as an image if you like.

A lot of work went into this. Two hundred philosophy books sitting unread in your wishlist says nothing about your taste — it usually just means one afternoon of bulk importing.

Settings. Some personal settings.

4. How books get in

This is where most of the work went, because real life offers no single door.

Where there’s an open interface, use it. WeRead turned out to have an official developer gateway, far more stable than the cookie-scraping approaches. Exports from Goodreads or Notion can be read directly.

Where there isn’t one, use the obvious thing: a screenshot. Zhangyue, JD Read and locally stored ebooks offer no public route, so Readnest takes a picture of your shelf and lets the phone read the text off it itself. That happens on your device; the photo goes nowhere.

For library paperbacks, photograph the cover or the spine, then pull author, publisher and page count from a public catalogue.

Duplicates merge automatically, and missing fields get filled in — but anything you wrote yourself, a rating, a note, a summary, is never overwritten. That’s on purpose. Better to leave one field empty than to wipe something you wrote.

Every platform’s own category names go through a shared vocabulary on the way in. The “self-help business” you typed by hand folds into “Management”. That lesson came straight from the Notion table.

5. The AI parts

I apply one rule to everything that uses a model: give it the things humans can’t do, not the things they can do in a second.

Reading reports are the clearest case. My first approach was lazy — hand the data over and let it write. What came back was “you read 38 books this year, 3.2 a month”, which is exactly what a reader can see for themselves by opening the app. Reading it taught nobody anything.

Now the split is: every number is computed locally, and the model only says what those numbers mean. Nine books spread across nine months and nine books crammed into one are two entirely different readers, and that judgement is worth handing over.

You don’t have to assemble reports yourself, or remember to go looking. The annual report generates itself the first time you open the app in the new year, same for the monthly one, and you get a nudge when it’s ready.

6. What data leaves your phone

This part has to be clear, because it’s the difference between trusting an app and not.

What stays on your device: shelf, progress, notes, statistics, settings — all local. Screenshot recognition runs on your phone; the image is never uploaded. Any key you enter is stored in the system keystore, not in plaintext.

Only two things ever go out, and both can be turned off:

Letting AI tidy up shelf recognition results. The image you selected goes to the model service you configured. Every import needs its own tap of consent, you can switch it off at any time, and switching it off drops you back to on-device recognition.

Generating a reading report. What goes out is aggregate statistics plus the title, author, category, status and rating of relevant books in that period — so the report can name actual books instead of speaking in generalities. Note text and highlights are never sent.

One more thing worth stating: there is no analytics SDK in the app, no advertising SDK, no crash reporting. The sole exception is that the recognition SDK sends Google operational metrics carrying no image content. That’s the SDK’s own behaviour, the app can’t disable it, and it’s spelled out in the privacy policy. Don’t use screenshot import and no such request is ever made.

7. Where it stands


Version	v1.0.1, Android package on GitHub Releases
Languages	English, 中文, Deutsch, Français, Español
Themes	12 — six solid colours, six textured backgrounds
License	MIT
iOS	Signing configured, not on the App Store yet

Looking back, the reason I built this is almost embarrassingly simple: I just didn’t want to maintain that table by hand any more.

But the real change wasn’t the time saved. It was that once looking back costs nothing more than opening the app, you actually do it. A year in review, written for you by you, sitting in the archive tab.

Readnest is free. Source and downloads are on GitHub, and the product page has more screenshots.

If you’re carrying your reading records between apps too, I hope it takes a little off your plate.

交叉熵:语言模型的损失函数,为什么只能是它

Mon, 05 Oct 2026 00:00:00 +0000

这篇是 3Blue1Brown 系列 Compression is Intelligence 的第二篇。Part 1 把熵重新讲了一遍,这里直接接上:熵之后,语言模型的损失函数到底是什么,以及为什么是它。原文链接:But what is cross-entropy? — Compression is Intelligence Part 2(B站官方双语版)。这个视频的讲解让我对交叉熵有了完全不一样的认识，妙不可言啊。

一、从一个20年前的实验说起

故事要从一篇 2002 年的论文《Language Trees and Zipping》开始。它做的事情相当离谱:把一堆不同语言写的文本扔进通用压缩器(gzip),看压缩之后的文件大小,就能把语言分好类、甚至恢复出语言的谱系树。

没有训练,没有词典,没有统计任何词频,甚至连语言学先验都不需要。只靠压缩。

这个想法很值得琢磨:压缩器为什么能看出语言关系?因为它本质上在估计数据的统计规律——一份文本能被压得多狠,取决于它有多”可预测”。两种语言越像,拿 A 的上下文去压 B,压缩比就会越接近单独压 A。

换句话说,“压缩距离”天然就是一个分布之间的差异度量。这个度量后面会变成一个名字:交叉熵。

不过先说好,视频里自己也提了:压缩距离不是严格的交叉熵,gzip 也远没达到 Shannon 极限,这只是一个经验估计。但方向是对的,而这一步足够把我们从”压缩”带到”语言模型训练”门口。

二、编码一个玩具消息

自然语言的分布复杂到没法描述,所以视频从一个玩具例子起步:给一个远程机器人发指令,只有四种符号——上、下、左、右。

第一版分布(也就是老规矩):

符号	概率
上	1/2
下	1/4
左	1/8
右	1/8

Part 1 已经推过结论:最优编码里,每个符号分配的比特数就是 −log₂(概率)。

符号	概率	比特数
上	1/2	1
下	1/4	2
左	1/8	3
右	1/8	3

−log₂ 这个形式第一次看到确实别扭。Shannon 把它叫做事件的信息量。Grant 甚至开玩笑说,希望历史上把它叫作 log base 1/2(即以 1/2 为底的对数),因为它其实就是在问”把东西对折几次,才能折到这么多”。

然后计划改了。从此刻起,新的分布是:

符号	概率
上	1/8
下	1/8
左	1/4
右	1/2

注意这句:编码和解码系统都写死了,用的是旧分布的最优方案。

于是每次发消息要花多少比特?加权求和:

1/8 · 1 + 1/8 · 2 + 1/4 · 3 + 1/2 · 3 = 2.625 bits / 消息

2.625 这个数,就是旧分布相对于新分布的交叉熵。

它回答的问题很具体:一个为某种上下文优化的压缩方案,换个上下文还能用吗?能用,但要多花多少?

三、交叉熵公式

把符号一般化:真实分布记作 p(实际发生的是什么),编码所依据的分布记作 q(我们按什么方案编码)。

H(p || q) = -Σ pᵢ log₂ qᵢ = Σ pᵢ · (-log₂ qᵢ)

每一项 −log₂ qᵢ 是”按 q 给符号 i 定的价”,乘上 pᵢ 是”这个符号实际出现的频率”,加起来就是平均要花的比特数。

写代码的话:

import numpy as np

p = np.array([0.125, 0.125, 0.25, 0.5])   # 真实分布(新)
q = np.array([0.5, 0.25, 0.125, 0.125])   # 编码所用分布(旧码本)

cross_entropy = -(p * np.log2(q)).sum()     # 2.625
entropy_p     = -(p * np.log2(p)).sum()     # 1.75
kl            = cross_entropy - entropy_p    # 0.875

三个数,后面全用得上。

四、p 和 q 不能对调

只留两个事件的分布,交叉熵的性质看得更清楚。

情形 A:q 是均匀分布(50/50),p 是偏斜的(90/10)。

0.9 × (-log₂ 0.5) + 0.1 × (-log₂ 0.5) = 0.9·1 + 0.1·1 = 1.00 bits

情形 B:反过来,q 偏斜(90/10),p 均匀(50/50)。

0.5 × (-log₂ 0.9) + 0.5 × (-log₂ 0.1) = 0.5·0.15 + 0.5·3.32 ≈ 1.74 bits

同样是 90/10 和 50/50 这两个分布,只是换了个角色,1 bit 直接变成 1.74 bits。

原因在 −log₂ q 这个”单价”上。q 偏斜意味着有个符号被定价成 3.32 bits;当 p 也均匀地往那个符号上摊 50% 的质量时,3.32 就被拉进了平均值。反过来 q 均匀时,所有符号单价都是 1,不管 p 长什么样,加权平均都等于 1。

所以 H(p||q) ≠ H(q||p)。写代码的时候,p 传错就出 bug。在语言模型里:p 是真实数据(下一个 token 到底是什么),q 是模型给的预测分布。顺序反了,你算的根本不是同一个东西。

五、最低点就是熵

把 p 钉死,让 q 动起来看交叉熵怎么变。

import numpy as np

p = np.array([0.125, 0.125, 0.25, 0.5])       # 固定真实分布
q_old = np.array([0.5, 0.25, 0.125, 0.125])   # 旧码本

ts = np.linspace(0, 1, 200)                   # q 从 p 走向 q_old
ce = np.array([-(p * np.log2((1-t)*p + t*q_old)).sum() for t in ts])

print(f"q=p 时: {ce[0]:.4f}   (= p 的熵)")     # 1.75
print(f"q=旧码本: {ce[-1]:.4f}")              # 2.625
print(f"差值(KL): {ce[-1] - ce[0]:.4f}")      # 0.875

q=p 时: 1.7500   (= p 的熵)
q=旧码本: 2.6250
差值(KL): 0.8750

曲线从下往上爬,最低点严格出现在 q = p 处,而那个最低值就是 p 的熵 H(p) = 1.75。

这条曲线的意义很大:熵不只是一个抽象数字,它是任何编码方案的比特数下界。q 偏离 p 一点,交叉熵就往上抬一点,抬了多少就是亏了多少。

顺带把 KL 散度定下来:

D_KL(p || q) = H(p||q) − H(p)

交叉熵减去熵,就是”用了次优编码方案,每个符号浪费的比特数”。上面的例子:2.625 − 1.75 = 0.875。机器学习中它常当分布之间的距离用——相等时为 0,差异越大越大,但不对称,不是真距离(不满足对称性)。

六、为什么公式非得是 `−log`

前面所有的例子都在展示交叉熵”是什么样”,这一节回答的是为什么只能是它。

语言模型的损失函数是”平均损失”。它应该满足什么性质?最自然的诉求是:只有当模型给出的分布 q 恰好等于真实分布 p 时,平均损失取最小值;其他任何 q 都更差。

那什么样的损失函数 F(q) 能做到这点?

模型对一组可能的下一个 token 给出分布 q,我们要最小化的是这个加权平均损失:

L(q) = Σ pᵢ F(qᵢ)     约束: Σ qᵢ = 1

这里有个细节容易忽略:L 是 q 的函数,但 p 是真实世界的固定事实——求导只对 q 求,pᵢ 是常数权重。这一点如果绕不过去,后面的推导就会卡住。

引入拉格朗日乘子 λ:

∂L/∂qᵢ = pᵢ F'(qᵢ) + λ = 0

对所有 i 成立,于是:

F'(qᵢ) = -λ / qᵢ     (所有 i 一致,说明 F' 就是 1/q 的形式)

积分:

F(q) = -λ log q + C

再取一个边界条件把常数定下来:当只有一个是真(q = p = 1)时,预测全对,损失该是 0。代入 F(1) = -λ·0 + C = 0,所以 C = 0。

于是 F(q) = −λ log q,取 λ = 1(这只是单位选择,等价于换底数),我们得到:

F(q) = -log q

平均损失就是交叉熵,而且只有它是这个形状。

也就是说,交叉熵不是被”选出来”的,是被”逼出来”的。只要承认”平均损失应该恰好在 q=p 时最小”,−log 是唯一答案。

这里有个我读到这里才真正体会到的东西:前几节里,交叉熵是从压缩那条路自然长出来的(信息量、比特数、编码);到这一节,它又是在机器学习那条路上,用约束优化从别处被推出来的。两条完全不相关的推理链,落在同一个公式上。

七、大语言模型的损失函数

现在把前面全部拼起来。

大语言模型做的事很简单:把文本切成 token,喂进模型,让模型输出下一个 token 的概率分布 q,然后跟真实数据里的下一个 token(即 p)比。损失就是模型给真实 token 打分的对数取负,再对所有 token 求平均:

L = - (1/N) Σ log q(true token)

写成交叉熵的形式,这就是”模型分布 q 相对于真实分布 p 的交叉熵”。第 6 节证明过,这个损失恰好在模型学会数据的统计规律(q 贴近 p)时最小。

几个细节:

代码里几乎都用 log 而不是 log₂。差一个常数因子 1/ln 2,会被学习率吸收,而自然对数求导干净得多。
预训练的终极目标,是让损失逼近语言的熵。语言的熵是下界(第 5 节那条曲线的最低点),永远到不了,但可以一直逼近。
形式上极其简洁,工程上极其复杂:批处理、优化器、数值稳定(softmax + log 容易出 NaN,得用 logsumexp),这些和数学无关,但都躲不开。

八、压缩即智能

回到标题。把第 7 节那句”模型视角下的平均信息量”换个说法:

模型的损失,就是它认为每个 token 值多少比特。把整个文本用这个”模型视角的比特数”加起来,就是一次压缩。

所以训练语言模型,数学上等价于训练一个最优文本压缩器。 这不是类比,是同一个目标函数的两种读法。

这条命题我现在的理解是:它在数学上严格成立——交叉熵既是压缩的期望比特数,又是预测的损失,证明路径互不相干却互相印证。但”智能”那半个词仍然是个教学框架:压缩得好不代表”懂”。Grant 真正在做的事,是把这两边放在同一个公式下面,让你自己决定这个等号要打多深。

知识蒸馏是另一个直接应用:让小模型在同一个位置,去逼近大模型给出的完整分布,损失就是小模型分布相对于大模型分布的交叉熵。好处是每个样本都能拿到整个分布的监督,而不只是一个 one-hot 标签——单个训练样本就能提供比标签多得多的信息。

九、参考资料与复现

原视频:But what is cross-entropy? — Compression is Intelligence Part 2(B站官方双语版)
系列第一篇:Reinventing Entropy — Compression is Intelligence Part 1(B站版)
系列播放列表:Compression is Intelligence
开篇实验论文:Language Trees and Zipping (Joshi et al., 2002)

正文里所有数字都可以用第三、六节的两段代码复现,纯 numpy:

交叉熵 2.625 bits/消息(旧码本 vs 新分布)
H(p) = 1.75 bits(新分布的熵,交叉熵下界)
D_KL = 0.875 bits(交叉熵 − 熵)
非对称性对照:H(p=90/10, q=50/50) = 1.00 vs H(p=50/50, q=90/10) ≈ 1.74

如果这篇笔记对你有帮助,欢迎交流。

Cross-Entropy: The Loss Function of Language Models, and Why It Must Be This One

Mon, 05 Oct 2026 00:00:00 +0000

This is the second post in 3Blue1Brown’s Compression is Intelligence series. Part 1 rewrote entropy from scratch, so this one picks up right there: after entropy, what on earth is a language model’s loss function, and why that one. Original video: But what is cross-entropy? — Compression is Intelligence Part 2 (also mirrored on Bilibili with Chinese subtitles). After this one, cross-entropy finally clicked for me in a completely different way — I used to just memorize the formula.

1. An Experiment From Twenty Years Ago

The story starts with a 2002 paper, Language Trees and Zipping. What it did is a bit wild: throw texts written in different languages into an off-the-shelf compressor (gzip), look at the resulting file sizes, and you can classify the languages and even recover the family tree among them.

No training, no dictionary, no word-frequency statistics, not even any linguistic prior. Just compression.

That idea is worth sitting with: why can a compressor see language relationships? Because at its core it is estimating the statistical regularities of the data — how far a text can be squeezed depends on how predictable it is. The closer two languages are, the more compressing B with A’s contexts looks like compressing A alone.

In other words, “compression distance” is naturally a measure of difference between distributions. That measure later gets a name: cross-entropy.

To be fair, the video says this itself: compression distance is not strictly cross-entropy, and gzip is nowhere near the Shannon limit — this is only an empirical estimate. But the direction is right, and it is enough to walk us from “compression” to the door of “language model training”.

2. Encoding a Toy Message

Natural language distributions are too complicated to describe, so the video starts with a toy: sending commands to a remote robot, with only four symbols — up, down, left, right.

The first version of the distribution (the old routine):

Symbol	Probability
up	1/2
down	1/4
left	1/8
right	1/8

Part 1 already derived the conclusion: in the optimal code, each symbol gets −log₂(probability) bits.

Symbol	Probability	Bits
up	1/2	1
down	1/4	2
left	1/8	3
right	1/8	3

The −log₂ shape feels awkward at first. Shannon called it the information content of an event. Grant even jokes that he hopes historians will call it log base 1/2, because it is really asking “how many times do you have to fold something in half to get this many pieces”.

Then the plan changes. From now on, the new distribution is:

Symbol	Probability
up	1/8
down	1/8
left	1/4
right	1/2

Notice this: the encoder and decoder are already frozen, built from the old distribution’s optimal scheme.

So how many bits does each message cost now? A weighted sum:

1/8 · 1 + 1/8 · 2 + 1/4 · 3 + 1/2 · 3 = 2.625 bits per message

2.625 is the cross-entropy of the old distribution with respect to the new one.

It answers a very concrete question: a compression scheme optimized for one context — can you still use it in a different one? Yes, but how much more will it cost?

3. The Cross-Entropy Formula

Generalizing the symbols: call the true distribution p (what actually happens) and the distribution the coding is based on q (what scheme we code with).

H(p || q) = -Σ pᵢ log₂ qᵢ = Σ pᵢ · (-log₂ qᵢ)

Each −log₂ qᵢ term is the “price q charges for symbol i”, multiplied by pᵢ — how often that symbol actually occurs. Sum them up and you get the average bits spent.

In code:

import numpy as np

p = np.array([0.125, 0.125, 0.25, 0.5])   # true distribution (new)
q = np.array([0.5, 0.25, 0.125, 0.125])   # distribution used for coding (old codebook)

cross_entropy = -(p * np.log2(q)).sum()     # 2.625
entropy_p     = -(p * np.log2(p)).sum()     # 1.75
kl            = cross_entropy - entropy_p    # 0.875

Three numbers; everything below uses them.

4. p and q Cannot Be Swapped

Reduce to a two-event distribution and the properties of cross-entropy show up much more clearly.

Case A: q is uniform (50/50), p is skewed (90/10).

0.9 × (-log₂ 0.5) + 0.1 × (-log₂ 0.5) = 0.9·1 + 0.1·1 = 1.00 bits

Case B: flipped — q is skewed (90/10), p is uniform (50/50).

0.5 × (-log₂ 0.9) + 0.5 × (-log₂ 0.1) = 0.5·0.15 + 0.5·3.32 ≈ 1.74 bits

Same two distributions, 90/10 and 50/50, just swapping roles — 1 bit becomes 1.74 bits.

The reason lives in the “unit price” −log₂ q. A skewed q means one symbol is priced at 3.32 bits; when p is also uniform and spills 50% of its mass onto that symbol, 3.32 gets pulled into the average. Flip it around: with a uniform q every symbol costs 1, so no matter what p looks like, the weighted average is still 1.

So H(p||q) ≠ H(q||p). In code, passing the wrong argument in for p produces a bug. In a language model: p is the real data (what the next token actually is) and q is the model’s predictive distribution. Reverse the order and you are not computing the same thing at all.

5. The Floor Is the Entropy

Pin p down and let q move; watch what cross-entropy does.

import numpy as np

p = np.array([0.125, 0.125, 0.25, 0.5])       # fixed true distribution
q_old = np.array([0.5, 0.25, 0.125, 0.125])   # old codebook

ts = np.linspace(0, 1, 200)                   # q slides from p toward q_old
ce = np.array([-(p * np.log2((1-t)*p + t*q_old)).sum() for t in ts])

print(f"q=p:      {ce[0]:.4f}   (= entropy of p)")     # 1.75
print(f"q=old:    {ce[-1]:.4f}")                       # 2.625
print(f"gap (KL): {ce[-1] - ce[0]:.4f}")               # 0.875

q=p:      1.7500   (= entropy of p)
q=old:    2.6250
gap (KL): 0.8750

The curve climbs from the bottom up, and its minimum sits exactly at q = p, where the value is the entropy of p, H(p) = 1.75.

That curve matters: entropy is not just an abstract number — it is a lower bound on the bits any coding scheme can use. Move q away from p a little and cross-entropy lifts a little; how much you lift it by is exactly how much you waste.

That gives KL divergence a definition along the way:

D_KL(p || q) = H(p||q) − H(p)

Cross-entropy minus entropy is “the bits wasted per symbol by using a suboptimal code”. In the example above: 2.625 − 1.75 = 0.875. In machine learning it is often used as a distance between distributions — zero when they agree, larger the further they drift — but it is not symmetric, so it is not a true distance (it fails symmetry).

6. Why the Formula Has to Be `−log`

Everything before showed what cross-entropy looks like; this section answers why it can only be this.

A language model’s loss is an “average loss”. What property should it have? The most natural one: the average loss is minimized exactly when the model’s distribution q equals the true distribution p, and every other q is worse.

So what loss function F(q) achieves that?

The model produces a distribution q over the possible next tokens, and what we minimize is this weighted average loss:

L(q) = Σ pᵢ F(qᵢ)     subject to: Σ qᵢ = 1

An easy detail to miss: L is a function of q, but p is a fixed fact of the real world — differentiate with respect to q only; pᵢ is a constant weight. If you get stuck on that, everything after breaks.

Introduce the Lagrange multiplier λ:

∂L/∂qᵢ = pᵢ F'(qᵢ) + λ = 0

This holds for every i, so:

F'(qᵢ) = -λ / qᵢ     (identical in form for all i, i.e. F' is a 1/q)

Integrate:

F(q) = -λ log q + C

Then a boundary condition pins down the constant: when there is only one possible outcome and the prediction is right (q = p = 1), the loss should be 0. Substituting, F(1) = -λ·0 + C = 0, so C = 0.

Thus F(q) = −λ log q; take λ = 1 (merely a choice of units, equivalent to changing the base) and we get:

F(q) = -log q

The average loss is cross-entropy — and it is the only function of this shape.

In other words, cross-entropy was not “chosen”; it was “squeezed out”. Once you accept that “the average loss should be minimized exactly at q = p”, −log is the only answer.

Here is what only hit me when I read to this point: in the earlier sections cross-entropy grew naturally out of the compression path (information content, bits, coding). In this section it is squeezed out of a completely different place on the machine learning path, via constrained optimization. Two entirely unrelated chains of reasoning land on the same formula.

7. The Loss Function of Large Language Models

Now put it all together.

What a large language model does is simple: chop the text into tokens, feed them in, have the model output a probability distribution q over the next token, then compare it with the real next token (that is p). The loss is the negative log of the score the model gives the true token, averaged over all tokens:

L = - (1/N) Σ log q(true token)

Written as cross-entropy, this is “the cross-entropy of the model’s distribution q with respect to the true distribution p”. Section 6 proved this loss is minimized exactly when the model has learned the statistics of the data (q close to p).

A few details:

Code almost always uses log rather than log₂. That is one constant factor, 1/ln 2, which the learning rate absorbs, while the natural log differentiates far more cleanly.
The endgame of pretraining is to drive the loss toward the entropy of language. That entropy is the lower bound (the bottom of the curve in section 5); you never reach it, but you can keep approaching it.
Mathematically it is extremely clean, engineering-wise extremely complex: batching, optimizers, numerical stability (softmax followed by log produces NaN easily — use logsumexp instead). None of that is about the math, but none of it can be avoided either.

8. Compression Is Intelligence

Back to the title. Rephrase the “average information content from the model’s viewpoint” of section 7:

The model’s loss is how many bits it thinks each token is worth. Add up those “model-viewpoint bits” over the whole text, and you have just done one pass of compression.

So training a language model is mathematically equivalent to training an optimal text compressor. This is not an analogy; it is two readings of one and the same objective function.

Here is how I now read the claim: it holds strictly in mathematics — cross-entropy is simultaneously the expected bits of compression and the loss of prediction, with proof paths that do not overlap yet corroborate each other. But “intelligence” is still half a teaching device: compressing well does not mean understanding. What Grant is really doing is putting both sides under one formula and letting you decide how deep to press on that equals sign.

Knowledge distillation is another direct application: have a small model approximate the full distribution the large model produces at the same position, with the loss being the cross-entropy of the small model’s distribution with respect to the large model’s. The win is that every sample receives supervision from an entire distribution instead of a single one-hot label — one training sample then carries far more information than the label alone.

9. References and Reproduction

Original video: But what is cross-entropy? — Compression is Intelligence Part 2 (mirrored on Bilibili with Chinese subtitles)
First in the series: Reinventing Entropy — Compression is Intelligence Part 1
Series playlist: Compression is Intelligence
Opening experiment: Language Trees and Zipping (Joshi et al., 2002)

Every number in the text can be reproduced from the two code blocks in sections 3 and 5, plain numpy:

Cross-entropy 2.625 bits per message (old codebook vs new distribution)
H(p) = 1.75 bits (entropy of the new distribution, the lower bound on cross-entropy)
D_KL = 0.875 bits (cross-entropy − entropy)
Asymmetry check: H(p=90/10, q=50/50) = 1.00 vs H(p=50/50, q=90/10) ≈ 1.74

If this write-up was useful to you, I’d love to hear from you.

千卡训练的隐形账单：通信不是瓶颈

Wed, 23 Sep 2026 00:00:00 +0000

上一篇 GPU 为什么会挨饿把视角从单点分析推到全链路优化。今天进入 Datawhale mlsysim 学习活动的第五阶段,问题换了一句话:从一台机器走到一千张卡,成本会怎么变? 对应教程:06 Scaling to 1000 GPUs、07 Geography is a Systems Variable、08 The $9M Question。今天跑下来有两个意外:规模扫描那张表里,通信开销不是随规模上升,而是从 1336.8ms 掉到 319.1ms;而可靠性部分的结论,交叉点位置比教程说的远得多。

一、先说结论

这一阶段跑的是 E1(3D 并行)、E2(规模扫描)、O1(可靠性与 checkpoint)、E4(选址)、E3(经济性),全程 Llama-3-70B + DGX H100 + InfiniBand NDR,mlsysim 0.1.2,纯仿真。五条值得单独记住的:

千卡训练最常被讲的瓶颈是通信。实测里通信开销占比只有 5.6%~7.4%,而且随规模上升不升反降——但这不是通信变好了,是教程那张表里混入了 batch 口径的变化。
引擎的集群 MTBF 是节点级串联模型,单节点 4285.7h,512 卡 67.0h。教程里”512 卡约 97h”是按 GPU 数直接除 MTTF 得来的另一个假设。
教程 §5 说”可靠性开销主导通信开销”。方向对,但交叉点在 2560 卡附近,不在 1024 卡;1024 卡时可靠性开销只有通信的 0.71 倍。
同一个集群 30 天作业,引擎里其实有两套可靠性账本。§5 用的是”写入 + 半个间隔回滚”;ReliabilityResult.goodput_ratio 用的是”写入/间隔 + 修复时间/MTBF”,在 1024 卡上给出的开销只有 §5 的 1/2.9。
选址:同样 32 卡、同样运行 30 天,波兰比魁北克多 61.1 倍碳。而经济性这边,30 天窗口里 Capex 恒占 83.4%——短期训练的钱几乎全是折旧,不是电费。

二、E2 规模扫描:那张表为什么会反着走

教程给实验参数设置:TP 固定为 8(节点内 NVLink),PP=1(不引入气泡),DP 靠节点数自然扩大,batch 取 max(64, n_gpus):

GPU	节点	通信 (ms)	扩展效率	步延迟 (ms)	吞吐 (tok/s)	通信占比
8	1	1336.8	92.4%	17,665.4	29.0	7.57%
32	4	393.4	93.6%	6,132.1	83.5	6.42%
64	8	236.2	94.4%	4,209.9	121.6	5.61%
128	16	280.4	93.4%	4,254.0	240.7	6.59%
256	32	302.4	92.9%	4,276.1	478.9	7.07%
512	64	313.5	92.7%	4,287.2	955.4	7.31%
1024	128	319.1	92.6%	4,292.8	1,908.3	7.43%

通信从 1336.8ms 降到 236.2ms,再爬回 319.1ms。扩展效率全程贴在 92%~94%,看起来规模扩展近乎完美。教程的解释是 Amdahl 定律下的通信开销”可控”。

但我没敢直接接受那张表。第一处不对劲的是 8→32 这一段:节点数翻了 4 倍,通信却降到 1/3.4。把通信拆成 TP 与 DP 两个分量就清楚了:

GPU	batch	DP	本地 batch	TP 通信	DP 通信	合计
8	64	1	64.0	1336.8	0.0	1336.8
32	64	4	16.0	334.6	58.8	393.4
64	64	8	8.0	167.6	68.6	236.2
128	128	16	8.0	167.6	112.8	280.4
512	512	64	8.0	167.6	145.9	313.5
1024	1024	128	8.0	167.6	151.5	319.1

两件事同时发生:

TP 通信与”本地 batch”严格成正比,每样本 20.9ms,不随节点数变化。而 max(64, n) 在单机阶段把全局 batch 顶到 64(本地 64),一跨节点本地 batch 立刻塌到 16、再到 8。TP 通信于是从 1336.8ms 一路掉到 167.6ms 封顶。
DP 通信只跟”rank 内梯度体积”有关,等于 params / TP,与全局 batch 无关,与参与方数量只呈弱单调(0 → 151.5ms 后基本饱和)。

也就是说,8 卡和 1024 卡的”通信”根本不是同一个东西:前者是一个节点里 8 张卡为了 64 个样本互相交换激活值,后者是 128 个节点同步梯度。两个量被同一列数字记录,才产生了”规模越大通信越少”的观感。换成分开计算通信开销,结论正常:

GPU	DP 组	DP 通信	TP 通信	合计
8	1	0.0	167.6	167.6
32	4	58.8	167.6	226.4
128	16	112.8	167.6	280.4
512	64	145.9	167.6	313.5
1024	128	151.5	167.6	319.1

通信随规模单调上升,但天花板很低(319ms,占 7.4%)。这个结论本身是对的——教程的”通信可控”站得住——只是它的论证路径有些问题。

2.1 顺带验一下气泡公式

教程背景栏给了气泡比例 (P-1)/(M+P-1)。8 GPU、TP=4、PP=2、扫 microbatch 数:

微批次 M	引擎 bubble_fraction	理论 (P-1)/(M+P-1)
1	0.5000	0.5000
2	0.3333	0.3333
4	0.2000	0.2000
8	0.1111	0.1111

四个点完全重合,P=4、P=8 也一致。引擎用的是理论计算:没有 1F1B 的额外收益,也没有交错流水线的压缩。所以引擎里的微批次只能换来”公式里的改善”,换不来调度算法的工程收益——这点在拿它做并行度决策时要记住。

2.2 E1 的 3D 并行:数字好看,但配置不成立

单节点 8×H100、batch=64、fp16,扫 TP×PP:

TP	PP	DP	步延迟 (ms)	气泡占比	通信 (ms)	效率
1	1	8	4,522.8	0.0%	549.1	87.9%
2	1	4	6,165.0	0.0%	426.3	93.1%
4	1	2	9,920.0	0.0%	651.3	93.4%
8	1	1	17,665.4	0.0%	1336.8	92.4%
1	2	4	9,078.7	50.0%	470.7	63.2%
4	2	1	25,638.6	50.0%	1145.6	63.7%
2	4	1	29,338.8	75.0%	763.6	55.7%

三个规律:PP 一启用,气泡直接吃掉一半或 3/4;TP 从 1 到 4 效率单调升(内存墙压力变小);TP=8 反而回落。

三、O1 可靠性: Young-Daly 公式该往哪用

教程背景栏写:单卡 MTTF 50,000 h，512 卡集群约 97 h。这是按 MTTF / N 直接算的。但引擎的 ReliabilityModel 不是这么实现的——它先算节点,再算集群:

节点 MTBF = 1 / (8/MTTF_gpu + 8/MTTF_nic + 2/MTTF_psu)
集群 MTBF = 节点 MTBF / 节点数

单节点 8 GPU + 8 网卡 + 2 电源,代入 MTTF 50,000 / 150,000 / 100,000 h:

节点 MTBF = 1 / (8/50000 + 8/150000 + 2/100000) = 4285.71 h

GPU	节点	集群 MTBF (h)	30 天故障数	故障平均间隔	Young-Daly 最优 (h)	goodput
64	8	535.7	1.34	22.3 天	4.226	99.59%
256	32	133.9	5.38	5.6 天	2.113	99.15%
512	64	67.0	10.75	2.8 天	1.494	98.76%
1024	128	33.5	21.50	1.4 天	1.056	98.17%
2048	256	16.7	43.01	0.7 天	0.747	97.27%

512 卡这行,教程说的”每天坏一次”是”30 天坏 10.75 次”(平均 2.8 天一次)的近似;而它背景栏的 97h,比引擎的 67h 高了 45%——因为 MTTF/N 只算了 GPU,没算网卡和电源,而这三类是串联关系:任何一个坏,同步训练整个停。

用 calc_young_daly_interval 复核 T_opt = √(2·C·M) 本身无误(1.494h 与手算一字不差),但这里的 C 有个陷阱:

假设 C	512 卡最优间隔
60 s(ReliabilityModel 默认参数)	1.494 h
141.2 s(70B+Adam 真实写盘耗时)	2.292 h

引擎内部这两处用的 C 并不一致:ReliabilityModel.solve(checkpoint_time_s=60) 是默认参数,CheckpointModel 算出的实际写盘是 141.2s。教程 §4 和 §5 都用默认 60s,但一个 988.4GB 的 checkpoint 显然不是 60 秒能写完的。把 C 换回真实值,最优间隔从 1.49h 变成 2.29h,两者相差一半。

3.1 Checkpoint 体积:14 bytes/param 从哪来

优化器	体积	bytes/param
sgd	282.4 GB	4.00
adam	988.4 GB	14.00
rmsprop	282.4 GB	4.00

Adam = fp32 权重 + fp32 一阶矩 + fp32 二阶矩 = 14 bytes/param,70.6B × 14 = 988.4GB,和引擎对得上。这个数决定了两件事:写盘是磁盘问题不是节点问题,以及间隔不能开太大。

3.2 写入带宽:141s → 2s

n_writers 扫描(间隔 1h):

n_writers	写入耗时	1h 间隔 MFU 损失	存储成为瓶颈
1	141.20 s	3.92%	True
8	17.65 s	0.49%	False
64	2.21 s	0.06%	False
256	1.98 s	0.05%	False

8 个 writer 之后曲线开始弯,64 之后基本压到 2s 不动——写入速度有一个硬件上限(引擎按 500 GB/s 文件系统上限 + 有效带宽折算,4 个 writer 就是 35.3s,说明上限在 4 左右开始生效)。

3.3 教程 §5 的两个口径:同一个作业,三种答案

教程 §5 的算法是:通信损失 = (1 − 效率) × 720h;checkpoint 损失 = 30天写入总时长 + 故障数 × 半个间隔。按 C=141.2s 重算:

GPU	通信损失	写入	回滚	可靠性合计	可靠性/通信	合计占 720h
64	40.4	4.4	4.4	8.7	0.22×	6.8%
256	50.9	8.7	8.7	17.4	0.34×	9.5%
512	52.7	12.3	12.3	24.6	0.47×	10.7%
1024	53.5	17.4	17.4	34.9	0.65×	12.3%
2048	54.0	24.6	24.6	49.3	0.91×	14.3%
2560	54.1	27.6	27.6	55.1	1.02×	15.2%
4096	54.2	34.9	34.9	69.7	1.29×	17.2%
8192	54.4	49.3	49.3	98.6	1.81×	21.3%

两个观察:

通信开销是平的,不是”随规模增长”。它在 7.4% 附近饱和(54.4h),因为步延迟里的固定项(计算 + 层税 + 本地 TP)不随节点数变。所以通信的”损失上限”是一个常数,而可靠性开销是单调上升的——交叉点必然存在,只是位置比教程说的远。
交叉点在 2560 卡附近,不在 1024 卡。教程的结论句”At scale, reliability overhead dominates communication overhead”方向没错,但按它自己给的表(1024 卡时 0.71×),dominates 说早了。

而引擎内部还有第三个数字。ReliabilityResult.goodput_ratio 用的公式是 1 − C/M − R/MTBF(C 是单次写入,R 是修复时间 300s),这是稳态占比,分母已包含 §5 那 7.4% 的通信损失,回滚也只计 300s/次而不是半个间隔:

GPU	§5 口径	§5 占比	goodput 口径	goodput 占比	比值
256	19.0 h	2.65%	6.13 h	0.85%	3.11×
512	26.9 h	3.74%	8.93 h	1.24%	3.02×
1024	38.1 h	5.29%	13.15 h	1.83%	2.90×
2048	53.9 h	7.48%	19.65 h	2.73%	2.74×

三个口径给出的”可靠性吃掉多少时间”差了 3 倍,原因是三处不同假设:分母要不要含通信损失、回滚按半个间隔还是按修复时间、写入按理想 goodput 的补数还是稳态占比。没有哪个是错的,但引用数字时必须说清口径——尤其”千卡训练会浪费 30%~50% 时间”这类说法,在这套模型里找不到支撑:§5 口径 1024 卡是 12.7%,goodput 口径是 1.83%。

值得记的一件事:§5 口径下,写入和回滚是严格相等的(每行 write=rollback)。原因是 Young-Daly 的最优间隔恰好让两项各占一半——C·(T/C)=1 与 f·(T/2)=1 在最优解处同时成立。所以”checkpoint 开销”这个词在这个模型里天然意味着”写入 + 同量回滚”,不是两个独立项。反过来也印证了上面的 C 陷阱:如果把最优间隔用默认 C=60s 算、而实际写入仍按 141.2s,这一等式立刻打破(64 卡:写入 6.7h vs 回滚 2.8h)——间隔被压得太短,checkpoint 次数变多,写入反超回滚。

四、E4 选址:61 倍不是 40 倍

E2 的配置(32 卡)、30 天,只换电网:

地区	能源	碳 (t)	总能耗 (MWh)	PUE	gCO₂/kWh
挪威	水电	0.2	17.1	1.06	10
魁北克	水电	0.3	17.1	1.06	20
法国	核电	0.9	18.1	1.12	50
美国平均	混合	7.7	18.1	1.12	429
德国	煤+风	7.0	18.1	1.12	385
波兰	煤电	20.9	25.5	1.58	820

波兰/魁北克 = 61.1 倍,波兰/美国 = 2.7 倍,波兰/冰岛 = 43.7 倍。

4.1 别只算运营碳

embodied_carbon_per_device 扫描(波兰,30 天):

制造碳 (kg/卡)	总碳 (t)	制造碳 (t)	运营占比
0	20.9	0.0	100.0%
300	30.5	9.6	68.5%
500	36.9	16.0	56.6%
1000	52.9	32.0	39.5%

默认值是 0——引擎只算运营碳。而制造碳跟选址无关,它是一笔固定摊入。这意味着”搬到水电地区”这个动作只对运营碳有效,对制造碳没有任何影响;反过来,硬件寿命延长的收益在低碳地区更不值钱,在高碳地区才明显。

五、E3 经济性:30 天的账单 83.4% 是折旧

读源码确认的 EconomicsModel 口径:

Capex(本次) = 单卡价 × 卡数 × infra_multiplier / 3年 × 运行天数 / 365
运维       = 全额 Capex × 5%/年 × 天数 / 365
电费       = PUE 后总能耗 × $0.12/kWh

H100 单卡 $25,000,infra_multiplier=2.0(含网络/冷却/机房/人力):

天数	Capex	电费	运维	TCO	碳 (t)	Capex 占比
7	$10,228	$506	$1,534	$12,268	1.8	83.4%
30	$43,836	$2,168	$6,575	$52,579	7.7	83.4%
90	$131,507	$6,503	$19,726	$157,736	23.2	83.4%
180	$263,014	$13,006	$39,452	$315,471	46.5	83.4%
365	$533,333	$26,373	$80,000	$639,706	94.3	83.4%

Capex 占比是常数 83.4%,和时长无关——因为 Capex、运维、电费都按期线性摊,三者比值固定。这是这套模型里”成本非线性”的一种表现形式:在 30 天这种短窗口里,几乎全部钱都花在一笔三年折旧上,而不是你真正烧掉的电。

规模扫描(30 天,infra 2.0):

GPU	TCO	Capex 占比	碳 (t)	碳 / 万元 TCO
8	$13,145	83.4%	1.9	1,473.8 t
32	$52,579	83.4%	7.7	1,473.8 t
128	$210,314	83.4%	31.0	1,473.8 t
512	$841,257	83.4%	124.0	1,473.8 t
1024	$1,682,514	83.4%	248.0	1,473.8 t

TCO 随规模严格线性,碳强度(每吨碳花多少钱)也恒定 1,473.8 t/万元——这套模型里规模本身不产生经济性,只放大绝对量。

infra_multiplier 是”完整数据中心成本”的开关:

mult	Capex(30d)	电费	运维	TCO
1.0(只算硬件)	$21,918	$2,168	$3,288	$27,373
2.0	$43,836	$2,168	$6,575	$52,579
2.5	$54,795	$2,168	$8,219	$65,181

电费不变,Capex 和运维翻倍。教程源码注释说 2.0~2.5 才对应完整机房成本,1.0 只算卡——默认值 1.0 是个偏乐观的口径,用它做预算容易少报一倍。

5.1 回到”$9M Question”:成本为什么不是线性

打卡要求说清”大规模训练的成本为什么不是线性的”。在这套模型里,我找到的非线性只在三个地方,而且都不是”越大规模越贵/越便宜”这种规模弹性:

折旧窗口。Capex 按 3 年直线摊,30 天的作业承担了 30/365/3 = 2.7% 的硬件寿命。同一个集群跑满一年,Capex 占比从 83.4% 掉到……还是 83.4%(因为三者都线性)。真正改变结构的是:如果硬件能连续跑 3 年而不是 30 天,那么 Capex 会被 12 倍摊薄,电费占比才会显著上升。 也就是说,成本非线性来自”利用率”,不来自”规模”。
可靠性。§5 口径下,30 天里 12.7%~22.5% 的时间被浪费——这部分对应的是已付的机器时。1024 卡集群 30 天的 $1.68M TCO 里,有约 $20 万~$38 万是纯浪费。这个量随规模超线性增长(1.81× @8192)。
选址。碳强度恒定,但碳本身不随规模线性——它跟 PUE 走,而 PUE 是地区属性。

一句话总结这一节:这套引擎里”规模”不是一个成本杠杆,它是一个时间维度的开关。成本结构由运行时长、折旧年限、可靠性决定,不由卡数决定。这跟直觉相反,但和”训练一个模型的钱主要是买时间”这个判断一致。

六、把两个口径合并看:这张图值得存下来

左图是 512 卡的间隔扫描:C=141.2s 时 Young-Daly 最优是 2.29h,30 天总开销 24.6h。取 0.5h 会写到 49.2h,取 8h 会回滚 43.0h——U 型两头都疼,但最优点附近很平(2.29h 到 4h 之间只差 4.5h),这说明工程上不必精确卡住 Young-Daly,把间隔控制在最优的 2 倍以内基本就够了。

右图是另一个墙:写入。141s → 2s 的收益,几乎全部来自 1→8 个 writer,64 之后曲线压平。所以”加存储带宽”和”调 checkpoint 间隔”是两件性质不同的事:前者是一次性投入换 1% 的 MFU,后者是零成本换 3~7% 的 30 天损失。

七、个人思考:今天最大的收获是”口径”

这次学习本身没什么新技术难度——但跑了以后我发现,最容易出错的地方不是公式,而是口径。四个具体例子:

通信随规模下降。表面是”网络很好”,实际是 max(64, N) 让 batch 口径在 8 卡和 32 卡之间断了一次。同一张表里混了两个不同的量,肉眼看不出来。
MTBF 是 67h 还是 97h。教程背景栏用 MTTF/N,引擎用节点串联。差 45%。两个都对,但对应不同的故障模型——前者假设只有 GPU 坏,后者假设网卡和电源也会把整个作业拖停。真实集群里后者更接近事实。
可靠性吃掉多少时间。§5 口径、goodput 口径、以及各自选用的 C,三者组合出 1.8%~12.7% 的范围。教程引文里的”10-30%”在这套模型里偏高。

我的处理方式是:引用教程结论前,先把它的口径复现出来。E2 那张表如果只看结论,”通信可控、可靠性主导”没错;但一旦要拿它做决策(比如”我该不该把 checkpoint 间隔从 2h 改成 1h”),口径错了 3 倍,决策就反了。

对这套工具的定位也清晰了:它是一个”把假设写出来”的计算器,不是一个”告诉你答案”的预言机。它的价值在于——每次给你一个数字,你都能追问”这里的 C 是多少、这里的分母含不含通信损失、这里的 batch 是全局还是本地”。这些追问本身,就是分布式系统设计的大部分内容。

八、参考资料与复现

教程链:00 Hello Roofline(第一篇)→ 01+02 Memory Wall & Two Phases(第二篇)→ 03 KV-Cache(第三篇)→ 04 Starving the GPU + 12 DSE(第四篇)→ 06 Scaling to 1000 GPUs+ 07 Geography+ 08 The $9M Question(本篇)。
打卡任务:Datawhale llm-algo-leetcode #137。
复现方法:pip install mlsysim,完整脚本见文末附录,均可直接运行(纯仿真,不需要 GPU)。
环境:WSL2 + conda 环境 mlsysim(python 3.11,mlsysim 0.1.2),全程无 GPU。

附:完整运行脚本(可直接复制运行)

"""
MLSysBook Task 5 · 附录复现脚本
E2 规模扫描 / E1 气泡公式 / O1 可靠性+checkpoint / E4 选址 / E3 经济学
环境: pip install mlsysim (本文实测 0.1.2) · 纯仿真, 不需要 GPU
"""
import warnings
warnings.filterwarnings("ignore")
import mlsysim
from mlsysim import Models, Systems
from mlsysim.solvers import (DistributedModel, ReliabilityModel, CheckpointModel,
                             EconomicsModel, SustainabilityModel)
from mlsysim.systems.types import Fleet
from mlsysim.systems.reliability import Reliability as REL
from mlsysim.infrastructure.registry import Grids

model = Models.Language.Llama3_70B
solver, rel, ckpt = DistributedModel(), ReliabilityModel(), CheckpointModel()
econ, sust = EconomicsModel(), SustainabilityModel()
JOB = 30 * 24

def fl(n):
    """DGX H100 节点(8 卡) + InfiniBand NDR"""
    return Fleet(name=f"{n}-GPU", node=Systems.Nodes.DGX_H100,
                 count=n // 8, fabric=Systems.Fabrics.InfiniBand_NDR)

# ── E2: 规模扫描, 并把通信拆成 TP 与 DP 两个分量 ──────────────────────
print("E2  通信为什么随规模下降? (TP=8 固定, batch=max(64,N))")
print(f"{'GPUs':>6} {'batch':>6} {'DP':>4} {'本地B':>7} {'TP通信':>8} {'DP通信':>8} {'合计':>8} {'效率':>7}")
for n in [8, 32, 64, 128, 256, 512, 1024]:
    r = solver.solve(model=model, fleet=fl(n), batch_size=max(64, n),
                     precision="fp16", tp_size=8, pp_size=1)
    print(f"{n:>6} {max(64,n):>6} {r.parallelism['dp']:>4} {max(64,n)/(n//8):>7.1f} "
          f"{r.tp_communication_latency.to('ms').magnitude:>8.1f} "
          f"{r.dp_communication_latency.to('ms').magnitude:>8.1f} "
          f"{r.communication_latency.to('ms').magnitude:>8.1f} {r.scaling_efficiency:>7.1%}")

# ── E1: 气泡公式验证 (P-1)/(M+P-1) ─────────────────────────────────
print("\nE1  气泡: 引擎 vs 理论式 (P-1)/(M+P-1)   8 GPU, TP=4 PP=2")
for mb in [1, 2, 4, 8]:
    r = solver.solve(model=model, fleet=fl(8), batch_size=64, precision="fp16",
                     tp_size=4, pp_size=2, microbatch_count=mb)
    print(f"  M={mb}: 引擎 {r.bubble_fraction:.4f}   理论 {(2-1)/(mb+2-1):.4f}")

# ── O1: 节点级 MTBF + Young-Daly + checkpoint 账本 ──────────────────
print("\nO1  节点级 MTBF 与 Young-Daly (C=60s 默认)")
node_mtbf = 1 / (8/REL.Gpu.mttf_hours + 8/REL.Nic.mttf_hours + 2/REL.Psu.mttf_hours)
print(f"  单节点 MTBF = {node_mtbf:.2f} h (8 GPU + 8 NIC + 2 PSU 串联)")
print(f"{'GPUs':>6} {'MTBF(h)':>9} {'故障/30d':>9} {'平均间隔(d)':>11} {'Young-Daly(h)':>14} {'goodput':>8}")
for n in [64, 256, 512, 1024, 2048]:
    r = rel.solve(fl(n), JOB, checkpoint_time_s=60.0)
    print(f"{n:>6} {r.fleet_mtbf.to('hour').magnitude:>9.1f} {r.expected_failures:>9.2f} "
          f"{JOB/r.expected_failures/24:>11.2f} "
          f"{r.optimal_checkpoint_interval.to('hour').magnitude:>14.3f} {r.goodput_ratio:>8.2%}")

print("\nO1  checkpoint 体积 (bytes/param) 与写盘带宽")
for opt in ["sgd", "adam"]:
    r = ckpt.solve(model=model, hardware=Systems.Nodes.DGX_H100.accelerator,
                   optimizer=opt, checkpoint_interval_hours=1.0)
    print(f"  {opt:<5}: {r.checkpoint_size.to('GB').magnitude:>7.1f} GB  "
          f"= {r.checkpoint_size.to('B').magnitude/model.parameters:.2f} bytes/param")
for w in [1, 8, 64, 256]:
    r = ckpt.solve(model=model, hardware=Systems.Nodes.DGX_H100.accelerator, optimizer="adam",
                   checkpoint_interval_hours=1.0, n_writers=w)
    print(f"  n_writers={w:>3}: {r.write_time_seconds.to('s').magnitude:>7.2f} s  "
          f"storage_bottleneck={r.storage_bottleneck}")

print("\nO1  两个口径: §5(写入+半个间隔回滚) vs 引擎 goodput")
for n in [256, 512, 1024, 2048]:
    d = solver.solve(model=model, fleet=fl(n), batch_size=max(64, n), precision="fp16",
                     tp_size=8, pp_size=1)
    rr = rel.solve(fl(n), JOB, checkpoint_time_s=60.0)
    ci = rr.optimal_checkpoint_interval.to("hour").magnitude
    w_s = ckpt.solve(model=model, hardware=Systems.Nodes.DGX_H100.accelerator, optimizer="adam",
                     checkpoint_interval_hours=ci).write_time_seconds.to("s").magnitude
    sec5 = JOB/ci*w_s/3600 + rr.expected_failures*(ci/2)
    gp = (1 - rr.goodput_ratio) * JOB
    print(f"  {n:>5} GPU: §5={sec5:>6.1f}h ({sec5/JOB:>5.2%})  goodput={gp:>5.2f}h ({1-rr.goodput_ratio:>5.2%})"
          f"  比 {sec5/gp:.2f}x  通信损失={(1-d.scaling_efficiency)*JOB:>5.1f}h")

# ── E4: 选址 ────────────────────────────────────────────────────────
print("\nE4  同一台 32xH100, 30 天, 换电网")
base = None
for g in [Grids.Norway, Grids.Quebec, Grids.France, Grids.US_Avg, Grids.Germany, Grids.Poland]:
    r = sust.solve(fleet=fl(32), duration_days=30, datacenter=g)
    base = r.carbon_footprint_kg if base is None else base
    print(f"  {r.region_name:<22} {r.carbon_footprint_kg/1000:>6.1f} t  "
          f"PUE {r.pue:.2f}  {g.carbon_intensity_g_kwh:>5.0f} g/kWh")

# ── E3: 经济学 ──────────────────────────────────────────────────────
print("\nE3  TCO: 规模扫描 (30 天, infra 2.0)")
print(f"{'GPUs':>6} {'TCO':>13} {'Capex占比':>10} {'碳(t)':>8} {'碳/万元TCO':>12}")
for n in [8, 32, 128, 512, 1024]:
    r = econ.solve(fleet=fl(n), duration_days=30, infrastructure_multiplier=2.0)
    print(f"{n:>6} {r.tco_usd:>13,.0f} {r.capex_usd/r.tco_usd:>10.1%} "
          f"{r.carbon_footprint_kg/1000:>8.1f} {r.carbon_footprint_kg/(r.tco_usd/1e4):>12.1f} t")

如果这篇笔记对你有帮助,欢迎交流。

The Hidden Bill of Thousand-GPU Training: Communication Is Not the Bottleneck

Wed, 23 Sep 2026 00:00:00 +0000

The previous post, Why Does the GPU Starve, pushed the view from single-point analysis to whole-pipeline optimization. Today I enter the fifth stage of the Datawhale mlsysim learning task, which reframes the question: going from one machine to a thousand GPUs, how does the cost change? The tutorials are 06 Scaling to 1000 GPUs, 07 Geography is a Systems Variable, and 08 The $9M Question. Two surprises: in the scale sweep table, communication overhead does not grow with scale — it falls from 1336.8 ms to 319.1 ms; and the reliability conclusion crosses over much later than the tutorial claims.

1. Results first

I ran E1 (3D parallelism), E2 (scale sweep), O1 (reliability & checkpoints), E4 (site selection), and E3 (economics) with Llama-3-70B + DGX H100 + InfiniBand NDR on mlsysim 0.1.2, pure simulation. Five things worth remembering separately:

Communication is the bottleneck everyone quotes at thousand-GPU scale. Measured, it is only 5.6%~7.4% of step time and falls rather than rises with scale — but that is not because the network gets better; the tutorial’s table mixes two different batch conventions into one column.
The engine’s cluster MTBF is a node-level series model: one node 4285.7 h, 512 GPUs 67.0 h. The tutorial’s “512 GPUs about 97 h” is another assumption obtained by dividing MTTF by the GPU count directly.
The tutorial’s §5 verdict is “reliability overhead dominates communication overhead”. The direction is right, but the crossover sits near 2560 GPUs, not 1024 — at 1024 GPUs reliability loss is only 0.71x communication loss.
For the same cluster’s 30-day job the engine carries two reliability ledgers. §5 uses “write time + half an interval of rollback”; ReliabilityResult.goodput_ratio uses “write/interval + recovery/MTBF”. At 1024 GPUs it reports one third of §5’s number.
Site selection: the same 32 GPUs, the same 30 days, Poland emits 61.1x Quebec’s carbon. On the economics side, Capex is a constant 83.4% of a 30-day TCO — short training jobs are almost entirely depreciation, not electricity.

2. E2 scale sweep: why the table runs backwards

The tutorial’s sweep parameters: TP fixed at 8 (intra-node NVLink), PP=1 (no bubbles), DP grows with node count, batch = max(64, n_gpus):

GPUs	Nodes	Comm (ms)	Efficiency	Step (ms)	Throughput (tok/s)	Comm share
8	1	1336.8	92.4%	17,665.4	29.0	7.57%
32	4	393.4	93.6%	6,132.1	83.5	6.42%
64	8	236.2	94.4%	4,209.9	121.6	5.61%
128	16	280.4	93.4%	4,254.0	240.7	6.59%
256	32	302.4	92.9%	4,276.1	478.9	7.07%
512	64	313.5	92.7%	4,287.2	955.4	7.31%
1024	128	319.1	92.6%	4,292.8	1,908.3	7.43%

Communication drops from 1336.8 ms to 236.2 ms, then climbs back to 319.1 ms. Efficiency sits at 92%~94% throughout — near-perfect scaling. The tutorial explains this as Amdahl’s-law communication being “manageable”.

I did not accept that table directly. The first red flag was the 8→32 step: nodes quadruple, communication falls to a third. Splitting communication into its TP and DP components makes it clear:

GPUs	batch	DP	Local batch	TP comm	DP comm	Total
8	64	1	64.0	1336.8	0.0	1336.8
32	64	4	16.0	334.6	58.8	393.4
64	64	8	8.0	167.6	68.6	236.2
128	128	16	8.0	167.6	112.8	280.4
512	512	64	8.0	167.6	145.9	313.5
1024	1024	128	8.0	167.6	151.5	319.1

Two things happen at once:

TP communication is strictly linear in local batch — 20.9 ms per sample, independent of node count. But max(64, n) pushes the global batch to 64 on a single node (local 64), and the moment you cross into multiple nodes the local batch collapses to 16, then 8. TP communication therefore falls from 1336.8 ms to a floor of 167.6 ms.
DP communication depends only on the gradient volume per rank — params / TP — independent of global batch, and only weakly monotonic in participant count (0 → 151.5 ms, then essentially flat).

So “communication” at 8 GPUs and at 1024 GPUs is not the same quantity: the first is 8 GPUs exchanging activations for 64 samples inside one node; the second is 128 nodes synchronizing gradients. Two different quantities recorded in one column is what produced the “bigger scale, less communication” impression. Re-sweeping with a fixed batch gives the normal result:

GPUs	DP group	DP comm	TP comm	Total
8	1	0.0	167.6	167.6
32	4	58.8	167.6	226.4
128	16	112.8	167.6	280.4
512	64	145.9	167.6	313.5
1024	128	151.5	167.6	319.1

Communication rises monotonically with scale but the ceiling is low (319 ms, 7.4%). The conclusion itself holds — the tutorial’s “communication is manageable” is defensible — but its reasoning path is not clean.

2.1 Checking the bubble formula along the way

The tutorial gives the bubble fraction as (P-1)/(M+P-1). With 8 GPUs, TP=4, PP=2, sweeping the microbatch count:

Microbatches M	Engine bubble_fraction	Theory (P-1)/(M+P-1)
1	0.5000	0.5000
2	0.3333	0.3333
4	0.2000	0.2000
8	0.1111	0.1111

All four points coincide; P=4 and P=8 agree too. The engine uses the textbook naive form — no 1F1B benefit, no interleaved-scheduling compression. So microbatches in this engine buy you only “improvement in the formula”, not the engineering gains of scheduling algorithms. Worth remembering when using it for parallelism decisions.

2.2 E1’s 3D parallelism

Single node 8×H100, batch=64, fp16, sweeping TP×PP:

TP	PP	DP	Step (ms)	Bubble share	Comm (ms)	Efficiency
1	1	8	4,522.8	0.0%	549.1	87.9%
2	1	4	6,165.0	0.0%	426.3	93.1%
4	1	2	9,920.0	0.0%	651.3	93.4%
8	1	1	17,665.4	0.0%	1336.8	92.4%
1	2	4	9,078.7	50.0%	470.7	63.2%
4	2	1	25,638.6	50.0%	1145.6	63.7%
2	4	1	29,338.8	75.0%	763.6	55.7%

Three patterns: turning on PP eats half or three-quarters of the step in bubbles; TP from 1 to 4 raises efficiency monotonically (memory-wall pressure drops); TP=8 falls back.

3. O1 reliability: where the Young-Daly formula should point

The tutorial’s background text says: single-GPU MTTF 50,000 h, a 512-GPU cluster about 97 h. That is MTTF / N directly. The engine’s ReliabilityModel does not implement it that way — it computes the node first, then the cluster:

node MTBF = 1 / (8/MTTF_gpu + 8/MTTF_nic + 2/MTTF_psu)
cluster MTBF = node MTBF / node count

One node with 8 GPUs + 8 NICs + 2 PSUs, using MTTFs of 50,000 / 150,000 / 100,000 h:

node MTBF = 1 / (8/50000 + 8/150000 + 2/100000) = 4285.71 h

GPUs	Nodes	Cluster MTBF (h)	Failures / 30 d	Mean interval	Young-Daly optimal (h)	Goodput
64	8	535.7	1.34	22.3 days	4.226	99.59%
256	32	133.9	5.38	5.6 days	2.113	99.15%
512	64	67.0	10.75	2.8 days	1.494	98.76%
1024	128	33.5	21.50	1.4 days	1.056	98.17%
2048	256	16.7	43.01	0.7 days	0.747	97.27%

The tutorial’s “something fails roughly every day at 512+ GPUs” is an approximation of “10.75 failures in 30 days” (one every 2.8 days). And its background figure of 97 h is 45% higher than the engine’s 67 h — because MTTF/N counts only GPUs, not NICs and PSUs, and those three component classes are in series: any one of them failing stops the whole synchronous job.

calc_young_daly_interval implements T_opt = √(2·C·M) correctly (1.494 h matches hand calculation exactly). But the C used here has a trap:

Assumed C	Optimal interval at 512 GPUs
60 s (ReliabilityModel’s default argument)	1.494 h
141.2 s (real write time for 70B + Adam)	2.292 h

The engine does not use the same C in both places: ReliabilityModel.solve(checkpoint_time_s=60) is a default parameter, while CheckpointModel computes an actual write time of 141.2 s. Tutorial §4 and §5 both use the default 60 s — but a 988.4 GB checkpoint clearly does not fit in 60 seconds. Restore the real C and the optimal interval moves from 1.49 h to 2.29 h, a factor of half.

3.1 Checkpoint size: where 14 bytes/param comes from

Optimizer	Size	bytes/param
sgd	282.4 GB	4.00
adam	988.4 GB	14.00
rmsprop	282.4 GB	4.00

Adam = fp32 weights + fp32 first moment + fp32 second moment = 14 bytes/param; 70.6B × 14 = 988.4 GB, matching the engine. This single number drives two consequences: writing is a storage problem, not a node problem, and the interval cannot be made very large.

3.2 Write bandwidth: 141 s → 2 s

Sweeping n_writers (interval 1 h):

n_writers	Write time	MFU penalty at 1 h	Storage bottleneck
1	141.20 s	3.92%	True
8	17.65 s	0.49%	False
64	2.21 s	0.06%	False
256	1.98 s	0.05%	False

After 8 writers the curve bends; after 64 it flattens at about 2 s — there is a hardware ceiling on writes (the engine uses a 500 GB/s filesystem cap plus an effective-bandwidth factor; 4 writers already gives 35.3 s, so the ceiling starts biting around 4). “Add storage bandwidth” is therefore a real but quickly exhausted gain, not an unbounded lever.

3.3 The tutorial’s §5: two ledgers, three answers

The tutorial’s §5 algorithm: communication loss = (1 − efficiency) × 720 h; checkpoint loss = total write time over 30 days + failures × half an interval. Recomputed with C = 141.2 s:

GPUs	Comm loss	Write	Rollback	Reliability total	Rel/comm	Total % of 720 h
64	40.4	4.4	4.4	8.7	0.22×	6.8%
256	50.9	8.7	8.7	17.4	0.34×	9.5%
512	52.7	12.3	12.3	24.6	0.47×	10.7%
1024	53.5	17.4	17.4	34.9	0.65×	12.3%
2048	54.0	24.6	24.6	49.3	0.91×	14.3%
2560	54.1	27.6	27.6	55.1	1.02×	15.2%
4096	54.2	34.9	34.9	69.7	1.29×	17.2%
8192	54.4	49.3	49.3	98.6	1.81×	21.3%

Two observations:

Communication loss is flat, not “growing with scale”. It saturates near 7.4% (54.4 h), because the fixed terms in step latency (compute + layer tax + local TP) do not change with node count. So communication has a cap on its loss, while reliability loss rises monotonically — a crossover must exist, it just sits much later than the tutorial says.
The crossover is near 2560 GPUs, not 1024. The tutorial’s closing claim — “at scale, reliability overhead dominates communication overhead” — is directionally right, but by its own table (0.71× at 1024 GPUs) the verb “dominates” is used a factor of two too early in scale.

The engine carries a third number internally. ReliabilityResult.goodput_ratio uses 1 − C/M − R/MTBF (C = single write time, R = recovery time 300 s). That is a steady-state share whose denominator already includes §5’s 7.4% communication loss, and whose rollback is counted as 300 s per failure rather than half an interval:

GPUs	§5 ledger	§5 share	Goodput ledger	Goodput share	Ratio
256	19.0 h	2.65%	6.13 h	0.85%	3.11×
512	26.9 h	3.74%	8.93 h	1.24%	3.02×
1024	38.1 h	5.29%	13.15 h	1.83%	2.90×
2048	53.9 h	7.48%	19.65 h	2.73%	2.74×

The three accounts differ by 3x on “how much time reliability eats”, for three different reasons: whether the denominator contains communication loss, whether rollback is half an interval or recovery time, and whether write cost is measured against ideal goodput or as a steady-state share. None is wrong, but you must state the convention when quoting — claims like “thousand-GPU training wastes 30%~50% of its time” find no support in this model: §5 gives 12.7% at 1024 GPUs, the goodput ledger gives 1.83%.

One thing worth keeping: in §5’s ledger, write and rollback are exactly equal (write = rollback in every row). That is because the Young-Daly optimum happens to split the two equally — C·(T/C) = 1 and f·(T/2) = 1 both hold at the optimal solution. So “checkpoint overhead” in this model inherently means “write plus an equal amount of rollback”, not two independent terms. The C trap from above breaks the identity in exactly the way you would expect: compute the optimal interval with the default C = 60 s but still write 141.2 s of real data, and at 64 GPUs write (6.7 h) overtakes rollback (2.8 h) — the interval is squeezed too short, so checkpoint counts rise and write dominates.

4. E4 site selection: 61x, not 40x

The E2 configuration (32 GPUs), 30 days, only the grid changes:

Region	Energy	Carbon (t)	Total energy (MWh)	PUE	gCO₂/kWh
Norway	Hydro	0.2	17.1	1.06	10
Quebec	Hydro	0.3	17.1	1.06	20
France	Nuclear	0.9	18.1	1.12	50
US average	Mixed	7.7	18.1	1.12	429
Germany	Coal+wind	7.0	18.1	1.12	385
Poland	Coal	20.9	25.5	1.58	820

Poland/Quebec = 61.1x, Poland/US = 2.7x, Poland/Iceland = 43.7x. Tutorial 07’s repeated “40x” is the purity gap (820/20 gCO₂/kWh); the actual carbon gap also folds in PUE (1.58 vs 1.06) and energy mix (25.5 vs 17.1 MWh) — so the real number is larger than the purity gap.

4.1 Do not count operational carbon only

Sweeping embodied_carbon_per_device (Poland, 30 days):

Embodied (kg/GPU)	Total carbon (t)	Embodied (t)	Operational share
0	20.9	0.0	100.0%
300	30.5	9.6	68.5%
500	36.9	16.0	56.6%
1000	52.9	32.0	39.5%

The default is 0 — the engine counts only operational carbon. Manufacturing carbon is independent of location; it is a fixed amount amortized in. Which means “move to a hydro site” only affects operational carbon and leaves manufacturing carbon untouched; conversely, extending hardware lifetime is worth less in a low-carbon region and only pays off in a high-carbon one.

5. E3 economics: 83.4% of a 30-day bill is depreciation

The EconomicsModel conventions, confirmed in source:

Capex(period) = unit cost × GPU count × infra_multiplier / 3 years × run days / 365
maintenance  = full Capex × 5%/year × days / 365
electricity  = post-PUE energy × $0.12/kWh

H100 at $25,000 each, infra_multiplier = 2.0 (network, cooling, facility, staff):

Days	Capex	Electricity	Maintenance	TCO	Carbon (t)	Capex share
7	$10,228	$506	$1,534	$12,268	1.8	83.4%
30	$43,836	$2,168	$6,575	$52,579	7.7	83.4%
90	$131,507	$6,503	$19,726	$157,736	23.2	83.4%
180	$263,014	$13,006	$39,452	$315,471	46.5	83.4%
365	$533,333	$26,373	$80,000	$639,706	94.3	83.4%

The Capex share is a constant 83.4%, independent of duration — because Capex, maintenance, and electricity are all prorated linearly, so their ratio is fixed. This is one form of “non-linear cost” in this model: in a short window like 30 days, almost all the money goes into a three-year depreciation, not into the electricity you actually burn.

Scale sweep (30 days, infra 2.0):

GPUs	TCO	Capex share	Carbon (t)	Carbon per $10k TCO
8	$13,145	83.4%	1.9	1,473.8 t
32	$52,579	83.4%	7.7	1,473.8 t
128	$210,314	83.4%	31.0	1,473.8 t
512	$841,257	83.4%	124.0	1,473.8 t
1024	$1,682,514	83.4%	248.0	1,473.8 t

TCO scales strictly linearly with GPU count, and carbon intensity per dollar spent is constant at 1,473.8 t per $10k — in this model, scale produces no economics at all, it only multiplies the absolute amount.

infra_multiplier is the switch for “full datacenter cost”:

mult	Capex (30 d)	Electricity	Maintenance	TCO
1.0 (hardware only)	$21,918	$2,168	$3,288	$27,373
2.0	$43,836	$2,168	$6,575	$52,579
2.5	$54,795	$2,168	$8,219	$65,181

Electricity is unchanged; Capex and maintenance double. The tutorial’s source comments say 2.0~2.5 corresponds to full facility cost and 1.0 to cards only — the default of 1.0 is an optimistic convention, and budgeting with it under-reports by roughly a factor of two.

5.1 Back to the “$9M Question”: why cost is not linear

The check-in task asks why large-scale training cost is not linear. In this model I found non-linearity in only three places, and none of them is a scale elasticity of the “bigger is cheaper/expensive” kind:

The amortization window. Capex is spread straight-line over 3 years, so a 30-day job absorbs 30/365/3 = 2.7% of the hardware’s life. Running the same cluster for a full year, the Capex share goes from 83.4% to…… still 83.4% (because all three terms are linear). What actually changes the structure is: if the hardware runs continuously for 3 years rather than 30 days, Capex is amortized 12x more, and only then does electricity rise as a share. So the non-linearity comes from utilization, not from scale.
Reliability. Under §5’s ledger, 12.7%~22.5% of 30 days is wasted — time on already-paid-for machines. Of the $1.68M TCO for 1024 GPUs over 30 days, roughly $200k~$380k is pure waste, and that grows super-linearly with scale (1.81× at 8192).
Site selection. Carbon intensity per dollar is constant, but carbon itself is not proportional to scale — it tracks PUE, which is a regional attribute.

One line for this section: in this engine, “scale” is not a cost lever — it is a switch on the time axis. Cost structure is set by run duration, amortization horizon, and reliability, not by GPU count. That runs against intuition, but it matches the judgement that “training a model is mainly buying time”.

6. Putting the two ledgers together: this chart is worth keeping

Left: the 512-GPU interval sweep. With C = 141.2 s the Young-Daly optimum is 2.29 h, 24.6 h of loss over 30 days. Picking 0.5 h writes 49.2 h; picking 8 h rolls back 43.0 h — a U shape that hurts on both ends, but flat near the optimum (2.29 h to 4 h differs by only 4.5 h). Practically: you do not need to nail the Young-Daly optimum, keeping the interval within 2x of it is close enough.

Right: the other wall — writing. The 141 s → 2 s gain comes almost entirely from 1→8 writers; past 64 the curve flattens. So “add storage bandwidth” and “tune the checkpoint interval” are different kinds of spend: the first is a one-time investment for about 1% MFU, the second is free and buys 3~7% of a 30-day loss.

7. Personal note: today’s biggest takeaway was “conventions”

This stage was not technically hard — but after running it, what I found is that the easiest thing to get wrong is not the formula, it is the convention. Four specific cases:

Communication falling with scale. On the surface, “the network is good”; in fact max(64, N) breaks the batch convention between 8 and 32 GPUs. Two different quantities in one column, invisible to the eye.
Is MTBF 67 h or 97 h? The tutorial’s background uses MTTF/N; the engine uses node series. A 45% difference. Both are right, but they model different failure assumptions — the first assumes only GPUs fail, the second assumes a NIC or PSU can also stall the whole job. The latter is closer to reality.
How much time does reliability eat? §5’s ledger, the goodput ledger, and whichever C you choose combine into a range of 1.8%~12.7%. The “10-30%” quoted in the tutorial’s key-insight box runs high for this model.

My habit going forward: before quoting a tutorial’s conclusion, reproduce its conventions first. E2’s table, taken as a conclusion, is “communication manageable, reliability dominates” — fine. But once you use it to decide (“should I change the checkpoint interval from 2 h to 1 h?”), a 3x convention error flips the decision.

The positioning of this tool is now clear too: it is a calculator that writes its assumptions down, not an oracle that hands you answers. Its value is that every number it gives you can be interrogated — what is C here, does the denominator include communication loss, is this batch global or local? Those questions are, for the most part, what distributed systems design consists of.

8. References and reproduction

Tutorial chain: 00 Hello Roofline(part 1) → 01+02 Memory Wall & Two Phases(part 2) → 03 KV-Cache(part 3) → 04 Starving the GPU + 12 DSE(part 4) → 06 Scaling to 1000 GPUs + 07 Geography + 08 The $9M Question (this post).
Check-in task: Datawhale llm-algo-leetcode #137.
Reproduction: pip install mlsysim; the full script is in the appendix below, all runnable directly (pure simulation, no GPU needed).
Environment: WSL2 + conda env mlsysim (python 3.11, mlsysim 0.1.2), no GPU throughout.

Appendix: full script (copy and run)

"""
MLSysBook Task 5 - appendix reproduction script
E2 scale sweep / E1 bubble formula / O1 reliability+checkpoint / E4 geography / E3 economics
Env: pip install mlsysim (0.1.2 for this post) - pure simulation, no GPU needed
"""
import warnings
warnings.filterwarnings("ignore")
import mlsysim
from mlsysim import Models, Systems
from mlsysim.solvers import (DistributedModel, ReliabilityModel, CheckpointModel,
                             EconomicsModel, SustainabilityModel)
from mlsysim.systems.types import Fleet
from mlsysim.systems.reliability import Reliability as REL
from mlsysim.infrastructure.registry import Grids

model = Models.Language.Llama3_70B
solver, rel, ckpt = DistributedModel(), ReliabilityModel(), CheckpointModel()
econ, sust = EconomicsModel(), SustainabilityModel()
JOB = 30 * 24

def fl(n):
    """DGX H100 node (8 GPUs) + InfiniBand NDR"""
    return Fleet(name=f"{n}-GPU", node=Systems.Nodes.DGX_H100,
                 count=n // 8, fabric=Systems.Fabrics.InfiniBand_NDR)

# -- E2: scale sweep, with communication split into TP and DP components --
print("E2  Why does communication fall with scale? (TP=8 fixed, batch=max(64,N))")
print(f"{'GPUs':>6} {'batch':>6} {'DP':>4} {'localB':>7} {'TPcomm':>8} {'DPcomm':>8} {'total':>8} {'eff':>7}")
for n in [8, 32, 64, 128, 256, 512, 1024]:
    r = solver.solve(model=model, fleet=fl(n), batch_size=max(64, n),
                     precision="fp16", tp_size=8, pp_size=1)
    print(f"{n:>6} {max(64,n):>6} {r.parallelism['dp']:>4} {max(64,n)/(n//8):>7.1f} "
          f"{r.tp_communication_latency.to('ms').magnitude:>8.1f} "
          f"{r.dp_communication_latency.to('ms').magnitude:>8.1f} "
          f"{r.communication_latency.to('ms').magnitude:>8.1f} {r.scaling_efficiency:>7.1%}")

# -- E1: bubble formula check (P-1)/(M+P-1) --
print("\nE1  Bubbles: engine vs theory (P-1)/(M+P-1)   8 GPU, TP=4 PP=2")
for mb in [1, 2, 4, 8]:
    r = solver.solve(model=model, fleet=fl(8), batch_size=64, precision="fp16",
                     tp_size=4, pp_size=2, microbatch_count=mb)
    print(f"  M={mb}: engine {r.bubble_fraction:.4f}   theory {(2-1)/(mb+2-1):.4f}")

# -- O1: node-level MTBF + Young-Daly + checkpoint ledger --
print("\nO1  Node-level MTBF and Young-Daly (C=60s default)")
node_mtbf = 1 / (8/REL.Gpu.mttf_hours + 8/REL.Nic.mttf_hours + 2/REL.Psu.mttf_hours)
print(f"  single-node MTBF = {node_mtbf:.2f} h (8 GPU + 8 NIC + 2 PSU in series)")
print(f"{'GPUs':>6} {'MTBF(h)':>9} {'fail/30d':>9} {'mean intv(d)':>13} {'Young-Daly(h)':>14} {'goodput':>8}")
for n in [64, 256, 512, 1024, 2048]:
    r = rel.solve(fl(n), JOB, checkpoint_time_s=60.0)
    print(f"{n:>6} {r.fleet_mtbf.to('hour').magnitude:>9.1f} {r.expected_failures:>9.2f} "
          f"{JOB/r.expected_failures/24:>13.2f} "
          f"{r.optimal_checkpoint_interval.to('hour').magnitude:>14.3f} {r.goodput_ratio:>8.2%}")

print("\nO1  Checkpoint size (bytes/param) and write bandwidth")
for opt in ["sgd", "adam"]:
    r = ckpt.solve(model=model, hardware=Systems.Nodes.DGX_H100.accelerator,
                   optimizer=opt, checkpoint_interval_hours=1.0)
    print(f"  {opt:<5}: {r.checkpoint_size.to('GB').magnitude:>7.1f} GB  "
          f"= {r.checkpoint_size.to('B').magnitude/model.parameters:.2f} bytes/param")
for w in [1, 8, 64, 256]:
    r = ckpt.solve(model=model, hardware=Systems.Nodes.DGX_H100.accelerator, optimizer="adam",
                   checkpoint_interval_hours=1.0, n_writers=w)
    print(f"  n_writers={w:>3}: {r.write_time_seconds.to('s').magnitude:>7.2f} s  "
          f"storage_bottleneck={r.storage_bottleneck}")

print("\nO1  Two ledgers: Sec5 (write + half-interval rollback) vs engine goodput")
for n in [256, 512, 1024, 2048]:
    d = solver.solve(model=model, fleet=fl(n), batch_size=max(64, n), precision="fp16",
                     tp_size=8, pp_size=1)
    rr = rel.solve(fl(n), JOB, checkpoint_time_s=60.0)
    ci = rr.optimal_checkpoint_interval.to("hour").magnitude
    w_s = ckpt.solve(model=model, hardware=Systems.Nodes.DGX_H100.accelerator, optimizer="adam",
                     checkpoint_interval_hours=ci).write_time_seconds.to("s").magnitude
    sec5 = JOB/ci*w_s/3600 + rr.expected_failures*(ci/2)
    gp = (1 - rr.goodput_ratio) * JOB
    print(f"  {n:>5} GPU: Sec5={sec5:>6.1f}h ({sec5/JOB:>5.2%})  goodput={gp:>5.2f}h ({1-rr.goodput_ratio:>5.2%})"
          f"  ratio {sec5/gp:.2f}x  comm_loss={(1-d.scaling_efficiency)*JOB:>5.1f}h")

# -- E4: geography --
print("\nE4  Same 32xH100, 30 days, only the grid changes")
for g in [Grids.Norway, Grids.Quebec, Grids.France, Grids.US_Avg, Grids.Germany, Grids.Poland]:
    r = sust.solve(fleet=fl(32), duration_days=30, datacenter=g)
    print(f"  {r.region_name:<22} {r.carbon_footprint_kg/1000:>6.1f} t  "
          f"PUE {r.pue:.2f}  {g.carbon_intensity_g_kwh:>5.0f} g/kWh")

# -- E3: economics --
print("\nE3  TCO: scale sweep (30 days, infra 2.0)")
print(f"{'GPUs':>6} {'TCO':>13} {'CapexShare':>11} {'Carbon(t)':>10} {'t per $10k TCO':>17}")
for n in [8, 32, 128, 512, 1024]:
    r = econ.solve(fleet=fl(n), duration_days=30, infrastructure_multiplier=2.0)
    print(f"{n:>6} {r.tco_usd:>13,.0f} {r.capex_usd/r.tco_usd:>11.1%} "
          f"{r.carbon_footprint_kg/1000:>10.1f} {r.carbon_footprint_kg/(r.tco_usd/1e4):>17.1f} t")

If this note was useful, I’d be glad to talk.

给开源教程修了个 demo

Wed, 16 Sep 2026 00:00:00 +0000

前一篇 KV-Cache 的显存账本里,我用公式算清了显存的去向。教程本身写得很好,但我在读它的第三篇配套实验(Tutorial 03: KV-Cache)时发现:§5 的 Paged Attention demo 对三种配置输出的结果完全一样。于是我开了个 issue,写了个 PR 重写它。这篇记录整个过程。

一、demo 怎么了:一个”跑不坏”的实验

教程 §5 想演示 PagedAttention 的价值:把 KV-Cache 切成固定大小的块,按需分配,比静态预留整段显存省得多。但 demo 跑出来是这样:

System	Max Users	Throughput	Frag	Speedup
Static (baseline)	32	3225 t/s	0.0%	1.3×
Paged (16 tok)	32	3225 t/s	0.0%	1.3×
Paged (64 tok)	32	3225 t/s	0.0%	1.3×

三种页大小,结果实验结果完全相同。它没有坏（没有报错）但它什么都演示不了。这是最危险的一种 bug:“看起来在跑”和”真的在演示原理”是两回事。

二、原因:四个巧合叠在一起

排查后,问题不是单点故障,而是四件事撞在一起:

#	原因	结果
1	`max_batch_size=32` 被原样传给三种配置,且内存推导出的上限(~130)比它大	`min(上限, 32)` 恒等于 32,批量上限把结果钉死
2	`seq_len=4096` 能被 2048/16/64 全部整除	三种页大小算出相同的填充后大小,末页碎片都是 0%
3	引擎只建模”末页”内部碎片,静态分配的真实浪费(按 `max_seq` 预留、churn 下的外部碎片)不在模型里	碎片栏对三行都一样
4	`speedup_vs_static` 是固定的 0.6 因子(来自 Kwon et al. 2023),与分配策略无关;吞吐又只由内存受限的步时间决定	加速比、吞吐对分配策略零响应

第 4 条最致命:就算去掉批量上限和整除问题,这个 demo 的输出也不会随页大小变化——因为被评测的引擎本身不区分分配策略。结论:这个 demo 需要重写,而不是修补。

三、重写:让实验自己讲出理论

PR 的思路是放弃调用引擎模型,自己写解析仿真:两个小分配器共享同一个 KV 预算——

静态连续分配:每个请求预留一整段 max_seq,简单但浪费;
分页块池:固定大小的块按需分配,块表记账;

喂给它们同一条长尾分布的请求流(固定随机种子,可复现),于是上面的四个问题各自有了实验:

内部碎片:静态预留 77.8% 浪费 vs 16-token 分页 0.4%,并发用户 65 → 314;
外部碎片:churn 之后,静态 first-fit 碎片率 97.1%、10 个新 8K 请求只进得去 0 个;分页 10/10 全进;
页大小扫描:8→1024 token 的页,碎片与块表开销的权衡;
前缀共享:8 个请求共享 1K system prompt,峰值 KV 减 58%。

每个结论都能在表格里直接看到,数字全部可复现。PR 提交后,维护者 Vijay Janapa Reddi 很快回复:“diagnosis is spot on and the new demo actually shows the idea”,然后他 push 了一个小修,合了进去。

四、合并:更诚实的代码

合并前,维护者在我提交之上又做了一轮修改。逐条看,每一条都值得学:

1. 截断点:在第一个放不下的请求处停止,而不是喂完整条流。

我原来的写法是把 10 万条请求全部喂给分配器——包括内存耗尽之后到达的那些。PagedAllocator.admit() 内部会拒绝放不下的请求,于是后续的短请求跳过被拒的长请求继续进入,分配器”挑食”,数字偏乐观。维护者改成:

def fill(alloc, lengths):
    """Admit requests in arrival order; stop at the first one that does not fit."""
    for t in lengths:
        if not alloc.admit(t):
            break
    return alloc

到达顺序截断。它把实验变得更加”诚实”:真实系统里请求不会因为你内存满了就不来,分配器也不该在拒了长请求之后偷偷收下短请求。

2. 数字下修,拥抱不好看的真相。

我的版本:314 users、1.78× 吞吐加速。修改合并后:259 users、1.47×。少的那部分正是”挑食”喂出来的。

五、复盘:这次被改写的三条收获

demo 代码是”看起来在跑”的重灾区。 报错、崩溃会立刻被发现;而每行都输出数字的实验,可能什么都没演示。写 demo 前先问:这个输出会随自变量变化吗?能变多少?
诚实的实验要按到达顺序截断。 “喂完整条流”会赋予分配器不该有的选择权。任何排队/分配类仿真,截断点都要放在第一个失败处,否则你测的是”挑食的分配器”,不是”分配器”。
建模粒度决定结论的适用边界。 引擎把加速比固定成 0.6 因子,意味着”分配策略收益”这个维度在引擎里是失真的——结论只能在这个粒度内成立。看懂一个工具把哪些东西解耦,比背它的输出更重要。

六、参考资料

Issue:#2134 Paged Attention demo returns identical results for every page size
PR:#2135 fix(mlsysim): redesign §5 Paged Attention demo in KV-Cache tutorial(merged)
相关笔记:KV-Cache 的显存账本——教程 03 的学习记录,就是它带我发现了这个 demo 的问题。

如果这篇复盘对你有帮助,欢迎交流。

Why Does the GPU Starve: The Data Wall, Speculative Decoding, and Design-Space Exploration

Wed, 16 Sep 2026 00:00:00 +0000

In The KV-Cache Memory Ledger, I flipped the lens from “time” to “space” and accounted for where the memory goes. Today I moved on to the fourth stage of the Datawhale mlsysim study activity, widening the view from single-point performance analysis (Tasks 1-3) to systematic optimization. This maps to three tutorials: 04 Starving the GPU, 12 Design Space Exploration, and the optional Module 2: Advanced Single-Node Analysis. The core question is blunt: when GPU utilization sits at 40%, the problem usually is not the GPU — so where is the bottleneck hiding?

1. From Single-Point Bottlenecks to a Whole-Pipeline View

The first three posts were all single-point analyses: roofline tells us whether a workload is compute- or memory-bound, the memory wall quantifies how much faster a hardware upgrade really is, and the KV-Cache ledger accounts for memory. But in a real system, a training step is never the GPU working alone — it is a pipeline:

storage I/O (move samples from disk to RAM) -> CPU preprocessing (decode, augment) -> GPU compute

The three stages run in series, so end-to-end throughput is set by the slowest one:

t_step = max(t_io, t_cpu, t_gpu)      # step time is set by the slowest stage
U      = t_gpu / t_step               # GPU utilization

That is the core intuition of the Data Wall: no matter how fast the GPU is, if the stages feeding it cannot keep up, the end-to-end throughput does not move. This stage uses three mlsysim models to check the pipeline segment by segment.

2. E1: Find the GPU “Ceiling” First

First, establish the baseline: how fast is the pure GPU compute of ResNet-50 on an A100, batch=256, FP16? This becomes the baseline every other stage is compared against.

model = mlsysim.Models.Vision.ResNet50
hardware = mlsysim.Hardware.Cloud.A100
solver = SingleNodeModel()
profile = solver.solve(model=model, hardware=hardware, batch_size=256, precision="fp16")

Measured result:

Metric	Value
Step latency	13.97 ms
Throughput (ceiling)	18,323 img/s
Bottleneck	Compute

The GPU can “eat” 18,000 images per second. Keep this number — it is the ruler for every comparison that follows.

3. E2: Can Storage I/O Keep Up?

The data the GPU needs must first move from storage into memory. Say each image is a 500KB JPEG (the ImageNet average); at batch=256 one step needs 256 x 500KB = 128MB, and the GPU wants a fresh batch every 13.97 ms. Convert that to a bandwidth requirement:

demand = 256 x 500KB / 13.97 ms ~ 9.16 GB/s

DGX A100 storage rides PCIe Gen4 x16, 32 GB/s theoretical. Check with DataModel:

Metric	Value
Workload demand	9.16 GB/s
Storage supply (PCIe Gen4 x16)	32.00 GB/s
Bus utilization	28.6%
Stalled?	False

Storage I/O is not the bottleneck — supply has more than 3x headroom. First segment green.

4. E3: CPU Preprocessing — the Real Reason the GPU Starves

The first two segments are fine, so the bottleneck must be the CPU. Say 8 workers at 250 MB/s each — 2 GB/s of preprocessing throughput total. Compare the CPU transform time against the GPU step time with TransformationModel:

Metric	Value
CPU transform time	64.0 ms
GPU step time	13.97 ms
CPU is bottleneck	True
GPU utilization	21.8%
Slowdown factor	4.58x

The GPU needs only 14 ms to do its work, but the CPU needs 64 ms to prepare the next batch — the GPU idles 78% of the time. That is the GPU starving: the GPU can process tens of thousands of samples per step, but 8 CPU workers can barely prepare a fraction of that.

4.1 Bigger Batches Starve It Harder

Sweep batch from 16 to 1024 (CPU throughput fixed at 2 GB/s — larger samples only widen the gap):

Batch	GPU Step	CPU Xform	Binding	GPU Util
16	1.36 ms	4.00 ms	CPU	33.9%
32	2.20 ms	8.00 ms	CPU	27.5%
64	3.88 ms	16.00 ms	CPU	24.2%
128	7.24 ms	32.00 ms	CPU	22.6%
256	13.97 ms	64.00 ms	CPU	21.8%
512	27.43 ms	128.00 ms	CPU	21.4%
1024	54.34 ms	256.00 ms	CPU	21.2%

4.2 Add Workers, Find the Crossover

How many workers does it take to feed the GPU? Sweep the worker count (batch=512):

Workers	CPU throughput	GPU utilization
8	2 GB/s	21.4%
16	4 GB/s	42.9%
32	8 GB/s	85.7%
64	16 GB/s	100.0%
128	32 GB/s	100.0%

GPU utilization is almost exactly proportional to the worker count, until it hits the 100% ceiling. The crossover is computable: at batch=512 the demand is 512 x 500KB / 27.43 ms ~ 9.3 GB/s; at 250 MB/s per worker:

N_workers >= 9.3 GB/s / 250 MB/s ~ 37 workers

So roughly 37 workers is where the CPU just stops dragging its feet. That is the value of the “data wall” concept: when GPU utilization is low, inspect the supply side first — don’t blame the GPU.

The standard engineering fixes: add workers, offload preprocessing to the GPU with NVIDIA DALI, pre-decode the dataset, aggressive prefetching. The idea is always the same — keep the CPU one step ahead of the GPU.

5. E4/O1: Speculative Decoding — Let an 8B “Draft” for a 70B

There are algorithmic ways to win big with small models too. Autoregressive decoding is serial: one token at a time, and every step reads the whole model’s weights from HBM (decode is memory-bound — see Part 2). Speculative decoding flips that: a small draft model proposes K tokens, then the target model verifies them in one parallel forward pass and accepts what matches.

Expected output per round:

E(alpha) = 1 + alpha(1-alpha^K)/(1-alpha)   # expected accepted tokens
speedup  = E(alpha) / (1 + K*r)             # r = draft per-token time / target per-token time

The key: the target verifying K draft tokens is one parallel forward pass, not K serial decode steps; the draft may not guess many tokens at once, but it is cheap. So one expensive 70B stream read gets amortized over several accepted tokens.

5.1 E4: Llama-3-70B with an 8B Draft

Measured with ServingModel (H100, seq_len=2048, batch=1, alpha=0.75):

Config	ITL	Speedup
Baseline (no spec decoding)	43.15 ms	1.00x
8B draft, alpha=0.75	20.53 ms	2.10x

E(0.75)=3.05: each round drafts 4 tokens and accepts 3.05 on average. The verification step is fundamentally one 70B HBM stream read (memory-bound), and the 8B draft proposal is cheap — amortize one stream read over ~3 accepted tokens and the 2.1x comes almost for free.

5.2 O1a: Sweeping the Acceptance Rate from 0.5 to 0.9

alpha	E(alpha)	Speedup
0.50	1.94	1.33x
0.75	3.05	2.10x
0.90	4.10	2.82x

The higher the acceptance rate, the steeper the gain — the better the draft guesses, the more stream-read cost each round amortizes.

5.3 O1b: How Big Should the Draft Be? Measurement Overturns a Common Intuition

Run drafts of different sizes (alpha fixed at 0.75):

Draft model	Params	Speedup
GPT-2	1.5B	2.77x
Llama-3-8B	8B	2.10x
Llama-3-70B	70B (=target)	0.62x

There is a gap between this and the usual story (“a draft too small -> low acceptance, too large -> exploding overhead, 8B is the compromise”): in the mlsysim engine, the acceptance rate alpha is a parameter you pass in directly — it does not automatically vary with the draft model’s size. With alpha fixed at 0.75, a smaller draft has only upsides: r shrinks, the denominator shrinks, speedup climbs — so GPT-2 wins; the 70B draft is as slow as the target itself, so proposals are slow and verification is slow, a pure overhead of 0.62x (slower than not using spec decoding at all).

The “8B” argument actually depends on something the engine does not model: in reality, alpha is a function of how close the draft’s distribution is to the target’s — a draft too small guesses poorly (low alpha), a draft too large is slow on its own (high r). By decoupling alpha from model size, the engine lets you study either effect in isolation, then combine them yourself. The real-world trade-off is “guesses well enough” vs “runs fast enough”, and the engine helps you understand the two forces separately before composing them.

6. E5: Design Space Exploration — Declarative Search Instead of Nested Loops

The third ruler is system-level. Given a cluster, which config wins on throughput across batch_size, tensor_parallelism, pipeline_parallelism, and friends? Hand-written nested for-loops work, but they get messy once dimensions multiply. mlsysim’s DSE engine takes a declarative search space and exhaustively evaluates every combination.

This run: Llama-3-70B, FP16, Frontier_8K cluster, space batch_size in [1024, 2048, 4096, 8192] x TP in [1, 2, 4, 8] x PP in [1, 2, 4] — 48 configs total, objective: maximize effective throughput:

Rank	TP	PP	Batch	Throughput	Step latency
1	8	4	8192	15,381 tok/s	17,043 ms
2	8	1	8192	15,245 tok/s	4,299 ms
3	8	2	8192	14,405 tok/s	9,099 ms

Three observations:

TP=8 and batch=8192 sweep the podium — the search lands on the boundary of the space. On this 8K-GPU cluster, larger TP means each card holds a smaller slice of the weights (less memory pressure), while intra-node NVLink is fast enough that AllReduce cost amortizes; larger batch means the weight read is spread over more samples. Both dimensions help monotonically, so the optimum sits in a corner, not inside the space.
PP is the latency tax: at the same TP8 x batch8192, going from PP=1 to PP=2 doubles step latency (4.3s -> 9.1s); PP=4 reaches 17s. The pipeline bubble is diluted at large batch, but the throughput gain (15,245 -> 15,381, only 0.9%) is dwarfed by the latency cost.
A detail the engine exposes on its own: the search keeps warning batch_size (1024) < dp_size (8192): some ranks will be idle. Frontier_8K has 8192 GPUs, and the data-parallel rank count is decided by cluster size — even batch=8192 only gives each rank 1 sample (local_batch=1). In other words, the cluster’s parallelism (8K ranks) has squeezed out the per-card batch dimension. The throughput numbers look great, but that is the credit of 8192 cards; at cluster scale, it is communication and parallelism that cap throughput, not single-card compute.

Designing the search space matters as much as the search itself: the same problem with a different space (batch from 1 to 64) gives a completely different answer — the optimum only ever lives inside the range you draw.

7. O3: Inference-Time Compute — o1 Moves the Compute Demand Back to Inference

The last topic comes from Module 2’s “Reasoning Wall”. Models like o1 run K hidden reasoning steps (chain-of-thought) before answering, and Inference-Time Compute shifts the compute demand from training back to inference. Simulate Llama-3-8B on H100 with InferenceScalingModel:

K (hidden reasoning steps)	Total latency	Energy/query	TTFT share
0 (no reasoning chain)	0.071 s	0.050 kJ	100%
8	2.15 s	1.50 kJ	3.3%
32	8.38 s	5.87 kJ	0.8%

Two key observations:

TTFT does not move at all — prefill only processes the prompt itself; hidden reasoning happens in the decode phase. So at K=32 the TTFT share drops from 100% to 0.8%, and latency is almost entirely generation.
Total latency and energy scale linearly with K: at K=32 one answer costs 8.38 seconds and 5.87 kJ. The systems impact is disruptive — the same GPU cluster can only serve 1/K of the concurrency. “Inference-time compute scaling” buys quality on the business side; on the systems side it is the most expensive optimization there is.

7.1 Memory Wall or Compute Wall? Where I Disagree with the Tutorial

The Module 2 summary page makes a claim: “Inference shifts back toward being compute-bound as generation length dominates prefill” — inference-time compute supposedly moves the bottleneck from the memory wall “back” to the compute wall. There are two layers to that sentence; I agree with the first and disagree with the second.

The first layer is right: compute demand really does shift from training to inference. o1 turns FLOPs that used to be spent on training into extra tokens generated at inference time. That is the essence of inference-time compute scaling, and it is the biggest systems impact — capacity planning goes from “how big should the training cluster be” to “how many FLOPs will inference burn”.

The second layer I push back on: “with generation length dominating, inference becomes compute-bound” — I think it is still stuck on the decode memory wall. Three reasons:

Hidden reasoning tokens are just ordinary autoregressive decode. K reasoning steps are not “more compute” — they are “more decode steps”: each one streams the weights from HBM once and emits a token, structurally identical to generating a normal answer. Decode is memory-bound — measured in Part 2 (ITL is priced by HBM bandwidth). K steps of reasoning just walk that memory-bound road longer; the physics of each step does not change.
The measurements support the memory wall, not the compute wall. If the bottleneck really flipped to compute, we should see prefill-like regime behavior; but the measurements show (a) the only compute-bound segment — prefill (TTFT) — did not move at all; (b) total latency scales strictly linearly with K, a constant per-step cost, which is exactly what “every token priced by an HBM stream read” looks like — not what a compute-bound regime looks like.
When would the author’s claim hold? Under large-batch serving. Decode’s arithmetic intensity rises with batch (see the roofline analysis in Part 2): at high enough concurrency the GPU’s compute gets saturated and decode genuinely shifts from memory-bound to compute-bound. But that is a system-throughput shift; for a single request at low concurrency, inference-time compute is just “longer decode”, still memory-bound. This is the other side of the same coin as Part 3’s KV-Cache conclusion: long sequences are priced by memory, not by compute.

8. My Take: Three Rulers and “Find the Bottleneck First”

Task 4 condenses the single-point skills from the first three posts into a methodology. I read it as three rulers:

Supply side (the Data Wall): when throughput stalls, look at the narrowest stage of the pipeline first; low GPU utilization is not the GPU’s fault;
Algorithm side (speculative decoding): you can go faster without new hardware — small model proposes, target verifies, serial decoding becomes parallel verification;
Search side (DSE): when the config space gets too big to reason about by hand, let a declarative search find the optimum.

The most valuable part is not any single formula but the “find the bottleneck, then optimize” ordering. The 4.58x slowdown in E3, the 2.10x speedup in E4, and the boundary solution in E5 all came from locating the constraint first. Optimizing blindly without knowing the bottleneck wastes effort on stages that were never the problem — like adding NVMe to storage while the real constraint sits in CPU preprocessing.

A second takeaway from Section 5.3: the draft-size result disagrees with common intuition not because anyone is wrong, but because the modeling granularity differs. The engine treats alpha as an independent knob; in reality alpha depends on draft quality. Reading the engine source and seeing what it decouples predicts “what changes if I change the conditions” better than memorizing conclusions.

9. References & Reproduction

Tutorial chain: 00 Hello Roofline (Part 1) -> 01 Memory Wall + 02 Two Phases (Part 2) -> 03 KV-Cache (Part 3) -> 04 Starving the GPU + 12 Design Space Exploration + Module 2 (this post).
Check-in task: Datawhale llm-algo-leetcode #136.
Reproduction: pip install mlsysim; the three scripts in the appendix all run directly (E1-E3 and E4/O1/O3 need no cluster; E5 uses the built-in Frontier_8K cluster from mlsysim.Systems.Clusters).
Environment: WSL2 + conda env mlsysim (Python 3.11, mlsysim 0.1.1), no GPU involved — pure simulation.

Appendix: Full Run Scripts (Copy & Run)

Script 1: E1-E3 Data-Pipeline Analysis

import mlsysim
from mlsysim.solvers import SingleNodeModel, DataModel
from mlsysim.solvers import TransformationModel

from mlsysim.solvers import SingleNodeModel
from mlsysim.core.units import Q_
from mlsysim.show import table, info

model = mlsysim.Models.Vision.ResNet50
hardware = mlsysim.Hardware.Cloud.A100
solver = SingleNodeModel()

# Baseline: ResNet-50 on A100, batch 256, FP16
profile = solver.solve(model=model, hardware=hardware, batch_size=256, precision="fp16")

info("GPU Compute Baseline",
     Model=model.name,
     Hardware=hardware.name,
     Batch_size=256,
     Step_latency=profile.latency.to('ms'),
     Throughput=f"{profile.throughput:.0f} img/s",
     Bottleneck=profile.bottleneck)

from mlsysim.solvers import DataModel

sample_size = Q_("500 KB")  # Average ImageNet JPEG
batch_size = 256

# Data demand = batch_size x sample_size / step_time
step_time_s = profile.latency.to("s").magnitude
data_per_step = (batch_size * sample_size.to("GB")).magnitude
demand_rate = Q_(data_per_step / step_time_s, "GB/s")

data_solver = DataModel()
data_result = data_solver.solve(workload_data_rate=demand_rate, hardware=hardware)

info("Storage I/O Check",
     Data_demand=f"{demand_rate:.3f}",
     Storage_supply=f"{data_result.supply_bw:.2f}",
     Utilization=f"{data_result.utilization:.1%}",
     Is_stalled=data_result.is_stalled)

from mlsysim.solvers import TransformationModel

transform_solver = TransformationModel()
cpu_throughput = Q_("2 GB/s")  # 8 workers x 250 MB/s each

t = transform_solver.solve(
    batch_size=256,
    sample_size_bytes=sample_size,
    cpu_throughput=cpu_throughput,
    accelerator_step_time=profile.latency
)

info("CPU vs GPU Pipeline",
     CPU_transform_time=t.transform_time,
     GPU_step_time=t.accelerator_step_time,
     CPU_is_bottleneck=t.is_bottleneck,
     GPU_utilization=f"{t.accelerator_utilization:.1%}",
     Slowdown_factor=f"{t.slowdown_factor:.2f}x")

rows = []
for bs in [16, 32, 64, 128, 256, 512, 1024]:
    p = solver.solve(model=model, hardware=hardware, batch_size=bs, precision="fp16")

    t = transform_solver.solve(
        batch_size=bs,
        sample_size_bytes=sample_size,
        cpu_throughput=cpu_throughput,
        accelerator_step_time=p.latency
    )

    binding = "Transformation" if t.is_bottleneck else p.bottleneck
    rows.append([
        bs,
        f"{p.latency.to('ms').magnitude:.2f} ms",
        f"{t.transform_time.to('ms').magnitude:.2f} ms",
        binding,
        f"{t.accelerator_utilization:.1%}"
    ])

table(["Batch", "GPU Step", "CPU Xform", "Binding", "GPU Util"], rows)

rows = []
for n_workers in [8, 16, 32,64, 128]:
    cpu_tp = Q_(f"{n_workers * 250} MB/s")

    p = solver.solve(model=model, hardware=hardware, batch_size=512, precision="fp16")

    t = transform_solver.solve(
        batch_size=512,
        sample_size_bytes=sample_size,
        cpu_throughput=cpu_tp,
        accelerator_step_time=p.latency
    )

    rows.append([n_workers, cpu_tp.to('GB/s'), f"{t.accelerator_utilization:.1%}"])

table(["Workers", "Throughput", "GPU Util @ bs=512"], rows)

Script 2: E5 Design Space Exploration

import mlsysim
from mlsysim.engine.dse import DSE
from mlsysim.solvers import DistributedModel, EconomicsModel
from mlsysim.engine.pipeline import Pipeline

# Our fixed baseline constants
model = mlsysim.Models.Language.Llama3_70B
fleet = mlsysim.Systems.Clusters.Frontier_8K

# The evaluation function accepts a dictionary of parameters and returns a structured object
def evaluate_config(params):
    pipe = Pipeline([DistributedModel(), EconomicsModel()])

    return pipe.run(
        model=model,
        fleet=fleet,
        batch_size=params["batch_size"],
        tp_size=params["tp"],
        pp_size=params["pp"],
        precision="fp16",
        efficiency=0.45,
        duration_days=30
    )

# 1. Define the dimensions of your search grid
space = {
    "batch_size": [1024, 2048, 4096, 8192],
    "tp": [1, 2, 4, 8],
    "pp": [1, 2, 4]
}

# 2. Initialize the Design Space Explorer
dse = DSE(
    space=space,
    objective="maximize: DistributedModel.effective_throughput"
)

# 3. Search! (Tests all 4 * 4 * 3 = 48 combinations analytically)
print("Starting search...")

result = dse.search(evaluate_config)

# 4. Analyze the best configuration
best_params = result["best_params"]
best_throughput = result["best_objective"]

print(f"\nBest Configuration: TP={best_params['tp']}, PP={best_params['pp']}, Batch={best_params['batch_size']}")
print(f"Max Throughput: {best_throughput:,.0f} tokens/sec")

print(f"\nTop 3 Configurations:")
print(f"{'TP':>4} | {'PP':>4} | {'Batch':>6} | {'Throughput':>12} | {'Latency':>10}")
print("-" * 45)

for candidate in result["top_candidates"][:3]:
    p = candidate["params"]
    dist_res = candidate["result"]["DistributedModel"]

    throughput = dist_res.effective_throughput.m_as("1/s")
    latency = dist_res.step_latency_total.m_as("ms")

    print(f"{p['tp']:>4} | {p['pp']:>4} | {p['batch_size']:>6} | {throughput:>12,.0f} | {latency:>10.1f}ms")

Script 3: E4/O1 Speculative Decoding + O3 Inference-Time Compute

import mlsysim
from mlsysim.solvers import ServingModel, InferenceScalingModel

hw = mlsysim.Hardware.Cloud.H100
target = mlsysim.Models.Language.Llama3_70B
draft8b = mlsysim.Models.Language.Llama3_8B
draft_gpt2 = mlsysim.Models.Language.GPT2
solver = ServingModel()

# E4: baseline vs speculative decoding (8B draft, alpha=0.75)
base = solver.solve(target, hw, seq_len=2048, batch_size=1)
spec = solver.solve(target, hw, seq_len=2048, batch_size=1,
                    draft_model=draft8b, draft_acceptance_rate=0.75)
print(f"baseline ITL: {base.itl.to('ms').magnitude:.2f} ms")
print(f"spec ITL:     {spec.itl.to('ms').magnitude:.2f} ms")
print(f"speedup:      {base.itl/spec.itl:.2f}x")

# O1a: acceptance-rate sweep
for a in [0.5, 0.75, 0.9]:
    r = solver.solve(target, hw, seq_len=2048, batch_size=1,
                     draft_model=draft8b, draft_acceptance_rate=a)
    print(f"alpha={a}: ITL {r.itl.to('ms').magnitude:7.2f} ms  speedup {base.itl/r.itl:.2f}x")

# O1b: draft-model size sweep
for name, dm in [("GPT2(1.5B)", draft_gpt2), ("Llama3-8B", draft8b), ("Llama3-70B", target)]:
    r = solver.solve(target, hw, seq_len=2048, batch_size=1,
                     draft_model=dm, draft_acceptance_rate=0.75)
    print(f"draft={name}: ITL {r.itl.to('ms').magnitude:7.2f} ms  speedup {base.itl/r.itl:.2f}x")

# O3: inference-time compute (K=0/8/32)
isolver = InferenceScalingModel()
for k in [0, 8, 32]:
    r = isolver.solve(mlsysim.Models.Language.Llama3_8B, hw, reasoning_steps=k)
    print(f"K={k}: total {r.total_reasoning_time.to('s').magnitude:.3f} s  "
          f"energy {r.energy_per_query.to('kJ').magnitude:.3f} kJ")

If this note helps you, feel free to reach out.

GPU 为什么会挨饿：数据墙、投机解码与设计空间探索

Wed, 16 Sep 2026 00:00:00 +0000

上一篇 KV-Cache 的显存账本把视线从”时间”转向”空间”,算清了显存的去向。今天进入 Datawhale mlsysim 学习活动的第四阶段,把视角从”单点性能分析”(Task 1-3)扩展到系统性优化。对应三篇教程:04 Starving the GPU、12 Design Space Exploration、以及选读的 Module 2: Advanced Single-Node Analysis。核心问题很直接:GPU 利用率只有 40% 时,问题往往不在 GPU 本身——那瓶颈到底藏在哪?

一、从”单点瓶颈”到”全链路流水线”

前三篇我们一直在单点分析:roofline 判断负载是计算受限还是内存受限,内存墙量化硬件升级的收益,KV-Cache 算清显存账本。但真实系统里,一个训练步从来不是 GPU 单独完成的,它是一条流水线:

存储 I/O(把样本从磁盘搬进内存)→ CPU 预处理(解码、增强)→ GPU 计算

三个环节串联,整体吞吐由最慢的一环决定:

t_step = max(t_io, t_cpu, t_gpu)      # 步耗时由最慢环节决定
U      = t_gpu / t_step               # GPU 利用率

这就是数据墙(Data Wall)的核心直觉:GPU 再快,如果前面的环节喂不饱数据,整体吞吐量也上不去。 本阶段就用 mlsysim 的三个模型,把这条流水线逐段检查一遍。

二、E1:先找到 GPU 的”天花板”

第一件事是确定基准:ResNet-50 在 A100 上、batch=256、FP16,纯 GPU 计算到底多快?这将是后续所有环节的对照”基线”。

model = mlsysim.Models.Vision.ResNet50
hardware = mlsysim.Hardware.Cloud.A100
solver = SingleNodeModel()
profile = solver.solve(model=model, hardware=hardware, batch_size=256, precision="fp16")

实测结果:

指标	数值
单步延迟	13.97 ms
吞吐(天花板)	18,323 img/s
瓶颈类型	Compute(计算受限)

意味着 GPU 每秒能”吃”下 1.8 万张图。记住这个数,它是后面所有对比的标尺。

三、E2:存储 I/O 扛得住吗

GPU 需要的数据要先从存储搬进内存。假设每张图是 500KB 的 JPEG(ImageNet 平均水平),batch=256 时一个 step 的数据量是 256 × 500KB = 128MB,而 GPU 每 13.97ms 就要一份——换算成带宽需求:

需求带宽 = 256 × 500KB ÷ 13.97ms ≈ 9.16 GB/s

DGX A100 的存储走 PCIe Gen4 x16,理论 32 GB/s。用 DataModel 验证:

指标	数值
工作负载需求	9.16 GB/s
存储供给(PCIe Gen4 x16)	32.00 GB/s
总线利用率	28.6%
是否阻塞	False

结论:存储 I/O 不是瓶颈,带宽富余 3 倍以上。流水线第一段绿灯。

四、E3:CPU 预处理——GPU 挨饿的真凶

前后两段都没问题,那瓶颈只剩 CPU。假设 8 个 worker、每个 250 MB/s,即 CPU 总共只有 2 GB/s 的预处理吞吐。用 TransformationModel 对比 CPU 变换时间和 GPU 步时间:

指标	数值
CPU 预处理时间	64.0 ms
GPU 步时间	13.97 ms
CPU 是否瓶颈	True
GPU 利用率	21.8%
减速因子	4.58×

GPU 干活只要 14ms,可 CPU 准备好下一批数据要 64ms——GPU 有 78% 的时间在空等。这就是”GPU 挨饿”:GPU 每步能处理上万个样本,但 8 个 CPU worker 只能备好不足千个。

4.1 batch 越大,饿得越狠

把 batch 从 16 扫到 1024,CPU 瓶颈被拉得更紧(CPU 吞吐固定 2 GB/s,样本变大只会让差距更大):

Batch	GPU Step	CPU Xform	瓶颈	GPU 利用率
16	1.36 ms	4.00 ms	CPU	33.9%
32	2.20 ms	8.00 ms	CPU	27.5%
64	3.88 ms	16.00 ms	CPU	24.2%
128	7.24 ms	32.00 ms	CPU	22.6%
256	13.97 ms	64.00 ms	CPU	21.8%
512	27.43 ms	128.00 ms	CPU	21.4%
1024	54.34 ms	256.00 ms	CPU	21.2%

4.2 加 worker,找到交叉点

那要多少个 worker 才能喂饱 GPU?扫一下(batch=512):

Workers	CPU 吞吐	GPU 利用率
8	2 GB/s	21.4%
16	4 GB/s	42.9%
32	8 GB/s	85.7%
64	16 GB/s	100.0%
128	32 GB/s	100.0%

GPU 利用率几乎严格正比于 worker 数,直到撞上 100% 的天花板。交叉点可以算出来:batch=512 时需求带宽 512 × 500KB ÷ 27.43ms ≈ 9.3 GB/s,每个 worker 250 MB/s,所以:

N_workers ≥ 9.3 GB/s ÷ 250 MB/s ≈ 37 个

也就是约 37 个 worker 时 CPU 刚刚好不再拖后腿。“数据墙”这个概念的价值就在这:GPU 利用率低的时候,先检查供给端,别急着怪 GPU。

工程上消除数据墙的标准手段:加 worker、用 NVIDIA DALI 把预处理搬进 GPU、对数据集做预解码、激进预取。核心思想都一样——让 CPU 永远比 GPU 快一步。

五、E4/O1:投机解码——让 8B 给 70B”打草稿”

算法层面也有”以小博大”的机会。LLM 自回归解码是串行的:一次只生成一个 token,每步都要把整个模型权重从 HBM 读一遍(Decode 是内存受限的,见第二篇)。投机解码(Speculative Decoding)的思路是:让一个小 draft 模型先草拟 K 个 token,再用目标模型一次性并行验证,接受通过的部分。

一轮的期望产出:

E(α) = 1 + α(1-α^K)/(1-α)        # 期望接受 token 数
speedup = E(α) / (1 + K·r)        # r = draft 单 token 时间 / target 单 token 时间

关键点:target 模型验证 draft 的 K 个 token 是一次并行前向,而不是 K 次串行解码;draft 模型虽然一次也猜不了几个 token,但它便宜。于是把一次昂贵的 70B 流读摊到多个被接受的 token 上。

5.1 E4:LLaMA-3-70B 配 8B draft

用 ServingModel 实测(H100,seq_len=2048,batch=1,α=0.75):

配置	ITL	加速比
基线(无投机解码)	43.15 ms	1.00×
8B draft,α=0.75	20.53 ms	2.10×

E(0.75)=3.05:平均每轮草拟 4 个、接受 3.05 个。校验步本质是一次 70B 的 HBM 流读(内存受限),而 8B draft 提案很便宜,于是把一次流读摊到了 ~3 个被接受 token 上——2.1 倍的收益就这么来的。

5.2 O1a:接受率 α 从 0.5 扫到 0.9

α	E(α)	加速比
0.50	1.94	1.33×
0.75	3.05	2.10×
0.90	4.10	2.82×

接受率越高收益越陡——draft 猜得越准,一轮摊掉的流读成本越多。

5.3 O1b:draft 模型该多大?实测推翻了一个常见直觉

用不同尺寸的 draft 模型跑(α 固定 0.75):

Draft 模型	参数量	加速比
GPT-2	1.5B	2.77×
Llama-3-8B	8B	2.10×
Llama-3-70B	70B(=target)	0.62×

这里和常见说法(“draft 太小→接受率低,draft 太大→开销爆炸,8B 是折中的选择”)有差异:在 mlsysim 的引擎里,接受率 α 是用户直接给定的参数,它不会随 draft 模型的大小自动变化。固定 α=0.75 时,更小的 draft 只有好处没有坏处:r 更小,公式里分母更小,加速比更高——所以 GPT-2 反而最快;而 70B draft 跟 target 一样大,提案慢、验证也慢,白白多付一轮开销,跌到 0.62×(比不用还慢)。

“8B 合适”的论证其实依赖一个引擎没有建模的环节:现实中 α 是 draft 模型与 target 分布接近程度的函数——draft 太小就猜不准(α 低),draft 太大自己就慢(r 高)。引擎把 α 和模型大小解耦成独立因素,反而方便单独研究其中一个效应。真实世界的规律应该是:在”猜得准”和”跑得快”之间取折中,而引擎让你先分别理解这两个力,再自己组合。

六、E5:设计空间探索——用声明式搜索替代嵌套 for 循环

第三把尺子是系统级的。给定一个集群,在 batch_size、tensor_parallelism、pipeline_parallelism 等维度上,哪种配置吞吐最高?手写嵌套 for 循环也可以,但维度一多就乱。mlsysim 的 DSE 引擎接受一个声明式的搜索空间,自动遍历所有组合。

本次实验:Llama-3-70B,FP16,Frontier_8K 集群,搜索空间 batch_size ∈ [1024, 2048, 4096, 8192] × TP ∈ [1, 2, 4, 8] × PP ∈ [1, 2, 4],共 48 种组合,目标最大化有效吞吐:

排名	TP	PP	Batch	吞吐量	步延迟
1	8	4	8192	15,381 tok/s	17,043 ms
2	8	1	8192	15,245 tok/s	4,299 ms
3	8	2	8192	14,405 tok/s	9,099 ms

三个观察:

TP=8 和 batch=8192 包揽了前三——搜索撞到了搜索空间的边界。在这个 8K-GPU 集群上,TP 越大,每张卡持有的权重越小(访存压力越小),而节点内 NVLink 通信足够快,AllReduce 的代价被摊薄;batch 越大,单次权重搬运摊到的样本越多。两个维度都是单调受益,所以最优解落在角落,而不是空间内部。
PP 是延迟的代价:同样 TP8×batch8192,PP 从 1 提到 2,步延迟从 4.3s 翻到 9.1s;PP=4 更是 17s。流水线气泡(Pipeline Bubble)在大 batch 下虽被摊薄,但吞吐收益(15,245→15,381,仅 0.9%)远小于延迟代价。
一个引擎主动暴露的细节:搜索时引擎反复警告 batch_size (1024) < dp_size (8192): some ranks will be idle。Frontier_8K 有 8192 个 GPU,数据并行维度的 rank 数由集群规模自动决定——batch=8192 都只够每个 rank 分到 1 个样本(local_batch=1)。换句话说,这个集群的并行度(8K ranks)已经把单卡 batch 的空间压没了。吞吐量数字好看,但那是 8192 张卡的功劳;在集群尺度上,压住吞吐的往往是通信与并行度,而不是单卡算力。

搜索空间的设计本身也很关键:同样的问题,换一个空间(batch 从 1 到 64)结论会完全不同——最优解永远只在你画的范围内。

七、O3:推理时计算——o1 把算力需求从训练搬回推理

最后一个话题来自 Module 2 的”推理墙”(The Reasoning Wall)。o1 这类模型在回答前会先做 K 步隐藏推理(chain-of-thought),推理时计算(Inference-Time Compute)把算力需求从训练阶段搬回了推理阶段。用 InferenceScalingModel 模拟 Llama-3-8B 在 H100 上:

K(隐藏推理步)	总延迟	能耗/query	TTFT 占比
0(无推理链)	0.071 s	0.050 kJ	100%
8	2.15 s	1.50 kJ	3.3%
32	8.38 s	5.87 kJ	0.8%

两个关键观察:

TTFT 完全不变——prefill 只处理 prompt 本身,隐藏推理是在 decode 阶段发生的。所以 K=32 时 TTFT 占比从 100% 掉到 0.8%,延迟几乎全部来自生成。
总延迟与能耗随 K 严格线性暴涨:K=32 时一次回答要 8.38 秒、5.87 kJ。对系统设计的影响是颠覆性的——同样的 GPU 集群只能服务 1/K 的并发。商业上”推理时计算缩放”用延迟和能耗换质量,系统上它是代价最昂贵的优化手段。

7.1 内存墙还是算力墙?我的观点和教程作者不一样

Module 2 总结页给了一个判断:“Inference shifts back toward being compute-bound as generation length dominates prefill”——推理时计算让推理从内存墙”回到”算力墙。这句话有两层意思,我同意第一层,不同意第二层。

第一层是对的:算力需求确实从训练转移到了推理。o1 把原本花在训练上的 FLOPs,变成了推理时多生成的 token。这是”推理时计算缩放”的本质,也是它对系统设计最大的冲击——容量规划从”训练集群多大”变成了”推理要烧多少 FLOPs”。

第二层我不太认同:“生成长度主导后推理变回 compute-bound”,我认为仍然卡在 decode 的内存墙上。理由有三:

隐藏推理 token 就是普通的自回归 decode。 K 步推理不是”更多计算”,而是”更多解码步骤”——每一步照旧是权重从 HBM 流读一遍、生成一个 token,和普通回答的生成完全同构。而 decode 是内存受限的,这在第二篇里已经实测过了(ITL 由 HBM 带宽定价)。K 步推理只是把这段内存受限的路走得更长,没有改变它的物理机制。
实测数据支持”内存墙”而不是”算力墙”。 如果瓶颈真的变成了算力,我们应该看到 prefill 式的机制变化;但实测里 (a) 唯一一段算力受限的环节——prefill(TTFT)——完全没动;(b) 总延迟随 K 严格线性,每步代价恒定,这正是”每个 token 都由 HBM 流读定价”的特征,而不是算力受限的特征。
作者的观点在什么条件下才成立?大批量服务。 Decode 的算术强度随 batch 上升(见第二篇的 roofline 分析):并发足够大时,GPU 算力被充分占用,decode 确实会从内存受限转向算力受限。但那是”系统吞吐”意义上的转变;单请求、低并发下,推理时计算只是”更长的 decode”,依然是内存墙。这和第三篇 KV-Cache 的结论是同一个硬币的两面:长序列的成本由内存定价,而不是由算力定价。

八、个人思考:三把尺子与”先找瓶颈”

Task 4 把前三篇的单点技能收束成一套方法论,我理解成三把尺子:

供给端(数据墙):吞吐上不去先看流水线最窄环节,GPU 利用率低 ≠ GPU 不行;
算法端(投机解码):不换硬件也能加速——用小模型提案、目标模型验证,把串行解码摊成并行验证;
搜索端(DSE):配置空间太大时别靠经验拍脑袋,声明式搜索自动找最优解。

这套方法最有价值的部分不是任何一个公式,而是“先找瓶颈,再优化”的顺序。E3 里 4.58× 的减速、E4 里 2.10× 的加速、E5 里边界解的出现,都是先定位约束再动手的结果。反过来,在不清楚瓶颈的情况下盲目优化,很容易在非瓶颈环节上浪费力气——就像给存储加 NVMe,而真正的瓶颈在 CPU 预处理。

另外一个体会来自 5.3 节:draft 大小的实测和常见直觉不一致,不是谁对谁错,而是建模粒度不同。引擎把 α 当作独立参数,现实里 α 依赖 draft 质量。读引擎源码、看清它把哪些东西解耦,比背结论更能预测”换个条件会怎样”。

九、参考资料与复现

教程链:00 Hello Roofline(第一篇)→ 01 Memory Wall + 02 Two Phases(第二篇)→ 03 KV-Cache(第三篇)→ 04 Starving the GPU+ 12 Design Space Exploration+ Module 2(本篇)。
打卡任务:Datawhale llm-algo-leetcode #136。
复现方法:pip install mlsysim,三个脚本见文末附录,均可直接运行(E1-E3、E4/O1/O3 不依赖集群;E5 使用内置集群 Frontier_8K,见 mlsysim.Systems.Clusters)。
环境:WSL2 + conda 环境 mlsysim(python 3.11,mlsysim 0.1.1),全程无 GPU,纯仿真。

附:完整运行脚本(可直接复制运行)

脚本 1:E1-E3 数据流水线分析

import mlsysim
from mlsysim.solvers import SingleNodeModel, DataModel
from mlsysim.solvers import TransformationModel

from mlsysim.solvers import SingleNodeModel
from mlsysim.core.units import Q_
from mlsysim.show import table, info

model = mlsysim.Models.Vision.ResNet50
hardware = mlsysim.Hardware.Cloud.A100
solver = SingleNodeModel()

# Baseline: ResNet-50 on A100, batch 256, FP16
profile = solver.solve(model=model, hardware=hardware, batch_size=256, precision="fp16")

info("GPU Compute Baseline",
     Model=model.name,
     Hardware=hardware.name,
     Batch_size=256,
     Step_latency=profile.latency.to('ms'),
     Throughput=f"{profile.throughput:.0f} img/s",
     Bottleneck=profile.bottleneck)

from mlsysim.solvers import DataModel

sample_size = Q_("500 KB")  # Average ImageNet JPEG
batch_size = 256

# Data demand = batch_size x sample_size / step_time
step_time_s = profile.latency.to("s").magnitude
data_per_step = (batch_size * sample_size.to("GB")).magnitude
demand_rate = Q_(data_per_step / step_time_s, "GB/s")

data_solver = DataModel()
data_result = data_solver.solve(workload_data_rate=demand_rate, hardware=hardware)

info("Storage I/O Check",
     Data_demand=f"{demand_rate:.3f}",
     Storage_supply=f"{data_result.supply_bw:.2f}",
     Utilization=f"{data_result.utilization:.1%}",
     Is_stalled=data_result.is_stalled)

from mlsysim.solvers import TransformationModel

transform_solver = TransformationModel()
cpu_throughput = Q_("2 GB/s")  # 8 workers x 250 MB/s each

t = transform_solver.solve(
    batch_size=256,
    sample_size_bytes=sample_size,
    cpu_throughput=cpu_throughput,
    accelerator_step_time=profile.latency
)

info("CPU vs GPU Pipeline",
     CPU_transform_time=t.transform_time,
     GPU_step_time=t.accelerator_step_time,
     CPU_is_bottleneck=t.is_bottleneck,
     GPU_utilization=f"{t.accelerator_utilization:.1%}",
     Slowdown_factor=f"{t.slowdown_factor:.2f}x")

rows = []
for bs in [16, 32, 64, 128, 256, 512, 1024]:
    p = solver.solve(model=model, hardware=hardware, batch_size=bs, precision="fp16")

    t = transform_solver.solve(
        batch_size=bs,
        sample_size_bytes=sample_size,
        cpu_throughput=cpu_throughput,
        accelerator_step_time=p.latency
    )

    binding = "Transformation" if t.is_bottleneck else p.bottleneck
    rows.append([
        bs,
        f"{p.latency.to('ms').magnitude:.2f} ms",
        f"{t.transform_time.to('ms').magnitude:.2f} ms",
        binding,
        f"{t.accelerator_utilization:.1%}"
    ])

table(["Batch", "GPU Step", "CPU Xform", "Binding", "GPU Util"], rows)

rows = []
for n_workers in [8, 16, 32,64, 128]:
    cpu_tp = Q_(f"{n_workers * 250} MB/s")

    p = solver.solve(model=model, hardware=hardware, batch_size=512, precision="fp16")

    t = transform_solver.solve(
        batch_size=512,
        sample_size_bytes=sample_size,
        cpu_throughput=cpu_tp,
        accelerator_step_time=p.latency
    )

    rows.append([n_workers, cpu_tp.to('GB/s'), f"{t.accelerator_utilization:.1%}"])

table(["Workers", "Throughput", "GPU Util @ bs=512"], rows)

脚本 2:E5 设计空间探索

import mlsysim
from mlsysim.engine.dse import DSE
from mlsysim.solvers import DistributedModel, EconomicsModel
from mlsysim.engine.pipeline import Pipeline

# Our fixed baseline constants
model = mlsysim.Models.Language.Llama3_70B
fleet = mlsysim.Systems.Clusters.Frontier_8K

# The evaluation function accepts a dictionary of parameters and returns a structured object
def evaluate_config(params):
    pipe = Pipeline([DistributedModel(), EconomicsModel()])

    return pipe.run(
        model=model,
        fleet=fleet,
        batch_size=params["batch_size"],
        tp_size=params["tp"],
        pp_size=params["pp"],
        precision="fp16",
        efficiency=0.45,
        duration_days=30
    )

# 1. Define the dimensions of your search grid
space = {
    "batch_size": [1024, 2048, 4096, 8192],
    "tp": [1, 2, 4, 8],
    "pp": [1, 2, 4]
}

# 2. Initialize the Design Space Explorer
dse = DSE(
    space=space,
    objective="maximize: DistributedModel.effective_throughput"
)

# 3. Search! (Tests all 4 * 4 * 3 = 48 combinations analytically)
print("Starting search...")

result = dse.search(evaluate_config)

# 4. Analyze the best configuration
best_params = result["best_params"]
best_throughput = result["best_objective"]

print(f"\nBest Configuration: TP={best_params['tp']}, PP={best_params['pp']}, Batch={best_params['batch_size']}")
print(f"Max Throughput: {best_throughput:,.0f} tokens/sec")

print(f"\nTop 3 Configurations:")
print(f"{'TP':>4} | {'PP':>4} | {'Batch':>6} | {'Throughput':>12} | {'Latency':>10}")
print("-" * 45)

for candidate in result["top_candidates"][:3]:
    p = candidate["params"]
    dist_res = candidate["result"]["DistributedModel"]

    throughput = dist_res.effective_throughput.m_as("1/s")
    latency = dist_res.step_latency_total.m_as("ms")

    print(f"{p['tp']:>4} | {p['pp']:>4} | {p['batch_size']:>6} | {throughput:>12,.0f} | {latency:>10.1f}ms")

脚本 3:E4/O1 投机解码 + O3 推理时计算

import mlsysim
from mlsysim.solvers import ServingModel, InferenceScalingModel

hw = mlsysim.Hardware.Cloud.H100
target = mlsysim.Models.Language.Llama3_70B
draft8b = mlsysim.Models.Language.Llama3_8B
draft_gpt2 = mlsysim.Models.Language.GPT2
solver = ServingModel()

# E4: 基线 vs 投机解码(8B draft, alpha=0.75)
base = solver.solve(target, hw, seq_len=2048, batch_size=1)
spec = solver.solve(target, hw, seq_len=2048, batch_size=1,
                    draft_model=draft8b, draft_acceptance_rate=0.75)
print(f"baseline ITL: {base.itl.to('ms').magnitude:.2f} ms")
print(f"spec ITL:     {spec.itl.to('ms').magnitude:.2f} ms")
print(f"speedup:      {base.itl/spec.itl:.2f}x")

# O1a: 接受率扫描
for a in [0.5, 0.75, 0.9]:
    r = solver.solve(target, hw, seq_len=2048, batch_size=1,
                     draft_model=draft8b, draft_acceptance_rate=a)
    print(f"alpha={a}: ITL {r.itl.to('ms').magnitude:7.2f} ms  speedup {base.itl/r.itl:.2f}x")

# O1b: draft 模型大小扫描
for name, dm in [("GPT2(1.5B)", draft_gpt2), ("Llama3-8B", draft8b), ("Llama3-70B", target)]:
    r = solver.solve(target, hw, seq_len=2048, batch_size=1,
                     draft_model=dm, draft_acceptance_rate=0.75)
    print(f"draft={name}: ITL {r.itl.to('ms').magnitude:7.2f} ms  speedup {base.itl/r.itl:.2f}x")

# O3: 推理时计算(K=0/8/32)
isolver = InferenceScalingModel()
for k in [0, 8, 32]:
    r = isolver.solve(mlsysim.Models.Language.Llama3_8B, hw, reasoning_steps=k)
    print(f"K={k}: total {r.total_reasoning_time.to('s').magnitude:.3f} s  "
          f"energy {r.energy_per_query.to('kJ').magnitude:.3f} kJ")

如果这篇笔记对你有帮助,欢迎交流。

Fixing a Tutorial Demo: An Open-Source PR Retrospective

Wed, 16 Sep 2026 00:00:00 +0000

In The KV-Cache Memory Ledger, I accounted for where the memory goes with formulas. The tutorial itself is well written, but while reading its third companion experiment (Tutorial 03: KV-Cache) I found that the §5 Paged Attention demo printed identical results for all three configurations. So I opened an issue, wrote a PR to rewrite it. This post documents the whole ride.

1. What Was Wrong with the Demo: an Experiment That Couldn’t Fail

Section 5 wants to show the value of PagedAttention: carve the KV-Cache into fixed-size blocks, allocate on demand, and waste far less than statically reserving whole segments. But the demo printed this:

System	Max Users	Throughput	Frag	Speedup
Static (baseline)	32	3225 t/s	0.0%	1.3x
Paged (16 tok)	32	3225 t/s	0.0%	1.3x
Paged (64 tok)	32	3225 t/s	0.0%	1.3x

Three page sizes, identical results. It did not break (no error), but it demonstrated nothing. This is the most dangerous kind of bug: “it looks like it runs” and “it actually demonstrates the idea” are two different things.

2. Cause: Four Coincidences Stacked on Top of Each Other

It was not a single point of failure; four things collided:

#	Cause	Result
1	`max_batch_size=32` was passed identically to all three configs, and the memory-derived limit (~130) was larger	`min(limit, 32)` is always 32; the batch cap pins the result
2	`seq_len=4096` divides evenly into 2048/16/64	All three page sizes compute the same padded size, so last-page fragmentation is 0% for all
3	The engine models only last-page internal fragmentation; static allocation’s real waste (reserving `max_seq` per request, external fragmentation under churn) is not in the model	The Frag column is identical across rows
4	`speedup_vs_static` is a fixed 0.6 factor (from Kwon et al. 2023), independent of allocation strategy; throughput is decided solely by memory-bound step time	Speedup and throughput have zero response to the allocation strategy

#4 is the killer: even with the batch cap and the divisibility problem removed, the output still would not vary with page size — because the solver being demoed does not distinguish allocation strategies at all. Conclusion: this demo needed a rewrite, not a patch.

3. The Rewrite: Let the Experiment Tell the Theory Itself

The PR’s idea was to drop the engine call and replace it with a self-contained analytical simulation: two small allocators sharing one KV budget —

Static contiguous: every request reserves a whole max_seq slot; simple but wasteful;
Paged block pool: fixed-size blocks allocated on demand, with a block table;

Fed the same long-tailed request stream (fixed random seed, reproducible), so each of the four problems above got its own experiment:

Internal fragmentation: 77.8% waste for static reservation vs 0.4% for 16-token pages; concurrent users 65 → 314;
External fragmentation: after churn, the static first-fit allocator hits 97.1% fragmentation and admits 0 of 10 fresh 8K requests despite free space; paged admits 10/10;
Page-size sweep (8→1024 tokens): the trade-off between waste and block-table bookkeeping;
Prefix sharing: 8 requests sharing a 1K system prompt cut peak KV by 58%.

Every conclusion is directly visible in a table, and every number is reproducible. After the PR landed, the maintainer Vijay Janapa Reddi replied quickly: “diagnosis is spot on and the new demo actually shows the idea” — then he pushed a small fix and merged.

4. The Merge: More Honest Code

Before merging, the maintainer made one more round of changes on top of my commit. Each one is worth studying:

1. The cutoff: stop at the first request that does not fit, instead of feeding the whole stream.

My version fed all 100,000 requests to the allocator — including the ones arriving after memory was exhausted. PagedAllocator.admit() internally rejects requests that do not fit, so short requests after a rejected long one kept getting in; the allocator “picked its food” and the numbers flattered. The maintainer changed it to:

def fill(alloc, lengths):
    """Admit requests in arrival order; stop at the first one that does not fit."""
    for t in lengths:
        if not alloc.admit(t):
            break
    return alloc

Cut off in arrival order. It makes the experiment more “honest”: in a real system, requests keep coming whether you have memory or not, and an allocator should not quietly accept short requests after refusing long ones.

2. Shrink the numbers; embrace the less pretty truth.

My version: 314 users, 1.78x throughput speedup. After the merge: 259 users, 1.47x. The difference is exactly what the “food-picking” had fed. The maintainer preferred a plain but true number over a pretty but cheating one.

5. Three Takeaways from Being Edited

Demo code is ground zero for “looks like it runs”. Crashes and errors get caught instantly; an experiment that prints numbers on every row can be demonstrating nothing. Before writing a demo, ask: does this output actually vary with the independent variable, and by how much?
Honest experiments cut off in arrival order. Feeding the whole stream gives the allocator a choice it should not have. In any queuing/allocation simulation, stop at the first failure; otherwise you are measuring a “food-picking allocator”, not the allocator.
Modeling granularity decides how far a conclusion travels. A fixed 0.6 speedup factor means the “allocation-strategy benefit” dimension is distorted inside the engine — conclusions only hold within that granularity. Understanding what a tool decouples beats memorizing its output.

6. References

Issue: #2134 Paged Attention demo returns identical results for every page size
PR: #2135 fix(mlsysim): redesign §5 Paged Attention demo in KV-Cache tutorial (merged)
Related post: The KV-Cache Memory Ledger — my study notes for tutorial 03; that is where I ran into this demo.

If this retrospective helps you, feel free to reach out.

LT Blog

Readnest:我的阅读记录,为什么要搬出 Notion

一、起因:那张表

二、一开始就定下的三件事

三、Readnest 现在什么样

四、书是怎么进去的

五、AI 那一块

六、什么数据会离开你的手机

七、现在的状态

Readnest: Why I Moved My Reading Log Out of Notion

1. Where it started: the table

2. Three things I decided before writing any code

3. What Readnest looks like now

4. How books get in

5. The AI parts

6. What data leaves your phone

7. Where it stands

交叉熵:语言模型的损失函数,为什么只能是它

一、从一个20年前的实验说起

二、编码一个玩具消息

三、交叉熵公式

四、p 和 q 不能对调

五、最低点就是熵

六、为什么公式非得是 −log

七、大语言模型的损失函数

八、压缩即智能

九、参考资料与复现

Cross-Entropy: The Loss Function of Language Models, and Why It Must Be This One

1. An Experiment From Twenty Years Ago

2. Encoding a Toy Message

3. The Cross-Entropy Formula

4. p and q Cannot Be Swapped

5. The Floor Is the Entropy

6. Why the Formula Has to Be −log

7. The Loss Function of Large Language Models

8. Compression Is Intelligence

9. References and Reproduction

千卡训练的隐形账单：通信不是瓶颈

一、先说结论

二、E2 规模扫描:那张表为什么会反着走

2.1 顺带验一下气泡公式

2.2 E1 的 3D 并行:数字好看,但配置不成立

三、O1 可靠性: Young-Daly 公式该往哪用

3.1 Checkpoint 体积:14 bytes/param 从哪来

3.2 写入带宽:141s → 2s

3.3 教程 §5 的两个口径:同一个作业,三种答案

四、E4 选址:61 倍不是 40 倍

4.1 别只算运营碳

五、E3 经济性:30 天的账单 83.4% 是折旧

5.1 回到”$9M Question”:成本为什么不是线性

六、把两个口径合并看:这张图值得存下来

七、个人思考:今天最大的收获是”口径”

八、参考资料与复现

附:完整运行脚本(可直接复制运行)

The Hidden Bill of Thousand-GPU Training: Communication Is Not the Bottleneck

1. Results first

2. E2 scale sweep: why the table runs backwards

2.1 Checking the bubble formula along the way

2.2 E1’s 3D parallelism

3. O1 reliability: where the Young-Daly formula should point

3.1 Checkpoint size: where 14 bytes/param comes from

3.2 Write bandwidth: 141 s → 2 s

3.3 The tutorial’s §5: two ledgers, three answers

4. E4 site selection: 61x, not 40x

4.1 Do not count operational carbon only

5. E3 economics: 83.4% of a 30-day bill is depreciation

5.1 Back to the “$9M Question”: why cost is not linear

6. Putting the two ledgers together: this chart is worth keeping

7. Personal note: today’s biggest takeaway was “conventions”

8. References and reproduction

Appendix: full script (copy and run)

给开源教程修了个 demo

一、demo 怎么了:一个”跑不坏”的实验

二、原因:四个巧合叠在一起

三、重写:让实验自己讲出理论

四、合并:更诚实的代码

五、复盘:这次被改写的三条收获

六、参考资料

Why Does the GPU Starve: The Data Wall, Speculative Decoding, and Design-Space Exploration

1. From Single-Point Bottlenecks to a Whole-Pipeline View

六、为什么公式非得是 `−log`

6. Why the Formula Has to Be `−log`