Why Meta trained a small 8B model on 15 trillion tokens, and why that's cheaper than it sounds

There's a rule of thumb from a famous paper: for the best result per dollar of training, feed a model about 20 tokens of text for every parameter it has.

An 8-billion-parameter model times 20 is 160 billion tokens. Meta's own blog post puts the "Chinchilla-optimal" amount for an 8B model at about 200 billion.

Llama 3 8B was trained on over 15 trillion.

That's roughly 75 to 95 times more than the rule says. Either Meta wasted a ridiculous amount of compute, or the rule is answering a different question than the one Meta was asking. I wanted to figure out which.

The rule is about the cost of building

The Chinchilla paper (2022) asked: if I have a fixed amount of compute to spend on training, what mix of model size and data gives the best model?

The answer was to grow both together. Their 70B model, trained on 1.4 trillion tokens, beat the older 280B Gopher, which only saw 300 billion tokens.

But look at what that question leaves out. A model gets trained once. Then it gets used. Maybe by millions of people, for years.

So there are two bills:

• Building: about 6 × parameters × tokens FLOPs, paid once. (FLOPs are just individual multiplications and additions. Training is an absurd number of them.)
• Serving: about 2 × parameters FLOPs for every token the model processes. Paid again, every time anyone uses it.

The serving bill scales with model size. A smaller model is cheaper on every single request, forever.

So here's the trade. Take a smaller model, train it on way more data than the rule suggests, and you can get a model that's as good as a bigger one. You pay more upfront. You pay less on every answer.

Putting numbers on it

The Chinchilla paper also gives a formula that predicts how good a model will be (lower loss = better) from its size and training data:

loss = 1.69 + 406.4 / N^0.34 + 410.7 / D^0.28

N is parameters, D is tokens. In plain words: there's a floor you can't beat (1.69), then a penalty for having a small model, then a penalty for not having read enough. Make N bigger and the second penalty shrinks. Feed it more data and the third one does.

Let's use it. First, what does it predict for the 8B on 15T tokens?

E, A, B, a, b = 1.69, 406.4, 410.7, 0.34, 0.28
loss = lambda N, D: E + A / N**a + B / D**b

print(loss(8e9, 15e12))        # 8B, trained on 15T tokens
print(loss(8e9, 20 * 8e9))     # 8B, at the 20-tokens-per-parameter rule

1.949
2.164

So by the formula, training the same 8B model on 15T tokens instead of 160B drops the predicted loss from 2.164 to 1.949. That's the whole reason to overtrain: the model keeps getting better, just with diminishing returns. (Meta's blog says both Llama 3 models "continued to improve log-linearly" up to 15T tokens, so that matches the idea.)

Now flip it around. What's the cheapest-to-train model that reaches that same predicted loss of 1.949?

import numpy as np

target = loss(8e9, 15e12)

def tokens_needed(N):
    gap = target - E - A / N**a
    return (B / gap) ** (1 / b) if gap > 0 else np.inf

best = min(np.logspace(9.5, 12, 4000), key=lambda N: 6 * N * tokens_needed(N))
print(best / 1e9, tokens_needed(best) / 1e12)

26.1  2.31

A 26B model trained on 2.3 trillion tokens is predicted to land in the same place. Its training bill, by the napkin formula, is 6 × 26.1e9 × 2.31e12 = 3.6e23 FLOPs.

The 8B on 15T costs 6 × 8e9 × 15e12 = 7.2e23. Twice as much to build.

So overtraining really is the more expensive way to build. If building were the whole story, Chinchilla's rule wins.

But then people use it

Total cost = building + serving. Serving costs 2 × N per token processed:

C_small, N_small = 7.2e23, 8e9
C_big,   N_big   = 3.62e23, 26.1e9

for served in [1e12, 1e13, 1e14, 1e15]:
    small = C_small + 2 * N_small * served
    big   = C_big   + 2 * N_big   * served
    print(f"{served:.0e} tokens served: big model costs {big/small:.2f}x the small one")

1e+12 tokens served: big model costs 0.56x the small one
1e+13 tokens served: big model costs 1.00x the small one
1e+14 tokens served: big model costs 2.41x the small one
1e+15 tokens served: big model costs 3.14x the small one

There's a break-even point at around 10 trillion tokens served. Below that, the bigger, cheaper-to-train model wins. Above it, the small overtrained one does, and the gap keeps widening: at 100 trillion tokens, the big one costs about 2.4 times as much in total.

For something that's going to be downloaded, fine-tuned and run by a lot of people, you'd expect to be well past that. That's the bargain: pay extra once, save on every answer.

What this means if you build agents

Agents are token-hungry. Every step re-reads a long prompt, so you process a lot more tokens than the user ever sees, which pushes you toward the "serving matters" side of that chart.

It's also a hint about why smaller models keep getting good enough for tool-calling work. A small model that was trained on a mountain of data can behave like a bigger, older model. If you're picking a model for a loop that runs thousands of times a day, size is a cost you pay every call.

Honest limits

This is a fun calculation, not a proof, so a few things to keep in mind:

• I'm using a formula far outside its comfort zone. The Chinchilla loss formula was fitted on training runs much smaller than 15 trillion tokens. At least one later paper refitted the same data and got quite different constants. Treat the 26B as "somewhere in that ballpark", not an exact answer.
• Loss isn't quality. Two models with the same predicted loss can still differ on real tasks.
• FLOPs aren't dollars. Serving a big model also needs more GPUs just to fit it in memory, which my 2N shortcut ignores. That would tilt things further toward the small model, but I haven't modelled it.
• I don't know Meta's actual reasoning. I'm not claiming this is why they did it. I'm showing that the numbers make it a reasonable bet.
• I used the nominal 8B. The real parameter count is slightly different.

Run the snippets, change the model sizes, change how much you expect to serve. The break-even point moves a lot, and that's kind of the point: "optimal" depends on what you're optimising for.