What it actually costs to train an LLM: GPUs, days and dollars, worked out step by step

You've seen the headlines. "Training a big model costs tens of millions of dollars."

Okay, but why that number? Where does it come from? I wanted to count it myself, with a formula short enough to fit on a napkin, and then check the answer against something real.

Turns out you can get surprisingly close. And the places where you don't get close are the most interesting part.

Four words you need

No jargon after this, promise.

• Parameters: the numbers inside the model that get adjusted during training. "8B" means 8 billion of them. Bigger usually means more capable, and more expensive.
• Tokens: the little text chunks a model reads. Roughly three quarters of a word each. Training data is counted in tokens.
• FLOPs: floating point operations, aka individual multiplications and additions. Training is basically an absurd number of these.
• GPU-hour: one GPU running for one hour. Rent 1,000 GPUs for 10 hours and you've used 10,000 GPU-hours.

The napkin formula

Here it is:

training FLOPs  ≈  6 × parameters × tokens

That's it. Model size times data size times 6.

Where does the 6 come from? The usual explanation: for each token, the model does about 2 operations per parameter going forward (making a prediction) and about 4 going backward (figuring out how to improve). 2 + 4 = 6. It's a rule of thumb, not exact, but it's the one everybody uses.

Quick sanity check, because I don't trust a formula until it matches something:

• Chinchilla, a famous 70B model trained on 1.4 trillion tokens: 6 × 70e9 × 1.4e12 = 5.9e23 FLOPs. The paper's own table says about 5.8e23.
• Llama 3.1 405B, trained on 15.6 trillion tokens: 6 × 405e9 × 15.6e12 = 3.8e25. Meta's paper says 3.8e25.

Matches. Good enough for a napkin.

Turning FLOPs into hours

Now, how fast does a GPU chew through FLOPs?

An NVIDIA H100 (the chip these big runs use) has a peak of about 989 TFLOP/s in the number format used for training. That's 989 trillion operations per second. Sounds huge. It is.

But "peak" is the speed on the spec sheet, in a perfect world. Real training spends time moving data around, waiting on other GPUs, and so on. Meta said their best setup reached over 400 TFLOP/s per GPU, which is about 40% of peak. (Their paper puts utilisation at 38 to 43%.)

So:

GPU-hours = FLOPs / (speed per GPU × 3600)

Napkin vs what Meta reported

Meta published GPU-hours for their models. So I could check my napkin against the real thing.

(I used the nominal sizes, 8B, 70B and 405B. The real counts are a bit different.)

The 405B result is fun: Meta reported 30.8M GPU-hours, and my napkin at 400 TFLOP/s said 26.3M. Not bad.

But look at the 8B. Meta reported 1.3M hours, and my napkin said 0.5M. Off by 2.6x.

So I flipped the question around. If Meta's hours are right, what speed did the GPUs actually deliver? Dividing the work by the reported hours gives:

• 8B: about 154 TFLOP/s, 16% of peak
• 70B: about 273 TFLOP/s, 28% of peak
• 405B: about 341 TFLOP/s, 35% of peak

The bigger the model, the better the GPUs were used. My guess, and it is a guess, is that small models don't keep a huge GPU busy as efficiently, and that the 6ND rule skips some real costs (attention, restarts, evaluation runs). I haven't dug into Meta's logs, so take that as speculation. The lesson I'd actually take: the napkin is great for order of magnitude, and the weak spot is always GPU speed. Use a range, not one number.

Okay, but dollars

GPU-hours times price per GPU-hour.

Big caveat: Meta owns its GPUs, so this isn't what Meta spent. It's what you'd pay to rent that much compute, and rental prices vary a lot. I'm using $2 to $4 per H100-hour as an assumption, not a quote.

That's just the final training run, by the way. It doesn't include failed experiments, data work, salaries, or the cost of all the smaller models you train first to figure out what to build.

"How many GPUs" is a question about time

Here's the part people miss. GPU-hours is a total. How many GPUs you need depends on how long you're willing to wait:

GPUs needed = GPU-hours / hours you can wait

The 8B run at 1.3M GPU-hours:

• On 1,000 GPUs: 1,300 hours, about 54 days.
• On 2,048 GPUs: about 635 hours, around 26 days.

(Those cluster sizes are my examples, not what Meta used for that model.)

For the 405B: 30.84M GPU-hours spread over 16,384 GPUs is about 1,880 hours, roughly 78 days, if every GPU worked the whole time. Real runs don't, so treat that as a rough ballpark.

Try it yourself

Here's the whole thing as a script. Change the numbers.

def train_cost(params, tokens, tflops_per_gpu, dollars_per_gpu_hour, n_gpus=None):
    flops = 6 * params * tokens
    gpu_hours = flops / (tflops_per_gpu * 1e12) / 3600
    out = {"FLOPs": f"{flops:.2e}",
           "GPU-hours": f"{gpu_hours:,.0f}",
           "cost": f"${gpu_hours * dollars_per_gpu_hour:,.0f}"}
    if n_gpus:
        out["days"] = f"{gpu_hours / n_gpus / 24:.1f} days on {n_gpus} GPUs"
    return out

# a 1B model, 20 tokens per parameter (Chinchilla's rule of thumb), one H100 at 150 TFLOP/s
print(train_cost(1e9, 20e9, 150, 2))
print(train_cost(1e9, 20e9, 150, 4))

{'FLOPs': '1.20e+20', 'GPU-hours': '222', 'cost': '$444'}
{'FLOPs': '1.20e+20', 'GPU-hours': '222', 'cost': '$889'}

So a small 1B model, on a sensible amount of data, is about $450 to $900 of rented GPU time. That's a "weekend project and a credit card" number, not a "nation-state" number. It's the 8B going to 70B going to 405B that moves you from "hobby" to "company" because the work multiplies each time.

And for fun, I measured my own laptop CPU while running small experiments: about 30 GFLOP/s on a tiny model. Same 1B model:

1.2e20 FLOPs / 3e10 FLOP/s = 4e9 seconds  ≈  127 years

So no, I won't be training that one on the couch.

What I'd take away

• 6 × parameters × tokens is a good napkin for training work. It reproduced both Chinchilla's and Llama 3.1 405B's published compute.
• GPU speed is the shaky number. Real training reached roughly 16% to 35% of an H100's peak in Meta's reported hours. Always give a range.
• Dollars depend on rent, and rent is an assumption. The reported GPU-hours are the solid part.
• Doubling the model, or the data, doubles the bill. Doubling both quadruples it.

Honest limits

I used nominal model sizes, rented-GPU prices I picked myself, and Meta's reported hours as ground truth. The explanation for why the 8B ran at a lower share of peak is my guess. And the 6ND rule is a simplification: real runs have extra costs it ignores.

If you want to check my arithmetic, the numbers in the table come straight from the two Llama model cards and the Llama 3 paper. Run the script with your own model size and see what a "small" model costs. I was surprised how small.