The Token Sticker Price Is a Trap

Jan Iłowski's Price per 1M tokens is meaningless is a useful correction to a bad procurement habit. His argument is simple: two models can advertise a neat $X per 1M tokens price, but that number stops being comparable when tokenizers, hidden reasoning, output length, cache pricing, and actual task success move independently.

My read is that the title is intentionally sharp, but the underlying lesson is right. Price per million tokens is not meaningless as an input. It is meaningless as a decision rule. If a team is choosing models from a spreadsheet without measuring cost per completed task on its own workload, it is not buying cheaper AI. It is buying uncertainty with a cleaner unit label.

Answer Snapshot

The Unit Is Not Neutral

The first problem is tokenization. Iłowski gives a concrete example: the same text from his post counted as 160 tokens for GPT-4o and 200 tokens for GPT-4 1106-preview using tiktokenizer. I am not treating that calculator output as a universal constant, but the mechanism is real. OpenAI's own tiktoken guide says different models use different encodings, and token counts matter because API usage is priced by token.

Anthropic makes the point even more directly in the Claude Sonnet 5 documentation. The docs say Sonnet 5 uses a new tokenizer, that the same input produces approximately 30% more tokens than Sonnet 4.6 depending on content, and that this affects token counts, context capacity, output budgets, and per-request cost even when per-token pricing is unchanged.

That turns a clean price table into a measurement problem. A model can look cheaper per token while making the same prompt larger. Another can look expensive per token while using fewer tokens to complete the same task. Once that happens, the unit price is still true, but the comparison is incomplete.

Task Cost Is The Better Question

The stronger part of Iłowski's post is the move from token price to task cost. He uses Artificial Analysis data to compare selected models by input/output price, Intelligence benchmark score, and cost per benchmark task. In his table, GPT-5.5 is listed at a higher nominal output price than Claude Opus 4.8 max, yet a lower cost per benchmark task. DeepSeek V4 Pro max is shown as a cost-efficiency outlier, while GLM-5.2 max is much cheaper per token than the GPT and Claude examples but not proportionally cheaper per task.

I would not overfit to those exact rows. Model prices change, router prices can differ from first-party prices, and a benchmark mix is not a substitute for a product's actual workload. But the measurement direction is right. Artificial Analysis says its cost-per-task metric uses input, cached, reasoning, and answer-token prices multiplied by tokens consumed across the workload. It also notes that models producing longer answers or more reasoning tokens can cost more per task even at identical per-token prices.

That is the practical distinction. The buyer should care less about how many tokens a vendor sells for a dollar and more about how much verified work a dollar buys. If the task is customer support triage, the unit is a resolved case. If it is code review, the unit is a useful finding that survives human review. If it is document extraction, the unit is accepted structured output, not generated tokens.

The metric I would put in front of every AI budget review is outcome per dollar at a declared quality threshold. Token price belongs underneath that metric, not above it.

Hidden Work Changes The Bill

The second problem is that visible output is no longer the whole story. Iłowski points at "thinking" tokens as a cost driver: reasoning can improve quality, but the hidden chain of work can dominate the bill. I would phrase the risk even more broadly. The bill also includes retries, tool calls, cache writes, long context, guardrail refusals, formatting cleanup, and human review time.

This is where I found TensorZero's analysis useful as independent context. TensorZero reports that identical inputs can produce materially different token counts across providers and content types, and argues that teams need to measure what they actually send rather than compare list prices alone. Its exact workload is not everyone's workload, but the conclusion matches the operational problem: the cheapest provider changes when the input shape changes.

Public technical discussion points in the same direction. In a Hacker News thread about an LLM pricing tool, one commenter immediately asked how to compare cache pricing across providers because key-value caching can become a major part of actual token usage. A separate HN thread about AI costs gets into serving-cost math and assumptions for GLM-5.2. I would not use comment threads as settled economics, but they show the right kind of skepticism: price, provider routing, cache behavior, throughput, and self-hosting cost are different questions.

The Source Is Right, But It Needs A Boundary

The critique I would add to Iłowski's post is that "cost per benchmark task" can become its own false comfort. It is better than price per token, but it is still a proxy. A benchmark tells me something about a model under a published harness. It does not tell me whether the model follows my schema, respects my latency budget, handles my edge cases, avoids my failure modes, or passes review from the people who own the workflow.

That matters because token efficiency can be purchased in ways a product may not want. A model that gives shorter answers may be cheaper but less useful. A model that reasons longer may be expensive but reduce rework. A model that routes through a cheaper provider may look good until availability, privacy posture, moderation behavior, or support quality becomes the real constraint.

So my version of the lesson is narrower and more actionable: do not replace token-price worship with benchmark worship. Use public benchmarks to form a hypothesis. Then run a workload-specific eval that measures success rate, latency, retry rate, review cost, cache hit behavior, and total dollars per accepted result.

What I Would Do In Practice

If I were designing the buying process, I would start with a small task catalog. Separate cheap routine work from ambiguous, high-stakes, long-context, and customer-visible work. Run each class against candidate models with the same prompts, same tool permissions, same retrieval context, and the same acceptance rubric.

Then I would log more than tokens. I would log prompt tokens, output tokens, reasoning or thinking tokens when exposed, cache reads and writes, tool calls, latency, retries, refusal rate, human override rate, and accepted-output rate. Only then does the pricing table become useful, because it can be multiplied against observed behavior instead of imagined behavior.

The product pattern is model routing with evidence. Cheap models earn routine traffic when they pass. Expensive models earn escalation traffic when they reduce failures enough to justify the cost. Local and open-weight models get considered where privacy, volume, or latency makes them compelling. Frontier models stay in the path where their extra capability proves itself.

My Bottom Line

Iłowski's post matters because it attacks a number that feels objective but can be misleading in production. The price per million tokens is easy to copy into a spreadsheet. The real cost is harder: how many tokens the model creates from your input, how much hidden work it does, how often it succeeds, and how much cleanup the organization has to fund afterward.

I would still keep token prices in the model card. I just would not let them make the decision. The decision should be cost per accepted task, measured on the work the product actually performs. Anything less is pretending that the meter is the same thing as the bill.

Originally published at markhuang.ai