- The fact that OpenAI documents theirs is already a big improvement over Anthropic. But, also, the OpenAI tokenizer got more efficient when they last updated it, rather than less. https://mdstudio.app/o200k-base-tokenizer
- Yeah, Anthropic's current tokenizer in Sonnet 5/Opus 4.8/Fable 5 is much worse than OpenAI's. Also, OpenAI has been using their current o200k_base from the day GPT-4o came out over two years ago. Just a few of my own tests:
- A ~2000-2002 legacy C++ game codebase at about ~90kloc: GPT 1.12M, Claude 2.2M
- A ~30kloc TypeScript codebase: GPT 260K, Claude 437K
In the end, GPT's current tokenizer is ~1.6x-2x better than Claude's current one, depending on your data. And you can check for free for both, for OpenAI just use the open-source libraries, for Anthropic - you have to use their count_tokens endpoint as they don't publish the tokenizer, but the endpoint is free (and allows requests over 1M tokens as well).
- > DeepSeek and GLM are left out of the tables entirely: we only have rough characters-divided-by-four estimates for them, not real tokenizer counts, and this post is about measured numbers.
lolwut. The open-weight models are inscrutable black boxes for which we can't possibly get real token counts? Typical lazy clanker, BSing their way out of doing the whole job.
- It's a bit unrelated, but I've been wondering if LLM providers are using cache read costs to preserve the illusion of constant output token prices. In reality, the longer your context the more expensive output tokens are, but anthropic and openai both have flat output token pricing.
In practice though as a result of cache reads over multiple turns you will end up paying quadratic pricing anyway.
- We've switched the default model in playcode.io among Opus 4.8, Opus 4.6, Sonnet 4.6, and Sonnet 5. I must admit, Opus 4.8 is quite expensive, and the costs accumulate quickly. Opus 4.6 is about 50% cheaper, while Sonnet 5 is significantly more affordable. According to the data, Sonnet 5 is about 2-3 times cheaper. Fable 5 is unaffordable at all...
Today, I tested Sol 5.6 on various tasks. It performs similarly to Opus 4.8 but is still noticeably more expensive than Sonnet 5. Although Sonnet 5 isn't the top model, it's quite effective for creating typical websites for small and medium businesses. However, they will increase the price starting September 1, as their free offer is ending.
I'm also actively testing Grok 4.5. There's something promising about it. The design is mediocre, in my opinion, but it operates quickly and reliably without any deadloops. Usually, Grok models would fail or loop, but this one is stable.
Overall, I really want a benchmark based on real tasks.
- The real elephant in the room is pricing for KV cache writes and reads. That makes all the difference for tasks with large context.
by luciana1u
1 subcomments
- Anthropic's tokenizer being 2x less efficient means they're essentially charging you for whitespace. premium whitespace, mind you — each space character gets its own attention head.
- @dang, why is this flagged?
by ricardobeat
3 subcomments
- I have reduced usage of Fable and Sonnet 5 to a minimum. Fable in particular is amazing at creative tasks, but not worth the cost for almost everything else. I can have Opus 4.6/4.7 running non-stop without hitting quota, vs maybe 20 minutes of Fable usage.
by iLoveOncall
1 subcomments
- Very unpalatable completely LLM-written article, but on top of that a lot of the fundations and conclusions are completely wrong, the main one being this one:
> You will see people claim Claude uses 2x to 4x the tokens of GPT. Our measurements do not support that, and overstating it would undercut the real point.
It's not because a single prompt represents only 1.7x the number of tokens that a model doesn't use 4x as many tokens as another, when running as an agent. This doesn't take at all the number of tokens of the output into account, and the number of tokens of the potential tool calls from this output, which directly feeds back to input tokens.
The article also has a very small test set (16 documents), all of very small length (15K tokens at most, when models go up to 1M in context and agents routinely exceed this and have to summarize).
Complete garbage article.
by zaptheimpaler
0 subcomment
- Wow this is great, it explains what I've seen. My Codex sub seems to last way longer than Claude in real codebases, and Claude eats up a ton of context in the initial read. I thought the harness might be the cause but it seems like tokenization is probably the bulk of it.
by karma_daemon
0 subcomment
- The "Honest size of the effect"
Honestly, haha
- The real price is how often a model uses subagents that scan your whole repository multiplied by thinking level.
by socketcluster
1 subcomments
- It's a terrible metric, it's kind of like companies paying employees by the hour for white collar work.
Most people here probably don't know what it was like to work a contract job and being paid based on actual deliverables.
The incentive of AI companies is to create as many tokens as possible to solve any given problem. Just like your incentive as a software engineer is to create as much complexity as possible in order to use up as many hours as possible.
This is why big tech companies have millions of lines of code... They've got thousands of engineers rapidly churning out tokens.
The difference in number of tokens I use in my day job vs side projects is massive. You can see the inefficiency quantified.
Show me the incentive, I show you the result.
- This piece focuses on the cost differences from the tokenizer, which do matter, but I wish they emphasized more that even adding the tokenizer to your calculation doesn't provide you with a good way to calculate cost for agentic coding tasks.
Other traits where models differ that have an even greater impact on your total spend:
* How much context do they load in to solve a given task?
* How long do they spend thinking to get equivalent results?
* How many times do they stop and ask you for input, and are you there to respond to them before the cache runs out?
* Etc.
Incorporating the tokenizer just makes a very imprecise measurement of cost a little bit more precise, but in my own experience I have not found that the token cost is a significant driver of task cost whether or not you incorporate the tokenizer. Everything else about the model's behavior has a much larger impact.
- Yes, and verbosity (thinking) is a huge factor.
by semiquaver
1 subcomments
- Are there any advantages of the new tokenizer? Does it have a larger or smaller vocabulary or just differently weighted?
by charcircuit
0 subcomment
- This article doesn't address the inference side cost. Not all tokens cost the same. If the response is predictable you get a few output tokens for for the price of 1. The further an output token is the cost of generating it grows linearly due to attention.
- this is surprisingly high delta. to make matters worse, reasoning tokens account for the majority of tokens and they are completely opaque so it's hard to tell how much of that is prose or code
by dinobones
1 subcomments
- Just start pricing in bytes input/output. This whole "token" and "tokenizer" thing is an implementation detail that shouldn't even be leaking out into the API.
Providers change tokenizers all the time with model updates, and it's often not even possible to query/figure out how text is tokenized without actually just sending the LLM a request.
Just switch to charging for bytes of intelligence. Please. Claude Shannon figured this out decades ago.
- Honestly, I find performant pricing where they test each model on the same task is much more useful than figuring out the tokens or using the input token tax.
The reason being is that the only tokens I feel I really control are the input tokens, but the whole program seems to just run itself and they just charge you what they want to charge you and it’s more of a black box.
Very interesting article though.
- Meta: why are the scrollbars hidden / invisible? Awful UX :(
by sscaryterry
1 subcomments
- My take from this is that Anthropic is screwing us again. I hope AMD shows them up again.
- Where is it getting more tokens in the same source file? Like if you kept minimizing what you're feeding it, what accounts for the difference?
- An individual token, and the level of energy it represents (electricity, or relative effectiveness per model) increasingly seems the space of obsfucation.
This space can be increasingly avoided by becoming, and remaining, efficient and effective with prompts.
- Is it on topic to complain about the various claude-isms in this article? I don't know any actual humans that write titles like "Two floors the rate card hides".
I find my brain disengages once I suspect something of being written by an LLM. If the author didn't put much effort into writing it, should I expect them to have put much effort into fact-checking it?
Edit: this specific title has been deleted from the article. That was not my point! Please put in more effort into writing things that you want others to read! Rather than putting in low effort but being better at hiding it.
- [flagged]
- [flagged]
by ignoramous
1 subcomments
- Tokenzier aside, a report shared on reddit found that the GPT 5.6 (edit: 5.5) series are incredibly thrifty with CoTs, resulting in cheaper bills than GLM 5.2 (let alone Opus/Fable): https://www.reddit.com/r/ZaiGLM/s/rUoG5adkPh
Chattiness remains an open issue for some of the SoTA open weights & (to a lesser extent) Claude.
Hacker News