AI / Claude Sonnet5 Interview questions
Why does Claude Sonnet 5's tokenizer change matter for migration?
Sonnet 5 uses a new tokenizer that maps the same input text to roughly 30% more tokens than Sonnet 4.6's tokenizer did, meaning token counts, cost estimates, and context-window capacity calculations calibrated against Sonnet 4.6 don't transfer directly.
Because per-token pricing itself is separate from this change, the practical effect is that an equivalent request costs more in absolute terms even without any pricing change, and a max_tokens value that was generous on Sonnet 4.6 can become tight on Sonnet 5 purely from this token-count shift, independent of the separate thinking-by-default change.
The recommended fix is mechanical but essential: re-run token counting against Sonnet 5 specifically, using the token counting API on representative prompts, rather than reusing figures measured against Sonnet 4.6 or assuming the tokenizer change is negligible.
More Related questions...
