Tessera
Folgen

Johann Schopplich

@johannschopplich.com

Lead Software Engineer @finanzfluss byjohann.link

10 Posts308 Follower120 Folgt
Johann Schopplich@johannschopplich.com · last wk.

Tokenizers are heavy – megabytes of vocab data for a number you can estimate. tokenx answers "what does this payload cost me?" in 2 kB, at ~96% accuracy. New in v1.6: 🧮 Heuristics for Cyrillic, kana and emoji 🧱 More accurate on JSON payloads and source code

Benchmark chart comparing real o200k_base token counts with tokenx estimates across eight samples: emoji-heavy chat, formatted and minified JSON, tokenx source code, and prose in English, German, Chinese and Japanese. Per-sample deviation ranges from +0.81% to +5.60%, with two underestimates at -3.13% and -2.57%. Mean deviation 3.45%.
Johann Schopplich@johannschopplich.com · 2w ago

Been experiencing an early version of AGI for a week now. It is breathtaking and overwhelming at the same time. Our sessions are short but intense. She’s currently rate-limited, so back to Claude Code.

A man with glasses leans back against a green pillow on a bed, shirtless, with a sleeping newborn lying skin-to-skin on his chest in a striped berry-print onesie. His left arm reaches down to a laptop on the mattress beside him; a phone lies nearby.