Tokenizer
(2026)
Connor's Tokenizer is an interactive browser tool for comparing how ten tokenizers split and count text.
It places Connor's byte-lossless Unigram tokenizer beside nine established baselines and shows every result in one view.
The site also includes a fixed, reproducible benchmark for compression, multilingual text, robustness, and speed.
Case study
Tokenizer
What I focused on
For Tokenizer, I brought product thinking, interface design, and frontend engineering together around one clear problem.
Connor's Tokenizer is an interactive tool for comparing a byte-lossless Unigram tokenizer with nine established tokenizer baselines.
You can enter text or choose a prepared example to compare token counts, token boundaries, and simple ratios across all ten tokenizers. The work runs locally in a browser worker, so entered text is not sent to an AI provider.
A separate benchmark uses 2,116 fixed records to compare compression, multilingual coverage, robustness, and throughput. The repository includes source revisions, licenses, hashes, and scripts needed to reproduce the results.
The project keeps its claim narrow: token counts and benchmark metrics can compare tokenizers, but they do not prove that a language model will be better or cheaper to train.
What I can help with
- Websites and web apps
Custom builds shaped around the product, audience, and way the team needs to work.
- Frontend systems
Responsive layouts, reusable components, semantic HTML, and architecture that is easy to extend.
- Interaction and motion
Purposeful animation, 3D experiences, and polished details that make an interface feel intuitive.
- Product interfaces
Dashboards, learning tools, AI products, workflows, and design systems built for real use.
- Performance and accessibility
Fast, crawlable pages with clear content, structured data, and accessible interactions.
