quicktok speeds up token counting for AI workflows
quicktok is a tool that turns text into tokens faster before the text is sent to an AI model. It is written in C++ and is designed to produce the same token numbers as OpenAI’s tiktoken for the same input. In shared tests on an Apple M1 chip using one thread, the native quicktok version ran about 4 to 11 times faster than the Python version of tiktoken.
It was also about 2 to 3.6 times faster than bpe-openai. It supports cl100k, o200k, GPT-OSS, Llama-3, and Qwen2.5/3 token formats. The speedup comes from more efficient data structures, cached checks, and a custom pretokenizer instead of a general .
The benchmark can be repeated from the repository with make bench-compare.
Key points
- quicktok turns text into tokens for AI model workflows.
- It is built to match tiktoken’s token numbers for the same input.
- The shared benchmark showed about 4 to 11 times faster speed than Python tiktoken.
- It supports cl100k, o200k, GPT-OSS, Llama-3, and Qwen2.5/3 token formats.
- It may help large s spend less time on ing and text preparation.