GPT Tokenizer (real cl100k_base BPE)
WebEverything runs locally in your browser — nothing is uploaded
This is real BPE tokenization, not the “about four characters per token” estimator. That approximation happens to be roughly right for English prose, and wrong in exactly the cases that matter: Chinese (a character is often 1–2 tokens), code (indentation and punctuation are split by BPE alone), and billing or context budgeting (the tokens you miscount are the money you miscount). It follows the official cl100k_base rules: split the text with tiktoken’s pattern, then for each piece repeatedly merge the adjacent pair with the lowest rank, and map each rank back to a token id. So the output matches OpenAI’s: `"hello world"` gives `[15339, 1917]`. The vocabulary (about 1 MB) loads only on this page, on demand, unpacked with the browser’s own `DecompressionStream` — no third-party library and no network. Everything runs in your browser; the text you paste never leaves the page.
Features
- Real BPE: the official cl100k_base pattern plus lowest-rank pair merging
- Lists every token’s id and text, one per line
- When a Chinese character is split across two tokens, the half UTF-8 sequence is shown as hex instead of mojibake
- Reports token count, character count and UTF-8 bytes
- Computes bytes per token, handy for billing and context budgets
- Splits Chinese, English and code by the official rules
- The vocabulary loads on this page only, so other pages stay light
How to use
- Paste the text you want tokenized
- The token, character and byte counts appear immediately
- Each token’s id and text are listed below
- Turn off the per-token list or the byte rows if you do not need them
FAQ
- Is this real tokenization or an estimate?
- Real. It loads the full cl100k_base BPE vocabulary (about 1 MB) and splits and merges by tiktoken’s official rules, matching OpenAI token for token. Check it yourself with `"hello world"` → `[15339, 1917]`.
- Why does a four-character Chinese phrase come out as more than two tokens?
- Because cl100k_base contains tokens only for individual Chinese characters; a two-character word that is not itself a token gets split into three, such as `3574 244 98220` (the first of these is half of a character, which is why it is shown as hex). The pair `[57668, 53901]` belongs to a different, shorter input — the two examples are easy to mix up.
- Does it support GPT-4o’s o200k_base?
- Only cl100k_base (the one GPT-3.5 / GPT-4 use) is provided. It does not claim to offer a vocabulary it does not have — another vocabulary would add about 1 MB of data, and can be added when there is demand.
- Does it use the network?
- No. The vocabulary ships with the page and is unpacked in your browser. Your text is used for local computation only and is never sent anywhere.
Related tools
Word Counter
TextCount Chinese characters and English words separately, alongside characters, lines, real UTF-8 bytes, reading and speaking time, and SMS segment counts.
Occurrence Counter
TextCount how often each character, word, line or given string appears, sorted by frequency with percentages. Spaces and breaks are labelled.
Text Inspector
TextFind invisible zero-width characters, BOMs and lookalike letters, and report normalization form, line endings and four character counts. All local.
URL Parser
WebParse a URL into scheme, host, port, path, query parameters and fragment, with each parameter shown raw and decoded. Runs entirely in your browser.
User-Agent Parser
WebParse a user agent into browser, version, OS, device type and engine, and see which rules matched. Pattern matching only. Runs entirely in your browser.
HTTP Status Codes
WebLook up an HTTP status code for its meaning, 1xx–5xx class, typical uses and common traps. Runs entirely in your browser.