LLM Token Counter
Prompt engineers shipping retrieval pipelines hit the context limit in staging and discover the 30-page PDF they stuffed into every request overflows GPT-4o's window well before production traffic even starts. Product teams comparing API bills cannot tell whether a 40-dollar invoice came from a few long prompts or thousands of short ones until someone counts the tokens.
Paste any text and this counter estimates the token total with two independent heuristics (characters divided by four, words multiplied by 1.33) and reports the realistic min-to-max band. The usage table then maps that estimate onto ten popular models — GPT-4o, GPT-4.1, o3, Claude Sonnet 4, Claude Opus 4, Gemini 2.5 Pro, Gemini 2.5 Flash, Llama 4 Maverick, DeepSeek V3, and Mistral Large — showing how full each context window would be and the rough input cost at published API prices.
| Model | Window | Full % | Remaining | Input $ |
|---|
How the estimate is calculated
Modern LLMs tokenize text with byte-pair encoding (BPE), a learned vocabulary that merges frequent character sequences into single tokens. English prose under cl100k-style vocabularies averages close to four characters per token and roughly 1.33 tokens per word, which is why both ratios appear in the estimate. The tool runs both and averages them: characters divided by four gives one view, words times 1.33 gives another, and the mean smooths out cases where either ratio drifts.
The min-to-max band uses characters divided by five on the low end and characters divided by three on the high end. Those bounds cover the spread seen across real prompts — English sits near the low end, code and structured JSON land mid-range, and CJK text or heavily punctuated data pushes toward the high end. If your content is unusual, treat the band, not the point estimate, as the number to plan against.
Character, word, line, and UTF-8 byte counts are exact — they come straight from string inspection with no heuristic involved. Byte count uses the standard TextEncoder, so multi-byte characters like emoji or accented letters contribute more than one. The method note under the stats restates the formula so you can sanity-check the estimate against a vendor tokenizer when precision matters for billing.
Before and after: a typical RAG prompt
Input — a 600-word retrieval prompt (abbreviated):
System: You are a support assistant. Answer only from context.
Context:
[doc 1] Billing cycles run on the 1st...
[doc 2] Refunds process within 5 days...
[doc 3] Enterprise plans include SSO...
Question: Can an enterprise customer get a
refund outside the 5-day window?Output — the estimate reported for that prompt:
Tokens (est.): 812 Range: 640 – 960
Characters: 3,250 Words: 610
GPT-4o (128k): 0.63% full, ~127,188 remaining
Claude Sonnet 4 (200k): 0.41% full, ~199,188 remaining
Estimated input cost on GPT-4o: ~$0.0020Who counts tokens
Prompt engineers
- Check that a system prompt plus few-shot examples leaves enough room for the expected response before shipping to production.
- Compare which model's context window comfortably fits a long retrieval payload without splitting it.
- Estimate cost per request across models when deciding whether a cheaper tier handles the workload.
Production
- Model monthly API spend by multiplying per-request token estimates by expected traffic volume.
- Set max_tokens limits that keep total input plus output under the window during load tests.
- Spot prompts that grew past the limit after a documentation or catalog change and need chunking.
Workflow
- Use the estimate in a CI check that fails a PR when a prompt template exceeds a token budget.
- Paste internal or unreleased text safely — estimation runs locally with no network request.
- Document token budgets in the prompt registry alongside owners and review dates.
Common token-counting mistakes
- Counting only the user turn. System prompts, tool schemas, few-shot examples, and conversation history all count as input tokens. A 200-token question sitting on top of a 25k-token system prompt bills like a 25k request, not a 200-token one. Always paste the fully assembled prompt.
- Treating the estimate as billing-exact. Heuristics track English prose well, but CJK text, emoji, and minified code shift the characters-per-token ratio. When an invoice matters, run a final count with the vendor's official tokenizer or a library like tiktoken before forecasting spend to the cent.
- Ignoring output budget. Context windows cover input plus generated output. Filling 128k of input on a 128k-window model leaves nothing for the response and triggers a truncation or error. Reserve 10 to 20 percent headroom for output on chat and summarization workloads.
- Assuming one number works for every model. A prompt that fits GPT-4.1's 1M window may overflow DeepSeek V3's 64k. Use the per-model rows to pick the right tier per workload instead of defaulting to the largest window available.
Best practices for token budgets
- 1.Set a token budget per feature (for example 8k input, 2k output) and enforce it with a CI check on the prompt template, so regressions surface in review rather than in production logs.
- 2.Chunk long documents and retrieve only the relevant sections; retrieval-augmented pipelines stay under the window more reliably than sending entire files on every call.
- 3.Log prompt tokens per request in your application telemetry alongside the model id, so you can reconcile vendor invoices against your own counts.
- 4.Prefer the smallest model that meets quality bar; GPT-4o and Gemini 2.5 Flash handle many classification and extraction tasks at a fraction of Opus or o3 pricing.
Frequently asked questions
How accurate is this token estimate compared to the real tokenizer?
For English prose the estimate typically lands within 10 to 15 percent of the real BPE count, because most English text averages close to four characters per token and 1.3 tokens per word. Code, non-Latin scripts, emoji, and long runs of digits or punctuation skew the ratio — CJK text can approach one token per character, while minified code packs more characters per token than prose. The min and max range shown (characters divided by five, characters divided by three) brackets the realistic band for mixed content, and you should treat the middle number as a planning figure rather than a billing-grade count.
Why do different models show different token counts for the same text?
They do not, in this tool — the estimate is model-independent, since the character and word ratios come from the same underlying tokenizer family used across most modern LLMs. What differs per model is the context window and pricing, which is why the usage table recalculates for each one. When the same prompt genuinely counts differently in a real product (for example OpenAI cl100k_base versus Anthropic BPE), that comes from vendor-specific vocabularies, and no offline estimate can replicate both exactly without shipping each proprietary tokenizer.
What is a context window and why does it matter?
The context window is the maximum number of tokens a model can attend to in a single request, including the system prompt, conversation history, your input, and the generated output. GPT-4o offers 128k, GPT-4.1 and Gemini 2.5 Pro offer 1M, and Claude Sonnet 4 sits at 200k. When total tokens exceed the window, the provider truncates history or rejects the call outright. The usage table in this tool shows how full each window would be for your prompt, so you know before pasting whether a long document fits or needs to be split.
How is the estimated input cost calculated?
Cost equals tokens divided by one million, multiplied by the model input price per million tokens. For example 50,000 tokens on Claude Sonnet 4 at 3 dollars per 1M works out to 0.15 dollars. Output pricing is listed in the model reference but not folded into the per-prompt figure, because output length depends on generation settings you control (max tokens, temperature, stop sequences). Models without public API pricing, such as open-weight Llama releases, show no cost figure.
Should I count tokens for system prompts and tool definitions too?
Yes. Anything the provider bills as input — system messages, few-shot examples, tool or function schemas, retrieved documents, and the conversation so far — all land in the input token count. Teams are often surprised that a 2k-token system prompt plus a 30k-token retrieved document dwarfs the 200-token user question. Paste the full assembled prompt into this counter, not just the user turn, to get a number that matches what the API will actually meter.
What happens when a prompt approaches the context limit?
Most providers return a context-length error before truncating silently, though some chat products drop the oldest messages instead. A practical buffer is 10 to 20 percent headroom for the output you still need — if the window is 128k and you want a 4k response, keep input under roughly 124k. For long-document workflows, retrieval-augmented pipelines that fetch only relevant chunks stay under the limit far more reliably than stuffing whole files into every request.
Does this tool send my text anywhere?
No. Token estimation runs entirely in the browser with local string math; nothing is uploaded to a server. That makes it safe to paste internal documents, unreleased code, or customer data when checking whether a prompt fits a context window. No text is stored after you close or reload the page.
Related text and developer tools
Exact word and sentence counts for the same text
Character and byte limits for social posts and forms
Strip HTML tags to clean plain text before pasting into a prompt
Clean up JSON payloads that get embedded in prompts
Build API calls that wrap your LLM requests in CI
Check tabular data before loading it into a prompt