Latency works as a silent tax on every product built on Claude. You can ship the smartest agent in the world, but if the first token takes three seconds to appear, users will think it is broken. In 2026 Opus 4.8 and Sonnet 4.6 are in production and agent fan-out is the default architecture, so latency tuning is no longer a nice-to-have but a requirement. This is the playbook we use internally and recommend to teams building on the Claude API through codexkey.
What a Claude API request's time is made of
First you need a shared vocabulary. Three numbers are worth tracking:
- TTFT (time to first token): the wall-clock time from the moment your client sends the request to the moment the first streamed token arrives. This is what the user perceives as "did anything start?".
- Total time: TTFT plus the time spent generating the rest of the response. As a rough model:
total = TTFT + output_tokens / tokens_per_second. - Tail latency (p99): the slowest 1% of requests. The tail decides whether your timeouts fire and your queues back up. A median of 800ms with a p99 of 12s is a worse system than a median of 1.2s with a p99 of 2.5s.
If you only watch averages, you are flying blind. Track all three per model, per region and per prompt template.
Per-model TTFT: what we measured
We run continuous probes from an EU PoP against the codexkey gateway, which sits in the same region. The numbers below are 30-day medians on small prompts (under 1K input tokens) with no caching:
- gpt-5.6-luna: TTFT around 250ms, generation around 180 tokens/sec.
- Sonnet 4.6: TTFT around 380ms, generation around 95 tokens/sec.
- Opus 4.8: TTFT around 700ms, generation around 55 tokens/sec.
There is no Haiku tier in our lineup. The cheapest model is gpt-5.6-luna, but it is a different family: the request format is compatible, while behavior and prompts may differ, so check it against your own task.
The pattern is consistent: the bigger the model, the longer the TTFT and the slower the stream. Opus is roughly 3× slower per token than gpt-5.6-luna. For interactive UIs this is the biggest knob you have. A classification step that runs on Opus "because the team standardized on Opus" is probably costing you a second of perceived latency with no gain in quality.
Why streaming halves perceived latency
A non-streaming request blocks until the full response is generated. For a 500-token answer on Sonnet that is roughly 380ms + 500/95 ≈ 5.6s before the UI shows anything. Streaming changes the picture: rendering starts at TTFT (380ms), and the user reads the response as it is generated. Total time stays the same, but perceived latency drops from 5.6s to 380ms. Users consistently rate streamed responses as "fast" even when the total time is longer than a non-streaming alternative.
In 2026 there is essentially no reason to call the Messages API without stream: true on a user-facing path. The only exceptions are batch jobs and tool-call orchestration, where you need the full message before acting.
Prompt caching cuts TTFT by about 50% on cached blocks
Anthropic's prompt caching is usually pitched as a way to save money: cached input tokens are billed at roughly 10% of the normal rate. Its effect on latency is just as strong. When a request hits a cached prefix, TTFT drops by roughly 50%, because the model does not have to re-process the cached tokens. On Sonnet, a 20K-token system prompt that normally adds ~600ms to TTFT collapses to ~300ms once cached.
Mark a long system prompt, a tool schema or a stable few-shot block with cache_control: { type: "ephemeral" } and TTFT starts falling from the second request. The five-minute TTL is short, but in a busy production system you will hit it on almost every request.
Response length: the biggest lever
The most underused optimization is simple: generate fewer tokens. Every output token on Sonnet costs ~10ms of wall time. Squeezing a response from 800 tokens to 300 saves five seconds. Three concrete techniques:
- Set
max_tokenstightly. Do not passmax_tokens: 4096"just in case". Pass the real ceiling for the task: 200 for a classification reason, 500 for a summary, 1500 for a code edit. - Ask for brevity. Add "Respond in under 100 words" or "Output only the JSON object, no explanation" to the system prompt. Models in 2026 are very good at following length constraints.
- Use stop sequences. If you know the response ends with
</answer>or}, pass it as a stop sequence. The model halts immediately instead of generating trailing whitespace or a closing pleasantry.
Parallel calls: agent fan-out with Promise.all
If your agent needs to call the model three times for independent sub-tasks (extract entities, score sentiment, draft a summary), do not queue them up. Sequential calls add their TTFTs together, while parallel calls hide them behind the slowest one.
const [entities, sentiment, summary] = await Promise.all([
client.messages.create({ model: "gpt-5.6-luna", ... }),
client.messages.create({ model: "gpt-5.6-luna", ... }),
client.messages.create({ model: "claude-sonnet-5", ... }),
])
The total time is now max(TTFT_a, TTFT_b, TTFT_c) + max(generation_a, b, c) instead of the sum. For a three-step agent this typically halves total latency.
Geography matters more than it seems
Anthropic's primary inference is hosted in the US. A round-trip from Frankfurt to us-east-1 adds roughly 90ms per request before any model work begins. For a streaming UI that makes 10 sequential calls in a session, that is nearly a second of pure network latency.
The codexkey EU PoP terminates the TLS handshake in-region and proxies to the model over a warm, pooled connection. For European clients this typically removes 60-120ms from TTFT compared with calling Anthropic directly across the Atlantic. It is the cheapest latency win available: no code changes, just a different base URL.
Code: from baseline to a concise, cached answer
Here is the same task (summarize a 5K-token document) in four variants, from baseline to the fastest, against https://api.codexkey.online/v1:
import Anthropic from "@anthropic-ai/sdk"
const client = new Anthropic({
apiKey: process.env.CODEXKEY_API_KEY,
baseURL: "https://api.codexkey.online/v1",
})
// 1. Baseline: no streaming, no cache, no max_tokens discipline
// Measured: TTFT 4.2s (because non-streaming TTFT == total time)
await client.messages.create({
model: "claude-sonnet-5",
max_tokens: 4096,
messages: [{ role: "user", content: longDocument + "\n\nSummarise." }],
})
// 2. Add streaming. Total time unchanged, perceived TTFT drops to 0.5s
const stream = await client.messages.stream({
model: "claude-sonnet-5",
max_tokens: 4096,
messages: [{ role: "user", content: longDocument + "\n\nSummarise." }],
})
// 3. Add prompt caching on the document
await client.messages.stream({
model: "claude-sonnet-5",
max_tokens: 4096,
system: [
{ type: "text", text: longDocument, cache_control: { type: "ephemeral" } },
],
messages: [{ role: "user", content: "Summarise the document above." }],
})
// 4. Add concise instruction + tight max_tokens
await client.messages.stream({
model: "claude-sonnet-5",
max_tokens: 300,
stop_sequences: ["</summary>"],
system: [
{ type: "text", text: longDocument, cache_control: { type: "ephemeral" } },
],
messages: [{
role: "user",
content: "Summarise in under 150 words, wrap in <summary>...</summary>.",
}],
})
How to measure
Tune what you measure. Two metrics are worth instrumenting on every request:
const t0 = performance.now()
let ttft: number | null = null
let outputTokens = 0
const stream = await client.messages.stream({ ... })
for await (const event of stream) {
if (event.type === "content_block_delta" && ttft === null) {
ttft = performance.now() - t0
}
if (event.type === "message_delta") {
outputTokens = event.usage.output_tokens
}
}
const total = performance.now() - t0
metrics.histogram("claude.ttft_ms", ttft, { model, cached })
metrics.histogram("claude.total_ms", total, { model, cached })
metrics.histogram("claude.output_tokens", outputTokens, { model })
Capture TTFT and total separately. Tag the measurements by model, by whether the cache hit, and by prompt template. Without this you cannot tell whether your changes helped.
Bottom line: 4.2s → 1.6s on a real workload
The example workload above (summarizing a 5K-token document with Sonnet) takes 4.2 seconds of total wall time at baseline and 1.6 seconds after streaming, caching, a tight max_tokens and serving through the codexkey EU PoP. That is a 62% reduction in total time and a 91% reduction in perceived TTFT (4.2s to 380ms).
None of these techniques is exotic. They are all available in the official SDK, documented in Anthropic's API reference, and need no infrastructure changes beyond pointing your baseURL at https://api.codexkey.online/v1. The teams that ship the fastest Claude-powered products in 2026 are not using a secret model. They are just disciplined about TTFT, output length and cache hit rate.
If your p99 is still measured in seconds, start with streaming, then caching, then output discipline, then geography. In that order. The first two will give you most of the win.