How OpenAI keeps ChatGPT fast as agentic coding rewrites the load profile

A QCon performance engineering talk from OpenAI reveals the infrastructure decisions behind ChatGPT's latency targets—and how the product's shift from chat to agent altered every assumption.

In plain terms: A performance engineering talk by Martin Spier (OpenAI), published on InfoQ, documents how ChatGPT's infrastructure adapted as the product shifted from simple chat to agentic coding workflows. The short version: agentic requests don't look like chat requests, and everything optimized for short-turn conversations needed rethinking.

The core challenge is request shape. Agentic coding sessions are longer, they stall mid-stream waiting on tool call results, and they create bursty load patterns on inference backends built for conversational rhythm. The team's response involves adaptive streaming, speculative caching, and per-session load balancing that accounts for expected context length at request time - not just at scheduling time.

[Under the hood] Agentic workflows dramatically increase code change volume at ChatGPT's scale, per Spier. Latency sensitivity concentrates at tool-call boundaries rather than generation speed—which inverts the optimization priority from the chat era. Cold-start latency on tool resolution becomes the critical path, not token generation throughput.

So what: The SLA assumptions and infrastructure patterns from 2023 ChatGPT don't apply to 2026 ChatGPT Work. Teams building on top of OpenAI APIs or similar platforms need to model their request patterns against agentic load profiles—burst shape, context length distribution, and tool-call frequency—not average chat latency benchmarks.