OpenAI Says It Cut ChatGPT Storage Read Latency From ~120ms to ~10ms

OpenAI has published the second half of an engineering write-up on Habitat, the online storage platform behind ChatGPT, the API, Codex and internal services. The company says Habitat now holds more than 500 petabytes, serves over 1 billion users across 40 geographic regions, and that its largest single Azure Cosmos DB account stores tens of petabytes and handles more than 20 million queries per second. The most interesting part is not the scale numbers but the reversal at the center of the story: OpenAI moved from shared multi-tenant storage to isolated single-tenant containers, then moved back.

What Habitat is

According to the post, written by OpenAI technical staff members Jon Lee, Hrachya Hakobyan and Chaomin Yu, Habitat began as a Python client library and became a distributed service layer that sits between product code and the underlying stores. Part I covered the service evolution; Part II covers the storage substrate, which is Azure Cosmos DB, alongside Valkey for caches, blob storage, and change-data-capture feeds into Kafka, Databricks and other downstream systems.

The platform's job is to let product teams read and write data in milliseconds without understanding the database. OpenAI frames its operating order of priorities as reliability first, then performance, then cost.

The multi-tenancy round trip

The post describes three phases, and the honest accounting of each is the most useful part for anyone running a shared data platform.

In phase one, Habitat used a single default Cosmos DB container per account and leaned on the database's documented per-logical-partition-key limit of 10,000 request units per second as natural back pressure. OpenAI says this broke in three ways. Because early API choices exposed partition key selection to clients, different product teams collided on generic keys like user_id and workspace_id, so unrelated features shared one key's throughput budget. Limits are actually enforced at the physical partition, not the logical one, so a single hot key could degrade colocated workloads — the company says niche features disrupted core login and signup flows. And because a container's provisioned throughput is spread roughly evenly across its physical partitions, splitting a partition on storage growth halves each child's effective throughput. OpenAI calls this the "weak partitions" problem: for blob-like datasets, growing storage alone shrinks serving capacity even when request rates are flat.

Phase two was isolation. New critical use cases got dedicated containers, plus a script that kept provisioned throughput in sync with partition count. OpenAI says this prevented many site-wide outages by narrowing the blast radius to individual features. But it transferred a planning burden onto every product team. In a large shared container, a feature going from zero to a million QPS barely registers against aggregate load; in a freshly provisioned container sized for internal dogfooding, the same launch can represent orders of magnitude more load than the baseline. The company describes teams shipping features and forgetting to pre-scale, then scaling containers during live incidents. Capacity planning, it says, failed to scale organizationally.

What Azure built to make shared tenancy work

Phase three returns to a shared container by default, enabled by changes OpenAI says it worked with the Azure Cosmos DB team to get: throttling the hottest keys first so one tenant cannot drain a physical partition; quality-of-service tiering that throttles low-priority bulk and offline work before interactive user requests; and a revised serverless mode. OpenAI says the existing serverless option would have solved weak partitions but capped physical partitions at 5,000 RU/s, which was unworkable for throughput-heavy use cases. The new mode allows 10,000 RU/s per physical partition and adds RU-usage-based splits in addition to storage-based ones, removing the need for manual container scaling.

OpenAI reports no provisioning or noisy-neighbor incidents in the months since. That is a short, self-reported window with no availability figures attached, and the post does not say whether these Cosmos DB capabilities are generally available to other Azure customers or remain specific to OpenAI's deployment. For teams reading this as a pattern to copy, that distinction matters: the fix depended on a vendor shipping features on request.

When a managed service stops being managed

The reliability section is unusually candid. OpenAI says a power outage hit the primary Azure region for one of its largest Cosmos DB accounts in late 2024, and the failover exercise revealed that its biggest accounts would take several hours to move to a secondary region — something neither team had tested at that size. About a month later, failing back triggered a second incident: a connection stampede overwhelmed the original region's Cosmos DB gateway because the gateway-mode client reconnected all clients to that region even though only some physical partitions had moved back.

The conclusion OpenAI draws is that at sufficient scale, managed services become co-managed. The two teams now share dashboards and tests, run a production-sized non-production account, iterate on SDKs together, and can page each other directly rather than going through support tiers. The post lists a year of joint work on control-plane and gateway scalability, metadata caching in the gateway, connection pooling to reduce pressure from Habitat's Python process fleet, and VPC-level separation of critical and offline workloads.

Getting reads to 10 milliseconds

Reads are more than 90% of Habitat traffic, and OpenAI describes a three-layer strategy. First, place data near users — which it says is mostly limited to data-residency cases, because many regions lack the datacenter capacity, power or cooling to host a proportional share of its storage footprint. That creates a real tradeoff the post names explicitly: reserve scarce regional capacity for storage and lower latency, or for serving compute and accept longer round trips.

Second, replicate only small, latency-critical datasets widely. Authentication metadata, needed on every request, lives in accounts replicated to as many as all 40 serving regions; others get at least one region per continent. This forced OpenAI to build a transparent data movement and routing layer earlier than planned.

Third, cache everything else. Habitat uses a Valkey-based lookaside cache near request processing, with invalidations propagated globally within seconds via Cosmos DB's change feed — an approach OpenAI compares to Meta's TAO. The post walks through why TTL-only caching forces a bad choice between staleness and hit rate, and why write-time invalidation alone breaks on partial failures and cross-region races.

Finally, OpenAI rewrote the Habitat service layer in Rust, replacing Python. Together, the company says global p50 reads fell from roughly 120 ms to roughly 10 ms this year. Note what is not in that claim: no p99 or tail-latency figure, and no breakdown of how much each of the three changes contributed. OpenAI acknowledges the slowest requests still fetch uncached data from distant regions, and says the deeper fix — placing data near where it will be read in the first place — depends on future capacity.

The post ends mid-section on change data capture, which OpenAI says handles millions of changes per second across thousands of containers and more than a hundred Kafka topics, feeding secondary indexes, data movement, analytics and real-time features. Cost is barely discussed despite being the third priority in the company's stated order, and no commercial terms with its cloud provider are described.

What to watch

• Whether the hot-key throttling, QoS tiering and revised serverless mode reach general availability for other Cosmos DB customers, or stay bespoke to OpenAI's accounts.
• Tail latency. A 12x p50 improvement says little about p99, which is what users notice; OpenAI concedes cold reads still cross regions.
• Durability of the "no noisy-neighbor incidents" claim past a few months, particularly through a large launch that moves aggregate load sharply.
• The data-placement roadmap: OpenAI ties further latency gains to new regional capacity and storage efficiency, both outside its direct control.
• The concluding CDC material, which the published text cuts off — the operational burden of discovering accounts, containers and regional CDC deployments sounds like the next abstraction Habitat has to hide.