OpenAI says Habitat, its online storage platform, now responds in milliseconds to more than 1 billion users across 40 geographic regions and stores over 500 petabytes, after changes that cut global median read latency from roughly 120 ms to roughly 10 ms. OpenAI detailed the work in the second part of an engineering series published Oct. 23.
Habitat backs OpenAI products including ChatGPT and Codex, and its storage layer runs on Microsoft's Azure Cosmos DB. OpenAI's largest Cosmos DB account stores tens of petabytes and serves more than 20 million queries per second. Part I of the series covered Habitat's evolution from a Python client into a service; Part II covers scaling the storage underneath it.
The current design came out of three phases. OpenAI started with one shared Cosmos DB container in which clients chose their own logical partition keys, such as user_id or workspace_id. That produced key collisions and noisy neighbors competing for the same physical partition limits. As data grew, partitions split and throughput per partition shrank: a 10,000 RU/s partition split in two left each child with 5,000 RU/s.
Key facts
- 01OpenAI's Habitat storage platform serves more than 1 billion users across 40 regions and stores over 500 petabytes.
- 02Global median read latency fell from roughly 120 ms to roughly 10 ms, a 12x improvement this year.
- 03OpenAI's largest Azure Cosmos DB account stores tens of petabytes and serves more than 20 million queries per second.
- 04OpenAI rewrote the Habitat service from Python to Rust and added a Valkey-based distributed cache.
- 05OpenAI is building an in-house exabyte-scale storage system on Rockset's RocksDB Cloud core with Raft replication.
Phase two gave teams dedicated single-tenant containers. Isolation improved, but every container needed its own capacity planning. OpenAI's example: a shared container provisioned at 100 million RU/s and running at 80% utilization could absorb a launch that jumped from 1,000 RU/s in load testing to 1 million RU/s in production. A new dedicated 10,000 RU/s container could not.
OpenAI then moved back to a shared container by default, this time with hot-key throttling, request-tier quality of service that throttles offline workloads first, and an improved serverless mode. That mode raised per-partition limits from 5,000 to 10,000 RU/s and splits partitions based on RU usage as well as storage. OpenAI reported no provisioning or noisy-neighbor incidents in the months after the change.
Reliability became a joint project with Microsoft after an incident. In late 2024, a regional outage forced a failover that took several hours for the largest accounts, and about a month later a connection stampede during failback overwhelmed the original region's gateway. The two companies now monitor the stack together, test on a production-sized non-production account and share engineering access.
Reads make up more than 90% of Habitat requests, so most of the latency work targeted them. OpenAI places data near users where capacity allows, mainly for data residency, and replicates small latency-sensitive datasets widely, in some cases to all 40 regions. It added a distributed cache built on Valkey, invalidated through the Cosmos DB change feed and modeled on Meta's TAO, and rewrote the Habitat service from Python to Rust.
OpenAI's change data capture platform handles millions of changes per second across thousands of containers, feeds more than 100 Kafka topics and runs on thousands of workers across regions. OpenAI contributed batched, in-memory checkpointing back to Azure's CDC SDK.
Next, OpenAI said it is building a custom in-house exabyte-scale online storage system optimized for bulky Habitat data types. The system combines the RocksDB Cloud core from Rockset with Raft-based replication, implements the Cosmos DB document model, and uses disaggregated storage to reduce flash requirements at the cost of lower peak QPS.
Working on something we should cover, or seeing a story we missed? Send leads, documents, or feedback to hello@aichatdaily.com. For sensitive tips, see our secure tips page for Signal and PGP options.
Spotted an error? Email hello@aichatdaily.com with the URL and the issue, or read our full corrections policy.
