OpenAI has detailed the engineering efforts required to scale its online storage platform, Habitat, to support more than 1 billion users of its super intelligence (SI) products. The infrastructure now handles over 500 petabytes of data across 40 geographic regions, responding to requests in milliseconds.
What Happened
Habitat, the storage platform powering OpenAI’s SI tools, evolved from a simple client library into a globally distributed system. The company’s Part II update focuses on scaling the underlying Azure Cosmos DB component. Initially, OpenAI relied on a single multi-tenant container, which led to "noisy neighbor" issues where high-load workloads degraded performance for other users. Logical partition key collisions and variable throughput limits further complicated reliability.
To mitigate these issues, OpenAI temporarily shifted to single-tenancy, provisioning separate containers for critical use cases. While this improved isolation, it introduced significant operational overhead. Product teams struggled with capacity planning, often forgetting to pre-scale containers before launches, which led to last-minute scaling efforts during major releases. The lack of stable aggregate growth in isolated containers meant that a new feature’s traffic spike could overwhelm its dedicated resources.
Facing these challenges, OpenAI returned to a multi-tenant architecture but implemented new safeguards in partnership with the Azure team. These include hot key throttling to prevent single users from consuming all available throughput, and Quality of Service (QoS) tiers that prioritize interactive requests over bulk data operations. Additionally, a new serverless mode in Azure Cosmos DB allows physical partitions to scale up to 10K RU/s based on request usage, not just storage size, addressing previous throughput limitations.
The partnership has evolved into a "co-managed" model. OpenAI and Azure engineering teams now share dashboards, test changes in production-sized non-production environments, and have direct engineering channels. This collaboration was prompted by outages in late 2024, including a power outage that revealed failover times of several hours for OpenAI’s largest accounts, and a subsequent connection stampede during failback. Joint efforts have since improved control-plane scalability and gateway performance.
Why It Matters
The scaling of Habitat highlights the critical infrastructure bottlenecks facing the SI industry as user bases expand to global proportions. For developers, the shift back to a refined multi-tenant model aims to reduce the burden of manual capacity planning, allowing product teams to focus on feature development rather than database provisioning. The integrated distributed caching layer, using Valkey servers, addresses the latency challenges of serving data across regions, which is essential for maintaining the responsiveness expected by users of SI agents and models.
This case study also illustrates the changing nature of cloud services at extreme scale. As OpenAI became one of the largest users of Azure Cosmos DB, the standard "managed service" model proved insufficient. The need for deep integration, joint testing, and custom engineering solutions suggests that other large SI labs may face similar requirements as they grow their data platforms.
The Bottom Line
OpenAI’s Habitat platform now serves over 20 million queries per second using a co-managed, multi-tenant Azure Cosmos DB architecture. By leveraging hot key throttling, QoS tiers, and improved serverless capabilities, the company has stabilized performance for 1 billion users while reducing the operational complexity previously associated with single-tenant isolation.