OpenAI Cosmos DB scaling: Deploying Multi-Tenant Architecture for 1B ChatGPT Users

OpenAI announced on October 23 2026 that its Habitat storage platform has been re-architected to use Azure Cosmos DB containers as the default back-end for all tenant data. The change follows a three-phase migration that began with a reliance on Cosmos DB primitives, moved through single-tenant isolation experiments, and culminated in a multi-tenant model that includes data-movement capabilities.

Multi-tenant design and logical partition limits

Cosmos DB partitions are identified by a logical key. OpenAI’s design caps the number of items per partition at 10 million to keep hot-key contention manageable. The team reports that the logical-partition-key limit is enforced by a custom wrapper that rejects writes that would exceed the threshold, preventing a single tenant from monopolising a partition.

Noisy-neighbor mitigation with hot-key throttling

When a tenant’s workload spikes, the wrapper applies a per-partition throttling policy that limits requests to 10,000 request units (RUs) per second. The policy is enforced at the Cosmos DB gateway, so the throttling is transparent to the application layer. The throttling mechanism is a direct response to the noisy-neighbor problem identified in Phase I of the migration.

Quality of Service (QoS) guarantees

OpenAI introduced a QoS layer that assigns a minimum RU quota to each tenant. The quota is calculated from the tenant’s historical usage and is enforced by a token-bucket algorithm. If a tenant exceeds its quota, the system automatically throttles the tenant’s requests, ensuring that no single tenant can degrade overall latency.

Serverless mode and Rust-based proxy

To reduce operational cost, OpenAI added a serverless mode that eliminates the need for pre-provisioned throughput. In this mode, Cosmos DB automatically scales RU consumption based on traffic. The team also rewrote the HTTP proxy layer in Rust, cutting proxy overhead by 30 % and lowering end-to-end latency for read-heavy workloads.

Global performance and data locality

The new architecture replicates only the smallest, most latency-sensitive datasets across all regions. Larger datasets are cached in an integrated Redis layer that sits in front of Cosmos DB. This hybrid approach keeps read latency under 20 ms for 95 % of requests while avoiding the cost of full multi-region replication.

Data movement and capacity planning

OpenAI’s data-movement capability allows tenants to migrate their data between partitions or regions without downtime. Capacity planning is now driven by a predictive model that estimates aggregate growth per tenant. The model uses historical request patterns to forecast future RU requirements, helping the team avoid over-provisioning.

What remains uncertain

The migration report acknowledges that aggregate growth is not stable; some tenants grow rapidly while others plateau. This uneven growth could force future re-partitioning or additional throttling policies. OpenAI has not yet disclosed how it will handle tenants that exceed the 10 million-item partition cap.

Implications for users and developers

The shift to Cosmos DB means that developers building on Habitat will see a consistent latency envelope across tenants, but they must also adapt to the new throttling thresholds. Applications that previously relied on high-throughput bursts may need to implement back-off logic to stay within the 10,000 RU limit. For users, the change promises lower costs and more predictable performance, but it also introduces a new layer of operational complexity.

Next steps and watch points

OpenAI will likely monitor tenant growth patterns closely and may introduce dynamic re-partitioning once the predictive model reaches production maturity. The company may also expose finer-grained QoS controls to developers, allowing them to negotiate RU quotas for critical workloads. Observers should watch for any announcements about scaling limits or changes to the serverless pricing model, as these will directly affect deployment cost.

Related coverage

Explore more on this topic