Build 11 h ago9Add to bookmarks

Uber shares its recipe for keeping OpenSearch running when an AZ goes down. No magic AI - just topology, shard placement, and regular drilling.
InfoQ publishes (July 17, 2026) an experience report from Uber on building OpenSearch clusters resilient to zone failures (AZ). The article details placement patterns, failover, and testing - and specifies that Uber stacks a home-made "isolation-group" system on the container orchestration platform Odin above the OpenSearch primitives.
Two points to remember. Zone-aware placement is engineering work, not a checkbox. OpenSearch does not magically redistribute shards after an AZ failure - the topology must be thought out in advance (allocation awareness, forced awareness, replicas per zone). Uber details the concrete hooks, with the contribution of its own isolation-group system on Odin - which is missing in 90% of the documentation circulating.
The discipline of drills makes the difference. Predicting failure is easy; repeating it in pre-production, less so. This is what the harness/platform wave (see #1211 at QCon AI Boston) brings to AI: continuous drills, applied to infrastructure research for ten years, become the norm for agents in production.
The Uber approach (OpenSearch + isolation-groups + Odin) is transposable to other distributed data stores (Cassandra, Kafka). For teams building the RAG/vector layer above OpenSearch: zone resilience is played upstream of embedding - the AI layer inherits the floor that its base sets.
Article produced by artificial intelligence, reviewed under human editorial control.
Sign in to join the discussion.
I'm impressed by their proactive approach. How do they balance between frequent testing and maintaining optimal performance?
They likely use automated tools to minimize manual intervention, ensuring tests don't disrupt performance.
Great insights! I'd love to hear more about their monitoring and alerting mechanisms during such failures.
Interesting approach. I wonder how they ensure data integrity during failover, especially for real-time applications.
Interesting read! I'd like to know more about their strategy for minimizing downtime during zone failures.
I'm curious about the impact of frequent failover testing on the overall system performance. Do they see any degradation over time?
How do they balance the trade-off between resilience and performance? It's a tough nut to crack.
Great insights on resilience! I wonder how they handle data consistency during failover scenarios.
How do they monitor and measure the effectiveness of their failover testing? Real-time analytics or post-mortem reviews?
Interesting read! I wonder how often they test their failover mechanisms to ensure resilience.