A note to our valued merchants, Dashers and customers
We want to acknowledge the service disruption that occurred early on July 9th and explain what happened. We know merchants rely on us to serve their customers, Dashers depend on us for consistent earnings, and customers count on us for timely deliveries. We're committed to transparency and to making our platform more reliable for everyone who depends on it.
What happened
On July 9, 2025, DoorDash experienced a service disruption primarily impacting our Dasher and customer experience for several hours in the early morning (PT). This post explains the technical root cause of the incident and the steps taken to mitigate it.
Understanding the Incident: The root cause
The incident began at 1:47 AM PT when our Dasher service circuit breaker opened. This was triggered by a performance degradation in the underlying storage system.
The problem started when a database table autostatistics job was completed. The database’s internal automated statistics process misjudged how data was being used, causing the database to choose an inefficient index for critical queries. Because of this, an inordinate number of rows were scanned per call, exhausting the database’s resources, causing numerous queries to be canceled due to statement timeouts, and resulting in cascading errors downstream.
Due to these system issues, Dashers were unable to check in or accept new deliveries, which meant we couldn’t reliably fulfill incoming orders. To protect the customer experience, we decided to temporarily pause order acceptance.
Our response and mitigation
Our teams promptly detected the issue and classified it as a high-severity incident. We executed a series of actions to restore platform functionality:
- The primary mitigation was to make the problematic index invisible to the query optimizer. This prevented inefficient query plans and helped stabilize CPU usage across the system.
- After the system stabilized, we gradually brought the Dasher backend back online, enabling Dashers to resume accepting shifts and completing deliveries.
- With Dasher supply recovering, we fully restored order acceptance shortly afterward. This marked the end of customer-facing impact and the full restoration of service.
Lessons learned and path forward
This incident, while disruptive, provided valuable insights and highlighted areas for improvement.
Areas for improvement
- Query Plan Visibility: While we were able to identify the query plan changes through manual inspection of logs and query plans, we recognize the need for more automated and visible indicators to surface critical plan regressions to our operators in real time. The lack of visibility delayed our mitigation efforts.
Action items in progress
To prevent recurrence and improve our resilience, we have prioritized the following action items:
- We have implemented mitigations to prevent recurrence of the statistical sampling issue and are actively exploring potential solutions for pinning query plans.
- We are making adjustments to improve performance and ensure more consistent system behavior, even under less-than-ideal conditions.
- We are analyzing and working to remove hard dependencies between services to ensure that a disruption in one area does not cascade to others (e.g., enabling drop-offs even if shift functionality is degraded).
We appreciate your patience and understanding during this incident. We are committed to continuously improving our systems and processes to ensure the reliability and stability of the DoorDash platform for Dashers, customers, and merchants.
