Aligning Dispatch to Reality: Using Reinforcement Learning to Adapt Real-Time Objective Weight 

Managing dispatch in a three-sided marketplace is a delicate balancing act; every assignment decision ripples through the DoorDash ecosystem, affecting customers, Dashers, and merchants. Historically, DoorDash used a static dispatch optimizer to handle these tradeoffs. However, static weights often struggle to adapt to the dynamic realities of different stores, regions, and times of day. In this post, we share how we replaced these heuristics with a production reinforcement learning (RL) system that dynamically adapts dispatch objective weights in real time. For the underlying dispatch platform architecture, see our companion post, How DoorDash Built a Next-Generation Real-Time Dispatch Platform.

Rather than replacing the optimizer, the policy acts as an alignment layer, observing local marketplace conditions and selecting an objective-weight multiplier that steers the tradeoff between delivery speed and operational efficiency while preserving existing safeguards and latency guarantees. This work was presented in our paper “Multi-Agent Reinforcement Learning from Delayed Marketplace Feedback for Objective-Weight Adaptation in Three-Sided Dispatch” at the International Conference on Machine Learning 2026 workshop, RLxF: Reinforcement Learning from World Feedback.

Why objective-weight adaptation?

Dispatch optimization involves competing goals. Prioritizing speed can reduce customer-facing delivery time but may reduce batching opportunities. Prioritizing batching can improve Dasher efficiency, but if applied in the wrong context, it can increase lateness or create poor pickup timing. A global weight cannot handle these local variances effectively for several reasons:

• Slack periods: Low volume means batching-compatible assignments can improve efficiency without hurting the customer experience.
• Congestion: High traffic requires prioritizing speed; over-batching during these times can create excessive delivery delays.
• Local variability: A store’s unique backlog, supply pressure, and courier availability demand localized tradeoffs that static, global weights cannot resolve.

This makes dispatch a natural fit for reinforcement learning. The system does not receive immediate human preference labels. Instead, it learns from delayed operational outcomes such as delivery duration, courier active time, and courier wait time.

Designing a hybrid RL-optimization architecture

To address marketplace variability, we designed objective weight adaptation with reinforcement learning, or OWA-RL, a hybrid architecture that treats RL as an alignment layer rather than a replacement for our production optimizer, as shown in Figure 1 above. The RL policy operates as the outer decision layer, observing store-level state and selecting an objective-weight multiplier. This multiplier acts as a tuning knob for the inner assignment optimizer, which remains responsible for complex constraints, feasibility, and operational safety. This design lets us leverage the adaptability of reinforcement learning while retaining the stability of our established production solver.

By constraining the RL agent to a small action space A = {0.8, 0.9, 1.0, 1.1, 1.2}, we ensure the policy can adapt locally to supply-and-demand conditions without overriding critical assignment safeguards. An action of 1.0 recovers the production baseline, while lower or higher values tilt the tradeoff toward batching efficiency or delivery speed, respectively.

Training on delayed feedback

The policy is trained offline from logged marketplace data. Each online dispatch run is joined with delayed fulfillment outcomes to construct transition tuples such as these: (s_t, a_t, r_t, s_{t+1}).

Reward construction uses both customer-facing and courier-side timing signals, as illustrated in Figure 2 below. We observe three metrics after delivery completion: customer-facing delivery duration, or ASAP; courier active time, or CAT; and excess courier active time, or XCAT, which measures CAT beyond the direct route. These delayed outcomes are attributed back to the dispatch runs that influenced them.

Because dispatch decisions can create network effects across nearby stores and Dashers, we aggregate rewards regionally rather than treating each store in isolation. This gives the policy feedback that better reflects marketplace-level outcomes, including the tradeoff between faster delivery completion and more efficient batching.

Offline RL with conservative value learning

We train a shared value function for decentralized store-level execution. The model is a lightweight discrete-action Q-network over the ASAP-weight multiplier action space. During inference, each store observes its local marketplace state and selects a multiplier that adjusts the speed-efficiency tradeoff passed into the production optimizer.

To improve offline training stability, we use double Q-learning targets to reduce maximization bias and add a conservative regularizer to reduce overestimation for actions that are weakly supported by logged data. This is especially important in production offline RL, where the policy must avoid assigning high value to actions for which it has not observed enough evidence. The conservative objective helps keep adaptation within well-supported regions of the logged data while preserving the optimizer’s existing feasibility and safety guarantees.

Production experiment

We evaluated the policy in a global switchback experiment against the production baseline. The policy improved marketplace efficiency without degrading customer-facing quality. As shown in Table 1 below, OWA-RL increased batching, reduced courier active time, and reduced courier wait time across all parts of the day, while ASAP and 20-minute lateness remained statistically unchanged.

The strongest operational gains appeared during dinner hours, from 5 p.m. to 9 p.m., when marketplace pressure is highest and batching opportunities are more valuable. In this window, the policy reduced courier-side time costs and increased batching while improving delivery quality. These results suggest that the RL alignment layer can adapt objective weights in real time while keeping the production optimizer’s safeguards intact.

Monitoring policy behavior after deployment

Online impact is only part of the story. We monitor action distributions to detect policy-behavior drift and state-feature distributions to detect marketplace drift. Because the policy is trained offline, monitoring ensures that marketplace conditions remain within the intended range for policy support. Large shifts in selected multipliers or supply-pressure features are key indicators that the policy behavior may need to be reevaluated.

Lessons learned

Several design choices made the system practical for production. Our key recommendations include:

• Use RL to adapt the optimizer, not replace it: The learned policy controls a low-dimensional objective-weight interface, while the optimizer handles feasibility and assignment.
• Keep the action space constrained: A small discrete action space makes offline learning safer, easier to monitor, and easier to roll back.
• Train from real marketplace feedback: Delayed operational outcomes provide a scalable signal for learning dispatch tradeoffs.
• Use conservative offline learning: Conservative value regularization helps reduce unsupported-action overestimation before deployment.
• Monitor both actions and states: Production RL systems need drift monitoring, not just experiment readouts.

Next up

This work demonstrates that reinforcement learning can safely adapt objective weights in a large-scale dispatch system when paired with a constrained production interface. Future work includes exploring additional decision layers, improving distribution-shift detection, and developing interpretability tools that connect policy decisions to changing marketplace conditions. By combining a learned policy agent with a production optimizer, we can make dispatch more adaptive while preserving the reliability required for real-world logistics.