Blog
Improving Experimental Power through Control Using Predictions as Covariate (CUPAC)
June 8, 2020
|
Jeff Li
In this post, we introduce a method we call CUPAC (Control Using Predictions As Covariates) that we successfully deployed to reduce extraneous noise in online controlled experiments, thereby accelerating our experimental velocity. 
Rapid experimentation is essential to helping DoorDash push key performance metrics forward. Whether improving our store feed ranking, optimizing the efficiency of our logistics system, or evaluating new marketing campaigns, it’s critical for DoorDash to maintain a robust experimentation methodology to ensure new releases improve business metrics in production. Without such rigorous validation, the practice of “test and learn” can quickly devolve into “ship and pray.”
To complicate matters, the metrics we care most about are often very noisy. One such metric is the total time it takes to prepare and deliver an order, which we call ASAP. ASAP is a key metric for us to monitor as delivery speed drives both customer satisfaction and retention. ASAP is very noisy as it varies by merchant type (e.g. quick-service restaurants prepare food quicker than steakhouses), the customer’s distance from the merchant, and current traffic conditions. Such variation lowers the probability of detecting improvements (i.e. the power of the test) driven by new product features and models in an experiment. This makes it difficult for us to conclude whether observed changes in ASAP are real or are merely fluctuations driven by random chance.
To mitigate this issue we developed and deployed CUPAC. CUPAC is inspired by the CUPED methodology pioneered at Microsoft (Deng, Xu, Kohavi, & Walker, 2013), extending it to leverage machine learning predictions built using inputs unaffected by experiment intervention. This approach has proved powerful in practice, allowing us to shorten our switchback tests by more than 25% while maintaining experimental power.
Related Jobs
Job ID: 3016665
Location
San Francisco, CA; Mountain View, CA; New York, NY; Seattle, WA
Department
Engineering
Job ID: 3014470
Location
San Francisco, CA; Sunnyvale, CA
Department
Engineering
Job ID: 3016533
Location
San Francisco, CA; Sunnyvale, CA; Seattle, WA
Department
Engineering
Job ID: 3014526
Location
San Francisco, CA; Seattle, WA; Sunnyvale, CA
Department
Engineering