Wanted: Data Scientists with Technical Brilliance AND Business Sense

When interviewing candidates for Data Science roles at DoorDash, we look for a quality of entrepreneurship as a key indicator of future success. When training in Data Science, the technical demands of the field require a focus on modeling and algorithms. The many applications for modeling in the real world justify this focus, but when it comes down to the needs of a business, we want to see practitioners who understand the real business impact of their models.

Last year I wrote Organizing Machine Learning: Every Flavor Welcome!, an article describing the values of DoorDash’s Data Science team. Since then we have hired over 20 data scientists to build data-driven products for advanced measurement and optimization. While many of these hires have machine-learning (ML) backgrounds, every single one of them has a strong entrepreneurial streak. By ‘entrepreneurial’ I mean that they take initiative in finding the highest impact data projects to work on, rather than expecting to be told what to work on.

Most technology companies prioritize technical ability during their data science interview process, but at DoorDash, we assess for both technical ability and business intuition. In this article I share why and what we are looking for in these interviews.

Entrepreneurism first, machine learning second

In previous places I have worked, both in finance and technology, I have seen data scientists make the same mistake over and over again: they skip the “why” and go straight to the “what” and “how”. Good entrepreneurs strive to understand the “why” and “so what” before diving into execution. We look for data scientists that take the latter approach.

To illustrate this distinction, here is an example conversation that I have seen take place between an internal business partner and a data scientist:

Business partner (BP): Hi DS, can you please build a model that can predict what food items should be recommended to a new user when they land on DoorDash’s website for the first time?

Data scientist (DS): Hi BP, I can build a personalization model using deep learning with 1,000 features.

BP: Thanks, let me know when it’s ready.

[One month later]

DS: The model is built.

BP: Great! Let’s run an A/B experiment.

[Two weeks later, after the experiment concludes]

DS: Hmm, looks like new-user conversion is not statistically significant in treatment versus control.

BP: That’s strange. Do you know why? Did the ML model not work?

DS: Not sure just yet, I’ll take a look at the data.

Let’s walk through everything that went wrong in this example:

1. Goal alignment: Every product and team needs to begin with a clear goal, which is typically a target value for a clearly defined metric. In this example, it was not clear what the team was ultimately trying to optimize. Was it the new-user conversion rate? Was it the total number of sales? Was it total revenue? Something else?
2. Opportunity sizing: Before working on a project, the team needs to estimate its potential impact. In this example, why build a new-user recommendation model for the web site if, suppose, 98% of new users experience DoorDash via a mobile app?
3. Exploratory data analysis: Before building a model, we need to interrogate the data to see if there is any predictive power in the feature set. For example, if we know nothing about a new visitor to a website, how are we going to be able to do any personalization? A simple matrix of scatter plots to assess correlation of variables with the target or a linear regression could determine if there is any predictive juice in the available data before committing to a full-blown ML model.
4. Prioritization: In a lean and efficient team, there should always be the need for prioritization to make sure each team member is working on the highest value project to advance the mission of the company. In the above example, there was no conversation about why this is important and merits pushing another project deadline to take this new work on.
5. A quick MVP: For any product it is best to get a minimum viable product (MVP) built quickly and cheaply to validate the hypothesis, before committing extensive time and resources. In this example, it is much preferable to build a toy model in one week and test it, before committing a whole month to build a complicated model with hundreds of features (none of which might be predictive).
6. Interpretability for sanity check: Whatever toy model is built, it should always be possible to extract the top one or two drivers (i.e. features) of the model performance. We can then check if the toy model’s results make some intuitive sense in the business context.
7. Experiment metrics: When designing an experiment, we should have more than one metric to look at when an experiment concludes. We should be able to look at different steps in the funnel, different slices and dices, and different cohorts, among many other potential metrics. In this example, the business partner and data scientist should have been able to identify that maybe the predictions were all the same, or not many users saw the new experience, or something else.
8. Regular check-ins: Rather than meeting at the start and end of a model build, it is better to check-in frequently (e.g. once or twice a week) to discuss latest findings and align on if any course corrections are necessary. In this example, maybe after one week, if we knew the backtest showed little up lift then the project could have been scrapped earlier.

Revisiting the example conversation from above, here’s how it might work out if the data scientist considered the both the business implications and the technical problem at hand:

BP: Hi DS, can you please build a model that can predict what food items should be recommended to a new user when they land on DoorDash’s website for the first time?

DS: Thanks BP. Can you give me more context please? What is the goal of this recommendation model? What is the proposed metric for measuring success? What is the precise hypothesis?

BP: Good questions, thank you for asking. We are trying to maximize new-user conversion rates. We think that a better web onboarding experience will increase new-user engagement and thus increase the chance of an initial conversion . Here are some background documents to learn more.

DS: Interesting, let me take a look at the documents and dig into the data a bit and get back to you.

[The next day]

DS: Hi BP, I looked at the data. The metric makes sense to me but I have low confidence in the currently proposed hypothesis. I don’t think developing this model is a good idea. Most new users go straight to the mobile app rather than our web site. Also, there is no useful information we have on new users when they land on the web site, so there’s no predictive power to capture via an ML model. The best thing to do is probably some high level rule to show whatever items are trending this week.

BP: Oh, that’s a shame. Oh well, thanks for looking into it.

DS: However, I did find something else that could help with new-user conversion. It looks like a lot of new users download the mobile app and then start filling out the sign-up page but don’t get to the end. Maybe we should introduce the sign-up page later, once they have engaged with the mobile app a bit more. For an MVP we could just delay it by a minute and run an experiment later. And later, if successful, we could build an ML model to find the earliest time to introduce each user to the sign-up page, based on their early engagement.

BP: Nice find! Interesting idea, let’s regroup with the broader team and prioritize this hypothesis.

What it means to be business owner

When people join our Data Science team, they are signing up for doing whatever they can in the business environment to help the company achieve (and even surpass) its goals. Typically, a company cares about the top line (growth) and the bottom line (profitability), and everything else ladders up towards these metrics. Obsessing over customer satisfaction, both when building external-facing products and internal tools, workflows, and management structures, is a precursor for ensuring growth and profitability.

On DoorDash’s Data Science team, in everything we do, we are focused on improving the customer experience (consumer, Dasher, merchant, or others who use our services) because we know this drives long-term growth and profitability. Each data scientist at DoorDash is accountable for how they spend their time and the impact they drive. They must be collaborative but still focus on the highest impact projects.

At the start of every year I challenge each data scientist to find and work on projects that deliver $10m of annualized incremental gross variable profit. This challenge helps the data scientist to filter their set of projects to only the highest value initiatives for the business. It also means that if there is nothing impactful enough to work on in their vertical, the data scientist should move to a different area of the business.

Data science principles

To help guide data scientists on how to build their models, we drew up a charter of principles. These principles help data scientists make prioritization and design decisions and will provide guardrails to our strategy over the coming years.

Principles (in priority order):

1. Impact: Model solutions should drive business metrics.
2. Expertise: We are industry leaders in measurement and optimization.
3. Platformization: Build generalizable models which can be new businesses lines.
4. Accountability: Ensure model decisions are fair, transparent, and interpretable.
5. Reliability: Models should have few outages and safe failovers.
6. Efficiency: Model outputs are fast and as cost-effective as possible.

What we look for in interviews

Throughout the data science interview process we are mostly trying to get signals on a candidate’s potential for Impact and Expertise, principles 1 and 2 above. The remaining principles, 3 through 6, are important but we believe they can be learnt on the job.

Most other data science interview processes only assess for expertise which, in our view, misses the more important attribute of being impact-driven. We see lots of candidates who are technically very capable but are more interested in building a model than solving a business problem, or who are not able to effectively operate under ambiguity, which is common in a fast paced environment.

At DoorDash, we first and foremost recognize data scientists for the impact they have on the business, regardless of whether they used complex ML to do so. However, over the course of a year we see, on average, about 70% of the impact of a data scientist comes from building new and more accurate ML models. But the model itself is not the goal, improving the business metrics with the model  is the goal, and ML is just one tool to achieve this ‚Äî a tool that the data scientist is an expert in.

Our interview process

Our interview process is straight-forward and consists of three stages:

1. Résumé screen: We review résumés for appropriate education and work experience.
2. Homework challenge: We share a dataset and set of tasks which include building a predictive model and writing a summary report.
3. One-one interviews: We schedule five interviews to assess business, technical, and values fit.

Our one-one interviews are different from most other data science interview processes in two respects:

1. We do not have a coding challenge. We believe that the homework challenge is a better representation of the type of work you will be doing here and gives us all the signal we need.
2. We have multiple consulting and analytics business case study interviews. Unlike other companies, we require our data scientists to have strong business intuition and the ability to operate under ambiguity.

Many candidates do well in the homework and technical one-one interviews and then poorly in the business case studies. We recognize that we are looking for a specific type of candidate, who are technically brilliant but also very business savvy. However, these candidates do exist and we will continue to keep the bar high to find these people.

Top 10 tips and tricks in the interviews

Finally, here are some ways to prepare for the one-one interviews and perform well in them:

1. Read data science articles from marketplace companies like ourselves to understand marketplace dynamics and experimentation. A lot of the techniques and customer considerations are similar across businesses.
2. When thinking through business problems and tradeoffs, consider the experience from all sides of the marketplace.
3. Do not be afraid to ask clarifying questions — this demonstrates engagement and ownership. Candidates will not be docked points for doing so, quite the opposite.
4. Candidates should feel free to take a moment to pause and structure their thoughts. The brief silence may feel awkward, but again it demonstrates good self-organization.
5. Try to think from first principles rather than shoe-horning the problem asked into a framework you are more familiar with. It may sometimes be appropriate but usually it can give you tunnel vision versus thinking more creatively.
6. Where possible, offer well-structured answers. This demonstrates clear communication and methodical thinking.
7. Be concise; it is better if candidates communicate an answer in one sentence versus several sentences. This again demonstrates clear thinking. It also gives more time to answer more questions to further impress.
8. Be specific in answers. For example, if listing a rate of conversion, state explicitly what the numerator and denominator are. Avoid providing vague examples of experiences.
9. Demonstrate a bias to action. It is ok to ask for more time and data, but if asked to make a call then make the best call given the existing information.
10. Be positive and try to enjoy the day. We are all aligned on candidates being successful and we will not try to trip them up. We will ask clarifying questions or try to steer candidates towards something else to maximize their chance of impressing!

There are more great pieces of advice in a previous article, The Undervalued Skills Candidates Need to Succeed in Data Science Interviews.

Conclusion

Our Data Science team at DoorDash is unique because of the emphasis we place on business impact and entrepreneurship, as well as technical brilliance. We know we are asking a lot of any single individual, but it makes for a more exciting and impactful Data Science team.

We are still a relatively nascent Data Science team and are looking to double our size in 2021. We are looking to add more data scientists to Dispatch, Dasher Engagement, Pricing, Ranking, Forecasting, Merchant Lead Scoring, and Fraud teams, as well as introduce data scientists to new areas of the company such as Merchant Services, Customer Support, Grocery, Dasher Positioning, and more! If you are technically brilliant but want to drive business impact above all else, then please apply to our team to join like-minded data science entrepreneurs!

Header photo by Dan Meyers on Unsplash.