Organizing Machine Learning: Every Flavor Welcome!

DoorDash’s principles and processes for democratizing Machine Learning

Challenges

1. ML is poorly defined: Is a linear regression in Excel ML? What about a toy random forest in a local Jupyter notebook? Where is the line between analytics and ML?
2. ML needs Engineering and Science: ML at technology companies requires performant optimal decision-making.
3. ML advances rapidly: Even over just the last five years we have seen modeling approaches and platforms and languages change almost every 18 months.
4. ML is trendy: many people view ML as magic and so everyone wants to work on it.

Vision

Build data-driven software for advanced measurement and optimization

Principles

1. Democracy: everyone can build and run an ML model given sufficient tooling and guidance.
2. Talent: we want to attract and grow the best business-impact focused ML practitioners.
3. Speed: if a cost-effective third party ML solution already exists then we should use it.
4. Sufficiency: if a function (typically Engineering) can implement a good-enough ML solution unaided then they should do so.
5. Incrementality: if a function (typically Data Science) can add enough incremental value to an ML solution then they should do so.
6. Accountability: each ML solution has a single technical lead acting as the technical decision-maker.

Organization

1. Reporting lines: ML Engineers report to Engineering managers and ML Data Scientists report to DS managers. ML Infrastructure reports into the central Data Platform team.
2. Hiring: Job descriptions and hiring processes for ML Engineers and ML Data Scientists are reviewed and approved by ML Council.
3. Technology: Strong investment in a centralized ML platform by Data Platform (workflow, provisioning, orchestration, feature stores, common data preparation, validation, quality checks, monitoring, etc.). Potential ML infrastructure technology (build/buy) decisions reviewed and approved by ML Council.
4. Execution:
   1. Any person(s) at the company can identify a use case for ML and draft a proposal (business problem, estimated impact versus build / maintenance cost, solution, team composition, single technical lead).
   2. The proposal is reviewed, amended, and approved by the pod’s / vertical’s cross-functional leads (PM, EM, DS Manager, Analytics Manager, etc.). The leads should approve the business problem, prioritization, and impact / cost.
   3. The proposal is reviewed, amended, and approved by the ML Council.
   4. All steps of the review will be transparent: ML Council and ML practitioners will meet weekly at ‘ML Review’ to review items and debate next steps. Decisions will be made at this ML Review and notes will be taken and emailed to all interested folks.

ML Council

1. Composition: the ML Council is composed of a group of experienced ML practitioners across the company, typically senior Engineering ML, Data Science ML, and Infrastructure ML folks. It is led by the ML Council Chair, who serves as the decision-maker for escalations. Rotates on some cadence e.g. every 12 months
2. Role: the role of the ML Council is to:
   1. provide balance between project-specific variability vs company wide uniformity, so that we are efficient as a company
   2. review and give feedback on all of new ML applications
   3. facilitate the cross-pollination of ideas and solutions
   4. create better visibility into common pieces (to feed into infra)
   5. encourage more proactive communication of data sources and solutions.
3. Responsibility: Typically the ML Council should ensure that if production performance is the biggest blocker to success then the tech lead is an ML Engineer. Otherwise if statistical performance is the biggest blocker to success then the tech lead is a Data Scientist. The ML Council should check solutions have enough support and where possible are part of the long term ML platform investment.
4. Autonomy: If the ML Council disagrees on the solution / team / lead, then the ML Council Chair tie-breaks and makes a decision.