Overhauling DoorDash’s Runtime Configuration Value Distribution with SquashFS

Runtime configuration values are a critical part of DoorDash’s engineering infrastructure, enabling teams to make changes quickly and safely without requiring a new deployment or service restart. These values are used to power feature flags, gradual rollouts, experiments, operational controls, and other mechanisms that let engineers quickly adjust application behavior at runtime.

This blog explains how we overhauled our runtime config value distribution to eliminate a scaling concern in one of DoorDash’s highest queries-per-second systems. This evolved architecture reduced latency in a major system bottleneck by more than 100-fold and produced a drop of more than five times in our average node Elastic Block Store input/output operations per second (EBS IOPS), ensuring a reliable and performant capability to support our needs on a massive global scale.

Previous architecture

Kubernetes is the primary platform used to orchestrate DoorDash services. Our prior architecture for runtime config value distribution was centered around a K8s DaemonSet that runs on every node across the fleet. This DaemonSet polls S3 for an updated .tar.gz archive containing the latest config values, then extracts them onto the K8s node hostPath. Each version of the .tar.gz is downloaded into its own directory, and a symlink points at the latest one that should be used. From there, hostPath-backed volumes with the config values are mounted to service pods at /srv/config-values. Note: There are different config value types internally that each use different mounts and volumes but, for the sake of simplicity, we refer to the stand-in throughout this post.

Each runtime config value is stored as a file regardless of its underlying type. From the perspective of a service, accessing the latest feature flag, experiment, or any other config value type becomes a fast and reliable Linux filesystem read of a specific file in /srv/config-values. Two caching layers make these file reads even faster — the node-local Linux kernel page cache and a pod-local internal cache of values. 

When a new version is ready, updating the symlink makes it immediately visible to services at /srv/config-values without interrupting reads. If for any reason the runtime DaemonSet fails to get the latest values from S3, the previous values remain in use and the system continues to operate reliably. The symlink keeps pointing at the last version that downloaded successfully; because downloads land in a temporary file that is only committed once complete, a failed or partial transfer can never replace a good version. A garbage collector periodically clears out older versions, but it never touches the active/symlinked dataset. The worst case scenario is stale values, not missing ones.

By design, every node receives every runtime config value. This allows pods to be efficiently scheduled to any node in the fleet without considering config availability, at the relatively small cost of storage efficiency. The vast majority of these values are not used; each pod typically only accesses a small subset of relevant files.

Diagram of Previous Architecture

Scaling concerns

Our old architecture supported our needs for years, until several scaling factors slowed the .tar.gz extraction beyond our timeout limits. When we bumped the timeout from 15 seconds to 30 seconds for getting the .tar.gz from S3 and extracting it to the K8s hostPath, we were regularly hitting the new 30-second timeout within a matter of weeks, as seen in Figure 2 below.

Our investigation of this reached the following conclusions:

• Runtime config value usage has grown dramatically across the company. The .tar.gz archives went from containing hundreds to tens of thousands of distinct values.
• Extracting this many files at once overloaded the underlying EC2 EBS disk volume.
• As our service footprint grew, this problem would only continue to worsen as demand for config values increased.

This latency issue was a major concern because failing to get the latest runtime config values to pods in a timely manner could slow incident mitigation.

Extraction Latency Bottleneck

Crux of the issue

The core problem was that every .tar.gz update required eagerly extracting the entire runtime config dataset onto every node, even though services only need access to a small subset of those values. This led to unsustainably large increases in latency caused by IOPS constraints.

Any long-term solution needed to avoid that unnecessary extraction work, particularly in light of the fact that the number of runtime config values was only going to keep growing. That could mean accessing values remotely, decompressing data on demand, or using another approach that did not require materializing every file on every update. This became a key consideration as we researched potential solutions.

Research and design

Finding a solution to this issue was challenging, as config distribution had many strict system requirements, including:

• Speeds comparable to native Linux file reads
• Virtually zero downtime
• No change to existing access patterns

Table 1 below outlines all notable approaches we considered during our research and design phase.

Potential Solutions to the Config Value Distribution Scaling Concerns

Out of all of the solutions we investigated, SquashFS offered the most benefits and the fewest drawbacks for fixing the config value distribution scaling concerns.

Overhaul using SquashFS

After selecting SquashFS, we began an overhaul of our runtime config distribution systems that would work with it. We prioritized read latency over image size, because the image is a single file each node downloads once per version; size impacts performance less than latency. After extensive benchmarking on our real-world runtime config values, we determined that SquashFS compressed with LZ4 using 128 KB blocks delivered the best average performance, as shown in Table 2. We then modified our config distribution service to upload a .squashfs image in this format to S3 alongside the existing .tar.gz archive containing the same data.

Benchmarks for SquashFS Performance using LZ4 with 128 KB blocks

The new architecture for SquashFS overlaps significantly with the prior architecture for .tar.gz. As with .tar.gz, we poll for the newest SquashFS images from S3 in the runtime DaemonSet. Instead of extracting the archive, however, we mount the .squashfs image as a loop device. This lets Linux treat the image file as if it were a read-only disk partition. The SquashFS driver can then expose its filesystem through Linux's normal filesystem interface, decompressing the necessary data blocks on demand as files are accessed rather than extracting everything up front.

Recall that every node receives every runtime config value, with each pod typically only accessing a small subset of relevant files. Because of this, a small number of on-demand decompressions will be performed relative to .tar.gz’s eager extraction of all values. This, alongside our two layers of caches stopping a majority of decompressions (the node-local Linux kernel page cache and a pod-local internal cache of values), is a major reason why SquashFS is so efficient for our use case.

From the perspective of service code, nothing changed. Service pods continue reading the same files at /srv/config-values, while the backing hostPath volume is changed to point at the SquashFS filesystem. K8s mount propagation made each filesystem mounted by the runtime DaemonSet visible to service pods that already were running, which allowed updates without pod restarts. Because SquashFS preserved the existing directory structure and filesystem interface, it could also reuse the existing pod-local caching infrastructure. This compatibility was a key design feature; it would be extraordinarily disruptive to coordinate a change to access patterns across thousands of services.

The existing reliability guarantees all still hold. Each new image is mounted into its own directory; the symlink only swaps over once that directory is fully in place, so pods never see a half-built version. Old versions are reclaimed by the same garbage collector used for .tar.gz. SquashFS needed only one change to the garbage collector: unmounting the directory before deleting it. That unmount of the previous version is lazy, so the directory goes away immediately, but the kernel keeps the underlying filesystem alive until the last in-flight read finishes. A pod reading a file mid-rotation is never interrupted.

Diagram of New Architecture

Rolling out SquashFS changes

Because runtime config values are critical to DoorDash, we took a cautious approach to the rollout. Over the course of several months, we gradually migrated service pod config volumes from the existing .tar.gz-backed hostPath to the new SquashFS-backed hostPath. Each rollout batch was announced across the engineering org and carefully monitored for regressions or performance impacts. After all services had successfully migrated to SquashFS, we officially deprecated the .tar.gz distribution path to eliminate its remaining resource overhead.

New architecture wins

Our new SquashFS-based solution delivered a number of benefits, including:

• Removal of a major scaling limitation for our runtime config value distribution.
• A more than 100x latency drop in the extraction bottleneck, with the latency plummeting from regular 30-second timeouts to averaging under 200 milliseconds, as shown in Figures 2 and 4.
• A more than 5x reduction in average K8s node EBS IOPS, as shown in Figure 5. 
• A more than 2x reduction in peak runtime DaemonSet CPU usage, as shown in Figure 6.

Latency Drop in Extraction Bottleneck

IOPS Usage Drop

CPU Usage Drop 

Acknowledgements

This project wouldn’t have been possible without the invaluable support of the following engineers: Luigi Tagliamonte, Tim Kapnik, Fredrik Nilsson, Bronson Mirafuentes, Ivan Radakovic, Alec Rabold, Rabun Kosar, Matt Gode, and Stephen Joel.