AI Engineer Interview Daily — 2026-09-02: ML System Design

🌏 中文版

Today's Topic

ML System Design is the section of AI Engineer interviews that most tests "systems thinking." The prompt usually starts with something as simple as "design a recommendation system," but what the interviewer actually wants to see is whether you can stitch together how data comes in, how features get computed, how the model gets deployed, and how you'd know it's still working after deployment — a complete, self-healing system, not just a model architecture diagram.

Today skips the tired "XGBoost vs. neural network" model-selection question and instead practices the skeleton interviewers actually care about: how a feature store prevents training-serving skew, how many stages a deployment needs to de-risk it, and which three layers monitoring should cover. This question type commonly shows up in onsite ML infra system design rounds, usually given 45-60 minutes.

Core Concepts Cheat Sheet

A Feature Store Solves "the Same Feature Computing Two Different Answers"

Training-serving skew is one of the most common and most insidious failure modes in ML systems: the same feature gets computed with one logic offline during training and a different logic online during inference, so the model receives inputs whose distribution doesn't match what it saw during training — prediction quality silently degrades without throwing any error. A feature store's core value is centralizing feature definitions so the online store (Redis/DynamoDB, low latency) and the offline store (Parquet/data warehouse, full historical lookback) share the exact same transformation logic and schema. In an interview, you should be able to explain how you'd verify that both sides compute the same value — not just say "I'd use a feature store."

Deployment Strategy Should Be Layered, Not a Binary "Ship It"

Canary (route a small traffic slice, monitor before expanding), shadow mode (run the new model in the background, log outputs without affecting production), A/B testing (route different segments to different models, judge by business metrics), and blue-green (swap entire environments, instant rollback) each solve a different problem — shadow verifies the new model "can it even run" without taking on business risk, canary verifies "does it cause harm," and A/B verifies "is it actually better." The scoring point in an interview is articulating that rollback triggers are automated (an automatic revert to the previous model when latency spikes or a key metric drops), not "someone eyeballs it and flips a switch when something looks off."

Monitoring Needs Three Independent Layers, Not Just Model Accuracy

Infra layer (QPS, latency, error rate, GPU/CPU utilization), data layer (feature availability, ingestion lag, input distribution drift), and model layer (prediction distribution, live accuracy, feedback-loop delay) are three places that can independently break, and monitoring them separately is what lets you actually localize the problem. The concrete way to detect drift is comparing the latest input distribution against the training-time reference distribution using a statistical test — PSI (population stability index) or a KS test are the common choices — and triggering an alert or automatic retrain once the score crosses a threshold. Naming these two specific tests carries far more weight than a vague "we monitor for data drift."

Latency Budgets Need to Be Broken Down to the Millisecond, Not Just "It Should Be Fast"

For a sub-100ms online inference request, you should be able to allocate the budget across each segment: feature lookup (Redis, 1-5ms), model inference (10-30ms depending on model size), network round-trip (10-20ms), and serialization (5-10ms). This breakdown demonstrates whether you've actually thought about where the bottleneck lives — often it isn't the model itself that's slow, but "unglamorous" segments like feature fetch or serialization eating most of the budget, which points to a different optimization (model compilation vs. caching vs. trimming payload size).

Today's Practice Problem

The Question

"Design a real-time product recommendation service for an e-commerce platform: user browsing and purchase behavior should be reflected in recommendations near-instantly, the system needs to support tens of millions of monthly active users, recommendation latency must stay under 100ms, and it must comply with GDPR constraints on personal data."

Source: Self-composed (adapted from System Design Handbook's ML System Design walkthrough) Difficulty: Advanced Round: Onsite system design (ML infra)

Breaking It Down

1. Clarify the problem first: Don't rush into an architecture diagram — first ask about functional requirements (is this ranking an existing product list, or generating personalized copy? How do you handle cold start for new users?) and non-functional requirements (what's the latency threshold? What's the traffic scale? Precision or recall priority? What hard constraints does GDPR impose on data retention and deletion?). The interviewer deliberately baked "near-instant" and "compliant" into the prompt to see whether you turn them into quantified design inputs, rather than nodding and moving past them.

2. Build a framework: Organize your answer as "estimate scale first, then layer the architecture, then deep-dive the highest-risk components." Give concrete numbers during estimation (e.g., tens of millions of MAU, ~20 recommendation requests per user per day → convert to peak QPS) so the interviewer sees your architecture decisions are grounded, not technology choices pulled out of thin air.

3. Go deep on the core: The three most important trade-offs in this design all hide in the details. First, the feature store has to support two completely different access patterns simultaneously — low-latency online reads and fully-recoverable offline training — so the online and offline sides must share the exact same schema and transformation definitions, or you get training-serving skew. Second, a new model can't ship in one step: it needs shadow mode to verify system stability, then canary to verify no negative impact, then A/B to verify business metrics, with an automated rollback trigger at every stage. Third, GDPR compliance isn't a checklist bolted on afterward — it has to shape the storage design itself: when a user requests deletion, the system needs to trace which features and which training snapshots that data flowed into, or the deletion isn't actually complete.

4. Wrap up: Close with one sentence that ties it together — "the core of this system isn't picking the best recommendation algorithm, it's making sure offline training and online serving share the same feature definitions, and that every model update passes through shadow → canary → A/B in sequence, with an automatic rollback to the last stable version if any stage shows an anomaly." That sentence signals what the interviewer is actually scoring: systems thinking, not model accuracy.

Sample Answer (What to Actually Say in the Interview)

Before I sketch the architecture, I'd like to confirm a few things: is this ranking an existing product catalog, or generating a personalized list in real time? How do we handle cold start for users with no history? I'll assume the latency bar is checkout-page level — under 100ms — and I'll assume tens of millions of MAU with roughly 20 requests per user per day, which puts peak traffic in the low thousands of QPS. On GDPR, I'll assume users can request deletion of their personal data, which means my feature store needs to trace the provenance of every feature back to its source data — otherwise a deletion request only clears the primary database and doesn't actually propagate.

Architecturally I'd split this into three layers: real-time events (clicks, purchases) flow in through Kafka, get processed by a streaming pipeline into near-real-time features (like product categories browsed in the last five minutes) and written to a low-latency online feature store like Redis; the same events also get batch-processed into historical features (like a user's 30-day category preference) and stored in a queryable offline warehouse for training. Both paths sharing the same feature definitions is the single most important step for avoiding training-serving skew — if online and offline compute "recent purchase count" with different logic, the model's inputs at inference time won't match what it saw during training, and prediction quality degrades silently with no obvious cause.

For deploying a new model, I wouldn't cut traffic over directly — I'd run shadow mode first to confirm the system holds up under real traffic without errors, then canary 5% of traffic to watch for negative signals, and only then use A/B testing with a business metric like conversion rate to decide on full rollout, with automated rollback conditions at every stage — for example, an automatic revert to the previous model if latency exceeds a threshold or the error rate spikes. For monitoring, I'd split it into three layers: infra layer watching QPS and latency, data layer watching feature availability and input distribution drift (quantified with something like PSI), and model layer watching whether the prediction distribution or live accuracy is degrading over time. Keeping these three layers separate is what prevents you from misdiagnosing "the data pipeline broke" as "the model got worse."

Self-Check List

Use this table to check whether your answer hit the key points:

Further Reading

• Machine Learning System Design Interview: Step-by-Step Guide 2026 — the core reference for today's article, with full scale estimation, an architecture diagram, and ten common follow-up questions
• ML System Design Interview Questions (PracHub, 295 questions, 79 companies) — filter real interview questions by company, useful for company-specific prep
• Refonte Learning: Machine Learning System Design Interview — another angle on the monitoring and A/B testing section, pairs well with the third core concept above

References

• Machine Learning System Design Interview: Step-by-Step Guide 2026 — primary source for the core concepts and practice-problem breakdown, including the feature store architecture, deployment strategy, and monitoring layering
• ML System Design Interview Questions (PracHub) — supporting source for the canary/shadow/A-B/blue-green classification in the "deployment strategy" section
• Refonte Learning: Machine Learning System Design Interview — supplementary source for the "monitoring needs three layers" section