AI Engineer Interview Daily — 2026-09-09: ML System Design

🌏 中文版

Today's Topic

Today is the ML System Design rotation. The biggest difference from a coding round is that the interviewer isn't checking whether you can build a model — they're checking whether you can narrate the full chain: how data flows in, how features get computed, how a model gets deployed, and how you'd know it's still healthy after launch. By 2026, ML system design interviews rarely ask "how do you train a model" in isolation anymore. Instead you get open-ended prompts like "design a recommendation system serving 50M DAU," and you're expected to string together the data pipeline, feature store, serving, monitoring, and retraining loop within 45 minutes, while articulating the trade-offs at every layer. This round shows up as a core onsite segment, and it's the line that separates "an engineer who can tune hyperparameters" from "an engineer who can ship to production."

Core Concepts Cheat Sheet

Offline/online consistency in the feature store

A feature store doesn't exist to make feature lookups convenient — it exists to guarantee that the features seen during training match the features seen during inference exactly. That mismatch is called training-serving skew, and it's the most common yet hardest-to-catch bug source in production ML systems. The offline store (usually batch-computed via Spark/Hive into a data lake) generates training data; the online store (Redis or similar low-latency KV) serves real-time lookups. Both must share the same feature definitions and the same point-in-time correctness logic, or the model's production behavior won't match its offline evaluation.

The candidate generation → ranking → re-ranking serving pipeline

No system runs a full model against every item when the pool is millions to tens of millions large — the standard pattern is a three-stage funnel. Candidate generation uses cheap methods (embedding similarity, rule-based filters) to narrow millions down to a few hundred or a few thousand candidates; ranking applies a heavier model to score those candidates; re-ranking adds diversity, business rules, and deduplication as post-processing. This three-stage architecture directly determines how your latency budget gets allocated — interviewers care a lot about whether you can state how many milliseconds each stage gets.

A/B testing and shadow deployment

Before a new model ships, it runs in shadow deployment — inferencing in parallel with the live model, but only the live model's output is actually returned to users. This lets you compare output distributions and latency between the two without taking on any user-facing risk. Once it's stable, it moves into A/B testing, where a small slice of traffic validates the new model against real online metrics (CTR, conversion, retention) — an offline AUC or NDCG improvement doesn't guarantee an online metric improvement, and that gap is a favorite follow-up for interviewers.

Monitoring for data drift and model degradation

The biggest risk after launch isn't a crash — it's silent degradation. Either the input distribution shifts over time (data drift) or the relationship between features and labels changes (concept drift), and accuracy slides without anyone noticing. Monitoring design needs two layers: statistical drift metrics on feature distributions (e.g. PSI, KL divergence) and rolling averages of online business metrics — both need alert thresholds, not just an offline evaluation score you check once a quarter.

Retraining cadence and the data flywheel

Retraining more often isn't automatically better — you're balancing the value of data freshness against the compute cost and rollout risk of each retrain. A well-built system feeds user interaction signals (clicks, purchases, negative feedback) back into training data, forming a data flywheel: more data leads to a better model, which leads to a better user experience, which generates more data. This is one of the clearest signals interviewers use to judge whether you actually think in production terms.

Today's Practice Question

The Question

Design an end-to-end machine learning system that serves real-time recommendations for a consumer-facing product (feed, products, videos). The system must handle high read traffic and continuously evolving content and user behavior. Given assumptions (refinable with the interviewer): traffic around 10k QPS, p95 latency target ≤150ms; inventory around 10M items, with daily additions and expirations; feedback signals include clicks, likes, and purchases as both implicit and explicit signals; privacy constraints include user consent, PII minimization, and right-to-erasure compliance. Explain and justify the design for: (1) data collection and event pipeline, (2) feature engineering and feature store (offline and online), (3) model training, labeling, and retraining strategy, (4) online serving architecture (candidate generation, ranking, re-ranking), (5) monitoring, alerting, and experimentation, (6) scalability, reliability, and cost considerations.

Source: OpenAI (Software Engineer, Technical Screen, via PracHub)　Difficulty: Advanced　Round: technical screen / onsite system design

How to Break It Down

1. Clarify first: Confirm the core user scenario (feed ranking? post-search recommendation?), the read/write ratio, whether 10k QPS is peak or average, whether the p95 150ms is end-to-end or just the serving layer, and how data retention and regulatory requirements (does GDPR's right-to-erasure affect the feature store design?) come into play.
2. Build a framework: Use a six-stage framework — data → features → training → serving → monitoring → retraining — to sketch the full data flow first, then go deep wherever the interviewer leans in. That way, even if you're interrupted to dig into one stage, you still have a complete map to return to.
3. Go deep on the core: The key trade-off in this question is the serving architecture — you can't run a heavy model against 10M items, so a two-stage funnel of candidate generation (embedding retrieval down to a few hundred to a few thousand candidates) plus ranking (a heavier model for precise scoring) is what makes the 150ms latency budget achievable. Also be explicit about how the offline feature store (for training) and online feature store (for inference) stay consistent, avoiding training-serving skew.
4. Close strong: Wrap up with "what breaks first at 10x traffic and how does it degrade gracefully" and "what metrics and alerts would prove the system is healthy after launch" — this is exactly the follow-up direction attached to the original question, and raising it before being asked signals you're thinking ahead of the interviewer.

Sample Answer (What You'd Actually Say)

Framing the problem: Before I sketch the architecture, I want to confirm a few assumptions — this is a feed-style recommendation scenario, read-heavy, 10k QPS is peak traffic, and the p95 150ms is end-to-end from the API gateway to the returned result. Given that scale, I'd split the system into an offline (training) path and an online (serving) path that share one feature store definition, to avoid training-serving skew.

Core architecture: On the data side, user behavior flows through Kafka into two paths — a batch job (Spark) computes offline features into the data lake for training, and a stream job computes online features written into Redis for real-time inference lookups, both sharing the same feature-definition code so the two sides never disagree. On the serving side, you can't run a heavy model against 10M items, so it's a two-stage funnel — candidate generation uses embedding similarity retrieval to narrow candidates down to a few hundred, then a ranking model scores them precisely. The latency budget is roughly 30ms for retrieval, 60ms for ranking, with the rest reserved for network and post-processing.

Monitoring and degradation: Before launch, I'd run shadow deployment to compare the new and old model's output distributions and latency, then move to A/B testing to check online CTR and conversion rather than relying solely on offline AUC. For monitoring, I'd watch both PSI on feature distributions and rolling averages of online business metrics, with alerts on both. At 10x traffic, the candidate generation stage would be the first to buckle, so I'd design in a cache for popular candidates and a fallback to rule-based ranking — better to accept some precision loss than to have the whole serving path go down.

Self-Check List

Use this table to check whether your answer covered the key points:

Further Reading

• Feature Store & Model Serving — System Design Space — A focused walkthrough of how feature stores keep "one shared feature definition" between training and serving, covering point-in-time correctness and training-serving skew in more depth than most interview prep material.
• How to Prepare for an AI/ML System Design Interview (2026 Roadmap) — Proposes a seven-component framework every ML system design answer should cover, useful for checking your own answer against missing pieces.

References

• Design an End-to-End ML System — PracHub — Source of today's practice question, its assumptions, and follow-up directions (OpenAI, Software Engineer, Technical Screen).
• ML System Design Interview Questions (2026) — PracHub — Source for the common question categories cited in the core concepts cheat sheet (feature store, A/B testing, monitoring).
• System Design Interview Prep 2026: Machine Learning & GenAI Guide — Fonzi — Supports the observation that 2026 ML system design interviews have shifted toward combined "feature store + serving + GenAI stack" question types.