AI Engineer Interview Daily — 2026-10-26: ML Fundamentals

🌏 中文版

Today's Focus

Monday is ML Fundamentals day. Today's focus is a scenario almost every MLE onsite eventually asks: a model validated well offline, but a few weeks after launch its performance drops — how do you diagnose it. On the surface this looks like a monitoring-tools question, but what it's really testing is whether you can keep "the world changed" (drift) cleanly separated from "your own system already had two inconsistent code paths" (skew) — because the fix for each is completely different. Interviewers want to hear a diagnostic sequence with real order to it, not three jargon terms thrown out at once.

Core Concepts

Data drift vs. concept drift vs. prediction drift

All three describe a different probability distribution shifting inside the same system. Data drift is the input feature distribution P(X) changing (say, the age structure of your user base shifts). Concept drift is the relationship between inputs and labels, P(Y|X), changing — the same feature combination now maps to a different real-world outcome, like how spending behavior's relationship to default risk shifted around a pandemic. Prediction drift is the model's output score distribution P(Ŷ) changing; it needs no labels to detect and is usually the earliest warning signal, but it's a symptom — it won't tell you on its own whether the root cause is data drift or concept drift. The classic interview trap is treating all three as the same thing. Be explicit: concept drift can only be confirmed once real (often delayed) labels come back, while data drift and prediction drift can be detected in real time.

Training-serving skew isn't drift

This is the detail interviewers most often use to catch people out: the fundamental difference between skew and drift is whether the world actually changed. Drift means the world changed and your model hasn't kept up. Skew means the world didn't change at all — it's that the feature your training pipeline computed in batch SQL/Spark and the same feature your serving path computes in real-time Python/Flink were never actually the same number. Offline AUC might read 0.97, but recompute it using the exact features your serving path actually receives and it drops to 0.68, because training used a statistic like "orders in the last 30 days" that simply isn't available at the point in time serving needs it. Misdiagnose this as drift and rush to retrain, and you just ship a new model with the exact same bug — nothing gets fixed.

PSI (Population Stability Index) tells you whether a distribution moved

PSI is the most common metric for comparing a feature's distribution during training against its distribution in the current time window: bucket the values into bins and sum up the change in each bin's share. The industry-standard thresholds are PSI < 0.1 = stable, 0.1–0.25 = a minor shift worth watching, and > 0.25 = significant drift that should trigger retraining or a rollback. Know PSI's limitation going in: it only tells you a distribution moved, never why. Binning choices, small sample sizes, and seasonality can all trigger false alarms — so a PSI breach is a signal to start investigating, not a conclusion on its own.

An offline metric isn't the same thing as an online business metric

Even once skew and drift are both ruled out, picking the wrong offline metric in the first place can decouple "looks good" from "works in production." Offline evaluation uses some fixed metric (AUC, F1) against a historical distribution, but the business actually cares about post-launch conversion, revenue, or the cost of misclassification — and in between sit data leakage, how you split your data in time, and which guardrail metrics your A/B test even tracks. A common interviewer follow-up is "how do you know this model actually works, not just that the score looks good" — the answer ties back to evaluating offline with a time-respecting split, then confirming the business metric with an A/B test rather than trusting a single offline number.

Today's Practice Question

The Question

You deployed a model, but two weeks later its performance has clearly dropped. How would you diagnose this, and how would you fix it?

Source: Spector AI Machine Learning Engineer interview (compiled from the knok.work interview guide)　Difficulty: Medium　Round: Onsite technical

How to Break It Down

1. Clarify first: Is the drop sudden or gradual (sudden points more toward a pipeline bug or skew; gradual points more toward drift)? Did anything change on the product side around deployment (a UI change alone can shift user behavior)? Do ground-truth labels arrive immediately or with a delay? What's the business cost of this drop versus the cost of retraining?
2. Build a framework: Lay out the investigation as four layers instead of dumping three terms at once — (1) check whether the data pipeline itself is broken (schema changes, dropped joins, unexpected nulls), (2) verify the training and serving feature-computation logic actually agree (the skew check), (3) compare feature distributions with PSI or a KS test (data drift), and (4) if delayed labels are available, check whether the relationship between predictions and real outcomes has shifted (concept drift). Drift only becomes a live hypothesis once the first two layers are ruled out.
3. Go deep on the technical core: The real substance here is that skew and drift need completely different fixes. Skew is a code bug — the answer is consolidating feature definitions into a shared feature store (like Feast) so training and serving call the same definition, which makes the two paths structurally unable to diverge again. Drift means the world actually changed — only then is retraining the right answer, and ideally through a continuous-training pipeline with a validation gate, so detecting drift doesn't automatically push a new model to production; it has to pass validation first.
4. Close it out: Land on one sentence — "diagnosing a performance drop isn't about picking a label to stick on it; it's ruling out pipeline issues and skew in order, and only what's left over is drift that actually needs retraining" — then add a line on prevention: consolidate feature definitions into a feature store to eliminate skew altogether, and use tiered PSI alerting (log minor shifts, only trigger retraining on major ones) to avoid over-sensitive false alarms.

Sample Answer (say it like this in the interview)

Rule out the cheapest explanation first. When performance drops, I'd check the data pipeline itself first — did the schema change, is a join silently dropping rows, has some upstream source been returning nulls for a while. These checks are cheaper than any statistical test, and in my experience, pipeline bugs plus skew combined show up more often than genuine concept drift. Next I'd rerun offline evaluation using the exact raw features serving actually receives. If the score drops back to the live level right there, that's training-serving skew, not drift — it means the feature-computation logic never matched between training and serving in the first place, and retraining won't fix that.

Only talk about drift once the distribution genuinely moved. With the pipeline and skew both ruled out, I'd compare the current time window's feature distribution against the training-period distribution using PSI, treating anything above 0.25 as significant drift. But PSI only tells me the distribution moved — it doesn't tell me whether the relationship with the label changed too. If delayed labels are coming in — something like default risk, where the real outcome only shows up after a while — I'd separately track whether the relationship between predicted scores and actual outcomes has shifted systematically. That's the direct evidence for concept drift.

Match the fix to the root cause. If it's skew, I'd consolidate the feature definition into a feature store so training and serving share the exact same logic, eliminating the problem structurally. If it's genuine drift, only then would I schedule retraining — behind a continuous-training pipeline with a validation gate, so a detected drift triggers a retrain candidate, but that candidate still has to pass validation before it replaces the live model; nothing switches over unsupervised.

Self-Check

Use this table to catch anything your answer might have skipped:

Further Reading

• Training-Serving Skew and Feature Stores Explained — Stacktrick — walks through a real churn-model example: 0.977 offline AUC versus 0.681 once recomputed with the actual live features, a concrete demonstration of the skew-versus-drift gap.
• Concept Drift vs Data Drift vs Model Drift: Key Differences — ML Monitoring Report — tabulates the three drift types against detection tools like PSI, KS tests, and ADWIN, a good next step into the algorithmic side of detection.

References

• spector-ai Machine Learning Engineer Interview: Questions & Prep — knok.work — source of today's practice question and its STAR-format sample answer.
• Machine Learning Engineer Onsite Interview: Round by Round — CoPilot Interview — source of the "Today's Focus" scenario, corresponding to question 6 in its ML fundamentals round.
• Streaming Machine Learning – Real-Time Features, Kafka, Flink — Data Expertise — backs the three-distribution definitions (P(X), P(Y|X), P(Ŷ)) in "Data drift vs. concept drift vs. prediction drift."
• Training-Serving Skew and Feature Stores Explained — Stacktrick — backs the core distinction and worked example in "Training-serving skew isn't drift."
• Monitor ML Models in Production (Drift): Practical Guide 2026 — khimananda.com — backs the PSI thresholds (0.1 / 0.25) and tiered-alerting strategy in "PSI tells you whether a distribution moved."