Metrics & Analytics Interview Guide: From North Star to Experiment Design

How Metrics Interviews Work

The Metrics round typically appears in big tech PM interviews as the Execution round, and also pops up as follow-ups within Product Sense and Strategy. There are three typical question formats:

Metric design questions: "What metrics would you use to measure the success of Instagram Reels?" — testing whether you can derive a coherent, non-contradictory set of metrics from a business objective, rather than casually naming DAU.

Data diagnosis questions: "YouTube's watch time dropped 10% this week. How would you investigate?" — testing the structure of how you decompose problems and the order in which you eliminate hypotheses, not whether you can write SQL.

Experiment design questions: "How would you validate that this new feature is effective?" — testing your depth of understanding of A/B testing, including sample size, experiment duration, and how to interpret results.

All three share a common thread: interviewers don't care whether you can calculate exact numbers; they care about the structure of your thinking and the logic behind your trade-offs.

North Star Metric: Why DAU Isn't a Silver Bullet

The North Star Metric is the single metric the entire product team aligns on. Its purpose isn't to "measure everything," but to "serve as the final arbiter when the team disagrees."

Three criteria for choosing a North Star:

1. Reflects user value, not just traffic. DAU tells you how many people opened the app, but not whether they got value. Spotify uses "weekly listening hours" instead of DAU because someone opening the app without playing a single song is meaningless to Spotify.

2. Can be influenced by team actions. Revenue is an important metric, but most feature teams can't directly influence it. If your North Star is "monthly revenue," there are too many layers between the team's daily work and that number. Choose a metric closer to team action — like "paid conversion rate" or "subscription renewal rate."

3. Leading indicators over lagging indicators. Churn rate is a lagging indicator — by the time you see it, users are already gone. "7-day retention rate" or "percentage completing core action in first week" are leading indicators that let you intervene before problems escalate.

A common interview trap is saying "I'd use DAU." DAU is almost never the best North Star because it doesn't distinguish high-value users from passersby. When the interviewer asks "Why not DAU?", you need to offer a specific alternative with reasoning.

Metric Trees: From North Star to Actionable Levers

The North Star by itself is too abstract to guide daily decisions. A metric tree's purpose is to decompose it layer by layer until every leaf node is a lever the team can directly operate.

For an e-commerce product with "monthly GMV" as North Star:

Monthly GMV
├── Visitors × Conversion Rate × Average Order Value
│   ├── Visitors
│   │   ├── Organic traffic (SEO, word of mouth)
│   │   └── Paid traffic (ads, affiliates)
│   ├── Conversion Rate
│   │   ├── Browse → Add to Cart (product page experience)
│   │   ├── Cart → Checkout (checkout flow)
│   │   └── Checkout → Payment Complete (payment success rate)
│   └── Average Order Value
│       ├── Average item price
│       └── Items per order (cross-sell effectiveness)

When drawing this tree in an interview, the key points aren't completeness but:

• Choose the right decomposition dimensions. Multiplicative decomposition (visitors × conversion × AOV) feels more structured than additive (category A + category B + category C).
• Identify the biggest lever. After drawing the tree, the interviewer will ask "Which would you optimize first?" The answer should be based on: which node has the most room for improvement at the lowest improvement cost. If conversion rate is only 1% while the industry average is 3%, then conversion rate comes first.
• Point out counter-metrics. For every metric you want to optimize, pair it with a metric you monitor simultaneously. When pushing up conversion rate, monitor return rate; when pushing up AOV, monitor repeat purchase rate. This demonstrates systems thinking.

Funnel Analysis: The Quick Interview Version

Funnel analysis is a foundational tool in Metrics interviews. You don't need to write SQL, but you should be able to quickly draw a funnel on the whiteboard, label conversion rates at each step, and find the step most worth improving.

Using a SaaS user activation funnel as an example:

When the interviewer asks "Which step would you improve first?", don't just pick the one with the lowest conversion rate. Consider:

• Absolute number impact. Improving sign-up conversion from 30% to 35% adds 500 users. Improving core feature usage from 50% to 55% adds only 60 users.
• Improvement feasibility. Homepage-to-signup conversion might be limited by traffic quality (you can't change the ads team's targeting strategy), but onboarding-to-core-feature conversion is entirely within your control.
• Downstream cascade effects. Improvements at the top of the funnel amplify absolute numbers at every downstream step.

A/B Testing: Design, Pitfalls, and Interview Answers

A/B testing questions in interviews aren't testing whether you can calculate p-values. They test your judgment about the experimental process.

Designing an experiment requires answering four questions:

1. What's the hypothesis? "We believe simplifying the checkout flow will increase conversion rate" — the hypothesis must be specific enough to be falsifiable.
2. What's the primary metric? Choose only one primary metric for decision-making. Track 2-3 secondary metrics and 1 counter-metric simultaneously.
3. How large a sample? You don't need to calculate exact numbers in interviews, but you should know the factors affecting sample size: baseline conversion rate, minimum detectable effect (MDE), significance level, and statistical power. The smaller the effect, the larger the sample needed.
4. How long to run? At least one full business cycle (usually a week) to cover weekday and weekend behavioral differences.

Three common pitfalls:

• Novelty effect: Users use new features more in the early period out of curiosity, inflating short-term metrics. Solution: run the experiment long enough (at least two weeks) or only look at the second week's data.
• Simpson's paradox: Overall results show version B wins, but when split by user group, version A wins in every subgroup. Usually caused by uneven traffic allocation. Mentioning this in an interview scores major points — it shows you know "looking at totals isn't enough, you need to look at segments."
• Peeking problem: Looking at results before the experiment is complete, then stopping when you see significance. This dramatically inflates the false positive rate. Solution: pre-define the experiment end date, or use sequential testing methods.

Data Interpretation: Handling Counterintuitive Results

Interviewers love this follow-up: "The experiment shows the new version's conversion rate increased 5%, but revenue didn't change. How do you explain that?"

Framework for handling counterintuitive data:

1. First check data quality. Any logging bugs? Bot traffic counted? Sample contamination (a user seeing both A and B versions)?
2. Segment the data. Split by user group (new vs returning), platform (iOS vs Android), region — the overall average might be masking differences between segments.
3. Find the causal chain. Conversion up but revenue unchanged could mean the conversion increase came from low-AOV users. Or the new version attracted more "try-it-once" users who convert but buy cheap items.
4. Decide whether to ship. Not all statistically significant results are worth launching. If the effect is too small (e.g., conversion increases 0.1%), the engineering cost of shipping and long-term maintenance might not be worth it.

Saying "I'd first rule out data quality issues, then segment to look at subgroup differences" already puts you ahead of most candidates.

Interview Tips

• First ask what stage the product is in. Early stage North Star might be activation rate; growth stage might be retention; mature stage might be ARPU. Different stages mean entirely different metric logic.
• Always pair with a counter-metric. Every time you say "I want to increase X," immediately follow with "while monitoring Y to ensure no degradation."
• Use specific numbers to give your answers weight. Don't say "conversion rate will improve"; say "I expect conversion rate to increase from 3% to 3.5%, based on industry benchmarks and our current UX friction."
• Acknowledge uncertainty. "This hypothesis needs a two-week A/B test to validate; currently it's my inference based on user interviews" — this is more persuasive than pretending to be certain.

Practice Question

Question

"Your SaaS product's paid conversion rate dropped from 5% to 3% this month. How would you investigate?"

Source: Google PM Execution Round　Difficulty: Medium　Round: execution round

Solution Framework

1. Clarify the question: Was it a sudden drop or gradual decline? Did it drop across all user groups or specific ones? Were there any concurrent changes (pricing, UI, marketing campaign ending)?
2. Build a framework: Decompose using a funnel — which step's conversion dropped? Did "trial→paid" drop, or did "signup→trial" drop, changing the quality of users entering the paid funnel?
3. Go deeper: Segment across dimensions — new vs returning users, traffic source, platform, region. Finding "who dropped" is more important than "how much it dropped."
4. Wrap up: Based on the diagnosis, propose 2-3 hypotheses, explain what experiments you'd run to validate, and how long it would take.

Sample Answer (How You'd Actually Say It in an Interview)

Elimination and segmentation. First I'd rule out data issues — confirm logging has no bugs, no bot traffic counted, and that the definition hasn't changed. Once data is reliable, I'd look at the timeline — did it drop suddenly one day (possible bug or event) or gradually over four weeks (possible structural issue). Then I'd segment across three dimensions: traffic source (Google Ads users vs organic), platform (iOS vs Android vs Web), and user type (individual vs team users).

Hypothesis building. Suppose I find the biggest drop is among new users from Google Ads. The most likely cause is the ads team recently changed their targeting strategy, bringing in more low-intent "browse and leave" users. In this case, the "conversion rate decline" isn't a product problem — it's a traffic quality problem. The solution is to confirm the targeting change with the marketing team and check whether CPA and LTV deteriorated in parallel. The second hypothesis: if all channels are declining, and the drop is at the "14-day trial → paid" step, then a recent product change may have affected aha moment reach rate — I'd check whether onboarding completion and core feature usage rates dropped simultaneously.

Action plan. Regardless of which hypothesis, I'd complete the segmented analysis within two days and identify the root cause. If it's traffic quality, short-term: adjust targeting; long-term: build a lead scoring model to filter low-quality traffic. If it's a product issue, first roll back the recent change and observe recovery, then design an A/B test for the improvement. I'd set up a weekly tracking dashboard until conversion rate returns above 4.5%.

Self-Check Checklist

References

• Lenny Rachitsky — How to Set Your North Star Metric — North star metric selection logic, with Spotify, Airbnb and other cases
• Amplitude — North Star Playbook — Complete operational manual from north star to metric trees; reading chapters 2-4 before the interview is sufficient
• Ronny Kohavi — Trustworthy Online Controlled Experiments — The authoritative A/B testing reference; novelty effect, Simpson's paradox, and peeking problem discussed in the interview all come from this book
• Mixpanel — Product Metrics Guide — Product metric categorization and funnel analysis frameworks commonly tested in Metrics & Analytics interviews
• Exponent — Execution Interview Guide — Covers metric tree decomposition, data interpretation, and A/B testing design structured answering methods for Metrics interviews