Product Builder Interview Daily — 2026-08-27: AI Product Design

🌏 中文版

Today's Topic

AI Product Design interviews don't test whether you can explain what an LLM is — they test whether, given that the model is guaranteed to make mistakes, you can design a system that catches those mistakes before they damage user trust. These questions show up often in interviews at AI-native startups and big-company AI product teams, where interviewers want to see you treat human-in-the-loop as part of the system's design, not a fire-fighting patch bolted on after something breaks.

Most candidates' blind spot is reducing an AI product to "model + UI" — debating which model to pick or how smart the interface should feel, without answering the more fundamental architecture question: where in the system does a human step in, what do they see, and how fast does their judgment feed back into the system? Today's practice is about answering those questions clearly.

Core Frameworks

The Four-Layer Framework: Memory / Retrieval / Reasoning / Control

In a recent interview, London-based PM Riddhi Bhasker argued that most complaints about "bad AI products" turn out to be architecture problems, not model problems. She splits an AI product into four independent architectural layers, and each one needs a deliberate design decision from the PM — the model won't make these calls for you:

The Three Trust Calibration Questions (how to place human-in-the-loop)

Bhasker gives every AI product PM a set of questions they need a clear answer for:

1. What confidence threshold triggers human review? (Not "should we review" — but "at what risk level and confidence score does review kick in.")
2. What does the reviewer actually see? (Enough to reconstruct the model's reasoning, not just whether the outcome was right or wrong.)
3. How fast does a human's judgment feed back into the model's future behavior?

If you can't answer all three, your product runs on hope, not a control layer.

Today's Practice Question

The Question

You work at a B2B AI customer-support platform. Your AI agent automatically replies to customers and closes tickets on their behalf. Last quarter's auto-close accuracy was 94%, but part of that 6% error rate is the AI marking a ticket "resolved" while the customer is still waiting on a refund — driving up complaints and dragging down NPS. As the PM for this feature, your boss wants the next version to define rules for when the AI can auto-close a ticket and when it must route to a human first. How do you approach it?

(Source: an original scenario built around human-in-the-loop design for an AI customer-support platform, drawing on Riddhi Bhasker's architectural framework and the data-driven spirit of Intercom's PR-review agent case.)

How to Break It Down

1. Clarify the problem: First ask how that 6% error rate breaks down — is it concentrated in specific ticket types, or spread randomly? Is the cost of a misjudged refund ticket disproportionately high?
2. Define the users: Separate two stakeholders — the internal reviewer (support lead) and the end user (customer) — since they have different tolerances for broken trust.
3. Apply the framework: Run the Trust Calibration questions. Design tiered confidence thresholds by ticket type and dollar amount instead of one global threshold, and make sure reviewers see the AI's reasoning — the conversation log plus the policy clause it cited — not just whether the ticket was closed.
4. Propose a solution: Route high-risk types (refunds, billing disputes) to human review every time; allow auto-close only for low-risk types (password resets, FAQs); let a confidence score decide the middle ground dynamically — tiered trust rather than a binary switch.
5. Define success: Use "share of auto-closed tickets that don't get reopened by a complaint" as the north star, not "auto-close rate" — otherwise you'll optimize for a metric that looks efficient while quietly eroding trust.

Sample Answer (say it like this in the interview)

Frame the problem first: I'd start by looking at how that 6% error rate breaks down, rather than rushing to adjust the overall automation rate. If the errors cluster in refunds and billing disputes — high-dollar, emotionally charged ticket types — the problem isn't "AI accuracy is too low." It's that we're applying the same confidence threshold to tickets with wildly different risk levels.

Break it down with the framework: I'd split tickets into three risk tiers. Low-risk types like password resets and FAQs can let the AI close on its own. High-risk types — refunds, billing disputes, complaints — always route to human review, no matter how confident the AI is. The middle tier gets decided dynamically by confidence score. At the same time, the reviewer interface can't just say "AI marked this resolved" — it needs to show which part of the conversation the AI cited and which policy it applied, or human review is just a rubber stamp.

State the trade-off clearly: This design lowers the overall automation rate in the short term, since high-risk tickets are forced through human review. But I'm betting that reopened-ticket rate and NPS are the real north star — a high automation rate that drives complaints back up makes this feature a net negative. Success means the share of tickets that get auto-closed and stay closed goes up, not the automation rate itself.

Self-Check

Use this table to check whether your answer hit the key points:

Today's Case Study

Intercom: letting AI auto-approve 19% of pull requests by trusting the system, not the model

Intercom published an internal experiment this year on letting an AI agent review and auto-approve engineering pull requests. AI agents now author more than 93% of their PRs, and 19% of those ship with no human review at all. They didn't start by trusting the model — they first ran a controlled pilot of over 100 PRs, which produced zero reverts and cut review time by 6-16x at the 75th percentile. After rolling out more broadly, AI-approved backend code has a 0.53% revert rate, versus 5.39% for human-approved code. The system's design closely echoes Bhasker's four-layer framework: the review agent is split into sub-tasks (one sub-agent each for description quality, whether the diff matches intent, security concerns, and logical correctness), any PR that's too large or unclear in scope gets bounced back for splitting, any engineer can request human review at any time, and every AI approval decision is fully logged and auditable.

Interview connection: This case answers the follow-up question "how do you prove AI review won't lower quality?" The answer isn't a promise based on intuition — it's a controlled pilot plus an auditable decision trail, which is exactly what the third Trust Calibration question — how human judgment gets systematically verified and recorded — looks like in practice.

Further Reading

• Riddhi Bhasker: trusting AI products at scale comes down to architectural decisions, not model capability — the original source for today's framework; the full interview also covers treating an LLM as "the most expensive employee in the system" when designing call costs.
• Cat Wu (Anthropic's Claude Code product lead) on how AI is reshaping the PM role — she interviews multiple PMs a week who want to move into AI product roles; this piece covers which skills get filtered out and which ones survive.
• How a Yelp AI PM designs with sample conversations instead of wireframes — a concrete walkthrough of using real conversations rather than a PRD as the first design artifact, good prep for "how do you prototype an AI product" questions.

References

• Riddhi Bhasker interview: AI Products Are Trusted at Scale — source for "Core Frameworks" and the Trust Calibration questions.
• Intercom Blog: AI is approving our pull requests — source for the data and design principles in "Today's Case Study."
• Lenny's Newsletter: How Anthropic's product team moves faster than anyone else — source for "Further Reading."