Skip to content
All tags

#experimentation

2 posts
product deep-dive

Value Validation for Digital Products: From Assumption Maps to the M3 Retention Baseline for AI

The unit of validation is an assumption, not an idea. Kohavi's data shows the industry median experiment success rate is ~10%, which means roughly 22% of 'winning' experiments at p<0.05 are false positives. Sean Ellis's 40% threshold has no publicly available dataset. AI product retention should be baselined at M3 rather than M0, and GRR splits from 23% below $50/mo to 70% above $250/mo.

#value-validation#product-market-fit#product-discovery#experimentation#pricing#product-management#ai-product
ai guide

RAG A/B Testing: A Scientific Approach to Comparing Pipeline Configurations

"Adding a Cross-Encoder feels better" is not a scientific evaluation. A/B testing tells you whether a change actually works, how much it helps, and which query types benefit.

#rag#ab-testing#experimentation#metrics#pipeline