Improving User Retention Through Experimentation

Why Retention Is the North Star

Retention is the single metric that separates products that genuinely work from products that appear to work. You can inflate acquisition numbers with paid campaigns. You can inflate activation numbers with clever onboarding. But you cannot fake retention. If users come back tomorrow, next week, and next month, your product is delivering real value. If they don't, everything else is a vanity metric.

The business case for retention is equally compelling. Research from Bain & Company consistently shows that a 5% increase in customer retention increases profits by 25–95%, because retained customers buy more, cost less to serve, and refer others. Retention is the mechanism by which product value converts to company value.

Understanding Your Retention Curve

Before running experiments, understand your current retention shape. Plot a cohort retention curve: for users who signed up in a given week or month, what percentage are still active after 1 day, 7 days, 30 days, 60 days, and 90 days?

Almost every product shows the same shape: a sharp early drop followed by a gradual decline that eventually flattens. The questions to ask are:

• How steep is the early drop? A 70% drop between Day 0 and Day 1 indicates an activation problem—users aren't getting to their first value moment.
• Where does the curve flatten? The long-term retention floor tells you what percentage of users have truly adopted the product into their workflow. This is your "retained" base.
• How quickly does it reach the floor? Products that reach their retention floor quickly have clearer value propositions and better fit with user needs.

Different retention problems require different experiments. An activation problem requires onboarding experiments. An engagement drop at Day 30 might require habit-forming features or notification experiments. A decline at Day 90 might indicate competitor switching or a missing feature.

The Aha Moment: Defining and Accelerating It

The "aha moment" is the specific action or moment when a new user first experiences the core value of your product. It's the moment the product clicks—when a user thinks "yes, I get it, this is useful for me."

Products with clear, fast aha moments have dramatically better retention than products where users have to work to discover the value. Facebook famously identified that users who connected with 7+ friends within their first 10 days retained at dramatically higher rates. Twitter found that following 30 accounts quickly was the strongest predictor of long-term engagement.

How to find your aha moment

Segment your retained users (90-day+ actives) from your churned users. Run a correlation analysis on their Day 1–7 behavior. Which actions do retained users take that churned users don't? The action with the strongest correlation is likely your aha moment.

Experiments to accelerate aha moment arrival

• Remove steps between signup and the aha action. Every step you remove from the path to the aha moment increases activation rate.
• Pre-populate the aha moment. If the aha moment requires users to have data in the system, can you import their data automatically or show a demo environment?
• Surface the aha moment earlier. Rather than making users navigate to it, can you make it the first thing they see?

Day 1 Retention: The Critical First Session

Day 1 retention—whether a user returns within 24 hours of signup—is the strongest predictor of long-term retention in most products. Users who come back on Day 1 are 3–5x more likely to still be active on Day 30.

Experiments to improve Day 1 retention

• End-of-session hook: Before users leave their first session, give them a specific reason to return. "Your first report is generating—we'll email you when it's ready" creates anticipation and an email touchpoint.
• Personalized welcome email: Send a personalized email within 2 hours of signup recapping what the user accomplished and suggesting exactly what to do next. Timing matters: emails sent within 2 hours outperform next-day emails by 40%+ on click-through rate.
• In-progress state: If possible, leave users with something visibly "in progress" or incomplete at the end of their first session. The Zeigarnik effect (humans remember incomplete tasks better than completed ones) creates pull to return.

Day 7 Retention: Building the Habit

Day 7 retention is where habit formation is decided. Users who reach Day 7 have typically had 2–4 sessions and have begun to understand the product's value. The goal at this stage is to reinforce the habit loop: cue, routine, reward.

Experiments to improve Day 7 retention

• Notification timing: For products where daily use makes sense, test different times of day for push or email notifications. Users have different routines—some prefer morning check-ins, others evening. Personalized notification timing (based on when the user actually tends to log in) outperforms fixed broadcast times.
• Feature depth progression: Design a deliberate feature unlock sequence where users discover more powerful capabilities in their first week. Each new capability they discover is an additional reason to return.
• Streak mechanics: For daily-use products, visible usage streaks (similar to Duolingo's streak counter) can increase Day 7 retention by 20–30% by making the habit streak itself a reward.
• Social connections: Products become stickier when users have social connections within them. Prompting users to connect with colleagues or follow other users during their first week increases Day 7 and Day 30 retention.

Day 30 Retention: Becoming Indispensable

By Day 30, users have either integrated your product into their workflow or they haven't. The experiments that matter most here are about depth of integration and switching cost.

Experiments to improve Day 30 retention

• Data imports and integrations: The more of a user's data and workflow lives inside your product, the higher the switching cost. Promote integrations with tools users already use (CRMs, project management tools, communication platforms) prominently in the first 30 days.
• Saved configurations and templates: Users who have customized your product to their workflow (saved filters, templates, dashboards, workflows) retain at significantly higher rates. Encourage this customization actively.
• Weekly digest emails: A weekly email summarizing what the user accomplished or what changed keeps your product top-of-mind for users who aren't daily-active. This can recover users who might otherwise drift to 30-day churn.
• Milestone celebrations: Acknowledging when users hit meaningful milestones (first month, 100th event tracked, first winning experiment) creates emotional connection and reminds users of the value they're accumulating.

Using Multi-Armed Bandits for Retention Experiments

Retention experiments have long measurement windows—you can't know if Day 30 retention improved until Day 30. This makes traditional A/B testing inefficient, because you're exposing users to potentially inferior experiences for a month before you can declare a winner.

Multi-armed bandits address this by continuously updating traffic allocation as early signals come in. Even before Day 30 data is available, Day 1 and Day 7 signals can guide traffic toward better-performing variants. This reduces the total cost of running retention experiments and reaches confident conclusions faster.

Thompson Sampling, the bandit algorithm used in Experiment Flow, is particularly well-suited to retention experiments because it maintains uncertainty estimates that naturally widen as the measurement window grows—preventing premature declarations of winners on early, noisy data.

Retention Metrics to Track

Run a retention dashboard that tracks these metrics weekly for each cohort:

• Day 1, 7, 14, 30, 60, 90 retention rates by signup cohort
• Session frequency (sessions per active user per week)
• Feature breadth (number of distinct features used per active user)
• Monthly Active Users (MAU) and DAU/MAU ratio
• For SaaS: Net Revenue Retention (NRR) by cohort

These metrics tell you not just whether users are retained, but whether they're deepening their engagement—which predicts future retention and NRR.