An activation backtest is credible when the source query, eligible cohort, observation windows, comparison group, exclusions, and contrary cases can all be reproduced.
Freeze the candidate before reading the result
Write the activation candidate, eligible cohort, early window, retained-use event, later window, user or account grain, and exclusions before calculating the comparison. This keeps a promising chart from silently changing the definition. Store the query or transformation version with the result so a reviewer can rerun the same state against the same data snapshot.
Build mature cohorts only. Every included entity needs enough elapsed time to complete both windows, unless the method explicitly handles censoring. Remove internal testers, deleted records, imports, bots, duplicated identities, and known migration artifacts through named rules. Report how many entities each exclusion removes instead of hiding the denominator change.
Publish the comparison table
The table should show eligible entities, activation reach, later retained use, retained use among activated entities, retained use among non-activated entities, and the absolute counts behind every rate. Add false positives that activated without later retained use and negative cases that retained without meeting the candidate. Those rows often reveal that the proposed state is optional, too broad, or measured at the wrong grain.
Run sensitivity checks on plausible windows and thresholds. A candidate that appears predictive only at one arbitrary cutoff deserves a lower confidence label. Compare signup periods, acquisition sources, plans, product versions, and account sizes when those differences affect eligibility or opportunity. The goal is a stable decision record, not the largest ratio found after many unrecorded attempts.
Keep association and intervention separate
Amplitude reported an association between top day-seven and top three-month performance across more than 2,600 companies. That benchmark supports investigating early behavior, while the team's own definition still needs a product-specific backtest. A correlation may reflect selection, customer fit, prior intent, plan, implementation support, or another shared cause. The backtest should end with the product decision it informs and the bounded intervention that could test causality.
Where the service stops
Reality Contact, LLC implements measurement records, but does not claim causality from correlations, choose product strategy, run experiments, contact users, replace the data platform, or promise retention changes. The buyer approves lifecycle definitions and exclusions, adopts the accepted queries for product review, investigates data exceptions, and chooses and runs the next activation experiment. This is analytics implementation and document preparation, and it does not replace statistical, privacy, security, legal, data-governance, or product review required by the buyer. Backtests describe the authorized historical data and accepted definitions; observed association does not establish that causing an event will change later retention.
Sources: Amplitude retention benchmark analysis; Mixpanel retention report documentation.