5-Second Range of Error Calculation
--
TL;DR — copy this Google spreadsheet to report your range of error in 5 seconds.
If you consider a measurement of user behaviour to be just a “sample” of the true, or natural, user behaviour, then you can use some simple equations to produce a range of error for your measurements in each sample.
To interpret sample results with varying sample sizes, you need to report a margin of error to know if there is actually a difference between the sample results.
In product management analytics, we typically want to measure ranges of error on a proportions (i.e. percentages, rates) of users. For example, 30% of users add a friend, plus or minus 2%. In math, this is called the confidence interval for the sample proportion.
Rather than Googling “range of percent error” every time, I recommend just having a Google Sheet on hand to make your analysis 5 seconds.
My example below is for product analytics, which typically looks at segments of users who perform different actions at different rates. But the same math applies to calculating CAC or LTV in marketing, any other data reported on in sample sizes. The second spreadsheet tab covers this case.
For my math geeks — I assume a normally distributed range of error symmetrical about the mean.
You can copy the formulas in this Google Sheet.
—
For example: here’s how many users in different segments return after 4 weeks of acquisition. The goal of this exercise is to calculate the 4-wk retention range of error (say, for a board presentation).
3 STEPS TO CALCULATE YOUR RANGE OF ERROR
1. Choose a z*-value that defines your desired confidence interval
For a risky business, I typically start with 80% confidence (1.28 z* score). That’s because I want the results to carry signal, but can’t afford to get blocked on a decision due to small amounts of data. A risky business must do that — take risks.
You must pick z* to satisfy the following conditions for the range of error to be valid:
- z* × proportion >= 10
- z* × (1-proportion) >= 10
In my examples, picking the the most extreme percentage of 32% (for Segment D), I get:
- 1.28 × 32% = 41% > 10% — TRUE
- 1.28 × (1–32%) = 87% > 10% — TRUE
2. Calculate the error interval for each segment
For example, for Segment A, this is:
- 1.28 × sqrt[ 40% × (1–40%) / 433] = 4.61%
Doing this for every segment, we get:
3. Calculate your error range by +/- your error interval in each segment
Some Google Sheet tips:
Set your z* value in it’s own cell so you can change it and compare results. If you can use a higher confidence interval, go for it.
To get the retention range, try:
=TEXTJOIN(“ “,TRUE, TEXT(PERCENTAGE — ERROR, “0%”), “ — “, TEXT(PERCENTAGE+ERROR, “0%”))
Check out Tab 1 cell F9 for an example of this formula in use.
