Online experiments can now track changes over time without losing their meaning.
What happened
Researchers have built a new way to check if people signing up for online tests change over time. This means results from short tests, like those used to check new website features, will better reflect the real users.
Why it matters
Companies that run online tests to decide on product changes usually do them quickly. This paper offers a way to make sure those quick tests still represent all users, even if the user base changes during the test. This could mean fewer bad product decisions based on skewed data. It helps ensure that what works for a small group of early testers also works for the broader customer base.
The signal
Watch whether companies start publishing A/B test results that explicitly state they used this new method for assessing sample representativeness.