The world is being quietly rearranged by people who write very long documents.


The title they went with Enhancing External Validity of Experiments with Ongoing Sampling Noisy translates that to

Online experiments can now track changes over time without losing their meaning.


Researchers have built a new way to check if people signing up for online tests change over time. This means results from short tests, like those used to check new website features, will better reflect the real users.
Companies that run online tests to decide on product changes usually do them quickly. This paper offers a way to make sure those quick tests still represent all users, even if the user base changes during the test. This could mean fewer bad product decisions based on skewed data. It helps ensure that what works for a small group of early testers also works for the broader customer base.
Watch whether companies start publishing A/B test results that explicitly state they used this new method for assessing sample representativeness.

If you insist
Read the original →