Applying the Scientific Method to Analytics

Subscribe to our monthly newsletter to get the latest updates in your inbox

All of us remember something about the “Scientific Method” from class in Middle School—ask a question, construct a hypothesis, test, etc. I’ve probably used the term incorrectly on many occasions, but as Oxford Dictionaries Online defines it, it’s "a method or procedure that has characterized natural science since the 17th century, consisting in systematic observation, measurement, and experiment, and the formulation, testing, and modification of hypotheses." For this method of testing work, there must be some constant variables on which scientists can rely. Gravity as a constant, or perhaps a sterile testing environment free of outside bacteria. It’s all about eliminating variables and determining what moved the needle and how.

Do you trust your gut? Or do you trust the data?

The desire to do this kind of testing is at the heart of what I see many clients ask for. They suspect something might be true, but can they trust the numbers? Is the testing environment stable? Are there any constants by which to measure change? In the world of digital analytics, like any science, if the tools are broken or not implemented correctly it’s like trying to run tests in a chaotic world. And as we like to ask at Analytics Pros, “Would you rather have bad data or no data?” Given those options we’d say, stick with no data and go with your gut. Even a stopped clock gives the right time twice a day, but a clock set to the wrong time is never correct.

First things first

What this means is that the early phases of implementation require some of the hard and unglamorous work for getting those things set up correctly. “Getting the lab prepped,” as it were. This is how you truly apply the Scientific Method to analytics, you start from the beginning. The cooler, fun stuff happens after that. I’ve seen it over and over. The questions start to get more interesting as the results come in. “What if we did this!?” This is the moment we love to see at AP—the light bulb moments. To do this, however, means first investing in that “lab.” Do you have the right tools? Are they set up correctly? No one ever cared about how good a lab was, they only remember the results. But, as any scientist will tell you, a good lab is a must that makes all subsequent discoveries possible.