Subscribe Bookmark


May 27, 2014

Scientific thinking in market research


Scientists aren’t always right. In fact, the good ones get used to being wrong most of the time. Solving problems no one else has solved often means you try a lot of approaches that don’t work. If you’re on a research and development team for a business, there’s also the chance that your solution is too expensive, doesn’t scale, or doesn’t have a market. Either way, figuring out all the ways you could be wrong about something is part of the job.

That’s why I’m not very patient when I talk to data analysts who are adamant about getting the “right” answers, or with people who say a software package is useless if it doesn’t implement their favorite statistical technique. Lengthy arguments where market researchers religiously defend different statistical techniques that will return the same answer 99 percent of the time make me tired because those debates miss the point. Whether it’s big data, deep learning, or natural language processing, there’s always an exciting new technique, and like any trend in any market, people will be obsessed with it until they’re not.

Market researchers shouldn’t be obsessed with right. They should be obsessed with better. In my opinion, the biggest opportunity for better in market research is not in the newest statistical techniques. It’s a diligent implementation of the oldest and least-sexy parts of scientific research.

Static Research Report == Missed Opportunity

If the end result of your market research projects are reports that are never seen again, you’re missing a valuable opportunity to learn about your business — and your customers — over time. The trend you missed in your last analysis may be coming around again. Maybe an older survey had a warning sign that your best-selling product has a new competitor — one you couldn’t see the first time but would be obvious now. There may be past successes in your market research that you could implement going forward, or maybe there’s a chance to learn from past mistakes. Either way, if you don’t know the market research history of your business, you’ll just repeat it without improving it.

Markets change fast. That may be why a lot of businesses ignore “old” data in favor of collecting new information. I would argue that rapid change is an important reason to study the past as well as the present. Past studies and the projections made from them give your business an opportunity to spot flaws in your thinking, or improve on methods that worked last time.

New analyses with old data and recreations of past studies with new data are valuable tools to identify how your market is changing. Storing past analyses in a way that makes it easy to re-create them means you can do it at a fraction of the cost of doing something completely new. Why wouldn’t you take advantage of an opportunity to learn that’s inexpensive and quick?

New Findings Need Context

Anyone who works in science is tired of new and surprising results appearing in the press as if they’re categorically true. One study based on a tiny sample with findings that can’t be replicated isn’t news to a scientist, but in business you sometimes have to make decisions based on small samples and imperfect information. Having results from past studies at your fingertips makes it much easier to place new results in context. Context means better information, and better information means better decisions. Research reports stored as documents without their data or the steps that were used to obtain the results can provide some context. But wouldn’t it be better to actually see the data and re-create the analyses side-by-side? 

JMP Can Help

As scientists are realizing that making it easy to reproduce study results is a critical piece of advancing science research, some are stepping forward to develop new tools and new best practices for reproducibility. JMP was originally developed for scientists and engineers, and many of these best practices have been built into the software for a long time. The JMP Scripting Language, the Data Table’s tools that make data changes transparent, and the live reports are all expressions of the philosophy that statistical analysis shouldn’t be mysterious, and statistical results should be reproducible. You can try JMP free, by the way.

Senior Member

 Yes! Having a reproducible workflow is so important (in science and everywhere!), saves so much time and is ridiculously easy in JMP :)

Thanks, Chelsea! When you're used to certain kinds of academia, you kind of take it for granted (I know I did), but a business environment is so different from an academic environment. We can learn a lot from each other!