What survey research can (and can't) do: Interview with survey expert Walt Paczowski
Aug 10, 2017 7:16 AM
Walt Paczowski knows surveys and believes they are very useful for making decisions. "They take us into a realm of possibilities, which is where decision makers often need to go," he says.
He leads seminars for JMP on how to get the most from survey data. In advance of a couple of recent talks, he gave us a sense of the opportunities and challenges involved in survey research.
What types of problems or questions can be addressed through survey research? What can't you find out?
In general, there is no limit to the type of problems or questions that can be addressed through survey research; all the problems and questions decision makers confront, whether in business or government, require information for their solutions.
Usually, the information needed is about possibilities: What will happen if a price is changed, a new product is offered, or a new policy is adopted? Surveys provide that information.
Databases of transactions, what is often called Big Data, also provide information, but this information is only about past transactions based on past decisions – basically, what did happen. This is valuable because we can use this past data to make predictions or forecasts assuming there will be no or minimal changes in behaviors or conditions. Nonetheless, the scope of what we know is limited. Surveys expand our scope of knowledge by allowing us to obtain and use information about opinions, preferences about new products or price points, and especially about future plans, all items not contained in databases.
It is true that surveys can also tell us about past behaviors, perhaps not as effectively as databases of transactions, which is one shortcoming of surveys. But most importantly, they take us into a realm of possibilities, which is where decision makers often need to go.
What types of organizations are making use of survey research these days? Has that changed over time?
Surveys are not restricted to just one type of organization, or even one type of question or problem. We normally think of consumer-oriented businesses, usually called B2C businesses, as the primary users of surveys to gain information about their customers' opinions and preferences. Business-oriented businesses, the B2Bs, also use surveys for the same reasons. Government agencies at the federal, state, and local levels use surveys to learn about their constituencies' preferences and issues. Charitable organizations learn about gift-giving trends. Non-profits learn about new services to offer. Community public service groups such as police, fire, and EMS learn about their performance and community relations. And the list goes on.
It is hard to tell how many and what types of surveys are done now as opposed to, say, a decade ago, but it should be safe to say that more are done now because of technology – more people have access to the internet, for example, and this helps a lot. The one factor that has changed, and it is a major issue, is the response rate for surveys – it is declining. Fewer people want to take the time to complete a survey for a host of reasons, which certainly impacts the information that surveys can provide.
Can you give us a sense of what you will be talking about in your seminar? Or perhaps share an example from the presentation?
The focus of my presentation is the extraction of what I call Rich Information from survey data, although the concept I discuss can be applied to Big Data as well. I view information as a continuum running from Poor to Rich Information. Data per se are at the Poor Information end of the spectrum, sometimes useful, but not always. The data are really raw, disorganized, chaotic building blocks that can be molded and shaped in whatever manner we wish to yield Rich Information, information that is useful, insightful, and actionable for decisions whether at the business, government, charitable, or public service levels.
This is almost a philosophy of doing data analysis: to reorganize the raw building blocks of data into Rich Information for decisions. The modeling and shaping are done with dynamic tools and advanced statistical methods, not by simple charts or tables or means or proportions that yield Poor Information. I illustrate this philosophy with a case study that highlights the use of the dynamic and statistical tools available in JMP.
Can you identify one common mistake people make who are doing survey analysis?
There are actually two common mistakes regardless of the source of the data, surveys or Big Data.
The first mistake is not recognizing that information is a continuum from Poor to Rich.
The second is viewing data per se as information rather than as building blocks to get Rich Information. Because people confuse raw data with information, they do not go beyond the obvious when they write their reports or make their recommendations to decision makers; they merely state the obvious with simple means and proportions with the view that this is information. Building Rich Information requires a different mindset that data are just items to be manipulated, not always to be taken at face value.
What would you describe as your biggest success in your life?
This is an interesting question because there are personal, social, and career levels to every life. On a personal level, I definitely have to say my two daughters and my wife. One daughter is an accomplished architect who designs beautiful homes, and the other is a PhD who works at a major hospital in the embryology area creating new lives. My wife is an EdD, former school superintendent, and now professor at Rutgers University training future school administrators, so she makes a big impact in the education area. On a social level, I have to say the many wonderful friends my wife and I have made and maintained over the years. And on a career level, I have to mention my time at Rutgers University training the next generation of leaders, decision makers, and, hopefully, some scholars.