The Survey Issue

Do you make most business decisions based on...

a. gut feeling or experience

b. data and information

c. other (reply in comments)

According to survey data from BI Survey, “58% of respondents say their companies base at least half of their regular business decisions on gut feel or experience [emphasis added] rather than being driven by data and information.” Hmm.

But is a survey, like the one above, even the right way to assess the state of company decision-making? Let’s roll up our sleeves here…

‍

SURVEY SAYS...

Surveys are a great way to understand the features of a group by assessing a sample of a population. If you were to ask the respondents in the survey above about the size of the company they work for, you'd likely get a reasonable depiction of where the survey sample is employed.

But a survey is too blunt an instrument to capture how “regular business decisions” are actually made within a company. Instead, it’s a decent way to get a sense of respondents’ perceptions or opinions of how decisions are made.

The survey is still valuable, of course—company leaders may indeed want to address exactly that: perceptions about how decisions are made. And the results may be especially valuable for the sponsor of the survey (a firm that supports the selection and implementation of business intelligence systems) as a marketing tool that raises valid questions about the use of data.

‍

NOT WHAT YOU THINK

Surveys are flawed.

Get this: the very act of surveying someone may create an intention to do something where one didn’t exist before. So if you ask a survey respondent whether they will buy your new product when it launches, they may say "yes" even though they had no intention of doing so a few minutes before. That means that your survey sample is no longer representative of the target market and you may make a decision based on flawed data.

Surveys, it turns out, are rife with bias. One example: surveyors may write questions that include subtle bias, like the order in which choices are presented. Another example: respondents may answer “yes” to ten questions in a row because they become bored after Question 4. And another: respondents may answer in a way that they believe is socially desirable, even if it does not reflect their attitudes or intentions. There are some good lists of bias types here and here, if you’re interested.

By the way, consumer panels are not a solution to survey quality problems. Panelists don’t always respond honestly; in fact, it’s not clear that they are even reading the questions.

‍

ARE WE DONE HERE?

So, if you are in the 42% of people whose companies base more than half of their decisions on data (wink, wink), should you give up on surveys forever?

No. Surveys have their place. They just need to be treated as the meta lens that they are, not as oracles.

At Spark, we believe in research that captures behavior, a far better predictor than asking people to speculate on future actions. Asking people about their behavior, whether in the past (Which five websites do you visit most often?) or the future (How likely are you to buy this product when it launches in October?) is rarely predictive, and there is data to prove it.

Surveys can be used in unconventional ways to predict behavior, however. Facebook, for instance, uses the act of filling out an employee survey as a source of predictive data: “People who don’t fill out either of our two annual surveys are 2.6 times more likely to leave in the next six months.” Now that’s a useful survey.

So if you use a survey, think of it as a way to...

• Capture basic data (Do you have a law degree?)
• Learn about attitudes (Rate the appeal of cheese on a scale from 1-10)
• Catalog respondents’ experiences, knowing they are subject to the constraints of memory (Have you ever used a wheelbarrow?)
• Get a sense of perceptions (In your company, what percent of regular business decisions are based on data?)

Or, even better, use surveys as behavioral data. The act of completing a survey about your brand or new product launch can be an indicator of interest and engagement. Repeating a survey periodically may be a way to track attitudinal changes that are revealed not by answers to questions but by the act of responding (or not responding) to certain questions.

‍

‍

OH, BEHAVE!

Want to debate the value of surveys vs a more behavioral approach to research? Game on!

Email us at hello@sparkno9.com

‍

‍