Monday, June 13, 2016

When small data is more important than big data

Have you ever completed a survey?

While surveys are sometimes conducted for other purposes, the usual reason for conducting a survey is to obtain information from a subset of the population, and use this aggregated information in decision making.

For example, if the Donald Trump campaign took a survey and discovered that the majority of people think he shoots from the lip way too often, the campaign would then conclude that Trump should be quiet. And Trump would immediately overrule his campaign staff. But I digress.

To get back on topic, I'm going to look at something that I wrote back in February. Here's an excerpt:

I cannot share the details of the two instances, although there is one that I'd REALLY like to share if I could. But both boil down to the same thing. In each case, Person A sent an email to Person B at a particular company. Not receiving a response, or an out of office message, Person A sent a follow-up message to Person B. After increasing frustration, Person A finally asked other people, "Why isn't Person B responding to my emails?" In both cases, it turned out that Person B had left the company, and the person's email account was not disabled.

Now I'm going to reveal a few things - but not everything - about one of these instances.

Person A was (and probably still is) an accounts receivable person at a company that I'll call Company X. Person B is the person who signed the contract with Company X for its annual service - a service that has an auto-renewal clause if you don't cancel at a particular time.

Filling in the blanks from my February post, Person A sent the annual renewal bill for the next year to Person B, who didn't answer his email because he had already left the company. Finally, after several months, Person A sent an obnoxiously-worded email to everyone that he knew at Person B's company, saying that the bill was overdue and was about to go to collection.

At this point the people at Person B's company decided, "Well, the new term hasn't started yet, so we just won't renew."

That's when they found out about the auto-renewal clause, and that the date to cancel had already passed.

So Person B's replacement paid the bill for the forthcoming year, then immediately sent a notice cancelling the service. Both sides were unhappy with the whole episode.

Several months later, employees at Person B's former company received a survey from Company X, asking for opinions of Company X's service. Person B's former coworkers just rolled their eyes - after all we went through, Company X is asking how we feel?

The results of the survey, of course, would be aggregated with everyone else's from the survey, and conclusions would be drawn from the aggregated data.

This is one of the data points that formed part of the aggregated data.


Now this data point in and of itself doesn't provide a lot of context - the person who completed this survey was so disgusted with Company X that most of the open-ended responses were left blank. (Why bother?)

But when all of the data is aggregated together, it will provide even less context. The aggregated data will simply report that in response to question 10, 1.62% or 2.3% or whatever of the respondents indicated that they would never recommend Company X. And then Company X will have to decide how to improve its performance in that area.

And I'd be willing to bet that Company X's solution won't please Person B's former coworkers. Company X doesn't have all the necessary data, despite its survey.

P.S. For more on surveys, see this Marketoonist cartoon.

blog comments powered by Disqus