Attention research firms - complex surveys are ineffective

This morning, one of my co-workers was grousing about having to read long emails. His standard for a long email? One that he has to scroll through to read all of it. I laughed as he said it (particularly since I've been known to write "War and Peace"-style emails).

I stopped laughing when I opted into a survey from a research firm. The firm promised a five-minute survey, so I figured, why not? The survey itself was easy enough, since it was multiple choice. But one of the questions was interesting. The question listed some different items, and you were supposed to choose the top three items in the list. (Click once and only once in the first item column, once and only once in the second item column, etc.)

The problem with the question? There were probably a couple of dozen items in the list. And they weren't single word items; each item was probably about three or four lines long.

The result? I had to - you've guessed this already - scroll down and up the screen to read all of the couple of dozen items before I could even make a single selection. And then you're supposed to agonize over them - so, is item 18 more important than item 16? And what about item 22?

I say that you're SUPPOSED to agonize over all of the couple of dozen items. By some strange coincidence, all three of my top three items happened to appear on the first screen of the survey page. Imagine that! (It could have been worse; I could have just chosen the top three items and been done with it.)

One would hope that the research firm guarded against this tendency by rotating the list of items for different survey respondents. But even if they did, what's the net result? The research firm is going to end up with a survey that doesn't reflect the respondents' true preferences.

So if you are running a survey for a research firm - or even if you're surveying Best. LOLcats. Ever!, please keep your surveys simple. It will improve not only the accuracy of your results, but also your response rate.

(Now - is this post too long?)