Be careful what you ask for - examine the goals you set

The AppsLab talks a lot about games - Jake Kuramoto returned to the topic last Friday - but when you think about it, games and quests and the like are really just engaging versions of goals. For example, take your annual performance review, with its goals for the coming year, and create a nice interface with percent complete and badges and stuff - you now have a quest.

But whether a program encourages you to reach the goals by giving you badges and virtual crops, or whether the program encourages you to reach the goals via less boring means, the "goal" of goals is to get something done. Incentives of some type are often used to complete the goal.

But before you set a goal, be sure that you examine what that goal is. You may get what you asked for.

Consider this recent post by Bruce Schneier, which links to this USA Today article. Employees at Crosby S. Noyes Education Campus received financial incentives in two separate years; teachers received $8,000 bonuses, and the principal received a $10,000 bonus. The school also received recognition from the U.S. Department of Education, and from the chancellor of the District of Columbia schools.

To receive these bonuses, they had to reach a goal; improve student test scores on standardized tests. And according to USA Today, they achieved that goal in dramatic fashion:

In 2006, only 10% of Noyes' students scored "proficient" or "advanced" in math on the standardized tests required by the federal No Child Left Behind law. Two years later, 58% achieved that level. The school showed similar gains in reading.

In this case, the goal was measured by looking at the test scores, and the financial and other rewards started flowing in.

There was only one problem.

A USA TODAY investigation, based on documents and data secured under D.C.'s Freedom of Information Act, found that for the past three school years most of Noyes' classrooms had extraordinarily high numbers of erasures on standardized tests. The consistent pattern was that wrong answers were erased and changed to right ones.

The details were enough to alarm some statisticians:

On the 2009 reading test, for example, seventh-graders in one Noyes classroom averaged 12.7 wrong-to-right erasures per student on answer sheets; the average for seventh-graders in all D.C. schools on that test was less than 1. The odds are better for winning the Powerball grand prize than having that many erasures by chance, according to statisticians consulted by USA TODAY.

"This is an abnormal pattern," says Thomas Haladyna, a professor emeritus at Arizona State University who has studied testing for 20 years....

Haladyna notes...that when entire classrooms at schools with statistically rare erasures show fast-rising test scores, that suggests someone might have "tampered with the answer sheets," perhaps after the tests were collected from students.

There were concerns from parents and others about students who were miraculously doing well, but there wasn't a great incentive to really investigate what was going on. After all, if you've praised Noyes for being a wonderful school, you don't want to turn around the next day and admit that it was all a fake. That doesn't look good.

So what happened? There was an incentive for schools to deliver good test scores...and the good test scores magically appeared. As Schneier notes:

The point is that whatever security measures were in place to prevent teacher cheating before the financial incentives and threats of firing wasn't sufficient to prevent teacher cheating afterwards. Because Rhee significantly increased the costs of cooperation (by threatening to fire teachers of poorly performing students) and increased the benefits of defection ($8,000), she created a security risk. And she should have increased security measures to restore balance to those incentives.

This is not limited to education. Take your average Facebook game. To increase the number of uses, games offer incentives for people to recruit more players to the game from their Facebook friends. Similarly, there are incentives to make new Facebook friends solely for the purpose of advancing in the game. (I did this myself just last week.)

What is the result of this goal? Backlash:

Sure people created fan pages entitled I don’t care about your farm, or your fish, or your park, or your mafia! Other people just hide those messages from friends via their live feed or worst of all they delete the friend entirely because they just do not want to deal with all those accomplishment posts.

Now most of my Facebook friends won't even see my Facebook game accomplishments. That's because I use Facebook's friend list feature to ONLY post those messages to a list of people who play Facebook games. And if you're not on that friend list, you won't see my messages.

However, if you're a Facebook friend of mine, and if you're not seeing these messages and want to see them, please message me via Facebook and I'll add you to my special "Farm City Planet" friend list.

And I'll even give you 100 virtual bacon cheeseburgers.