Posts Tagged ‘Game Reviews’
Jam and Game Reviews
For every one of us, making decisions is part of our daily human existence. Most of them are of little consequence –what to eat, what movie to see, what video game to buy– so we have developed an astonishing array of mental short-cuts to make these kinds of decisions comparatively quick, easy, and not too mentally taxing. We may eat what we have eaten and enjoyed in the past, and by and large we use simple decision rules such as “I like this genre” or “I like this developer” to choose movies or games.
Other decisions, though, are either much more important or much more public and thus we put more work into it. Whom should we date? What college should we attend? Which house should I buy? When faced with questions like these, many of us have probably drawn two columns on a piece of paper, labeling one “Pro” and one “Con” and then listing things in each column. When trying to decide whether to marry or stay a bachelor, famous biologist and five-time Counter-Strike world champion1 Charles Darwin did exactly that, producing the list below.

The list admittedly looks a bit sexist by today’s standards, but it illustrates the idea well.2 But is this sort of thing always a good idea? When video game reviewers ruminate over the merits of a particular title, they are often asked to consider standardized lists of features –graphics, sound, fun factor, multiplayer, value, extendibility, controls, and so on. Should they always try to analyze decisions across every possible variable? Is that the right way to review a game?
Researchers Timothy Wilson and Jonathan Schooler probably wouldn’t think so, or at least they could imagine situations in which this type of approach could lead to poor evaluations. And here’s the best part: jam was involved. Delicious, strawberry jam.
In their study3 the duo were intrigued by a Consumer Reports ranking of 45 different brands of strawberry jam.4 Panelists in the study were trained sensory experts (i.e., professional food tasters) who sampled each of the condiments and rated them on 16 characteristics including sweetness, aroma, bitterness, spreadability, and others.

Man, those testers at Consumer Reports are creepy.
This is the kind of thing Consumer Reports does all the time, but Wilson and Schooler were curious about something. Some of their previous research had suggested than when asked to analyze their reasons for making decisions or ratings, people tend to screw things up. The theory goes that we are often aware of our preferences for products (or art, or whatever), but when asked to explain WHY, we often feel obligated to include the most salient (that is, apparent) and plausible explanations. Even if we would have otherwise ignored them.
So if asked to explain why you like Red Dead Redemption so much, you may think about what should be included in the checklist for evaluating an action game, come up with “the weapons,” and then feel compelled to award or take away points for how the game’s weapons feel and work. The problem is, the most salient and plausible factors may not be the ones that are really responsible for how much you enjoy the game. The weapons in Red Dead Redemption are largely unremarkable –the game’s appeal lies almost entirely in other areas and any weight given to how cool the weapons are is inappropriate at best.
So, thinking along these lines, Wilson and Schooler wondered what would happen if they asked normal people to recreate Consumer Report’s jam ratings. And what would happen if you asked them to ponder the reasons for their ratings the same way the Consumer Reports experts did?
So they fed some college students the 1st, 11tth, 24th, 32nd, and 44th best jams from the report to find out. Those in the control group were just asked to taste the treats and make their rakings. Those students actually did pretty well –their rankings were very close to the professional taste testers’ rankings. But the group that was asked to write down the reasons for their ratings did far worse. They may have favored the jams that the experts thought were gross and scrunched their noses up at the ones the experts thought were great.
Why? Because the subjects started focusing on factors that didn’t really matter. Smucker’s had more chunks of fruit in it, so it gets a higher rating. Wait, what? Is chunkiness really important for them? Doesn’t matter; it sounds plausible so it got factored in. When the XBLA shooter Monday Night Combat came out, some people lamented the small number of maps. Same thing –one could argue that because the gameplay requires a very specific setup, you don’t need –or even want– a lot of maps.

Over thinking it?
Puny humans are pretty bad at combining an array of weighted factors so as to arrive at a rating or decision –it’s just not how our minds were designed. Jelly or game review guidelines that require us to over analyze our decisions or check them off against a standardized list of factors (graphics, sound, etc.) can exacerbate this limitation and lead us to consider what should be irrelevant information when making our ratings. This corrupts the rating process and takes us farther from our “true” feelings or evaluations.5 This is one reason why I prefer more organic, experience-based evaluations of games from message boards or podcasts rather than formal game reviews. I feel like I can listen to someone talk in an unstructured way about how much the enjoy a game and get a much better idea of how much I might like it. Just consider what’s important and ignore the rest.
Now, go get yourself some jam –whatever kind you think tastes good.
- If my notes are right [↩]
- If you’re curious, Darwin decided to marry and, in very short order, found himself a bride in Emma Wedgewood, who by all accounts turned out to be a wonderful wife. She was also his cousin, which apparently didn’t made it into neither the “pro” nor “con” column. [↩]
- Wilson, T. & Schooler, J. (1991). Thinking Too Much: Introspection Can Reduce the Qualities of Preference and Decisions. Journal of Personality and Social Psychology, 60(2), 181-192. [↩]
- Strawberry Jams and Preserves (1985, August). Consumer Reports, 487-489. [↩]
- This isn’t to say that meticulous decisions don’t have their place. I’d want a highly standardized process for deciding if a bridge is safe, for example, and as an Industrial-Organizational psychologist I make my livelihood off creating decision-making tools to make scrupulous, detailed accounts ratings for job candidates. But bridges and job requirements can be held constant and a standardized list of evaluations can be applied to them; games and jelly vary much more widely. [↩]
Hedonic Adaptation and Game Reviews
[Note: This article first appeared in my column on Gamasutra.com.]
My wife and I have a Father’s Day tradition where I get to celebrate the joy of parenthood by kicking everyone out of the house and playing video games for 12 hours straight.1 This year I decided to take a chunk out of my backlog by unwrapping Bioshock 2 and popping it in. Normally it would take me weeks of playing a game like this in one or two hour chunks when I could find the time, and I’d often look forward to these bite-sized gaming sessions. But this time I wanted to use my annual alone time to burn straight through as much as I could without stopping.

I loved the game at first and I envisioned myself playing until biology forced me to stop. A few hours later I was slinging plasmids and stomping splicers, but I was enjoying the game less and less. This made me think of something called “hedonic adaptation” that Dan Ariely had written about in his new book, The Upside of Irrationality. Had my playing Bioshock 2 for hours and hours straight diminished my enjoyment of the game?
Probably so, according to the research Ariely reports on. The theory is that people become less sensitive to pains or pleasures over time2. Ever notice that a bad smell fades the longer you’re exposed to it? Or how people who move from the Midwest to Southern California3 appreciate the weather a lot when they first get there but take it for granted a couple of years later? Same concept. It’s a good thing for unpleasant stimuli, but somewhat unfortunate when it diminishes our enjoyment of nice things like playing a good game.
The thing is, though, that researchers have found that this kind of adaptation can be short circuited simply by taking a little break. In one 2008 study by Leif Nelson and Tom Meyvis4 the subjects were all lucky enough to get a nice massage as part of the experiment. One group got a three-minute massage. A second group got a 80 second massage, took a twenty second break, then got another 80 seconds for a total of two minutes 40 seconds –less time with the magic fingers than the first group. So, given hedonic adaptation, which group do you think reported higher satisfaction with the massage and said they would be willing to pay more for it? Yep, the one that took a 20-second break. Because that short break stymied their adaptation to the pleasurable event and kind of “reset” their appreciation for it. Nelson and Meyvis also did the same trick with an unpleasant experience –listening to an annoying vacuum cleaner sound at high volume.5 Same thing: those who took a break listening to the sound found it sucked more.6
So think about the implications this has for video game reviewers. When a popular AAA game comes out, it’s not uncommon at all for reviewers to sit down and burn through it as fast as they can in order to satisfy readers’ expectations of a timely review. Sometimes game publishers even invite reviewers on-site to sequester themselves with the game just ahead of release and do literally nothing but sit in a room and play the game until they’re either done or start recreating the last third of “The Shining.”
Given hedonic adaptation, does a reviewer risk becoming numb to a game’s good points if he/she plays it for long stretches without a break? Do they amplify their perceptions of a game’s negative traits in the same way? Given that most of us, rare exceptions like my Bioshock 2 marathon aside, play games in smaller chunks and take long breaks between sessions, does this mean that we’re experiencing the game is a significantly different way than reviewers?
The science seems to suggest so. But I also wanted the opinion of a veteran game reviewer on whether or not it was a legitimate concern or so much poppycock so I asked Tom Chick, who has written bajillions of game reviews for a wide variety of outlets, including 1Up, GameSpy, Yahoo! Games, Gamespot, and others. He also currently serves as editor-in-chief for Fidgit.com and runs the Quarter To Three website. “I do think it’s a legitimate concern in the reviewing process.” Chick said. “I’ve only once done one of those review events where you go somewhere to play through a game before it’s released. It was for Half-Life 2. I was in a room at Valve over the course of two days and one night. I felt it negatively impacted my opinion of the game. I had to be careful to be analytical about the game rather than responding to the fact that I couldn’t take breaks to do other things.”
When asked about reviewers could counteract such demands on those occasions when Gabe Newell isn’t standing over them and poking them with a stick so that they keep playing, Chick said “When I’m charging through something for a review on a deadline, I’ll routine take breaks to either play another game, watch a movie, do some writing, or somehow break up the experience. Whereas I had previously put this down to having a short attention span, I’m now going to say that it’s to avoid acclimation.”
Indeed. Apparently those who say video games and YouTube are stunting our attention spans just aren’t seeing the upside.
- And ladies, before you cry foul, she gets to do something similar on Mother’s Day involving scrapbooking and a nice bottle of Chardonnay [↩]
- Don’t confuse this with diminishing sensitivity to gains/losses a la prospect theory; hedonic adaptation refers to changes in sensitivity to the same thing over time [↩]
- Sorry for the USA-centric reference, Rest of the World; just think bad weather to nice weather [↩]
- Nelson, L. and Meyvis, T. (2008). Interrupted Consumption: Adaptation and the Disruption of Hedonic Experience,” Journal of Marketing Research, 45 654-664. [↩]
- To recreate this effect, turn on your vacuum cleaner, lie down on the floor next to it, and stick your face right up in there. Go on, I’ll wait. Done? Okay. [↩]
- Pun intended. [↩]
