Can The Xbox One’s Kinect Read Your Mind?

Well, no. Of course not. That’s a silly question. Why would you even ask it?

That said, the updated supercamera on the Kinect 2.0 is capable of some pretty amazing things. Microsoft demonstrated how it can tell where you’re looking, estimate your heart rate from the color of your skin, and even infer your mood from your facial expressions. Finally, it has a sophisticated voice recognition system and the ability to see in the dark, which will come in handy when it wants to sneak into your bedroom at night and listen to your breathe for hours on end.

And though it hasn’t been discussed, I wonder if the Kinect ‘s high definition camera could be programmed to measure one other important biometric: pupil dilation. This would be both awesome and worrisome, because while not exactly a mirror into our souls, the eyes can reveal a lot about what goes on in our minds.

How Kinect sees you: a pulsing sack of meat and emotions. (Image from Wired's Kinect demonstration.)

How Kinect sees you: a pulsing sack of meat and emotions. (Image from Wired’s Kinect demonstration.)

Daniel Kahneman, famed psychologist and voice of Domino’s pizza’s The Noid,1 wrote in his book Thinking Fast and Slow2 about how pupil dilation is a good proxy for mental effort. In a series of experiments he asked people to take a large numer, then increment each digit in the number by one to form a new number. So 348 would become 459. They would then do the same to a new number, using a metronome to do one new number/sum every two seconds.

Try it yourself and you’ll see that the task gets very difficult pretty quickly.3 And if you had someone eyeball your eyeballs he or she would clearly notice your pupils growing larger and larger as the mental machinery behind them started to work harder and harder –right up until the point where you gave up, when they would snap back to normal size.

In what sounds less like science and more like an exhibit at the museum of contemporary art, Kahneman and his colleagues would train a camera on subjects during these experiments and broadcast an enormous image of their eye onto a television in the hallway. The pupils were about a foot wide and thus dilation was easy to measure. The results were pretty consistent: the more mentally taxed we are, the bigger our pupils get.

If the Kinect (or any camera) could detect pupil size, it would open up a whole new level of scaling game difficulty. A puzzle game could be made more and more difficult until you’re taxed just the right amount to get you in the zone –something psychologists call “psychological flow.” And in fact, we may not actually need the camera to be able to detect pupil sizes. One study that looked at psychological flow in piano players found that heart rate variability, respiration, and the movement of certain facial muscles were highly indicative of the state.4

Imagine playing a rhythm game like Guitar Hero and having the game adjust the speed of the note highway until you’re pushed just to the brink of your abilities based on how hard you’re concentrating on the task.

"I see that you're like super pissed off right now, Dave. Would you like me to order a case of Doritos?"

“I see that you’re like super pissed off right now, Dave. Would you like me to order a case of Doritos?”

Or what about knowing when to offer you a helping hand? If the Kinect can tell the point at which you’ve given up on a puzzle or sequence because your pupils shrink back to normal, it might offer you a hint. Possibly in a condescending tone.

Another, more unsettling implication would be that if the Kinect could tell when you are stressed and mentally taxed, it could use that opportunity to sell you stuff. Willpower is like a muscle that can be exhausted by any mental activity, and when it’s depleted we’re more likely to do dumb stuff like make impulse purchases or, one might imagine, place an impromptu Skype call to an ex-boyfrined when we really should know better.

Either way, I look forward to seeing all the biometric applications of the new Kinect. Should be terrifying.

Supporting me on Patreon helps ensure that new articles and podcasts are available for everybody.

Footnotes:

1. Or so I assume; I’m not going to bother fact checking that one.
2. Kahneman, D. (2011). Thinking, Fast and Slow. New York: Farrar, Straus and Giroux.
3. If not, try adding 2, 3 or even 4 to each digit instead of 1, BRAINIAC.
4. Manzano, O. et al. (2010). The Psychophysiology of Flow During Piano Playing. Emotion, 10(3) 301-311.

2 thoughts on “Can The Xbox One’s Kinect Read Your Mind?

  1. Oh yeah, the advertising opportunities are the reasoning behind this.

    One thought that I’ve not seen anyone offer up on the always on Kinect part of the Xbox one is that it could be used to pause an advert when no one is looking at it. I hate adverts before videos on the computer and always look away or walk round the room until it’s over. However if you have a camera that can tell the system that no one is looking at the screen, the advert can be paused until attention is returned.

    I suspect advertising companies would kill for that.

  2. Pingback: Dev Links: London Fog « DIYGamer

Leave a comment