Biometrics and gaming

[Donjonson]

I've been thinking about this a lot in the past few days, I know Gabe Newell has said that he reckons biometrics will have an impact on gaming in the future, and there was some stuff about aspects of it in last months Edge, does anyone have any thoughts or have any experience with this?
Being scared shitless by a skeleton attack TWWANG in Minecraft, as I was scrambling up a cliff to my hilltop home, made me think about it seriously.. :)
So obviously pulse rate could be used and might be the most practical way of measuring any kind of biological changes, resulting in modifications to music and graphics.. transitions in music and graphical effects are pretty normal as a result of in game actions though, I'm sure this is just the tip of the iceberg...
I presume there are already ways to differentiate between psychological states that share similar biological attributes.. like, I was thinking, when someone's scared their pulse rate rises and they start to sweat, similar things happen when a person is sexually excited or embarrassed though... I suppose this would be one of the main stumbling blocks? These things are presumably fairly unique to each person aswell...
The more I think about it the farther away the technology appears..

Any interesting articles or whathaveyou I should read/watch/listen to?

Edit: And there are mainstream motion sensors like the Wii, Move and Kinect, not sure how interested I am in this kind of biometrics.. I suppose it's the beginning though, it will just become more subtle with future devices

[EndelNurk]

I presume there are already ways to differentiate between psychological states that share similar biological attributes..

It would certainly be possible to use pulse rate or galvanic skin response as measures of generalised arousal. More specific and nuanced emotions would be harder to ascertain. As you say, many emotions bring about similar physiological responses. A strong theory of emotion argues that arousal emotions (anxiety, stress, but also excitement and others) are only the cognitive "naming" of the same physiological responses based on the environment. Therefore, the physiological response to meeting a beautiful person is similar to that of seeing a Tyrannosaurus bite of the roof of your toilet and then the brain ascribes a pleasurable or unpleasurable emotion to that state. The more nervewracking one being the former one, if you're anything like me.

Wasn't there a Wii sensor that was supposed to check pulse rate? I assume it was related to Wii Fit but I cannot remember off hand.

Valve have at least considered a huge number of techniques for measuring the emotional responses of players. There's a .pdf of a powerpoint slide given by the Valve psychologist here. As the slides suggest, many of the more complicated biometric techniques would be too tricky or too complicated to use in any meaningful way at the moment, even under controlled playtesting conditions. I think we might have to wait a little longer before they're part of the common user experience.

[Donjonson]

I'll try to remember the next time someone threatens me with a knife to think "it's ok brain, just name it pleasureable!" :) I get what your saying, yeah, the different states do seem very similar physically, maybe not emotionally though :)

Cheers for the PDF link :)
Yeah, the Wii vitality sensor-



I've never witnessed one in action, looks pretty ungainly though..

Maybe it'll make sense to use pulse monitoring or whatever in more open sandbox games, as things are usually less scripted.. although I'm thinking in terms of a fairly narrow range of game types..

[EndelNurk]

And here's a link saying they've had trouble getting it to work properly. Unfortunately, I think that's to be expected. It's the same sort of sensor that they use in hospitals to check your blood is oxygenated enough. Do we want games consoles to be able to tell us that we're about to die?

I wonder if the best way to do the things you're referring to, that is modifying audio and visual stimuli in response to the player, might be best achieved by observing the player avatar's movement rather than trying to observe the player themselves. If you're excited then you might be more twitchy and the way you control the avatar might change. Likewise, if you're scared then you might be running backwards or stop utterly motionless more than usual (depending on the game) - again, that would be very tricky to design and it's of course speculation on my part to suggest that players would even act noticeably differently.

[Donjonson]

And here's a link saying they've had trouble getting it to work properly. Unfortunately, I think that's to be expected. It's the same sort of sensor that they use in hospitals to check your blood is oxygenated enough. Do we want games consoles to be able to tell us that we're about to die?

I wonder if the best way to do the things you're referring to, that is modifying audio and visual stimuli in response to the player, might be best achieved by observing the player avatar's movement rather than trying to observe the player themselves. If you're excited then you might be more twitchy and the way you control the avatar might change. Likewise, if you're scared then you might be running backwards or stop utterly motionless more than usual (depending on the game) - again, that would be very tricky to design and it's of course speculation on my part to suggest that players would even act noticeably differently.

Yeah, I was thinking the same thing regarding player movement and reactions- I was definitely jumping a lot and moving the mouse around pretty frantically during the Minecraft attack.. I suppose this kind of thing already happens at a fairly basic level in most games- "player enters room/reaches certain point- cue music change"
It would be easy to invoke any reactions that are set to occur at any time though, with a system like this... I can imagine it being a bit all over the place..
I'll be interested to see how stuff like this might be implemented but at the end of the day Amnesia had me bricking it without monitoring my sphincter.

[Malawi Frontier Guard]

Yeah, the Wii vitality sensor-

I've never witnessed one in action, looks pretty ungainly though..

I always thought this solution was too awkward. However, I could imagine a pulse oxymeter in the back of a gamepad, so you could comfortably slide one of your fingers inside for measurement.

Alternatively, put that thing on a toe.

[Wizardry]

Interesting, but I can't see any of this being appropriate for CRPGs.

[Donjonson]

I always thought this solution was too awkward. However, I could imagine a pulse oxymeter in the back of a gamepad, so you could comfortably slide one of your fingers inside for measurement.

Alternatively, put that thing on a toe.

I don't know if I'd want to bother putting something on my toe every time I play Portal 23... if it came in a slipper though! I would buy two pairs.

I think pulse rate would be the easiest and most practical way... a mouse or gamepad with multiple detection points so that we can induce this kind of extreme reaction in grown adults... poor little bugger

[EndelNurk]

I think you're right, Wizardry. Probably tricky for most other genres as well. Donjonson makes a good point that games seem to be able to manage to draw us in pretty well without this sort of thing. I would love to know more about how it's being used in playtesting though, but then that's a personal interest of mine.

[Wizardry]

I think you're right, Wizardry. Probably tricky for most other genres as well. Donjonson makes a good point that games seem to be able to manage to draw us in pretty well without this sort of thing. I would love to know more about how it's being used in playtesting though, but then that's a personal interest of mine.

Well, I can see it working for first person action games, whether horror themed or not.

[icupnimpn2]

There are companies out there working on techniques for remote blood pressure monitoring that wouldn't require any sort of device to be worn on the body.

Here's one using a laser.

This is the direction the tech would have to go to be viable for gaming, I think. You need a sensor on the TV or game system -- not a clip on your finger that restricts movement.

[Donjonson]

There are companies out there working on techniques for remote blood pressure monitoring that wouldn't require any sort of device to be worn on the body.

Here's one using a laser.

This is the direction the tech would have to go to be viable for gaming, I think. You need a sensor on the TV or game system -- not a clip on your finger that restricts movement.

Ah, that's pretty interesting.. maybe things like Wii and Kinect will incorporate stuff like that at some point..

[Xercies]

I don't see how many advantages this would have, and it would be a little bit worrying if my game was taking hold of my pulse and stuff like that

[Peter Radiator Full Pig]

I would of loved to have this when I was 18. Seen what my tachycardia would of done to games. Constant battle music, no matter where you are, etc. Or just random spikes at random times. Could of been a laugh.
As for now, I dont think Id bother. Pretty sure you could do clever stuff, and obvous things like players health going down x amount in time y. Or the mouse going frantic, with an enemy close by as you guys said above.

[hamster]

Is this really necessary for anything that isn't an open ended game? Linear games allow the devs to dictate the play experience anyway. Even in open ended games, it shouldn't be too hard: simply log the number, frequency and interval of "fright" moments, such as baddies, or foreboding corridors, or whatever. These are the mechanics that the game is going to utilize to arouse the player anyway, so why not focus on them rather than player arousal?

[soldant]

It's the same sort of sensor that they use in hospitals to check your blood is oxygenated enough.

And they're fiddly as all hell when they want to be. Pulse oximetry probes are a pain, there have been plenty of times where I've sat there waiting for it to give me a proper reading or where it's done absolutely nothing for absolutely no reason. Especially Welch Allyn stuff. Sorry, went on a rant. On topic: using that stuff with gaming at an affordable price where the reading affects the game doesn't seem like a plan for success.

Also I think another thing to think about is whether or not people really want to hook themselves up to all this stuff just to play a game. I like my TrackIR kit for example, but having that plus a HOTAS joystick setup all plugged in and set up becomes a pain in the backside unless you've got a dedicated table and setup for it. I suppose "wireless" devices would help with that, but I can't imagine they'd work so well when your subject is moving around. The only practical one I can think of is an exercise heartrate monitor.

Finally, like others suggested, do we need to do any of this? I really don't think so, unless it's a "fitness game". Good games build tension without vital sign monitoring, they're the very cause of increased HR/BP/RR, it seems to me that hooking in monitoring just adds another unnecessary layer which is ultimately meaningless. Then again the way the games industry is going they almost alawys just use cheap sting moments anyway, so who cares?

[thegooseking]

We're looking into this as a research area. We do a lot of stuff with EEG, but I think that on its own is only about 70% reliable so we're trying to correlate it with GSR, EKG/ECG and stuff like that to get a picture. I saw a guy at last year's AISB conference giving a talk about real-time emotion recognition in games using all this stuff. I think it's important because AI is dumb. Even if the AI can 'see' you using something like Kinect, it can't pick up on all the subtle cues that will tell it how you're feeling, so we give it hints by direct measurement. We have plenty of interactive storytelling applications where characters will alter their behaviour based on how another character feels, or how the character thinks it will make the other character feel (basically theory of mind and empathy and stuff like that), but because we can't really tell how the player feels, the player is left out of that loop a bit. Which, when the player is supposed to be the most important character there, is kind of lame. Part of addressing that is using biometrics, and the other, I guess, is behaviour monitoring.

I'm not, myself, using this in my work, though it would be really useful. Right now I'm just assigning the player a role (i.e. they play one of the characters in the scenario) and assuming that they want what that role wants and feel how that character would feel in that situation. Not quite as exciting, but that's the first step.

[measurements]

Are you fellas aware of them mind reading gizmos where you just think of stuff? I see it being quite likely that we will interface with computers in the future with a mixture of motion detection, thought detection, eye movement detection and biometric systems. I think it would be great to use one of them breathing rate monitor thingies with a sniping game. The immersion from having gun steadiness relate to your breathing would be fantastic.

Sources of my daydreams:
http://www.ted.com/talks/tan_le_a_he...rainwaves.html
http://www.ted.com/talks/john_underk...a_gesture.html
Blocked video of sony breathing game from e3 2010...
Kinect, etc.

[thegooseking]

Are you fellas aware of them mind reading gizmos where you just think of stuff? I see it being quite likely that we will interface with computers in the future with a mixture of motion detection, thought detection, eye movement detection and biometric systems. I think it would be great to use one of them breathing rate monitor thingies with a sniping game. The immersion from having gun steadiness relate to your breathing would be fantastic.

We use the Emotiv headset in our work. It's supposed to be for mind-control of games, but we use it for detection rather than control (we have a proper EEG set, too, but we like working with the Emotiv because tech that we develop for that will be cheap enough for market applications: an Emotiv headset costs about £200 whereas a proper EEG set costs more like £2000).

[Donjonson]

We're looking into this as a research area. We do a lot of stuff with EEG, but I think that on its own is only about 70% reliable so we're trying to correlate it with GSR, EKG/ECG and stuff like that to get a picture. I saw a guy at last year's AISB conference giving a talk about real-time emotion recognition in games using all this stuff. I think it's important because AI is dumb. Even if the AI can 'see' you using something like Kinect, it can't pick up on all the subtle cues that will tell it how you're feeling, so we give it hints by direct measurement. We have plenty of interactive storytelling applications where characters will alter their behaviour based on how another character feels, or how the character thinks it will make the other character feel (basically theory of mind and empathy and stuff like that), but because we can't really tell how the player feels, the player is left out of that loop a bit. Which, when the player is supposed to be the most important character there, is kind of lame. Part of addressing that is using biometrics, and the other, I guess, is behaviour monitoring.

I'm not, myself, using this in my work, though it would be really useful. Right now I'm just assigning the player a role (i.e. they play one of the characters in the scenario) and assuming that they want what that role wants and feel how that character would feel in that situation. Not quite as exciting, but that's the first step.

Interesting stuff... food for thought alright, although I imagine that I don't express a whole lot of emotions when I'm playing, I probably look a bit zombified... :)