Surfacescapes: D&D On Microsoft Surface

Not strictly PC gaming, but it’s nevertheless fascinating. It shows Carnegie Mellon’s proof of concept application for running a D&D game on Microsoft Surface and, well, that zoomable fantasy world map is about the most alluring piece of nerd-kit I have ever seen. I mean, I love maps at the worst of times, but that is simple too much. What’s even more interesting about it is the way that in terms of interaction, it hybridises boardgame conceits – maps, dice, and physical surfaces – with videogame processes – having menus and automated computation. Pen and paper games become “screen ‘n’ finger” games?

Surfacescapes Demo Walkthrough from Visual Story TAs on Vimeo.


  1. Optimaximal says:

    Now if only Surface didn’t cost so much…

  2. Seraphim2150 says:

    Imagine Warhammer 40K with a system like that!

  3. DMJ says:

    No, no, no, no, no. The breakthrough that touchscreens desperately need is fingerprint-proof surfaces. Until that day they shall forever be promising but flawed.

    • Vinraith says:

      They could also stand to be about 100x more durable.

    • jalf says:

      Surface isn’t strictly speaking a touch-screen. It doesn’t register touch, they’ve got a bunch of cameras under it, registering shadows where your fingers are blocking out the light. And I’d imagine that solves a lot of the durability problems

    • Kakksakkamaddafakka says:

      This is a misunderstanding of what P&P is.

      I have no issues with using this technology for things like maps and whatnot, but recreating dices and total representations of environments is just plain wrong.

      The P&P games of late have been infested with an abundance of visual aids. The whole point of P&P is using your imagination. It’s like creating a book while you’re reading it.

      The Surface tech is cool though, no doubt about it, and it could certainly be awesome to use in P&P roleplaying, if used correctly.

  4. Ian says:

    I’d feel super fancy while still sucking at strategy games.

  5. Adamos says:

    now if they can make an affordable 22′-24′ monitor that will be on front of me on my desk and also support touch then i will be interested. Imagine playing sins of a solar empire and commanding a few hundred spaceships with the touchscreen

  6. Alexander Norris says:

    The virtual die-rolling is clunky. If they can find a way of implanting some sort of RFID chip or similar into their dice so that Surface can see which side is in contact with the table, they could just roll the dice on the table and have Surface work the maths out for them.

    Very neat proof of concept, though.

  7. TooNu says:

    This is like virtual reality, it was kinda exciting to hear about and see the first time but then just really sucked and went nowhere.
    Hopefully this leads on to something more improved down the road that doesn’t cost a house to buy.

  8. postx says:

    Fascinating! So many possibilities for future board gamers, especially for D&D and the stratey genre. It would also save a lot of time for setting up huge games.
    But for smaller games I can’t see much use.

    It’s still a wonder!
    Which needs some time to happen for the board gamers.

  9. Rosti says:

    Bah, this needs more work – seriously, that Dire Wolf totally provoked an attack of opportunity with it’s automated path!

    [/Geek pedantry]

    Also, wow! This tech has always looked ace, but that’s astounding.

  10. Spacewalk says:

    It’s both a technology thing and a fantasy thing. I am close to pizza-face heaven here.

  11. The Hammer says:


    That is one hell of a map…

    Total War. That’s all that needs saying…

  12. Tim says:

    I think I just blew a nerf fuse. Nothing seems nerdy any more. Not even my screensaver of Captain Janeway.

  13. MacD says:

    Just a couple of stray thoughts:

    -They should dump the grid and use circles for movement and area-of-effect.

    -the dice roll mechanic as-is is lame, out-of-bounds and has crap physics. But even if it where perfectly simulated, what they should have done was use REAL dice marked with UV reflecting ink, a (scavengenged from a wiimote) IR camera to track the number on the bottom of the dice and from there infer what the dice are showing up top (with 6-sided dice, a 1 on the bottom always means you rolled a six). Wouldn’t that be awesome? The physicality of real dice rolled with instantanious dice totalling software showing what you rolled….

    -why not use those miniatures to be your character controll object? A nice QCode on the bottom (or whatever marker) to identify it, and you’d just leave your miniature avatar on the ‘board’/surface. Missed oportunity, there.

    -the GM screen should not be running on the Surface. It should be run off a laptop/tablet/netbook connected to the Surface (TCP/IP or nullmodem or whatever) and hidden from the players. The players use the Surface, the GM might for certain actions, but should not break immersion for the players; let him do that hidden on his laptop.

    -love the radial menu’s…yet another reason to dump the grid and use concentric circles for AoE, LoS, etc.

    -whilst the clockwise/counterclockwise twist is nifty, they should have just nic ked the pinch gesture from Apple, or used a scrunch up/ pull out gesture for zooming in/out of the map

    -this setup is amazing for use as prop display (maps, notes taken from dead messengers, loot)

    -speaking of which, how about a looting mini-game where you see the dead enemy and you point-and-click to loot it? If you don’t click his necklace, you don’t get that magic ruby necklace! Click the sword to pick it up!

    -you could draw runes to perform magic :)

    I’ll leave it at that…but, man, the posibilities….and you could do most of this with a simple projector and camera to pick up where fingers are placed, right now! With the propper software of course (not trivial, but it wouldn’t neccessarily require Surface).

    • jalf says:

      -the dice roll mechanic as-is is lame, out-of-bounds and has crap physics. But even if it where perfectly simulated, what they should have done was use REAL dice marked with UV reflecting ink, a (scavengenged from a wiimote) IR camera to track the number on the bottom of the dice and from there infer what the dice are showing up top (with 6-sided dice, a 1 on the bottom always means you rolled a six). Wouldn’t that be awesome? The physicality of real dice rolled with instantanious dice totalling software showing what you rolled….

      Good grief, you’re overcomplicating matters. ;)

      The thing can already read barcodes. I don’t think it’s that far-fetched to get it to read the number facing down towards the surface with the existing cameras. No need for messing with IR or taking a Wii controller apart.

    • Gutter says:

      > (with 6-sided dice, a 1 on the bottom always means you rolled a six)

      face up = 7 – face down

      In fact, the same can be said about EVERY dice out there, bar the one that don’t show their value on the top face, like a D4.

      But generally, face up = (max dice value + 1) – face down.


    • MacD says:

      @Jalf: you don’t know much about image recognition technology, do you?

      @Gutter: yeah, I just couldn’t be bothered to write it all down :)

    • Carda says:

      “-They should dump the grid and use circles for movement and area-of-effect.”

      While that would be great if this were an original game, the concept as I see it is a way to play 4th edition D&D (you’ll notice they mention at-will powers at one point). That’s why they stuck with a grid, to match the gameplay mechanics of what was already there.

      So yeah, yours is a very good idea, it just doesn’t fit given the concept they were going for.

  14. Torgen says:

    Fascination at seeing this article quickly gave way to fail as I watched the video. :(

  15. We Fly Spitfires says:

    Another way for geeks to geek out whilst indulging in geeky gaming. Nice ;)

  16. Ging says:

    I’m pretty sure Surface already includes IR cameras, standard cameras don’t read so well through the projected screen. I seem to remember talk of them including rfid readers too.

    Though I’m not too sure it really needs the physicality of a dice to roll, it would be a nice touch but the odds are you’d spend quite a lot of time picking up a dice after it had skittered off the relatively smooth surface of Surface (there’s always one person who can’t roll dice and keep them on the table, that’d be so much worse off glass).

  17. JoeDuck says:

    FYI, our D&D 4th ed. campaign has been going on for almost a year (we are in session 35 I think) We play using a 32″ LCD TV on a makeshift stand so it acts as a table for our tactical combats. The cool part is that we use a piece of open software called MapTool (link to to not only display the map and the grid, but also have fog of war and line of sight calculations done on the fly.
    We also use real figures laying on the screen to mark our character’s position. In our first version the movement of the figures had to be input by hand into the program. Now we have automated the process by using a webcam that hangs over the TV. It looks at the user moving the figure and when he’s done, it inputs the movement into the program and recalculates the fog of war.
    Needless to say it’s cool as hell and D&D combat becomes more tactical and interesting, but the big problem is that now our DM is spending a crazy amount of time designing beautiful maps with line of sight information, multiple levels and dynamic lightning.
    Microsoft Surface used in this context looks very nice and the use of fiducials and animation in the map add yet another layer of coolness.
    It’s a blast to play D6D like this, but the work to prepare it…

  18. coreyvw says:

    I’ve been playing since the blue book and have always embraced changes like this! We’ve been using MapTool with a ceiling projector for years now and it’s been a blast. Definite prep on my end (DM) but it pays off. I LOVE LOVE LOVE the radial selectors on the players here! Straight from the ToEE video game but using glorious 4e!! As far as mini’s and dice, just keep the software full of options or plug ins/add-ons so the many varied and highly opinionated gamers (oh boy just like politics) everybody has their options. Like using real mini’s like our group does since we have many great hobbiest painters AND we like to roll real dice, nothing like it! But the option to use virtual dice and virtual mini’s should stand for sure especially imho for monsters since I, as DM don’t have 600 different monster mini’s or the time to paint them. As an uber DM and Player, when this hits the market and I stll don’t have a job, I will be calling Barney Frank for one of those no credit, no job, no problem loans so I too can play on the 4e Surface!

    • JoeDuck says:

      And again FYI, if you are interested in multitouch gaming and technology, the place to go to is It has loads of information on DIY surfaces and technologies. And if you are looking for cool videos and photos of fancy multitouch projects, the showcase part of the forum has some exciting projects.

  19. Fumarole says:

    It’s a proof of concept for fuck’s sake.

  20. itchyeyes says:

    I've been saying since almost the day they announced this thing that gaming would be a total killer app for Surface. My biggest disappointment with this D&D concept is that it doesn't make as much use of some of surfaces unique features, namely that ability to recognize physical items placed on top of it. For instance, instead of rolling a virtual die, they could use physical dice and Surface could read whatever's on the bottom of the dice to deduce what came out on top. This connection between physical and virtual is what makes Surface so appealing for these kinds of applications.

    Of course, none of this solves the problem that Microsoft is still selling the thing for >$5000 and almost exclusively to commercial venues, but that's something that time and Moore's law will deal with eventually.

  21. Psychopomp says:

    Do want

  22. kalgor says:

    people just don’t get that you play RPGs with your mind, not miniatures, maps, or microsoft surface.

    • Edgar the Peaceful says:

      i haven’t played D&D for years – actually I miss it, sniff – and I agree that primarily it all happens in the mind, but it’s always used maps to give flavour, both as props and as practical boards with miniatures.

      This would be fantastic if it could recognise physical miniature placement on top of its maps.

      It would never replace physical dice for me as the virtual roll is not as satisfying as the physical. Also, a nice set of dice are like artefacts. For me they’re the spirirtual / religious symbols of the game, and were crazily exotic and thrilling when I first got the game in the early 80s

  23. Ho Ho The Eskimo says:

    this would work great for 4e. Too bad 4e sucks.
    a computer run system is far too restrictive for a game like D&D. Might be fantastic for wargames, but D&D requires flexibility that you can only get from the imagination.

    • Alexander Norris says:

      Too bad you’re wrong.

      (See? Everyone can make unsubstantiated comments and add strictly nothing to the conversation.)

    • Stromko says:

      Except that he actually has a point. The rules are only a jumping-off point, the point of tabletop roleplaying games is that anything could happen depending on choice and circumstance. In a wargame, you stick to the rules because that’s the only fair way, but a roleplaying game without player choice and GM arbitration falls rather short of its potential.

      Nothing saying that someone couldn’t pull some levers behind the GM screen to make that touch interface work with a more open-ended game, but altogether I’m not really convinced. SecondLife has had a perfectly workable interface for quasi-tabletop games (complete with nigh-infinite resources for constructing models, maps and other visual aids), and that still hasn’t taken off.

      Ultimately the point of the thing is hanging out with your friends and having something to talk about, the only point of a visual aid is to minimize arguments and keep things straight. People aren’t going to be talking about the graphics years on, they’re going to be talking about that time the druid hit the carnivorous ape with oil, followed by the rogue hitting it with a flaming arrow and igniting the oil, which would’ve been all well and good if the ape hadn’t been grappling the dwarven cleric at the time. And said dwarven cleric got color-sprayed by the sorceror and was dazed for 3 rounds, while the ape managed to resist the effect and keep smacking him about while they were both burning.

      I would say that 4e lacks the ability for player creativity and GM arbitration and that you may as well play it strictly through a computer, but then I actually got a good GM and saw that said assumption was wrong.

  24. PenAndPaperOnly says:


    Exactly. That combat encounter with the dire wolf, for example.

    Also, this makes me want to replay ToEE.

  25. K says:

    People use actual physical models for D&D?
    We play totally in our minds, man.

  26. Rei Onryou says:

    Fuck cloth maps. I want this bundled with my games.

  27. Fenchurch says:

    All it does is make me want to play Planescape again. x-D

  28. Moorkh says:


    This is what I’ve always dreamt of for my hot-seat Warlords or other TBS sessions!

  29. john t says:

    Neat idea, well implemented. I’d either drop the dice entirely or use actual physical dice, because the virtual dice are clunky and slow.

    And yes, Warhammer 40k like this, please.

  30. Rod says:

    Now if you could program smart cards to hold Magic the Gathering decks and had a MTG game installed, people could just drop the card on the table and play. Auto shuffle, auto-mulligan, no rules problems, easy card trading, possible direct connection to the WOTC store. The only problem would be stopping your opponent seeing your hand but i’m sure a wee cardboard screen would suffice :)

  31. imirk says:

    Plus Friggin a Bazillion, although I could really go for something like a 30×42 to play a massive RTS ala Sins or TA

  32. Kommissar Nicko says:

    Looks pretty awesome, but it really doesn’t suit laziness at all. The amount of work that would have to go into making a campaign up would definitely put me off, but then again, so does the potential price tag.

  33. Wiggett says:

    To everyone paying out on the clunky dice,: This is a WIP, a proof of concept. Issues like how clunky the dice roll is or how lame the wolf graphic is are negligible. The full version will have these things tweaked! Also you see him refer to the player tools controlling the stats etc, they don’t say these tools can’t be minis.

    @MacD; I’m pretty sure in the video they both mention and show the DM screen is in fact on a separate monitor from the desk, though they show the walk grid for the wolf on the table as a demonstration.

    I think the concept is amazing, so much room for great story telling and computer aided interactivity. Also preparation time would be much better than you think, I spent a whole week preparing a foamcard town for a single encounter on one of my games. I wish I was able to replicate such a thing on an interactive desktop! Also splice in some films of npc interaction.

    In fact there is a suggestion; Cut scenes and Interactive visuals for NPC’s (like pre-determined speech etc)

  34. JoeDuck says:

    When we play using our version of this system (see above), the DM has a laptop with a view of the map without fog of war and where he can redraw parts if needed. It works pretty well and there are surprises every minute.
    Of course, this adapts very well to a tactical relatively reality based RPG like 4th ed. D&D in the first 10 levels, but it does not help if your terrain is abstract, nondescript or you are flying between planes of existence between clouds of demon souls, for example.
    Now what would be seriously awesome would be to make a cooperative RTS game where a team plays on one M$ surface controlling one army while in 10 mts away the other team is playing in another surface. It would demand a lot of coordination and teamwork, but bragging, beer drinking and good times seem very likely with this kind of setup.

    • postx says:

      Would love to see you guys play, do you have any sessions on video?

    • JoeDuck says:

      @postx: We had some photos around somewhere, we were going to release the code for the webcam control system as open source, but we actually did never get to preparing the page, the campaign got too interesting. I’ll ask the guys and maybe dig up something for you. :-)

  35. John says:

    I think I just had a nerdgasm.

  36. Wurzel says:

    It’s a very neat application, and certainly cool for your NWNs or baldur’s gates, but as a DM/GM I’d never use it, simply because of all the stuff you’d need to prpare for sessions. Maybe it’s because I like the white wolf/storyteller kind of system more, where the pcs have an incredible freedom of expression and combat could never work with a grid or w/e. I’m not sure how well this system could react to the PCs deciding to screw the quest and go pillage the nearby village after becoming emperors of the nearby goblin city (true story).

  37. laugh says:

    geeks and their stupid crap lol