Man Makes Open Source Kinect Drivers

By Quintin Smith on November 11th, 2010 at 10:35 am.

Let's see if I can get through this whole post without mentioning Kinect's disappointing lag.

Last week, open source hardware company Adafruit announced a $3,000 bounty for the first individual or group to hack Kinect, Microsoft’s hands-free controller extra for the Xbox 360. Kinect contains a depth sensor, RGB camera, multi-array microphones and a motorised pivot, so it can track you around a room.

Yesterday, Slashdot reported that Adafruit reported that they HAVE A WINNER! Some very “hacky” open source Kinect drivers were released by a guy called Hector a meagre 3 hours after the European Kinect launch, and can be acquired here if you’re technically minded. Hector says he intends to invest the $3,000 bounty into further hacking. Isn’t that sweet? Watch Hector’s video of Kinect running on a PC below.

It might be useful in the field of robotics! He wants to use it to track his laser! Hackers are amazing.

, .

71 Comments »

  1. pakoito says:

    Not man but spaniard. Pride in my country!

    • pakoito says:

      Also, his nickname is Marcan.

    • Premium User Badge

      Harlander says:

      Isn’t he both?

    • TXinTXe says:

      No. We spaniards are more than merely men. By the way, why the word “spaniards” instead of “spanish”? I know it’s to differenciate us from the latinamerican people, but… why that word?

  2. Premium User Badge

    Harlander says:

    Man, this is awesome.

  3. Meat Circus says:

    Look at you, Hacker, panting and sweating as you prance through my living room.

  4. Lewie Procter says:

    In theory, this should work pretty well for head tracking.

  5. Tei says:

    The XBox is a very old piece of hardware. It only has like 512MB. Thats less than some smartphones by today standards. You probably have more ram in your pocket with a usb pendrive.

    Since It lack resources, and the device runs monstly in software and has to compete with the game rendering.. is gimped. The kinect device is gimped in the Xbox.

    Running the Kinect in LINUX, FreeBSD and other OS’s may able some nice speed, that will let us do things are not possible with the XBox. Maybe will be possible to run Kinect in a PS3 with Linux.

    This type of stuff are done for the Heck of it (Lulz ) and not to manage some practical results, like create uses of Kinect for education. But such practical results can at some point exist.

    • Tei says:

      Note to self: Register the trademark E-Peen for usb pendrives.

    • Premium User Badge

      stahlwerk says:

      Is this an E-Peen in your pocket or are you just happy to see me?

    • Premium User Badge

      JB says:

      and think of the novelty shapes you could make them in.

    • televizor says:

      Well the PS3′s Other OS option was discontinued via a firmware update. But since the PS3 was hacked too maybe we’ll have a chance to see it. I’m curios if Sony’s own camera (the eye toy) knows depth perception or if that’s really needed in standard applications.

      Plus there’s that whole ” 8 feet of space in your living room” that makes Kinect not really suitable for desktops.

    • Deadend says:

      Yeah, but it was never the HARDWARE that drove the Kinect, but the crazy software MS came up with.
      That took years of work and a good number of people to teach the Kinect to do it’s tricks.
      But if MS ever decides to be cool and put out the Kinect API shit could get crazy (MS hinted at using the Kinect software in future devices and applications).

      The hardware itself is pretty much just a webcam with depth perception, the only thing that took development time on it was figuring out how to get the required accuracy levels cheap enough for mass retail.

      It’s not that I’m hating on cranking out drivers for it, but that there is a long way to go before it’s Kinect level of software going, as the Kinect can tell who is who and what hand is which, it’s some fancy calculations. I’m not sure any hackers are going to slog through what took MS the most time.

    • Nick says:

      iPeen.

  6. Kevin says:

    Curious to see if that can be re-purposed into a TrackIR stand-in for DCS-A-10 on the PC or something. As wonderful as TrackIR is for those types of games, using Kinect as an alternative would be better value for money, that much is certain.

    • Kevin says:

      Hell, that and Child of Eden might actually convince me to buy that blasted piece of technology.

    • D says:

      I guess you mean there would be some added advantage of having a depth sensor in the cam? I can’t really think of it myself – a face is pretty much the same depth, but I don’t know what kindof depths it’s capable of measuring.

      But TrackIR faking already exists so you can plug in your webcam, and it will use FaceAPI to track your face to use with for example the DCS games, Falcon, Arma etc. It’s actually quite recent so here is the link: http://facetracknoir.sourceforge.net/

    • dundee says:

      This kinect hack has the potential for allowing a different approach to integration in a game that supports head tracking. The problem with the track ir and even the noIR is that it is reliant on the Natural Point (encrypted) schema. There are patches to get around most of these but not all. For example Natural point has yet to release a 64bit upgrade for use with DCS AC-10c (beta)… which means no one on a 64bit can use freetrack and consequently noIR.

      The point being.. if there is different approach to headtracking, there could possibly be no relying on other (monopolizing?) commercial software developers. Of course at this stage looks like you’d have to buy a kinect possibly making the point, well… pointless.

      This could be cool!

    • BAReFOOt says:

      You can always write a 64-32-bit wrapper. Everything Wine for example, is wrapped on my 64 bit Linux box. And Flash too. Works well.

  7. Corrupt_Tiki says:

    Very smart bloke, I will just stick to minecraft and running from creepers, thanks.

    • Niels Walta says:

      This newfangled piece of technology combined with Minecraft will yield untold hours of digging-excitement! Running from creepers has never been this intense!

    • MadTinkerer says:

      Kinect + Minecraft = horrific tennis elbow from waving your arms to excavate all that cobble!

      Though being able to actually physically swing a sword at those frickin’ Creepers might be worth it. (I’m not the only one who knows how easy it is to take on Creepers in daylight, right? Not that I’m any less inclined to run and hide at night, but figuring out the daylight thing really helped.)

    • OptionalJoystick says:

      No need for Diamond Picks any more- it’s all about the Diamond Elbows.

  8. Premium User Badge

    stahlwerk says:

    What lag do you not speak of, Quintin? Was it in all games you played or only in some?

    • Quintin Smith says:

      The input Kinect sends to the hardware it’s attached to has a slight delay. If you move your hand, for example, it takes time for the hardware to respond. Hardly ideal for most games.

    • The Sombrero Kid says:

      the lag happens after processing on the 360, the lag will be much smaller with a decent pc than it is for 360 games, also amazingly the pc is capable of receiving the full 640×480 depth map rather than the xbox which can only process 320×240 because microsoft crippled the software implementation for some bizarre reason.

      i.e. Kinect on PC will be much much better than kinect on 360.

    • Premium User Badge

      stahlwerk says:

      @The Sombrero Kid, that’s interesting about the resolution. When I heard that it was halved for the launch, I assumed they just built in a cheaper part to push down cost. Do you know anything about the bit depth of the depth data?

      This thing could make an excellent, low cost, one-shot 3d scanner, if the depth resolution is any good.

    • Premium User Badge

      stahlwerk says:

      @Myself
      Wikipedia says the camera records at 16bit across a range of 1.2-3.5 meters which, assuming linear distribution and a totally awesome structured light pattern, means
      *breaks out calculator*

      2700 Millimeters / 65.535 quantisation steps = 0.041 millimeters per step.

      No. Way.

    • BAReFOOt says:

      Actually, the depth resolution would be logarithmic I think. And such high depth resolutions are entirely possible when the timing electronics are quick enough. No idea about that though.

  9. Dreamhacker says:

    Still, this doesn’t cure the 2 sec lag between input and reaction that has been observed time and time again in the Kinect, now does it?

    • Josh W says:

      No, but it’s ok, just set it up with a wiimote set and you can put all your faster motions on that; you don’t really want to be having to duck and dive at a seconds notice anyway.

      And just to emphasise that; the possibility of a wiimote, and the kinect depthfield stuff, working together to cover each others weaknesses, despite their creator companies being in competition! I love the PC!

  10. DevilSShadoW says:

    so, kinect for PC then?

  11. Baqueta says:

    Impressive work, especially the 3hr turnaround! It’s a bit of a shame that all the really cool kinect stuff – skeleton tracking, facial & speech recognition, etc – is done in software. No kinect for PC anytime soon. :(

    Hopefully this might galvanise some brave souls into creating an open source skeleton-tracking library though. If that happens then all sorts of cool projects could be created off the back of it…

    • BAReFOOt says:

      Skeleton tracking should already be available, as there are a lot of feature tracking libs available. They are done by universities, and they are very impressive.
      Speech recognition… well, I found 4 projects with the first search.
      Facial recognition… apparently there are tons of those, and some also work with detecting the rest of the body, and where you are looking at.

      Everything there. I don’t see why you could not integrate it.
      That’s why open source is so great: The spirit of programming is, that you don’t do anything twice. You do it once, right, and then you re-use it however you like.

  12. Vodka & Cookies says:

    As said this is just a driver not the software magic that makes Kinect work. I think it likely Microsoft will eventually release their own driver and SDK for Windows at some point down the road either when Kinect is a huge success or a dismal failure but not while it’s too early to tell either way at the moment.

  13. cliffski says:

    I’m not sure I get what has been hacked here. He has ‘hacked’ a driver that will display the input from the IR camera in the kinnect?
    That’s it?
    Because as I understand it, talking to people who code for it, that isn’t where the magic is. In fact, that’s a trivial element of it. The clever stuff, the REALLY clever stuff, is the software that works out that blob A is your left hand, and blob B is your left leg, and even better, deduces whose hand is whose with multiple people, and even does some clever gubbins to deduce what cannot be happening because it understands what part of your body is connected to what other part yada yada…

    Unless I misunderstand, this guy has written a webcam driver yes? In which case, I don’t think Microsoft could care less.

    • Matt Kemp says:

      An endian-swapping webcam driver, no less.

    • Mezkal says:

      Yes, you misunderstand. There is no magic involved with the MS Kinect/Natal SDK’s clever stuff. It uses the input derived from the cameras and mic array to generate body and audio ‘morphs’. ie, the shapes and audio signatures of people who stand infront of it.

      Biometrics is what Kinect/Natal really uses ti make sense of that data and there are already a metric shit ton of Biometrics open source solutions which will now be able to interface with Kinect on PC, many of them with greater precision and accuracy than what is made available through MS’s own Kinect SDK. In writing this open source driver, the main barrier to using Kinect on a PC is lifted.

      You’re just not smart enough to understand the importance of such an achievement. Internet : land of cheapseats feedback.

    • Bhazor says:

      The guys had three hours. At least give him an afternoon before you start expecting him to completely reverse engineer the bally thing. Honestly.

    • Premium User Badge

      stahlwerk says:

      I’d have wanted to write something similar to Mezkal, albeit a bit more on the good side of manners.

      @Cliffski granted, it’s not that much of an achievement, since USB is standardized and if you know how to poll a generic device or webcam, you know how to access the data stream coming from the cameras. I find it impressive for the short amount of time it took (3 hours after launch!), considering there is no documentation available for this device. Also, it opens the sensors up for usage with the open computer vision (OpenCV) suite, which already does a lot of things like face detection, eye tracking, motion sensing etc.
      Now for a Win7 64 bit driver, get hackin’ good spanish sir!

    • Okami says:

      Internet: land of people who’ll casually insult strangers over trivial misunderstandings.

    • Cooper says:

      Ah, good old measured responses.

      You’re right Cliffski, the clever stuff is the software that interprets the outputs of the kinect camera. As has already been said, this stuff exists as opensource as OpenCV. I would not be surprised if someone has OpenCV working with kinect within the week.

      That being said, 3 hours to hack a driver that can read the output of Kinect is pretty impressive, even if it wasn’t a particularly difficult task.

    • Tei says:

      @Mezkal cliffski is not some random internet dude, he is game dev

    • Mezkal says:

      @Tei : Then he should know better. The task was to write a driver, not an SDK.

    • BAReFOOt says:

      As I said: All that “clever stuff” already existed for a long time, and there are many open source projects covering every aspect.

      Have you learned nothing? There is nothing, absolutely nothing that Microsoft ever invented itself. They are only very good at taking something established from somewhere else, and acting as if they just invented it.
      Basic, DOS, Windows, Office, MSN, Xbox, DCOM, Active Directory, you name it.

      MS was never, isn’t and never will be a development and engineering company. They are a marketing company that clones stuff from others, and then tries to kill off the original before anyone notices. Lotus, Netscape, Java, Borland, again you name it.

    • BAReFOOt says:

      Oh, and every time I found out about something that looked like MS finally invented something for themselves, and started to give them props, I later found out that it’s just a crippled clone of a ISO standard or something like that. Really. Every single time.
      If MS ever really invents something itself, I will still give them props. I’m not a dick. But I will make very very damn sure I check everything there is for precedents, before doing so. Simply out of decades of experience, which mostly consisted of having been disappointed..

  14. Robsoie says:

    I don’t know how the kinect thing work, but with an open source driver could this system be used as some very cheap mocap device without any legal trouble ?
    Mocap studio costs are prohibitive for motion capture unless you have some high profile budget for the software you build.

    With that kind of cheap, it could mean that even indie software may benefit from mocap animation.
    Well, if the whole kinect system can be used as such.

    • Premium User Badge

      stahlwerk says:

      It may be feasible, as soon as someone writes a gesture and skeleton tracking system for it. But I think at the moment you’d be better off with a pair of wiimotes and a bunch of IR diodes (or markers and a spotlight), because the wiimote features 1024×768@100Hz IR tracking and has a rather mature set of drivers from the homebrew community.

  15. Cooper says:

    For an example of the potential of console toys when crossed with open source software:
    YouTube – Motion Control; OpenGL + OpenCV + Playstation Eye + Sixaxis

    • Cooper says:

      Ok, the stuff from about 5:00 onwards got me thinking about having this interface with something like Amnesia… That would be about the most awesome thing ever…

    • Quintin Smith says:

      Jesus. Yeah. Amnesia + Playstation Move would be stunning.

    • Premium User Badge

      stahlwerk says:

      Amnesia? Pfft. Minecraft!

  16. Joe Duck says:

    My killer non gaming PC app for gesture recognition? XBMC
    My killer gaming app? Mount and Blade, of course…

  17. Rick says:

    I’m sure that the robotics people already have a lot more advanced gear than Kinect to help them, but hey.

    • Premium User Badge

      stahlwerk says:

      If it is Good Enough®, then you can’t beat “off the shelf” in convenience and price.

    • DarkNoghri says:

      Most of the people in my graduate program focus area use off the shelf 30$ Logitech webcams.

  18. clownst0pper says:

    My there are some clever people. I can barely get Steam to load!

  19. Premium User Badge

    Stense says:

    And here I thought the Kinect worked by having a bunch of trained Hamsters painstakingly doodling over an over-head projector in real time.

    • Premium User Badge

      stahlwerk says:

      That was only true for the device that Pete Molyneaoux demoed. The final release SKU contains 3 gerbils (one for each dimension).

  20. _Jackalope_ says:

    I don’t understand. At no point during this video was I reffered to as a “bitches”, mocked for my noobness, had my mother insulted or told I am a homosexual and that all people of colour are somehow inferior.

    This is possibly the first video on youtube about videogames that I’ve watched where I’ve understood what is happening and not thought the guy making it was NOT a massive racist, homophobic jerk. That in itself was an achivement worthy of a cash prize!

  21. _Jackalope_ says:

    Doh. Stupid double negatives. This is what happens when you don’t go to bed aftr night shifts.

  22. Gary W says:

    I’m glad that someone has taken the time and trouble to do this. Now I’ll finally be able to unlock the full potential of all those robots, lasers and other assorted gadgets lying around the house.

    Also can’t wait to play “David Braben’s Kinectimals” on the PC in 2016.

  23. Premium User Badge

    Gabbo says:

    Has Microsoft acknowledged it’s happened yet? The last I heard was that they had publicly denied the hack had happened. Guess they’ll have to scramble to get their own half-assed pc drivers out at some [far] future date.

  24. KillahMate says:

    This will be either irrelevant, or Really Really Important. Time will tell.

    But oh, definitely worth it for XBMC if nothing else.

  25. Jimbo says:

    He could get another 8Gq of Memory with that kind of money!

  26. Premium User Badge

    Christian Dannie Storgaard says:

    I foresee an interactive film addition, like director’s comments except awesome.
    Image playing “Face Face Revolution”, trying to match Jim Jarrey’s facial movies while watching Ace Ventura or the head rocking of Will Farrell in A Night at the Roxbury and having the playback program yell encouraging phrases in the style of Mortal Kombat on every successful move… it will be godlike!