A (Difficult) Day In The Life Of An MMO Studio

By Dan Grill on May 3rd, 2011 at 3:29 pm.


We recently catapulted our correspondent Dan Griliopoulos over the ocean to have a nose at upcoming Roman Empire-themed MMO Gods & Heroes; a preview on that itself is due very soon, but first let’s try something a little different. Dan was kindly granted an extraordinarily high level of access to Austin, Texas developer Heatwave Interactive (staff at which worked on some of the earliest and most legendary MMOs) as they worked on the game during its late stages – their meetings, their design and marketing dilemmas, their thoughts on competition, painful decisions about to delay or not to delay… It’s rare insight into the hard work and heart ache inherent in creating an MMO, games of (in)famously sprawling scale and complexity.

It’s 6am. All through Austin the iridescent grackles are beginning their morning chorus; all through Austin, people are thinking of ways to kill grackles. Community manager Donna Prior is already up, moderating the Gods & Heroes beta forums, but the rest of Heatwave Interactive is still abed, dreaming of Romans, MMOs and the end of crunch time. The studio, in the basement of an isolated tower block is quiet until 8.00am when content designer Todd and lead designer Tim let themselves in. Whoever you ask in Heatwave – from the CEO to the guy who builds the servers right over to the handyman – says that Todd is there when they arrive, when they leave, at weekends…

…Gradually, the studio fills up. At 8.15 Theda (Marketing) and Phil (the Producer) get in, and put the coffee on. By 9.30am everyone’s in, checked their emails, found all the outstanding issues and, in the words of technical artist Jeremy Dombroski “jumped on the show-stopper bugs”. Now we’re ready for Scrum. For those of you not blessed enough to move in development circles (which includes us) Scrum is an organisational technique used to manage packs of rabid Tasmanian Devils, sacks of cats and game development teams. If you ever want to work in game production, getting familiar with Scrum is a really good idea. So, here’s a picture of Heatwave’s Scrum board:

The different post-it colours represent different categories of task; sound, client programming, community, art, server, estate quests, quests and so on… The different columns represent what level of completion they’re at; whether they’re in progress, blocked by something else or finished. The different marks on each post-it tell you what the task is, who owns it, how many days it should take to complete, and if the owner has found more work on the project. At the end of each two week period, the board is cleared and the Scrum process starts again.

For Scrum, the entire team assembles and talks through the progress of the previous day. The remote contractors come through first, Skyping in from places as far afield as St Petersburg, and the team follows, one-by-one. As post-its are moved to ‘complete’ there are cheers; as blockages emerge the team boos, though as the team are in crunch (again) everyone’s slightly frayed and quiet. When Bernie, the Senior Server Engineer, declares that he thinks his team have worked out how to get the server to build there are half-hearted cheers (and a single shout of “nuke it from orbit – it’s the only way to be sure”). When designer Todd moves the God Quests to ‘finished’ to he gets more honest cheers, as does the CEO Anthony when he declares he “smacked down some forum trolls”.

One of the most developer-intensive sections, and hence most full of bugs, is the new Estates section of the game. Heatwave bought Gods & Heroes from Perpetual Entertainment and, though the game was nearly complete, it was quite outdated; Perpetual had hit a game-breaking bug back in 2007 (since solved) and abandoned the game. Much of Heatwave’s work has been to understand Perpetual’s proprietary systems, polish the game up for launch, and introduce all the features that we’ve come to expect of an MMO in the last four years.

The most important new feature they’ve introduced is the Estate. You play a Roman aristocrat whose country estate has been destroyed and your family slaughtered; throughout the game, you’re seeking to avenge your family on the enemy’s gods. However, you’re also rebuilding your own estate which is absolutely huge; it’s got a barracks, a watch-tower, a temple and is set in a valley about the size of Stormwind in WoW – I mean, big. When you first encounter it, it’s in ruins, with your surviving household living in tents; but as you complete quests in the main world, you can rebuild and slowly improve the estate buildings, each of which unlocks buffs, items and henchmen, all shared with all your characters on that server.

After Scrum, I spot Phil, the producer, grabbing Jeremy to talk to one of the external contractors about the collision failures in the Estate. They talk quickly, agree on tactics, and are in done in 15 seconds. I grab Phil quickly, and get his story; his job is basically knocking heads together; “Before this I was with KPMG (one of the big four accountants). I was on the beach, having finished my latest project and I heard about these MMOs. I started playing them…” Cue involvement in the first MMO Meridian 59 and EverQuest, then joining Origin as a project manager. How big a change is the world of games from accountancy? “It’s become more structured, but Scrum is still elastic; I can’t imagine implementing CMM level 5 (a military or business organisational structure) here – game developers are not that super-regimented.”

At about 10am, Scrum finishes and the various teams head back off to their own sections. I first wander over to the art department, where a new hire is rebuilding the animations of the characters, to make them much more dynamic and faster. Jeremy, the technical artist is doing the “chicken wire and duct tape that holds all art side together”. Right now he’s stomping on a lot of bugs, planting trees, recalculating the grass where topography’s changed, and fixing the estate zone collision problems. “Honestly,” he says “I thought I’d be doing a lot more art.” At midday, there should be a break for lunch but I can’t see much sign of it – Jeremy just grabs some fruit and protein bars and goes back to his desk.

Technical artist and protein bar enthusiast Jeremy Dombroski

In the afternoon, I pop in on Todd Bailey, the content designer (the writer), the hardest-working man in the office, and a veteran of Origin. Increasingly, as the day goes on, I’m realising the wealth of MMO experience in this office; most people here worked on Ultima Online or the long-cancelled sequel UXO; A few, like Jeremy or the lead designer Tim Schubert, worked on earlier projects, including the first MMO Meridian 59. Todd’s a large, softly spoken man who is the only person handling the content of the game; “There’s currently 3000 quests in the game – I have to do all the dialogue, misspelled, confusing, bad dialogue, fix broken elements and so on. Perpetual still had a lot of time and work left, and one of the things still missing was any through story to the game. In the middle of the game, they forgot about the plot. We’re still only 90% on how the tools work.” Todd’s family are going camping that weekend; he’s planning to camp in the office, but from the amount he talks about Smores, he obviously wants to be with them. His kids that is, not the smores. I leave him going through his huge bug list, trying to fix the story, dropping new Estate quest chains in (and finding places to tie them in), and writing the copy for promotional videos.

Across the way from Todd is Mike, the techhead. His separately-powered cubicle is full of PCs. “I filled up the server room” he admits “and it’s more flexible just to build new $1000 PCs when I need them, with multiple OSes installed on virtual machines, so we can test all the possible language OS combinations quickly. It saves a lot of time and money. I put 200mm fans in, so it’s not so hideously noisy.”

Then it’s time for the once-a-week EP meeting, where all the top dogs get together to talk about the business aspects of the game. This takes place in the Blofeld-inspired boardroom, dimly-lit, with a fish-tank bubbling away at the back, adorned with the sign “Mykel not allowed to feed!!!” As the CEO Anthony says, this meeting “is about as down and dirty as you can get”; we’re really getting insights into the heart of an MMO company here.

Firstly, producer Phil shows off a spreadsheet demonstrating beta account creation; there are notable spikes at GDC and for their second beta event, with a huge spike when they did a stress test. It shows that from 20,000 beta keys taken, just 2500 accounts were created, and only 2000 of those logged in. Next, we get a look at the Facebook insights back end. This shows monthly visits have gone up by 327%, which Theda deduces means their Facebook advertising is working; the aim of this is, explicitly to push these hardcore fans towards the IGN boards and pass along the messaging. The Facebook ad numbers also show that players are being pulled in, not from other MMOs, but from other Roman or mythological crossovers – Spartacus or TV shows.

Our penultimate metric is from the installer, tracking P2P efficiency; how long it takes to download the client. Heatwave really don’t want players taking too long to download the game and, while the average is 50 minutes, there’s a long tail; this shows them that giving people the option for a direct download of the whole client works better for some gamers. Finally, Tony Williams, the COO (“I worked on over 300 games and managed to keep my name off all of them except one”), shows the country-tracking data; he works out from this that Canada is their fourth largest market (after the US, UK, and Germany) and Lauren, the distribution lady, expresses annoyance that she was told it wasn’t worth targeting.

Then tre’s a little bit of talk about the struggle to get bug reports through – apparently the back-end left by Perpetual reports manually, so they can’t track where in the game reports are coming from, a huge problem for bug-fixing. Anthony, the CEO bulls in “I can’t figure out how to get off Takenos (the starting island.) I mean, I couldn’t work that out. I want progress on this before we go into open beta.”

Community manager Donna Prior

Donna quickly runs us through their website (they’re using WordPress for the blog aspect, but the site customisations mean that they can’t use Xloc for automatic translation for Germany), Anthony talks about the credits for the manual (“everyone who ever worked on this gets credited”), Mykel (head of biz dev) talks about Prima pulling out of the official guide (“They don’t have the time or the bandwidth”), and then Anthony asks a big question.

“Do we need an extra two weeks?”

They all know what this involves; another fortnight without any income, but another fortnight of bug-fixing and polish. If they slip, then their release will avoid E3, where it will would be lost in the morass of NGP, Wii 2, and the big MMO announcements; if it comes out buggy, gamers will crucify it. After much haggling, the decision is… deferred. Tony tries to precipitate the decision by asking if they’re ready for the Gold Master, the disc that’s sent to the replicators and will appear in all the boxed copies. “We’re ready” says Mike Jones, who’s also in charge of development services, casually, like it’s the easiest thing in the world. When Anthony expresses surprise, Mike says “If you look what Rift did, they gave players 8MB on the disc, and the rest came down on patch. They were patching every day after that.” For MMOs, actual shops are really just advertising these days.

After that, they move onto new personnel; the company is staffing up very quickly, as Sony Online Entertainment has just laid off at least a hundred talented developers nearby. Tony asks if the new server engineers are getting on; Producer Phil says “the chemistry was a bit odd and they’re both quirky, but you expect that from server programmers.” Tony is very clear that if anyone’s not happy, they should talk to HR immediately. And it’s done.

Thankfully, this being a Thursday, the day finishes early at 4.30pm for happy hour; everyone has a drink and gets to go home early. Of course, every day’s different – On Monday, there’s a stand-up session, where everyone says what they’ll be working on that week. On Tuesdays there’s a marketing call, where Theda decides the direction of advertising and target audiences. Wednesdays, there’s a PR call with the international team. And on Friday there’s Show and Tell, where anyone can stand up and show off their achievements from that week.

The office gradually shuts down. Beer-lover Donna has decided to take us to a drafthouse (a low, gray strip mall building, trapped between motorways, that has over 200 of the best beers in the world), and everyone else drifts away. The usual die-hards stay in the office; Tim Schubert, who sometimes works to 3am; Phil, who doesn’t seem to have lost the KPMG work ethic; and Todd Bailey, camping out there, dreaming of Smores.

__________________

« | »

, , .

39 Comments »

  1. Jonathan says:

    Completely off-topic, but the grackles in TX are lovely — they’re the fancypants boat-tailed variety, and they’re comedians.

  2. Cooper says:

    I know this kind of ‘a day in the office of…’ piece is a dream of a number of games journos out there (and I imagine it’s not restricted to games – what music journalist would not want a day in a recording studio?).

    Excellent piece and here’s hoping it spins off a few more like it.

    • Arglebargle says:

      Yeah, very nice piece. I know a number of ex-Origin and ex-game industry workers who’ve left the field due to the the inordinate crunch time and attendant woes. Still, many cannot keep themselves from returning.

  3. smi1ey says:

    It sucks that MMO devs get so much crap from whiny players. Imagine developing for a game that’s never finished being developed. There’s no crunch week then a month of vacation. Just constant work! God bless these people. :)

    • bob_d says:

      It’s true, it’s non-stop work. But then again, I don’t know anyone (in the US at least) who offers a month off after a week or two of crunch. I know people who have had a year of crunch followed by a week off. More likely these days is months of crunch followed by layoffs.

  4. Gar says:

    I’ve never heard of Heatwave Interactive, and certainly didn’t realize they were based in Austin (where I’ve been living for the past 3.5 years). They seem like a pretty cool bunch. Which drafthouse did Donna take you to? Just curious, as I don’t think the Gingerman, Flying Saucer, or Draught House fit that description, and they are the only places I’ve been that have a huge variety of delicious beer…

  5. TomSavage says:

    Been rocking out with a priest myself and managed to get to lvl 7 over this weekend. Although the improvements coming are well pointed out here (and on their site, they’re very open about development!) the game is in great shape and I can’t wait to play when it launches on June 21st which is a few days before my birthday and I know what I’ll be getting myself for it! Gods and Heroes FTW!!!

  6. Reapy says:

    Every time I read a story like this is pretty much the reason I’ve steered away from game development. I guess there are places where it’s not crunch time all the time, but I guess really the part that hits home for me personally is that if it came down to work vs camping with my kids, camping with my kids would win every time.

    At the end of the day heatwave isn’t going to change your bed pan and let you hold your grandkids, and the satisfaction from making a random mmo / game will probably only live in your memory. But maybe that memory is stronger than losing 1 or 2 weekends of memories with your children. I dunno. We all do crunch time from time to time so this might just be a ‘rare’ moment lost, but still.

    Anyway, great story as usual rps :)

  7. Rii says:

    Fascinating article. I’ve not heard of the team or the game before, but this is a good way of reminding us proles of the people behind the games we enjoy, and even those we don’t.

  8. bob_d says:

    Ah Perpetual and Heatwave – they embody the sad story of modern AAA PC game development. Perpetual wasn’t almost done with the game – they were done. The “server bug” mentioned meant that no more than a dozen people could be on one server though, a problem which they couldn’t fix before running out of money and which obviously prevented it from being a workable MMO. It could have been released as a single player game, but since it was developed as an MMO, they wouldn’t have been able to recover their (substantial) investment in it.
    Heatwave started out with grand ambitions but financial issues that caused them to lay off a good portion of the company early on. They limped along doing iPhone games until this project. (I’m not sure how they ended up with “Gods and Heroes,” presumably they found further investors.)
    MMO development is so crazily expensive that broken/unfinished MMOs actually have commodity value. Another team can work to finish it off with a (relatively) small workforce, and if they can fix it, the potential payoff can be large. I’m seeing more and more developers in this situation, which, not too long ago, would have been unthinkable. Given the cost of single player AAA games, I’m sure we’ll be seeing more of those being treated the same way.

    • Rii says:

      “I’m seeing more and more developers in this situation, which, not too long ago, would have been unthinkable. Given the cost of single player AAA games, I’m sure we’ll be seeing more of those being treated the same way.”

      Question is, can an A-grade product emerge from such a process?

    • Baboonanza says:

      It doesn’t have to. You’d be buying the half finished game for a fraction of the actual dev costs so a B-grade / niche product finsihed by a smaller team can make money where the original team would have been hugely in the red. It probably helps that the new team will be less emotionally invested in the product and prehpas be able to see more clearly where the issues are (see APB).

      Its a bit like buying a half-finished building really, except the sale depreciation is much larger due to the substantially larger risk involved.

    • Rii says:

      I meant from, well, our perspective. It might be profitable, but then so is FarmVille.

    • bob_d says:

      @Rii: Well, that’s (one of) the important question(s). Is design by committee worse when you have two different committees at different times? (As opposed to what you sometimes have now, two [or more] different committees designing a game at the same time…)
      Having been on a team that worked on someone else’s AAA online game project after the original company collapsed, I’d say, if I’m going to be completely honest, that the game was “fixed” in a number of ways (because we had enough distance to figure out where it wasn’t working), but made the game worse in others. Our team had a slightly different vision for the game, but not the resources to completely remake it, so it ended up being a bit of an ungainly process, trying to shoehorn the existing game into a new shape. (Not to mention that the publisher had a third vision for the game…) Although in our case the original developers ended up with a really inconsistently developed game, in tone and gameplay, due to poor management, and we made an effort to make the game more consistent, a second development team could easily make a game feel more scattershot in design.

    • Josh W says:

      To my mind b-grade and niche are very different things; much better to go for a smaller audience who will love it than be ok for a lot of people. You’re relying on their apathy in not finding a game that is better for them. What is more, you might even form a marketing framework for your competitors, as you gather together the people who then go off to find what they really want!

  9. Peter Radiator Full Pig says:

    Dan Griliopoulos, I want to have your names babies.

  10. Torgen says:

    With the UO pedigree of some of the dev team, I was hoping for something more open than the usual “fantasy class-based tank / melee DPS / ranged DPS / healer” game, but it seems I’m to be disappointed. : (

    I’ll take a look in any case when/if they have an open beta, though.

    • Josh W says:

      Bearing in mind how much UO changed over the course of it’s run, they might sneak that stuff in there! Unless of course they were the guys on the team less interested in the emergence stuff.

    • Torgen says:

      UO lives on in my memory as it was 1998-99. I was in the original Mage Tower/UOVault gang.

  11. Xizor says:

    Great insight. More articles like this if you please :)

    On another note: While I realize the payoff on MMOs can be big. Considering the number of failed attempts how many MMOs actually made a profit? Has someone looked at that?

    • bob_d says:

      It’s complicated because a lot of companies fold before they can get a working product out the door (e.g. “Gods and Heroes”). The number of company/project closures has far, far exceeded the number of released, failed MMOs. Also, what qualifies as an “MMO”? Multiplayer web-based games? Low development costs there mean that the success rate is many, many times higher. For AAA projects, though, it seems like I’ve been seeing less than 5% of announced projects actually turn into finished, profit-making games. This was a result of the great “post-WoW MMO gold rush,” though; developers (and more importantly, investors) have a much better idea of how hard it is to make (and make money off of) MMOs now, so far fewer are jumping into AAA MMO development these days.

  12. Frosty says:

    Ah, Donna Prior. She used to work on the The Old Republic boards and was generally splendid and magnificent. The admins have been pretty much useless since she left. Shame.

  13. Shiny says:

    “Anthony, the CEO bulls in ‘I can’t figure out how to get off Takenos (the starting island.) I mean, I couldn’t work that out. I want progress on this before we go into open beta.’”

    This made me laugh, because I’ve also been in a number of situations where the boss turns playtester on a whim, finds some problem, and all of a sudden that problem is priority #1, brought up in a quite unrelated meeting or discussion.

    • bob_d says:

      Ha, oh gods, yes. I used to get a flurry of emails demanding my immediate attention from the CEO every time he play-tested the game…

  14. Carra says:

    That board screams SCRUM from miles.

    It ‘s a great way for our boss to impress prospectives. Look at the fancy, colorful board! Also seems to work with game journalists :)

    • BobsLawnService says:

      Oh yes. Scrum. As someone who uses Scrum in my day job all I could think looking at that board was “Poor bastards.”.

      Scrum is great for managers who love micromanaging their staff and plotting everything on a graph.

  15. Howl says:

    They need a SCRUM sticker with, “Put a FAQ on the website” as after a few minutes of trawling around it there was no indication of what kind of game it was or what it involves. I wouldn’t even know it was an MMO if I hadn’t seen this RPG article.

    Of course you can link it to your social networking sites, or even pre-order it. But what you’re actually pre-ordering is not mentioned.

  16. WJonathan says:

    Remind me to buy stock in Post It Notes when development starts on the next WoW.

  17. Josh W says:

    Even though I’ve heard of agile development before, I kept thinking you meant Scumm.

  18. BobsLawnService says:

    Who needs analysis and design – we’ve got Scrum.

Comment on this story

XHTML: Allowed code: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>