By Dan Grill on May 3rd, 2011 at 3:29 pm.
We recently catapulted our correspondent Dan Griliopoulos over the ocean to have a nose at upcoming Roman Empire-themed MMO Gods & Heroes; a preview on that itself is due very soon, but first let’s try something a little different. Dan was kindly granted an extraordinarily high level of access to Austin, Texas developer Heatwave Interactive (staff at which worked on some of the earliest and most legendary MMOs) as they worked on the game during its late stages – their meetings, their design and marketing dilemmas, their thoughts on competition, painful decisions about to delay or not to delay… It’s rare insight into the hard work and heart ache inherent in creating an MMO, games of (in)famously sprawling scale and complexity.
It’s 6am. All through Austin the iridescent grackles are beginning their morning chorus; all through Austin, people are thinking of ways to kill grackles. Community manager Donna Prior is already up, moderating the Gods & Heroes beta forums, but the rest of Heatwave Interactive is still abed, dreaming of Romans, MMOs and the end of crunch time. The studio, in the basement of an isolated tower block is quiet until 8.00am when content designer Todd and lead designer Tim let themselves in. Whoever you ask in Heatwave – from the CEO to the guy who builds the servers right over to the handyman – says that Todd is there when they arrive, when they leave, at weekends…
…Gradually, the studio fills up. At 8.15 Theda (Marketing) and Phil (the Producer) get in, and put the coffee on. By 9.30am everyone’s in, checked their emails, found all the outstanding issues and, in the words of technical artist Jeremy Dombroski “jumped on the show-stopper bugs”. Now we’re ready for Scrum. For those of you not blessed enough to move in development circles (which includes us) Scrum is an organisational technique used to manage packs of rabid Tasmanian Devils, sacks of cats and game development teams. If you ever want to work in game production, getting familiar with Scrum is a really good idea. So, here’s a picture of Heatwave’s Scrum board:
The different post-it colours represent different categories of task; sound, client programming, community, art, server, estate quests, quests and so on… The different columns represent what level of completion they’re at; whether they’re in progress, blocked by something else or finished. The different marks on each post-it tell you what the task is, who owns it, how many days it should take to complete, and if the owner has found more work on the project. At the end of each two week period, the board is cleared and the Scrum process starts again.
For Scrum, the entire team assembles and talks through the progress of the previous day. The remote contractors come through first, Skyping in from places as far afield as St Petersburg, and the team follows, one-by-one. As post-its are moved to ‘complete’ there are cheers; as blockages emerge the team boos, though as the team are in crunch (again) everyone’s slightly frayed and quiet. When Bernie, the Senior Server Engineer, declares that he thinks his team have worked out how to get the server to build there are half-hearted cheers (and a single shout of “nuke it from orbit – it’s the only way to be sure”). When designer Todd moves the God Quests to ‘finished’ to he gets more honest cheers, as does the CEO Anthony when he declares he “smacked down some forum trolls”.
One of the most developer-intensive sections, and hence most full of bugs, is the new Estates section of the game. Heatwave bought Gods & Heroes from Perpetual Entertainment and, though the game was nearly complete, it was quite outdated; Perpetual had hit a game-breaking bug back in 2007 (since solved) and abandoned the game. Much of Heatwave’s work has been to understand Perpetual’s proprietary systems, polish the game up for launch, and introduce all the features that we’ve come to expect of an MMO in the last four years.
The most important new feature they’ve introduced is the Estate. You play a Roman aristocrat whose country estate has been destroyed and your family slaughtered; throughout the game, you’re seeking to avenge your family on the enemy’s gods. However, you’re also rebuilding your own estate which is absolutely huge; it’s got a barracks, a watch-tower, a temple and is set in a valley about the size of Stormwind in WoW – I mean, big. When you first encounter it, it’s in ruins, with your surviving household living in tents; but as you complete quests in the main world, you can rebuild and slowly improve the estate buildings, each of which unlocks buffs, items and henchmen, all shared with all your characters on that server.
After Scrum, I spot Phil, the producer, grabbing Jeremy to talk to one of the external contractors about the collision failures in the Estate. They talk quickly, agree on tactics, and are in done in 15 seconds. I grab Phil quickly, and get his story; his job is basically knocking heads together; “Before this I was with KPMG (one of the big four accountants). I was on the beach, having finished my latest project and I heard about these MMOs. I started playing them…” Cue involvement in the first MMO Meridian 59 and EverQuest, then joining Origin as a project manager. How big a change is the world of games from accountancy? “It’s become more structured, but Scrum is still elastic; I can’t imagine implementing CMM level 5 (a military or business organisational structure) here – game developers are not that super-regimented.”
At about 10am, Scrum finishes and the various teams head back off to their own sections. I first wander over to the art department, where a new hire is rebuilding the animations of the characters, to make them much more dynamic and faster. Jeremy, the technical artist is doing the “chicken wire and duct tape that holds all art side together”. Right now he’s stomping on a lot of bugs, planting trees, recalculating the grass where topography’s changed, and fixing the estate zone collision problems. “Honestly,” he says “I thought I’d be doing a lot more art.” At midday, there should be a break for lunch but I can’t see much sign of it – Jeremy just grabs some fruit and protein bars and goes back to his desk.
In the afternoon, I pop in on Todd Bailey, the content designer (the writer), the hardest-working man in the office, and a veteran of Origin. Increasingly, as the day goes on, I’m realising the wealth of MMO experience in this office; most people here worked on Ultima Online or the long-cancelled sequel UXO; A few, like Jeremy or the lead designer Tim Schubert, worked on earlier projects, including the first MMO Meridian 59. Todd’s a large, softly spoken man who is the only person handling the content of the game; “There’s currently 3000 quests in the game – I have to do all the dialogue, misspelled, confusing, bad dialogue, fix broken elements and so on. Perpetual still had a lot of time and work left, and one of the things still missing was any through story to the game. In the middle of the game, they forgot about the plot. We’re still only 90% on how the tools work.” Todd’s family are going camping that weekend; he’s planning to camp in the office, but from the amount he talks about Smores, he obviously wants to be with them. His kids that is, not the smores. I leave him going through his huge bug list, trying to fix the story, dropping new Estate quest chains in (and finding places to tie them in), and writing the copy for promotional videos.
Across the way from Todd is Mike, the techhead. His separately-powered cubicle is full of PCs. “I filled up the server room” he admits “and it’s more flexible just to build new $1000 PCs when I need them, with multiple OSes installed on virtual machines, so we can test all the possible language OS combinations quickly. It saves a lot of time and money. I put 200mm fans in, so it’s not so hideously noisy.”
Then it’s time for the once-a-week EP meeting, where all the top dogs get together to talk about the business aspects of the game. This takes place in the Blofeld-inspired boardroom, dimly-lit, with a fish-tank bubbling away at the back, adorned with the sign “Mykel not allowed to feed!!!” As the CEO Anthony says, this meeting “is about as down and dirty as you can get”; we’re really getting insights into the heart of an MMO company here.
Firstly, producer Phil shows off a spreadsheet demonstrating beta account creation; there are notable spikes at GDC and for their second beta event, with a huge spike when they did a stress test. It shows that from 20,000 beta keys taken, just 2500 accounts were created, and only 2000 of those logged in. Next, we get a look at the Facebook insights back end. This shows monthly visits have gone up by 327%, which Theda deduces means their Facebook advertising is working; the aim of this is, explicitly to push these hardcore fans towards the IGN boards and pass along the messaging. The Facebook ad numbers also show that players are being pulled in, not from other MMOs, but from other Roman or mythological crossovers – Spartacus or TV shows.
Our penultimate metric is from the installer, tracking P2P efficiency; how long it takes to download the client. Heatwave really don’t want players taking too long to download the game and, while the average is 50 minutes, there’s a long tail; this shows them that giving people the option for a direct download of the whole client works better for some gamers. Finally, Tony Williams, the COO (“I worked on over 300 games and managed to keep my name off all of them except one”), shows the country-tracking data; he works out from this that Canada is their fourth largest market (after the US, UK, and Germany) and Lauren, the distribution lady, expresses annoyance that she was told it wasn’t worth targeting.
Then tre’s a little bit of talk about the struggle to get bug reports through – apparently the back-end left by Perpetual reports manually, so they can’t track where in the game reports are coming from, a huge problem for bug-fixing. Anthony, the CEO bulls in “I can’t figure out how to get off Takenos (the starting island.) I mean, I couldn’t work that out. I want progress on this before we go into open beta.”
Donna quickly runs us through their website (they’re using WordPress for the blog aspect, but the site customisations mean that they can’t use Xloc for automatic translation for Germany), Anthony talks about the credits for the manual (“everyone who ever worked on this gets credited”), Mykel (head of biz dev) talks about Prima pulling out of the official guide (“They don’t have the time or the bandwidth”), and then Anthony asks a big question.
“Do we need an extra two weeks?”
They all know what this involves; another fortnight without any income, but another fortnight of bug-fixing and polish. If they slip, then their release will avoid E3, where it will would be lost in the morass of NGP, Wii 2, and the big MMO announcements; if it comes out buggy, gamers will crucify it. After much haggling, the decision is… deferred. Tony tries to precipitate the decision by asking if they’re ready for the Gold Master, the disc that’s sent to the replicators and will appear in all the boxed copies. “We’re ready” says Mike Jones, who’s also in charge of development services, casually, like it’s the easiest thing in the world. When Anthony expresses surprise, Mike says “If you look what Rift did, they gave players 8MB on the disc, and the rest came down on patch. They were patching every day after that.” For MMOs, actual shops are really just advertising these days.
After that, they move onto new personnel; the company is staffing up very quickly, as Sony Online Entertainment has just laid off at least a hundred talented developers nearby. Tony asks if the new server engineers are getting on; Producer Phil says “the chemistry was a bit odd and they’re both quirky, but you expect that from server programmers.” Tony is very clear that if anyone’s not happy, they should talk to HR immediately. And it’s done.
Thankfully, this being a Thursday, the day finishes early at 4.30pm for happy hour; everyone has a drink and gets to go home early. Of course, every day’s different – On Monday, there’s a stand-up session, where everyone says what they’ll be working on that week. On Tuesdays there’s a marketing call, where Theda decides the direction of advertising and target audiences. Wednesdays, there’s a PR call with the international team. And on Friday there’s Show and Tell, where anyone can stand up and show off their achievements from that week.
The office gradually shuts down. Beer-lover Donna has decided to take us to a drafthouse (a low, gray strip mall building, trapped between motorways, that has over 200 of the best beers in the world), and everyone else drifts away. The usual die-hards stay in the office; Tim Schubert, who sometimes works to 3am; Phil, who doesn’t seem to have lost the KPMG work ethic; and Todd Bailey, camping out there, dreaming of Smores.