Music To My Thumbs: Transcribing Braid

Julian Benson asked if we'd like to see Braid transcribed into musical notation, and vice versa. We said yes. Here's how he got on.

Gaming is one of the few media without an instructional notation. The other arts have tools to relate concepts into symbols that can be digested at a glance, allowing great detail to be condensed into a conventionalised code. Musicians use musical notation, movie makers - story boards, writers - short hand, linguists - phonetics, and so on. If musicians were stuck with the same methods we use - strategy guides, walkthroughs, and videos - each composition would be an interminably long tome that lost all immediacy and comprehension - “Pluck the G-string on the third fret, then immediately after the A-string on the fourth fret, pause, play the G string again”. The other arts developed their own notation systems, yet we haven’t. Why?

Now, we may not need a short form, we’ve managed to get by without a gaming equivalent so far, but that is not to say it couldn’t be useful. Notation has advantages for both musicians and composers that aren’t available to us: the complexity and the pace of a piece can be judged in a glance; (it has lots of symbols; then it’s fast and complex). It’s easy to share ideas, tweak sounds, and write down ideas in brief:, you don’t need to detail the entire concept of the piece to communicate what one musician needs to do, a single instrument's part can be written in isolation. It’s also possible for someone to lay out an entire score; (instructions for many musicians playing different instruments) and see how all the mechanisms lock together.

Also, as I’ll show later, in developing a short form, we open up possibilities for our games which are difficult to accomplish currently.

The same can be done for games.

The seeds of the idea were born one night whilst listening to a piece by the minimalist composer Steve Reich, and playing a particular world in Braid in which your motions are locked to the level’s timeline. It’s a platformer that’s always introducing new temporal gameplay mechanics, but what makes this one relevant is that it essentially renders the level into a piece of music. As you move to the right time moves forwards, as you move to the left time moves backwards: all the elements of the level, the enemies, the fireballs, the hostile plants, play or rewind depending on the direction you move. Walking from extreme left to extreme right is like scrubbing a 12 second track on your computer.

At this point, I began scrubbing back and forth on the track on the Reich piece.

Watch on YouTube

[Steve Reich - ‘violin phase’]

In Steve Reich’s ‘Violin Phase’ three violins play the same short phrase of music repeatedly but out of sync with one another. To listen to it, it sounds like a mess of noise: each violin is playing a different note from the others causing disharmony.

Much like the three violins, all the elements in Braid’s levels, (the fireballs, the mindless goombas, the aggressive pot plants), appear to be in a state of discord. They’re firing, marching, and snapping and they’re doing it all at once.

But in both Braid and Reich there is an element that makes sense of their discord . In Reich, it’s the fourth violin. Whilst the other three step on each others’ toes, all playing different notes, the fourth plays a note simultaneously with one of the other violins thus amplifying that note above the cacophony. In this way it plays a different phrase of music from the other violins but never a unique note, one of the others is always playing that note also.

The audience is most aware of the music played by the fourth violin as its phrase is made up wholly of amplified notes. Whereas the other violins change their music infrequently, slowly moving through five phrases during the 15 minute piece, the fourth is shifting its phrase frequently, each time pointing to a pattern that was hidden in the noise.
In Braid, the player is the fourth violin. As players work their way to the collectible jigsaw pieces dotted about each level they highlight a pattern of movement that in this case avoids all the numerous aggressive elements.

Essentially, each level in Braid can be further broken down into sections: each one centred around attaining a single puzzle piece. By getting to these puzzle pieces the player is pointing out a new pattern in the level’s complexity of action. When the players get to the end of the level, and the audience reaches the end of the Reich piece, they no longer perceive that in-game world is a mess of conflicting noise. They’re now aware of all the patterns that weave through it.

It’s within these two examples that we have the necessary similarities to start applying musical notation to the game, to prove it’s possible that games can be communicated in short form. It's a little ham-handed - musical notes are clearly not meant for this - but we can make them fit for purpose.

First, though, we’re going to have to change a fundamental principle of musical notation. Inherent to that code system is rhythm and timing. Though, here, when transcribing for games, at least in this proto-version, the symbols have little to do with timing. What I mean is that two notes next to each other does not mean two actions that need to be performed in quick succession, only that one comes before the other. Instead, the focus is on conveying non-linear movement. Whereas musicians start at the left of the line of notes and play rightwards until they reach the end of the page, gamers do not usually work linearly. Rare is the game that simply has you play left to right with no back track whatsoever. You are often climbing up ladders, dropping down pits, and playing merry hell with linear directional movement. Yet, despite this - so long as there is a singular goal you are working towards - your movement never stops being linear and can be treated as such. Going up is still essentially moving forwards, if each step takes you along the path to the goal. As long as the transcription accounts for this direction then it can still be written. Though, that all said, I can’t wrap my head around notating a 3D space, which is why I only refer to 2D examples here.

These transcriptions are rough, haphazardly stitched together in Photoshop, requisitioning symbols without much thought beyond practicality. But that is the point. To see if it is practicable. Can we see gameplay in similar terms to how we see music-play? And, if we can, what can we use it for?

An explanation of this level without notation would be something like this (right, deep breaths everyone): to get to the jigsaw piece in the image above the player climbs the ladder to the right, activates the lever to the left -which operates the two doors next to the jigsaw piece, raising one and lowering the other - they then climb the ladder again, move to the left end of the platform (reversing time and bringing the falling goomba back up to jumping height), they jump on to the goomba, bounce back onto the platform, move to the right and jump on that goomba to vault the lowered door and finally reach the puzzle piece. (Breathe.)

As you can see, a straight written tutorial is confusing and bloated.

How about an annotated image?

Whilst managing to get the point across, it has a number of drawbacks: I’ve had to circle thin air to show where the second goomba will be rather than where they are when you jump. You can’t separate the annotations from the image without them becoming meaningless. Plus, it’s a mess to look at.

That’s all of the above instruction in seven notes. 100 words down to to 11 symbols.

Far more immediate, far cleaner.

This is how it works: if the note is full it means you need to interact with one of the level elements, if it’s hollow you don’t (I’ve not counted ladders as interactive because in games they’re simply a space where you can travel upwards, in the same way that falling is travelling downwards). The stem coming out of a note indicates upward or downward movement and the arrows above the notes is the direction you’re to move in. So, those first five symbols mean: climb up the ladder, move left and interact with the object on the middle floor (the lever), move right and go up the ladder again.

That sixth symbol means go right, but there is no interactive object to the right, instead, there is an empty down symbol, meaning, you must walk till you drop off the ledge.

The curved line join symbol at the top is used to convey that two actions are immediately linked, so where the drop symbol is joined to an interactive up with a flick, what’s happening is that you’ve hit the goomba and you need to direct your bounce to the right (if it were to the left the flick would be on the opposite side). The next symbol is another full flick meaning you must jump off the object ahead of you (the other goomba) and walk to the last interactive object, the jigsaw piece.

All that information is held in seven notes.

We need only transcribe the player’s movements because the level elements themselves are dumb: the enemies can’t see you, they don’t aim for you and so never differ in their patterns. In the same way, the three violins in Reich’s piece play with a complete disregard for the other instruments whilst the fourth - though technically dictated as much as the other three - is musically aware of the others in that its notes are are always amplifications of theirs. This is true to the point that often ‘Violin Phase’ is played by a single violinist taking on the role of the fourth and three tape players acting as the first three. The violinist is able to improvise patterns over the background noise of the other three, so long as they are playing the note in time with another violin they are free to make their own patterns. Likewise in gaming, the player can move anywhere in the space, but in order to progress to the goal of the level, certain movements in time with the dumb level elements are necessary

How can this, albeit limited example, be expanded to other games?

If we developed a universal notation system for gaming, or even a set of notation systems that accounted for genre, it could make the lives of many people involved in our industry easier. It would provide a succinct means to communicate ideas for mechanics, levels and systems in a language not specific to any one department.

Current game-making curriculum rely heavily on case studies. If we were able to capture basic design principles in a concise symbol form it makes it easier for students to study games in their own time, applying the abstract principles without the guiding hand of an instructor.

There are benefits for players also.

Braid works as a start point because it skirts the edge of Reich’s work, close enough that the notation of one can be co-opted to fit the other. However, with that in place we can easily expand to other games: VVVVVV and Super Meat Boy can both receive the same treatment, though for slightly different purpose. As games, they’re both the haven for speed runners - those players who aren’t simply content to complete a difficult platformer, they must sprint through it. Whereas an average player may be able to complete Meat Boy in a week speed runners have done it in just 22 minutes:

Watch on YouTube

Speed runners are the concert pianists of gaming. They practice and practice each stage of a game, finding a route through noise of each level, sounding out the optimal path, refining their pattern working at it until they have it note perfect.

Notation gives speed runners two things: an ability to view an entire level with all its interlocking movements and a language with which they can share their routes. Much like a conductor is able to lay out an entire concert score, with all its different parts, and comprehend the whole, a speed runner can see a completely transcribed level, the movement of all its parts. With that information they can see and devise a route through all the obstacles, all on paper: they can show it to people, other speed runners: it can be consumed faster than video, it can be tweaked an altered by the speed runner community.

Though, to be of real use to speed runners, then bringing timing into the notation system would be vital. As I said before, what I’ve detailed above is a rough and ready approach. Simply taking another art’s notation system in lieu of our own. I’m not saying this is the one we should use but using it to show something of what we can achieve with it. The real applications and possibilities for a notation system would not start appearing rightaway. Developing a system would be gathering a toolbox to see what people make with it. Think of it as a pen & paper Minecraft without Creepers and pet wolves.

However, considering we have the rudimentary notation system here and I’ve been harping on about the benefits of abstraction for a while now, how about something else a little off the wall. Take something that already works musically and do the reverse: transcribe a piece of music into a level of Braid.

Here we have the first line of Ode to Joy. Now originally this would be played bythe left hand and right hand of a pianist but we can separate it into player and game elements. It’s not an exact copy of the Beethoven piece: I’ve changed the standard crotchets (interactive ups) into jumps to the right. I’ve also changed the level of the fireballs and plants slightly to make for a more threatening level, and removed two problematic notes. That all said, you play it on a piano and it will still sound like ‘Ode to Joy’.

Looking between the score and the level we can see you must jump straight upwards immediately, that’s to dodge the plant that comes out of the pipe immediately to your left. Next, jump up the two platforms, over the ledge, and over the fireball that comes from the smaller pipe to the left. Jump over the plants down the ledges. Once you reach the lower platform, the first plant, which emerges at intervals from the pipe, attacks again - though you are well out of range. You must then jump up the next two platforms and jump from the pipe onto the finish flag before the second plant emerges from the pipe beneath your feet.

No, it’s not the most exciting level but it is ‘Ode to Joy’ as a game; (something I suspect Beethoven always wanted).

You can also see how, by transcribing from this way round, the original timing of the piece and its musical flow has found its way into the structure of the level. It retains its pauses, its dips and rises, and the enemies attack in time with the position of the player.Transcribing a more complex piece would bring with it a more nuanced level, one with lulls in the action and a rising crescendo both in the level’s geometry and its enemy placement.

Of course, if we were to continue pushing the short form model derived from musical notation, its limitations would become clear very quickly. We can’t lift it for our purposes wholesale: it can’t handle 3D space, unpredictable AI, or complex player actions for a start. What this example has shown is that it’s possible to transcribe our interactive medium into concise abstract symbols, that it’s within our reach as gamers to develop a new language for our games. The same example could have been made with many of the available systems of notations available. Though, each would have had their problems and their limitations. For the best fit we’d need to develop something wholly our own. It would take time, modern musical notation is the product of generations of development, but even a basic notation system beings to open up possibilities for us as designers and as gamers.

Also, as you can see. once we break something down into an abstract representation it allows us to link it to abstracts of other media more easily - music to games and vice versa is just the beginning: for instance, there’s a short form to transcribe dance movements which means it may be technically possible to perform games as a dance: Half Life ballet Kickstarter, anyone?

Many thanks to DragezeeY who created the Mario tileset used in the Ode to Joy image.