Interactive Music Score Engraving

This is a short post about a JavaScript library I've recently discovered for producing music scores called VexFlow. It includes a language for defining score layouts called VexTab. Although this is still under development, it is complete enough to give me about 95% of what I need for displaying scores of simple traditional melodies. The main features that are missing at the moment are:

• Changing time signature or key signature mid-stave
• First and second repeats
• Displaying quadruplets in compound time accurately

I have recently written an elm wrapper for VexTab and a translation from ABC notation into VexTab. I already have an editor for ABC notation that re-parses after every character (and thus makes it available to a player). It has thus been straightforward to add interactive score generation - i.e. the score grows as each character is typed. If you are interested, you can try it here.

Using Modules in Elm 0.17

Elm 0.17 introduces significant breaking changes from the previous release.  FRP has gone, as have signals, but what has improved is a more coherent architecture using the key new concept of subscriptions to external services.  These are managed for you by the elm runtime and will return Cmd messages to you in exactly the same way as (for example) a view component will return a message.  The overall architecture is nicely summarised in this picture.

There is no doubt that this is a major improvement, but unfortunately elm is still severely lacking in one crucial area - integration with the web platform API.  Evan has indicated how it is likely that this might be achieved in the future with a small set of examples which include web sockets and which rely on a still undocumented feature named Effect Managers.

The unfortunate result of this is that, if you want (say) to use a platform API such as web-audio, you are still encouraged to do so using ports, which provide a subscription to javascript services. These have major drawbacks - you are not allowed to build a library if you use them and are thus debarred from publishing it as a community package. Nor can you build a simple single artefact for distribution by other means - the javascript produced by your module must be hand-assembled alongside that produced by the calling program.

So, a good many developers will be forced down the road of using ports to access the platform API and producing their own modules as  'pseudo-libraries' for getting the job done.

Modules

Whereas there is good documentation for showing you how to build modules there is less available on the subject of how to incorporate a module into your program, particularly if it uses ports.  The rest of this article gives an explanation of how this might be done.

Suppose your module encapsulates a widget that plays back midi recordings on a suitable instrument. It might look like this, with a start/pause and stop button and a capsule which shows the proportion of the tune that has been played:

The module will use web-audio to actually create the sounds and so will use ports to a javascript service.  It will export a module definition looking like this:

  
    module Midi.Player exposing (Model, Msg (SetRecording), init, update, view, subscriptions)

This is all as expected. The only subtlety is that the module exposes the SetRecording message type that allows the calling program to tell it which recording to play. The messages that respond to the player buttons are hidden and act autonomously.

Main Program

The following section describes how the calling program that (somehow) gets hold of a MIDI recording via the MIDI message might integrate the player:

import

  
    import Midi.Player exposing (Model, Msg, init, update, view, subscriptions)

model

 
   type alias Model =
     { 
       myStuff :....
     , recording : Result String MidiRecording
     , player : Midi.Player.Model
     }

messages

The program needs to send a message to the player which describes the midi recording to play. Otherwise, all player messages must simply be delegated to the player itself:

  
    type Msg
      = MyMessage MyStuff
      | Midi (Result String MidiRecording )  
      | PlayerMsg Midi.Player.Msg   

initialisation

It is important that the calling program allows the player to be initialised. The let expression gets hold of the player initialisation command and then the Cmd map function turns the module-level message into a program-level message. The exclamation mark function requires some explanation - it is used here as a shorthand to convert the model into the (model, Cmd Msg) tuple.

  
    init : (Model, Cmd Msg)
    init =
      let
        myStuff = ....
          (player, playerCmd) = Midi.Player.init recording
      in
        { 
          myStuff = myStuff 
        , recording = Err "not started"
        , player = player
        } ! [Cmd.map PlayerMsg playerCmd]

update

It is assumed that the program issues a message somehow to get hold of a MIDI recording which it then saves to the model with an incoming Midi message once it receives the response. Thereafter, all module-level messages are simply delegated to the module:

  
    update : Msg -> Model -> (Model, Cmd Msg)
    update msg model =
      case msg of

        MyMessage stuff -> ...

        Midi result -> 
          ( { model | recording = result }, establishRecording result )    

        PlayerMsg playerMsg -> 
          let 
            (newPlayer, cmd) = Midi.Player.update playerMsg model.player
          in 
            { model | player = newPlayer } ! [Cmd.map PlayerMsg cmd]

where establishRecording sends a command to the player which establishes the recording to play:

  
    establishRecording : Result String MidiRecording -> Cmd Msg
    establishRecording r =
      Task.perform (\_ -> NoOp) 
                   (\_ -> PlayerMsg (Midi.Player.SetRecording r)) 
                   (Task.succeed (\_ -> ()))

view

To see the player widget, you have to map the message onto the player view:

  
    view : Model -> Html Msg
    view model =
      div [] 
        [  
        myView ..
        ,  Html.map PlayerMsg (Midi.Player.view model.player) 
        ]

subscriptions

Similarly, you must map the subscriptions onto those of the MIDI player (alongside any subscriptions the program requires for other purposes):

  
    subscriptions : Model -> Sub Msg
    subscriptions model = 
      Sub.batch 
        [  mySubs ...
        ,  Sub.map PlayerMsg (Midi.Player.subscriptions model.player)
        ]

Final Integration

The complete source of the MIDI player module can be found here. An example of a final html file that integrates the javascript from the player module with that of a calling program named MidiFilePlayer can be found here.

Is elm a game-changer?

I'm a server-side developer. Whenever I venture into the web side of things, I'm faintly horrified at what I find - three disparate languages, ad hoc structure, events firing from different widgets often with no clear overall effect and so on. For me, the most frightening is javascript - I am sure there is a very competent functional language in there struggling to get out, but it's difficult to find.

Elm solves these problems. Html, css and javascript are simply encapsulated as pure functions, and all you need do is combine them together in straightforward ways and you get no surprises. All messages are routed sequentially through one single place, so that you simply pattern match on the signal stream and you thus retain control of your application's behaviour. Elm code compiles down to a safe subset of javascript in such a manner that you virtually never see any kind of runtime exception.  Also you get a very simple interactive development environment with truly informative compiler error messages.

This means that I can now approach the web tier in the same way that I approach the server tier - for the first time I'm feeling comfortable. I suppose this is elm's main value proposition - you can control both your application's look-and-feel and its behaviour and it's all in the context of a simple mental model.

But I want to approach things from left-field.  I'm less interested in how the application looks than in how it sounds.  Audio has been the poor relation in web programming for such a long time now, and elm is no exception.  However, once Evan comes clean about his plans for wrapping the web platform API, I fully expect audio to blossom in elm.  What I find exciting is that it's now possible to build, quite quickly, relatively sophisticated tools and have them run in the browser in a manner that I would previously have thought to be impossible.

Functional Parsers

I've now written two fairly substantial parsers - one for MIDI and the other for the ABC notation (which describes musical scores).  Both use the wonderful elm-combine  parser combinator library from Bogdan Popa. This has proven itself to be a great choice - not just because you can write practical parsers with it but also because of the great support Bogdan provides.

If you come from a javascript background, you're perhaps not too sure of what a functional parser is.  A traditional imperative parser is monolithic - it will usually employ a lexer for building tokens from the individual characters and a single parser for checking that the token stream fits the grammar and for building a parse tree.  By contrast, a functional parser is built from lots of very tiny parsers, combined together.  For example, you may have one that just parses a particular character sequence such as the word 'import' or another that just parses integers and so on.

It turns out that you can combine these little parsers together very simply.  Just to give one example, the ABC notation has a concept of a tuplet which is a generalisation of a triplet (three notes taking the time allotted to two) or a quadruplet (four notes taking the time allotted to three) and so on.  For example, a triplet for three successive notes on a keyboard from middle C would be notated as (3CDE.

You might want to represent it in a parse tree like this to imply that a tuplet consists of a signature part (e.g. 3 or 4) and a list of notes (e.g. A,B and C):

    
   type Music =
      Tuplet TupletSignature (List AbcNote)
    | ....

another way of looking at this is that Tuplet is a constructor function that builds a Music type and which has the signature:

    
    Tuplet : TupletSignature -> (List AbcNote) -> Music

Now, a parser for this part of the ABC language might look like this:

    
    tuplet : Parser Music
    tuplet = Tuplet <$> (char '(' *> tupletSignature) <*> many1 abcNote

I'm well aware that this looks like hieroglyphics and that elm discourages the use of infix operators like these but in the case of functional parsers, I think their use is entirely justified and, once you get used to them, makes life considerably simpler.

To deconstruct this statement - there are three primitive parsers - for the left bracket character, the tuplet signature (i.e. 3 or 4) and for a note in ABC notation (i.e. A, B or C). The abcNote parser is used with the many1 combinator which means that one or more instances of a note are recognised, producing a list. The *> and <*> operators mean that you process the parser at their left followed by the one on their right in sequence. The difference being that the first of these throws the left hand result away whilst retaining the right hand side result, whilst the second retains both results. So at this stage, we've successfully parsed (say) a triplet and retained the '3' and the 'ABC'. Finally, the <$> combinator just applies the Tuple constructor function we defined earlier to the two results, building the Music type. (All types are, however wrapped inside a Parser type). This is an example of what is called the applicative style.

The full parser for ABC is merely extends this technique to cover the entire grammar.

Web-Audio Players

Once you have a parse tree, it is then a relatively simple job to translate it into a set of instructions for playing each individual note (at the correct pitch and for the intended duration).  This can then be dispatched to web-audio so that the tune can be played.  Web-audio is supported in recent versions of most 'modern' browsers like Chrome,  Firefox and Opera.  Unfortunately, elm 0.16 has no simple way of integrating with javascript APIs like these and so you have to resort to the frowned-upon technique of wrapping a javascript integration layer inside an elm interface.  The hints coming from elm-dev suggest that this might be solved in 0.17. 

The net result of all this?  You can accept an ABC tune from a user, parse it, and if it is valid, play it immediately.  All this takes place in the browser.  All the applications that I know about that do this sort of thing do the processing on the server before returning something playable to the client.  So, my contention is that elm allows you to attempt applications in the browser that simply were not possible before (or at least not without superhuman effort).

If all this has given you a thirst for learning the ABC notation, there's an interactive tutorial here and an ABC editor here.  The code is here.