CORS Support in Spray

My tradtunedb service is working nicely, but there is a problem with it.  In order to hear a midi tune being played, it has to be rendered.  At the moment, this is done by converting it to a wav file on the server - an expensive process which doesn't scale well.  A better solution is to render the tune in javascript on the browser - but this presents a further problem.  The architecture is such that the front- and back-end have different host names.  And this means that the browser will prevent an XMLHttpRequest to the back-end because of potential security breaches.

Why CORS?

Browsers all maintain a same-origin policy.  One reason this is necessary is that XMLHttpRequest passes the user's authentication tokens. Suppose he is logged in to theBank.com with basic auth and then visits untrusted.com. If there were no restrictions, untrusted.com could issue its own XMLHttpRequest to theBank.com with full authorisation and gain access to that user's private data.

But a blanket ban on cross-origin requests also prohibits legitimate use.  For this reason, most modern browsers support Cross-origin resource sharing (CORS). . This is a protocol which allows the browser to negotiate with the back-end server to discover whether or not such requests from 'foreign' domains are allowed by the server, and to make the actual requests.

When browsers issue requests, they always include the Origin header.  The essence of the protocol is that the server will respond with an Access-Control-Allow-Origin header if that Origin is acceptable to it.  Browsers will then allow the access to go ahead.  For example, here we have an XMLHttpRequest emanating from a script named midijs and hosted on a server running on localhost:9000:

Accept:*/*
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-GB,en-US;q=0.8,en;q=0.6
Cache-Control:no-cache
Connection:keep-alive
Host:192.168.1.64:8080
Origin:http://localhost:9000
Pragma:no-cache
Referer:http://localhost:9000/midijs
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/28.0.1500.71 Chrome/28.0.1500.71 Safari/537.36

And here is a response from a back-end server that accepts the Origin:

Access-Control-Allow-Credentials:true
Access-Control-Allow-Origin:http://localhost:9000
Content-Length:2222
Content-Type:audio/midi; charset=UTF-8
Date:Sun, 29 Sep 2013 09:13:03 GMT
Server:spray-can/1.1-20130927

This is all that is necessary for GET requests.  Security issues are greater for requests that are not used for pure retrieval and/or where user credentials are supplied.  Here, CORS provides additional headers and a 'preflight request' mechanism which asks permission from the server before it makes the actual request. For more details see http://www.html5rocks.com/en/tutorials/cors/

CORS Support in Spray

The M8 release versions of Spray have no CORS support.  Initial support for the CORS headers has now been integrated into the forthcoming release of Spray. These can be accessed if you develop your own CORS directive which makes use of respondWith directives in order to set the appropriate headers. For example, here's an approach based on that of Cristi Boarlu that works very well:

import spray.http._
import spray.routing._
import spray.http.HttpHeaders._

trait CORSDirectives  { this: HttpService =>
  private def respondWithCORSHeaders(origin: String) =
    respondWithHeaders(      
      HttpHeaders.`Access-Control-Allow-Origin`(SomeOrigins(List(origin))),
      HttpHeaders.`Access-Control-Allow-Credentials`(true)
    )
  private def respondWithCORSHeadersAllOrigins =
    respondWithHeaders(      
      HttpHeaders.`Access-Control-Allow-Origin`(AllOrigins),
      HttpHeaders.`Access-Control-Allow-Credentials`(true)
    )

  def corsFilter(origins: List[String])(route: Route) =
    if (origins.contains("*"))
      respondWithCORSHeadersAllOrigins(route)
    else
      optionalHeaderValueByName("Origin") {
        case None => 
          route        
        case Some(clientOrigin) => {
          if (origins.contains(clientOrigin))
            respondWithCORSHeaders(clientOrigin)(route)
          else {
            // Maybe, a Rejection will fit better
            complete(StatusCodes.Forbidden, "Invalid origin")
          }      
        }
      }
}

And this is how it can be used in my case to protect a route which generates a response for a path that requests a particular file type (e.g. midi):

path(Segment / "tune" / Segment / Segment ) { (genre, tuneEncoded, fileType) =>  
  get { 
    val contentTypeOpt = getContentTypeFromFileType(fileType:String)
    if (contentTypeOpt.isDefined) {
      respondWithMediaType(contentTypeOpt.get.mediaType) { 
        corsFilter(MusicRestSettings.corsOrigin) {
          val tune = java.net.URLDecoder.decode(tuneEncoded, "UTF-8") 
          val futureBin = Tune(genre, tune).asFutureBinary(contentTypeOpt.get)
          _.complete(futureBin)
        }
      }
    }
    else {
     failWith(new Exception("Unrecognized file extension " + fileType))
    }
  } 
} ~    

There is also a discussion started by Tim Perrett which hopes to be able to wrap routes within a cors directive which will just 'do the right thing' with preflight requests and appropriate access control headers as unobtrusively as possible.

If you want to try this out yourself, you can try one of the release candidates that were published on October 23rd.

Embedded YouTube Videos and Chrome

I have just discovered a related problem to do with rendering embedded YouTube videos in Chrome. I want to extend my trad tunes web site with a comments page attached to each tune. The foot of the page will contain a form allowing the user to add a comment to the page, and this is particularly useful if she finds a video where the tune is being played. YouTube allows you to embed the video into the page which it manages by means of an iframe. Unfortunately, when she posts the tune, Chrome won't display it until the page is refreshed (although all other major browsers will). Instead it reports:

The XSS Auditor refused to execute a script in 'http://localhost:9000/genre/agenre/tune/atune/comments' because its source code was found within the request. The auditor was enabled as the server sent neither an 'X-XSS-Protection' nor 'Content-Security-Policy' header.

What's happening is that the browser must run javascript code to render the iframe, and it again thinks that it emanates from the server that provided the page which is thus making a cross-origin request. In this case, the solution is to add a request header instructing the browser to suspend the XSS Auditor check when it process the page. This is achieved with the X-XSS-Protection header. For example, the Play Framework Action that processes a valid comment might respond like this:

  
    val comment = commentForm.bindFromRequest.get
    commentsModel.add(comment)
    Ok(views.html.comments(commentsModel.getAll, commentForm, genre, tune)(userName)).withHeaders("X-XSS-Protection"->"0")

When Chrome sees this header, it immediately renders the video. Note, this technique doesn't work if you issue a redirect instead of responding directly.

Spray Migration

Firstly, apologies for having been silent for such a long time - I've been looking after my parents who were ill and are now, thankfully, recovered. I'd like to say a huge thank-you to Pinderfields Hospital in Wakefield, Yorkshire for saving my dad's life.

Since I've been away, the Scala ecosystem has been moving on unabated. We now have the final release of Scala 2.10.2 and I've been anxious to upgrade my RESTful music application accordingly. This has been enjoyable, but has taken longer than expected, largely because Spray underwent a huge refactoring at version 1.0-M3 which incorporated lots of breaking changes. As of this writing, the current release is labelled 1.2-M8 and is consistent with Akka 2.2.0-RC1. Other libraries that I use have changed too, but Spray has caused the most impact because the application concentrates on content negotiation and Spray has altered this considerably.

The New Spray

So much is new.  The documentation has been completely overhauled and is now much more detailed. If you look at its navigation panel you can get an idea of the massive refactoring into a new module structure that has taken place. Both the github repository and the Spray repo have moved.  Packages have been renamed from “cc.spray...” to simply “spray...” and the group id has changed from “cc.spray” to “io.spray”.  There is a new DSL for testing routes. However, the changes with the biggest impact for me have been those to do with (un)marshalling (which has been moved to the new spray-httpx module) and those handling authentication.

Marshalling

You now use the Marshaller.of method to create a marshaller for a particular type (and which can be picked up implicitly in the routing). For example, if I want to create a marshaller for a handful of different binary types that I use represented by BinaryImage:

  import spray.httpx.marshalling._
  import MediaTypes._

  class BinaryImage(mediaType: MediaType, stream: BufferedInputStream)

  implicit val binaryImageMarshaller = 
     Marshaller.of[BinaryImage] (`application/pdf`,
                                 `application/postscript`,
                                 `image/png`,
                                 `audio/midi`) {
       (value, requestedContentType, ctx) ⇒ {
           val bytes:Array[Byte] = Util.readInput(value.stream)
           ctx.marshalTo(HttpEntity(requestedContentType, bytes))     
           }
     }

Alternatively, if you have a type A with no marshaller, but you do have a marshaller for a type B, you can provide a marshaller for A by delegating to B's marshaller if you can provide a function from A to B. The definition is:

  def delegate[A, B](marshalTo: ContentType*)
                    (f: A => B)
                    (implicit mb: Marshaller[B]): Marshaller[A]

In my case, nearly all the content is represented as a type Validation[String, A] where the String type represents an error and A represents the various types of source or intermediate content. You can provide a meta-marshaller for a Validation which either handles the error or invokes the marshaller for the underlying type:

implicit def validationMarshaller[B](implicit mb: Marshaller[B]) =
    Marshaller[Validation[String, B]] { (value, ctx) ⇒
      value fold (
        e => throw new Exception(e),
        s ⇒ mb(s, ctx)
      )
    }

It seems rather infra dig to have to throw an exception from a Validation. At the moment it is necessary to do so because a Marshaller has no access to the HTTP response headers and so it is not possible (say) to marshall an error to a string-like content type if you've already told the marshaller that it's only dealing in binaries. However, Mathias from Spray has proposed introducing alternative Marshallers, one of which will have access to the headers in an upcoming release so this small blemish can then be removed.

Unmarshalling

Unmarshalling goes in the opposite direction and can be used to construct a user-defined type from a form submission. The interface here has changed a little too. For example:

  implicit val AbcUnmarshaller =
    Unmarshaller[AbcPost](`application/x-www-form-urlencoded`) {
      case HttpBody(contentType, buffer) => {         
        val notes =  new String(buffer, contentType.charset.nioCharset)
        AbcPost(notes)
      }
    }

As you can see, there is now a nice symmetry between Marshallers and Unmarshallers. They can be used within routes like this:

  post {    
    entity(as[AbcPost]) { abcPost =>     
       // do something

And as with marshallers, there are a whole set of basic unmarshallers provided and you can delegate between unmarshallers if you wish.

Authentication

This is another area where the API has changed and here the documentation is lacking. In my case, I need to perform simple authentication of a user name and password by checking with a backend database. This is best achieved by providing a UserPassAuthenticator. You construct one of these by passing it a function that takes an Optional UserPass and returns a Future which contains an Optional result. The standard approach is to return an encapsulation of the user name in a BasicUserContext if the validation passes and return None otherwise:

  val UserAuthenticator = UserPassAuthenticator[BasicUserContext] { userPassOption ⇒ Future(userPassOption match {
        case Some(UserPass(user, pass)) => {
            try {
              println("Authenticating: " + user + ":" + pass)
              if (TuneModel().isValidUser(user, pass) ) Some(BasicUserContext(user)) else None
            } 
            catch {
              case _ : Throwable => None
            }
        }
        case _ => None
      })
    }

I also need to identify the administrator user who has special powers. I do this by first using the UserAuthenticator to establish the login credentials and then composing this future with a check to establish the administrator name:

   
    val AdminAuthenticator = UserPassAuthenticator[BasicUserContext] { userPassOption => {
      val userFuture = UserAuthenticator(userPassOption)
      userFuture.map(someUser => someUser.filter(bu => bu.username == "administrator"))  
      }
    }

Once you have an authenticator, you can use it in routes like this:

   get {     
     authenticate(BasicAuth(UserAuthenticator, "musicrest")) { user =>    
       _.complete("user is valid")    
     }
   }

Testing

As an example of the new testing DSL, here is how you can test the authentication behaviour:

package org.bayswater.musicrest

import org.specs2.mutable.Specification
import spray.testkit.Specs2RouteTest
import spray.http._
import spray.http.HttpHeaders._
import StatusCodes._
import spray.routing._
import org.bayswater.musicrest.model.{TuneModel,User}

class AuthenticationSpec extends RoutingSpec with MusicRestService {
  def actorRefFactory = system
  
  val before = insertUser

  "The MusicRestService" should {
    "request authentication parameters for posted tunes" in {  
       Post("/musicrest/genre/irish/tune") ~> musicRestRoute ~> check 
         { rejection === AuthenticationRequiredRejection("Basic", "musicrest", Map.empty) }
    }
    "reject authentication for unknown credentials" in {
       Post("/musicrest/genre/irish/tune") ~>  Authorization(BasicHttpCredentials("foo", "bar")) ~> musicRestRoute ~> check 
         { rejection === AuthenticationFailedRejection("musicrest") }
    }
    "allow authentication for known credentials (but reject the lack of a POST body as a bad request)" in {
       Post("/musicrest/genre/irish/tune") ~>  Authorization(BasicHttpCredentials("test user", "passw0rd1")) ~> musicRestRoute ~> check 
         { rejection === RequestEntityExpectedRejection }
    }
    "don't allow normal users to delete tunes" in {
       Delete("/musicrest/genre/irish/tune/sometune") ~>  Authorization(BasicHttpCredentials("test user", "passw0rd1")) ~> musicRestRoute ~> check 
         { rejection === AuthenticationFailedRejection("musicrest") }
    }
    "allow administrators to delete tunes" in {
       Delete("/musicrest/genre/irish/tune/sometune") ~>  Authorization(BasicHttpCredentials("administrator", "adm1n1str80r")) ~> musicRestRoute ~> check 
         { entityAs[String] must contain("Tune sometune removed from irish") }
    } 
  }
   
  def insertUser = {
   val dbName = "tunedbtest"
   val collection = "user"
   val tuneModel = TuneModel()
   tuneModel.deleteUser("test user")
   tuneModel.deleteUser("administrator")
   val user1 = User("test user", "test.user@gmail.com", "passw0rd1")
   tuneModel.insertPrevalidatedUser(user1)    
   val user2 = User("administrator", "john.watson@gmx.co.uk", "adm1n1str80r")
   tuneModel.insertPrevalidatedUser(user2)    
  }
   
}