The New York Times needed to modernize the infrastructure supporting its growing syndication service, so it turned to reactive programming darlings Scala, Play, and Akka.
The New York Times has been on the wrong side of the news lately, with changes in leadership and the leak of a sweeping internal report excoriating the publication's digital efforts.
Overlooked, however, is a fascinating blog by a pair of Times' technical team members. In their article, Suman Roy and Krishna Sundaresan reveal how the Times built an internal application — code-named Blackbeard — to manage the newspaper's enormous news service/syndication division. At the heart of Blackbeard are open-source technologies like Scala and Play, developed by Typesafe, which enables the Times to hit massive scale without breaking a sweat.
I caught up with these two to talk about Blackbeard and how the Times built out the system to accommodate its dizzying array of all the articles, commentary, images, and multimedia that are "fit to print" or — in this case — publish online.
TechRepublic: What were you trying to do with Blackbeard?
Roy: Our syndication business is growing quickly. We were relying on a desktop-based CMS that wasn't designed to work with our web publishing system. Our authors needed a new authoring and syndication tool. They needed a document editor where it was easy to track changes and had other features like real-time dashboards.
TechRepublic: At a high level, what were your goals with Blackbeard?
Roy: Our first goal was to eliminate wasted time waiting for the application to be deployed and the application server to start. Anyone should be able to have an instance of the app running locally in less than five minutes. We wanted faster idea prototyping, easier implementation of real-time features, and efficient article locking.
TechRepublic: Can you give me an example of what that looks like for a user of your application?
Sundaresan: Typically, multiple Blackbeard users edit or review an article in a very short period of time. We chose a model where one user exclusively locks the article while editing it and allows others to view it in read-only mode. In the past, we've implemented a lock as a row in a database table with a timestamp column. When a user opens an unlocked article, a row is inserted into the table. While this user has the article open (and has not gone idle), a periodic call updates the timestamp. Stale locks are deleted every few minutes.
This mechanism works well but results in a lot of writes to the database, connection overhead, and thread blocking from all the back end calls to update the timestamp. We wanted to implement locking without the connection and database overhead, and we were successful.
TechRepublic: I notice in your blog post that you chose a Play / Akka / Scala stack for your architecture. Why?
Roy: Since the team's expertise is in working with the JVM, we looked at languages and tools for that platform. We narrowed our choices down to two candidates: Grails and Play. While Grails has a much larger ecosystem, we found Play's websocket and web service APIs much simpler to use, and we loved Scala's elegance and expressiveness. Play's WebServices API allows us to seamlessly fetch data from other applications. Using Akka (with its Apache Camel integration), we're able to listen for article events published on ActiveMQ and then update client dashboards with just a few lines of code and in real time.
TechRepublic: What were some key lessons you learned in the process?
Sundaresan: We learned a few lessons along the way. The actors that syndicated content were initially configured to execute in a Future using Play's default execution context. This meant any network anomaly could result in both syndication and user requests being blocked because the controllers also shared the same execution context.
It turns out the solution was simple. We had the Futures execute in a different execution context. Maintaining state in an actor carries risks because the actors can crash with their states. The actor system takes care of recreating them with no effects on other parts of the system that refer to that actor.
At first glance, this is problematic for us, because each child actor keeps information about articles that a user is working on. But a quick read of the Akka documentation showed us that we could use the life cycle events (specifically the restart hooks) to ensure states are not lost.
The Play framework and the Scala language have made it fairly easy (and fun) for us to build this tool. You can can learn more about the details of the implementation from Victor Chan's presentation at the nescala conference.
If given the opportunity, what other questions would you ask Roy and Sundaresan about Blackbeard? Let us know in the discussion thread below.