At TrueAccord, we take our service availability very seriously. To ensure our service is always up and running, we are tracking hundreds of system metrics (for example, how much heap is used by each web server), as well as many business metrics (how many payment plans have been charged in the past hour).

We set up monitors for each of these metrics on Datadog, that when triggered, will page an on call engineer. The trigger is usually based on some threshold for that metric.

As our team grew and more alerts were added we noticed three problems with Datadog:

1. Any member of our team can edit or delete alerts in Datadog’s UI. The changes may be intentional or accidental, though our team prefers to review changes before they hit production. In Datadog, the review stage is missing.
2. Due to the previous problem, sometimes an engineer would add a new alert with uncalibrated thresholds to datadog to get some initial monitoring for a newly written component. As Murphy’s law would have it, the new alert would fire at 3am waking up the on call engineer, and it may not even indicate a real production issue, but a miscalibrated threshold. A review system could better enforce best practices for new alerts.
3. Datadog also does not expose a way to indicate that an alert should only be sent during business hours. For example, for some of our batch jobs, it is okay if they fail during the night, but we want an engineer to address it first thing in the morning.

To solve these problems, we made DogPush. It lets you manage your alerts as YAML files that you can check in your source control. So you can use your existing code review system to review them, and once they’re approved they get automatically pushed to DataDog — Voila! In addition, it’s straightforward to setup a cron job (or a Jenkins job) to automatically mute the relevant alerts outside business hours.
DogPush is completely free and open source – check it out here.

When we started working on TrueAccord, we had a limited understanding of various technical aspects of the problem. Naturally, one of those unclear aspects was the data model: what data entities we will need to track, what will be their relationships (one-to-one, one-to-many, and so on), and how easy it is going to be to change the data model as business requirements become known and our domain expertise grows.

Using Protocol Buffers to model the data your service uses for storage or messaging is great for a fast-changing project:

1. adding and removing fields is trivial, turning an optional field into a repeated field and so on. If we modeled our data using SQL, we will be constantly migrating our database schema.
2. the data schema (the proto file) serves as an always up-to-date reference documentation for the service’s data structures and messages. People from different teams can easily generate parsers for almost every programming language, and access the same data.

Continue reading “ScalaPB: TrueAccord’s Protocol Buffer code generator for Scala” →

At TrueAccord, we use Play to develop our backend. Given our development environment, we need to have part of our URL space routed to a different server written in Python. We initially thought of setting up a lightweight HTTP server like nginx that would act as a reverse proxy for both of our development servers, which is a reasonable solution. However, we also wanted to avoid having yet another moving part in our development environment and were curious if we could write something quick in Scala that could achieve this.

As it turns out, writing this little reverse proxy in Scala/Play is relatively straightforward. It’s also pretty impressive that with so few lines of code we get a reactive proxy server that streams the content continuously to the end client while chunks of it are still arriving from the upstream server. A more traditional (and time-intensive) implementation would have buffered the entire upstream response until it was complete and only then sent it to the client..

So, without further ado, here is the code:

In line 14, proxyRequest.stream returns a Future[(WSResponseHeaders, Enumerator[Array[Byte]])]. This means that at some point in the future, our closure at line 15 will get called and will be supplied two things: the headers returned from the upsteam server (WSResponseHeaders) and an Enumerator[Array[Byte]], which is a producer of arrays of bytes. Each array of byte that it will produce is a part of the response body from the upstream server. Conveniently, Play provides a Result constructor that takes producers like this and turns them into responses that can be served to the end client.

flattenMultiMap is a little helper function that converts the query string parameters from the collection type they are given by Play requests to the format expected by WS.url.

Pretty cool, eh?