When REST Gets Messy

Brian Cooksey / July 10, 2014

Back in January, we launched a new version of the Developer Platform – the place where companies can add their APIs to Zapier. Part of that project involved converting the existing views – standard Django forms – into Backbone.js views powered by a yet-to-be-written API. It was a fun project to be apart of, especially since it gave me a greenfield to try and build a RESTful API for the Backbone models to read/write to. What I didn't anticipate, however, was that there were going to be some awkward interactions to convert to resources. Here is a peek behind the curtains of where REST worked for us and where we had to cheat.

Blissful Beginning

Designing the API starts off great. There is an app, which has child triggers and actions. The URL structure for the endpoints falls naturally into /apps/<id> and /apps/<id>/triggers. Even better, each of those children are objects with their own properties that can be viewed, edited, and deleted. HTTP verbs map cleanly on to those actions.

With the first few resources complete, I get into a good rhythm. Find the next child resource, create two new endpoints at /items and /items/<id>, write some Django code to support the HTTP verbs I want, and boom; I have a working API for that resource. The existing Django forms prove useful for validation to boot. OK, so maybe I intentionally skip over a few pieces of the UI that don't quite fit this pattern, but that is no big deal. Later I will see how those are resources no problem. Besides, I have a bigger issue to tackle.

Needing Everything Now

If you take a look at the Platform, you see something like this when you click to view an app:

Notice that the page displays all the first-level children. This may not seem special at first, but remember I just split each of those children into separate resources. To acquire the data needed for that view, the frontend now needs to make a GET request for each child. That, or the listing endpoint needs to return verbose representations. What to do?

Welcome to the first point where we cheat REST a little. Backbone already makes a request to load the data for the app. In theory, that payload should include links to the child resources so the frontend can retrieve them with separate requests. However, for performance and simplicity, we don't want to wait on another 5-10 requests to load each child individually. So, we compromise. The /app/<id> endpoint returns a complete tree of an app and its descendants. To make it work seamlessly with Backbone, we override the parse() method to create the descendant models on load. When all is said and done, we have one network request give birth to a bunch of Backbone models, but each model continues life independently, doing PUTs and GETs to its own endpoint like normal.

Resource !== DB Row

With the site loading quickly and the bulk of the content converted to models, I turn my attention to a trickier section; the request logs. The log is a simple list of HTTP requests the app has made. It's useful for debugging while building your app. All it does is dump some data from Graylog (a logging service) onto the page. So how do I make this JSON blob fit into my nice REST API?

It sounds obvious now, but the answer is to make a request resource. When the user visits the list page, have a Backbone collection load /requests to get a list of recent requests. The backend adds IDs to the individual requests, so I am also able to have a /requests/<id> endpoint for Backbone to load a single request into a model.

What made this difficult to come up with? The data source. For this resource, the data comes from a different spot than the rest of the API. For everything else, the backend queries MySQL for the exact record it needs. In the case of the log, the backend actually queries for a chunk of log data and then manually parse out the request with the matching ID. Making a resource "on the fly" like this feels strange at first. Yet, Backbone does not care that the data comes from Graylog instead of MySQL. The API talks JSON and that is all Backbone wants.

Turning Processes into Resources

The final piece of the Platform that needs to be made over is the activation process. Now, to understand why this is a tough piece to tackle, let me briefly explain the old workflow:

1. When a user is ready to submit their app for activation, they click a link. A view runs some error checking and presents an html page with either a form to proceed or the errors
2. If all is OK with the app and the user submits the form, Zapier gets an email
3. The status of the app changes to pending

That is quite the process. Intensive error checking, posting forms not associated with any entity in our DB, emails being sent. The only resource-oriented portion of the whole thing is the last step where the app gets updated. What to do?

I'll be honest, this one takes me awhile. I have to really embrace the REST mindset to see how I can make this work. To do that, I begin asking myself a simple question about each step in the process: "what would I call this step in a conversation if I had to use a noun?"

The first breakthrough comes with the error checking. As I brainstorm words, I mumble to myself "audit." Hmmm, intriguing. An audit is a review of something to make sure it's up to spec. If I want to check my app for errors, GET /app/<id>/audit. The backend does the error check inline and don't persist the results, but Backbone doesn't care. It has an endpoint to query and it gets back JSON like usual. This actually fits quite nicely into the API.

Next up is that pesky form. It contains some contact details so we know who to reach out to with questions. Like the audit, this form does not correlate to any model or DB record on the backend. All the details get dumped into an email. In fact, this is actually similar to that request log problem I worked through earlier. In a sense, recording the form values into an email is persisting the data, just not to the usual spot. Backbone certainly won't notice as long as I can give it a consistent endpoint. So what noun to use? After some searching, I settle on "activation request", allowing POST /app/<id>/activation_request to shoot us that email. That's a big win for resources in my book!

The final bit to complete the process is to update the status of the app. Here is another spot where we decide to cheat. As soon as the activation request is sent, the app needs to move to the pending state. The proper way to do this in REST would be to have a second request like PUT /apps/<id> update the status. Unfortunately, enforcing that the second request occurs is difficult and more complex than updating the app's status server-side; as a side effect of sending us the activation request. It's not a clean solution (the creation of one entity also changes a property on a related entity), but it ensures our system stays in a valid state and that is more important to us than winning the Beautiful REST API pageant.

Where is the HATEOAS?

I'm aware that we only hit level 2 on the Richardson Maturity Model. By Roy Fielding's definition, this API is not a REST API. You can call it Resource-Oriented, but not RESTful. I am content with that distinction. Getting the frontend and backend communicating in terms of resources was a big gain in itself. To reach the next level of RMM, where hypermedia drives the process, seems like it would require a very different approach on the client side. I'm not sure how a framework like Backbone fits into that solution. I'm also not convinced that it makes sense for an internally consumed API.

By building the UI, we are already hard-coding the states the user can transition between. Having hypermedia links that match the rules enforced by the UI feels pedantic. If we want to get way out there, we could create the UI dynamically based off hypermedia links. It would be a cool technical feat, but it also means throwing out a ton of out-of-band information we possess given our position as author and consumer of the API. That sacrifice does not seem valuable at this point.

How is it Today?

After six months of battle testing, I'm happy with this style of architecture. It is comprehensible and most of the behaviors we need to support map cleanly to resources. Maintaining the API has also gone smoothly. Extending and tweaking to provide new functionality has been straightforward. If I had to do it again, I would still pick this solution.