The Argument for Real-Time REST APIs

Guest Author, March 5th, 2013

This guest post comes from Bryan Helmig, a self-taught hacker and co-founder of Zapier. By day he hacks on APIs, by night he hacks on APIs (its kind of his thing). Read more at Zapier’s blog and bryanhelmig.com.

ZapierAPIs enable incredible customization. No longer are customers locked into a specific interface or feature set: they can modify and augment core functionality as they see fit. It is incredibly liberating, but just deploying any old API isn’t enough. You need to really understand how people use your API. Luckily, you can group most API usage into one of two buckets: reading and writing.

For the most part, writing into an API is straightforward: you bundle all the data up and POST (or PUT/PATCH) it to an endpoint. This creates or modifies the record. Easy enough.

However, if you want to read data (specifically changing data), more nuance is required. Due to the fact that most APIs are “request based” (meaning, only give me data when I ask for it), you need to maintain some sort of state between requests. Any developer worth his salt will tell you that maintaining state gets complicated.

Unfortunately, most REST APIs do very little to help the average developer with this stateful problem. The API may have various endpoints that return data, but it just means means the developer is left to filter and sort through the data himself. It’s a lot of overhead to find new data.

In the end, your customers are spending more developer resources trying to use your API instead of simply getting value from the product. This is backwards. They should be spending money on the product because it makes their life easier and not wasting a developers time fighting some API.

However, you can help fix this by taking your product’s API to the next level: real-time.

What is Real Time?

Instead of forcing the developer to write complex code that requires periodic polling, which is just plain wasteful for everyone involved, the API instantly sends new data via a simple callback.

Real-time is extremely powerful as developers can get “push notifications” of events in the product. This level of fidelity means customers can react to your product instantly and integrate it more deeply into their processes.

Let’s use an example or two, take any CRM (like Highrise) for example:

The CRM may have a well-supported REST API, but it still suffers from the classic problem: customers have to constantly poll for new changes. This is bad because…

  1. 1. …it adds an artificial delay (which is horrible for warm leads).
  2. 2. …it wastes the customer’s and vendor’s resources to constantly check for data that didn’t change.
  3. 3. …it requires customers write/run/maintain complicated code that manages state.

What’s worse, the points are at odds! If a customer wants to lower their delay time by ~80% they’ll waste ~5x more resources doing it (if the API even allows such a high call rate).

Moving on to another example, this time Dropbox:

At the time of this writing, Dropbox has a powerful endpoint called /delta that lists changes since a specific point in time (basically, since you’ve last check). I assume this endpoint powers many applications, perhaps even their native desktop app. However, the first time around, the endpoint walks through the entire history of that particular Dropbox account. For an old, heavily used Dropbox account, the metadata alone can be hundreds of thousands of entries and hundreds of megabytes in size.

With a CRM like Highrise or an app like Dropbox, simple tasks like “do something with new things” would require a lot of wasteful work to maintain state. Perhaps enough to warrant a switch to a more flexible application, which is the worse case scenario.

Real-time would solve this. It’s also not that hard to implement, dozens of vendors do it already. Traditionally, they’ve been called webhooks and customers had to configure them individually by hand, but they can be better.

We like to call them “REST Hooks”.

What are REST Hooks?

We’re about to get a little nerdy, so hold on…

REST Hooks are a straightforward way to add real-time to an existing REST API without adding too much complexity or creating some arbitrarily unique protocol. Plus, it ensures existing customers of your REST API have an immediate understanding on how to use it. Finally, it reuses plumbing already in place for REST APIs like authorization and JSON/XML serialization, which helps lowers implementation friction.

Hooks can be minimally defined by two pieces of data:

  1. 1. What event should trigger a notification? (EG: contact.created or ticket.updated)
  2. 2. What URL should the notification be sent to? (EG: http://example.com/ping.php)

To implement REST Hooks, just add another resource to the existing API so customers can manage their hooks via standard GET/POST/PUT/DELETE verbs. So right alongside /api/v1/contacts and /api/v1/tickets would be /api/v1/hooks.

The real magic happens behind the scenes: whenever an event happens (IE: a contact is created or a ticket is updated), just check if there are any Hooks that reference that event. If there are, serialize the data and POST it to the attached URL.

Since REST Hooks adhere to the REST philosphy and not to some rigid specification, a lot more metadata can be stored as well (like shared secrets, complex filters, time to live, etc…). Event name and URL are just two required bits of information to make the system go.

Who does real-time right?

Some vendors already do this (and developers love them for it). A great example is Wufoo: they have stellar support for Hooks baked right into their API and have for years. Developers love them.

Wufoo doesn’t stand alone though. MyEmma in conjunction with Zapier added REST Hooks to their platform. ButSolve360 just added REST Hooks, and applications like PodioChargifyShopifyCampaignMonitor and dozens more have support baked right in or are adding it now.

The API world changes quickly and within a year or two you can be sure that REST Hooks will be the standard way to react to data within applications. The question is, will your API be one of the first to the party or will it come in last?

Both comments and pings are currently closed.

2 Responses to “The Argument for Real-Time REST APIs”

March 5th, 2013
at 1:53 pm
Comment by: Jordan

But isn’t checking for webhooks polling?

March 6th, 2013
at 12:56 am
Comment by: Sérgio Costa

Disclaimer: I work for Realtime.co

There are solutions that do this already. Realtime.co is one of them.

Follow the PW team on Twitter

ProgrammableWeb
APIs, mashups and code. Because the world's your programmable oyster.

John Musser
Founder, ProgrammableWeb

Adam DuVander
Executive Editor, ProgrammableWeb. Author, Map Scripting 101. Lover, APIs.