Netflix continues to see tremendous growth in its Netflix API. The API which saw humble beginnings in 2008 has taken a giant leap in usage, ever since Netflix started streaming functionality to devices. In what could be one of the largest usages of a public API, the current number of requests for its API is around 20 billion per month and increasing by the day, forcing Netflix to look at measures to decrease them.
In December, we covered how Netflix was able to scale significantly by moving its API and other services over to Amazon Web Services (AWS). In an interesting blog post, Director of Engineering Daniel Jacobson has given the API numbers and laid out plans on how they are going to tackle this deluge of API usage.
Just look at the numbers. A 37x growth in 13 months, 20+ billion API requests in January of this year and the number may even be higher by now. One way to look at these numbers would be continuously boost your infrastructure requirements to anticipate higher load/usage. Netflix, however, wants to get to the root of the issue and as highlighted in the blog entry, its has set two clear goals for its API redesign after seeing these numbers:
It is good to see this discussion in the open. Netflix acknowledges that the number of resources that are being served via its REST API is too many (around 20) and with overlapping needs, it results in several series of calls from the device to its API before all the information is obtained. Hence, the need to look at these resources closely and design them in such a way that with minimum number of requests, the device gets what it needs. The flip side to reducing the number of requests would be that the payload may start getting bigger. To address this, Netflix is looking at approaches like partial response and even providing a SQL-like variability for their database, so that devices get what they need.
Netflix is one of the best case studies of how APIs are becoming the cornerstone around which engineering makes its plans. It comes as no surprise that heavy usage of an API presents its own challenge in making sure that infrastructure and API design will be able to cater to unpredictable load in the future. Sometimes it pays to think about the basics like Netflix has done to look at reducing the number of requests and even payload size.
Image Credits : The Netflix Tech Blog