Cloud Computing May Decrease Your API Call Limit

Tomas Vitvar, March 31st, 2011

GoogleI recently wrote an AppsScript that sends an e-mail every time a new tweet is posted on a specified Twitter account and encouraged you to copy it and use it on your own. I run it in Google Spreadsheets every 5 minutes and after several weeks I started to receive error messages that the script exceeds a limit of 150 API calls per hour imposed by Twitter. The script always fetches the latest tweets from a Twitter account in one API call per a single run, thus according to my configuration settings it should make a maximum of 12 API calls per hour. Wonder what is wrong here?

Google Spreadsheets together with AppsScript is a Platform as a Service environment that you can use to automate tasks across Google and third-party services. You can utilize JavaScript together with a set of Google APIs to access various Google resources such as contacts or documents as well as use protocols such as OAuth. You do not, however, have any access to underlying infrastructure that empowers the execution of your code. For example, you can write a script that fetches data from multiple resources, integrates and aggregates the data and populates a spreadsheet with results. You can then access the results via a Web interface or programmatically by using Atom feeds, and you can configure your script as a job.

When you run your script as a job, the infrastructure automatically schedules its execution. As a result, the script may run every time with a different IP address. If there are multiple instances of your script (such as in my case several users who copied the script and run it regularly), it is likely that several of such instances will share one or more IP addresses throughout all executions. Now assume that your script accesses an API that has a call limit per some time interval accounted per an IP address. Executions of the script’s instances running from a single IP address in the same time interval will thus share the API’s call limit. And this is exactly what happens with my script.

In order to get more details on this issue, we have performed a simple experiment to get a distribution of IP addresses over time that Google infrastructure uses to run the scripts. We run a total of 16 instances of a script for a period of 20 days where each script instance makes a single API call to our service that logs the script’s IP address. We use 16 Google accounts that each have installed the script and run it as a job every 5 minutes.

GAE-ip-chart

The chart above shows the distribution of IP addresses during this experiment. For the period of 20 days there was a total of 120 IP addresses assigned to around 5,500 executions of the script.

Thanks to Milan Dojchinovski and Jaroslav Kuchar who helped with the experiment.

Both comments and pings are currently closed.

One Response to “Cloud Computing May Decrease Your API Call Limit”

April 1st, 2011
at 3:30 am
Comment by: Richard Hughes

I believe that this is a well known problem, and I have seen several discussions about it with respect to Google App Engine. In my view, it is a deficiency of the Twitter API for not being cloud-friendly – it is hardly an uncommon requirement to call Twitter from a cloud-hosted app now.

Some of the Twitter API limits are based on user accounts, which are OK, but the standard search is IP-address limited.

Follow the PW team on Twitter

ProgrammableWeb
APIs, mashups and code. Because the world's your programmable oyster.

John Musser
Founder, ProgrammableWeb

Adam DuVander
Executive Editor, ProgrammableWeb. Author, Map Scripting 101. Lover, APIs.