Yahoo’s YQL Makes the Internet Your Database

John Musser, April 30th, 2009

Yahoo Query LanguageYahoo has just released a major update to YQL, the Yahoo Query Language platform they first launched late last year as part of their Yahoo Open Strategy. YQL is a SQL-like programming interface to all Yahoo data that can also support non-Yahoo data as well (think of queries that look like: select id from flickr.photos.search where text=’car’). This week’s release adds a set of new features called Yahoo Execute which begin putting in place more pieces of a powerful cloud-based development platform.

As background, it helps to understand a bit about YQL. Their blog post announcing this new release gives a good, concise summary:

The Yahoo! Query Language lets you query, filter, and join data across any web data source or service on the web. Using our YQL web service, apps run faster with fewer lines of code and a smaller network footprint. YQL uses a SQL-like language because it is a familiar and intuitive method for developers to access data. YQL treats the entire web as a source of table data, enabling developers to select * from Internet.

Another way to look at it: if you consider Yahoo Pipes as the GUI-centric way to mashup data, YQL exists much closer to the programmer and code level.

As part of YQL, Yahoo also offers something called Open Data Tables (ODT), an extensible architecture for creating structured definitions of all sorts of online data, not just Yahoo data. This lets anyone make data YQL-accessible. There’s a growing community repository of these definitions over on Github with about 70 new ‘table definitions’ available there now for sources ranging from Yahoo to bit.ly to Zillow.

Now, with this week’s update, Yahoo builds on top of ODT via the “Execute” piece of the puzzle. Execute definitions let developers add arbitrary code that will run during the processing of any YQL statement. The code itself is server-side JavaScript with E4X (native XML support). Here’s a bit more on how this works:

With Execute, developers now have full control of how the data is fetched into YQL and how it’s presented back to the user. With Open Data Tables, developers can build tables that manipulate, change, and sign the URLs to access almost any protected content, allowing YQL access and combining data across a variety of different authenticated services such as Netflix or Twitter. Developers can call multiple services and data sources within Execute to join and mashup data however they desire, letting Yahoo! do the work rather than their applications. Data can be tweaked and manipulated into an optimal format for applications to consume.

This means developers can start writing server-side code on Yahoo’s infrastructure. Large scale joins and data manipulation can occur on Yahoo’s cloud, but the results can also can be integrated back as part of a developer’s app elsewhere. Or, with the Pipes YQL module, it can be integrated into Yahoo Pipes. For now the service provides read-only access to data and APIs, but it makes sense that they would add write capabilities before long.

As with other Yahoo platform services, as they’ve recently done with the BOSS search API, YQL may have both free and paid levels of access, with a reasonably high bar before hitting the paid level.

As Yahoo continues to add pieces to their Open Strategy it will be interesting to see how developers begin using these increasingly sophisticated tools.

Tags: Yahoo, cloud
Both comments and pings are currently closed.

14 Responses to “Yahoo’s YQL Makes the Internet Your Database”

May 1st, 2009
at 9:08 am
Comment by: The Product Guy’s Weekend Reading (May 1, 2009) « The Product Guy

[...] http://blog.programmableweb.com/2009/04/30/yahoos-yql-makes-the-internet-your-database/ Yahoo’s YQL Modular Innovation knitting together other Modular Innovations of the Web. [...]

May 2nd, 2009
at 10:09 am
Comment by: Recent Links at Consulting, Online Communities, and Social Media: Fast Wonder Blog

[...] Yahoo’s YQL Makes the Internet Your Database [...]

May 2nd, 2009
at 2:42 pm
Comment by: Top 10 YQL execute links

[...] http://blog.programmableweb.com/2009/04/30/yahoos-yql-makes-the-internet-your-database/ [...]

May 4th, 2009
at 2:33 am
Comment by: Yahoo’s YQL Makes the Internet Your Database « I.T News & Stuff

[...] Link [...]

May 4th, 2009
at 4:31 am
Comment by: YQL – Yahoo Query Language - Guru Stop

[...] You can also check this for more details. [...]

May 4th, 2009
at 9:18 am
Comment by: YQL make’s the internet your database « `wyldKard’s

[...] Read on… [...]

May 16th, 2009
at 8:22 am
Comment by: Yohay

Quite an interesting development. I’m wondering how I can utilize YQL for forexcrunch.com

July 9th, 2009
at 2:56 am
Comment by: Yahoo Extends YQL: Adds INSERT Command to the Internet

[...] a great improvement to its Yahoo Query Language (YQL) platform (our YQL API Profile). As you may remember from earlier this year, YQL is a SQL-like programming interface to lots of Yahoo data and APIs that can also support [...]

September 7th, 2009
at 10:54 pm
Comment by: This one’s for all you SQL gurus… « Ask the PropellerHeads

[...] ProgrammableWeb explains how YQL fits into Yahoo’s overall strategy of opening up data on the web in this post. [...]

September 11th, 2009
at 4:34 pm
Comment by: blog.data-directions.com » Blog Archive » This one’s for all you SQL gurus…

[...] ProgrammableWeb explains how YQL fits into Yahoo’s overall strategy of opening up data on the web in this post. [...]

February 16th, 2010
at 1:19 pm
Comment by: Google Buzz Now Available via Yahoo’s YQL

[...] time job, not mention trying to write code across all of them. Yahoo thought so too, and so they created Yahoo Query Language (YQL), which offers a simplified, SQL-like interface across not only Yahoo’s APIs, but third-party [...]

February 16th, 2010
at 2:53 pm
Comment by: YQL – Yahoo Query Language | GuruStop.NET

[...] You can also check this for more details. [...]

April 23rd, 2010
at 1:56 am
Comment by: Yahoo to Developers: YQL Will Soon Be The Gateway to All Yahoo Data Services

[...] also offers some advanced features like Execute definitions, which were highlighted in one of our previous posts on YQL. Related ProgrammableWeb ResourcesLearn more Yahoo Query Language API [...]

December 13th, 2011
at 12:25 pm
Comment by: elias

YQL only will be a valuable tool if yahoo decides to release the web service to the public, so that anyone can run a YQL web service. It does not make sense to put all of your data on yahoo’s servers. I have plenty of disk space I do not need yahoo! It will make life easier for anyone who want to extract data from any place, not only yahoo servers. Maybe I’ll take my time to develop such a thing.

Follow the PW team on Twitter

ProgrammableWeb
APIs, mashups and code. Because the world's your programmable oyster.

John Musser
Founder, ProgrammableWeb

Adam DuVander
Executive Editor, ProgrammableWeb. Author, Map Scripting 101. Lover, APIs.