Web Crawling Powerhouse Datafiniti To Provide Web’s Datasets via SQL

Garrett Wilkin, October 11th, 2011

DatafinitiIf there’s a new dawn in the age of search, Datafiniti is the sun. They are the first to take a new and profound approach to search in years. Most search engines return a list of links to web pages, but Datafiniti has much bigger plans. Instead of links, they return a set of data. A search for a burrito in Texas would give you a list of restaurants complete with goodies like reviews in addition to the important bits like name and address. The dataset itself has been out there for years, but its the aggregartion and presentation that’s different. With the Datafiniti API, it is now possible for developers to easily integrate that data into web applications.

This is incredibly profound because the task of normalizing and serving data sets is a big one.  For any given subject area, there are multiple APIs to learn and master in order to collect just the most relevant information.  This is a real problem for programmers and Datafiniti is making an earnest effort tp solve it.  They provide one simple API call that consists of a token and an SQL query.  The results can be provided in JSON or CSV. This makes it possible for programmers to use Datafiniti as a great simplifying normalization point, but it also makes the data accessible to the moderately tech savvy spreadsheet user.  Datafiniti will serve copy and pastable datasets that will likely find their way into slide show presentations.  There’s a page dedicated toward helping people up the syntax learning curve and a screencast to walk you through some typical queries.

An important aspect of this service is that its built on a powerful web crawling architecture developed by 80legs which is now a part of Datafiniti. In a recent conversation CEO Shion Deysarkar described the crawling system as “completely transient and ephemeral cloud computing grid” which allows Datafiniti to operate nimbly and cheaply. It also gives their service a true competitive advantage. They’ve got a serious engine under the hood that should keep the data set healthy on all metrics of coverage, recency, and accuracy.  A high level of quality is Datafiniti’s stategy for adoption.

This team has experience, tried and tested technology, and a game changing idea. I can’t wait to see where they take it. Datafiniti is one of more than 200 search APIs, but they really stand head and shoulders above the rest.

Both comments and pings are currently closed.

5 Responses to “Web Crawling Powerhouse Datafiniti To Provide Web’s Datasets via SQL”

October 11th, 2011
at 3:01 pm
Comment by: Shion Deysarkar

Thanks for the awesome post, Garrett! We’ll be working hard to deserve your praise!

October 11th, 2011
at 6:50 pm
Comment by: Ron Colburn

Nice, a hybrid of Google Squared and YQL. Sounds promising.

October 16th, 2011
at 8:01 am
Comment by: 48 New APIs: Datafiniti, EPA Envirofacts and Social Games Mechanics Platform

[...] realtime public transit information and a book trading community service. In addition we covered Datafiniti’s efforts to provide the web’s datasets via SQL queries. Below are more details on each of these new [...]

October 16th, 2011
at 9:55 am
Comment by: 48 New APIs: Datafiniti, EPA Envirofacts and Social Games Mechanics Platform | News of Business – Marketing | Forex | Gold | Money

[...] realtime public transit information and a book trading community service. In addition we covered Datafiniti’s efforts to provide the web’s datasets via SQL queries. Below are more details on each of these new [...]

October 16th, 2011
at 12:06 pm
Comment by: 48 New APIs: Datafiniti, EPA Envirofacts and Social Games Mechanics Platform | USEFUL WEB STORE

[...] realtime open movement information and a book trade village service. In further we lonesome Datafiniti’s efforts to yield a web’s datasets around SQL queries. Below are some-more sum on any of these new [...]

Follow the PW team on Twitter

ProgrammableWeb
APIs, mashups and code. Because the world's your programmable oyster.

John Musser
Founder, ProgrammableWeb

Adam DuVander
Executive Editor, ProgrammableWeb. Author, Map Scripting 101. Lover, APIs.