Twitter Search History Dwindling, Now at Four Days

Adam DuVander, August 12th, 2010

TwitterDevelopers looking for trends in Twitter search are finding it more difficult now that the micro-blogging site has decreased the search history to four days. Previously going back weeks and months, the backlog has steadily decreased, now too short for some types of applications. At the same time, the newer streams have become the go-to API for former search use cases.

Back when you searched Twitter with Summize, the history went back several months. Once Twitter acquired Summize, that continued. Though, as Twitter became more popular, the history decreased. It appears to fluctuate, based on how long it takes to hit its maximum storage. Twitter stores tweets in MySQL and recently dropped a planned move to Cassandra, a system open sourced by Facebook, which some believe could improve Twitter’s search performance.

Twitter refused to comment, citing a policy against making user statistics public. However, its own documentation lists the limit at 1.5 weeks. The same page claimed a history of one month, then three weeks, in early 2009. It was last updated in March. The drop to less than one week likely happened just last month. Damon Cortesi noted the change in a tweet.

Cortesi’s company makes RowFeeder, a tool that performs social media monitoring, adding references to a spreadsheet. The service uses Twitter, among other sources, and now uses the streaming API for the bulk of its work. However, “when new customers sign up,” Cortesi told us, “they ask if we can get back data.” For that operation, RowFeeder uses Twitter search, which is subject to the what is now a four day limit. Cortesi said sometimes it goes back up to five days.

Twitter streams, which we’ve covered previously, are a “push” technology. Rather than an application polling for the latest data, it registers to receive specific searches automatically. Then, when there is new content, Twitter sends it over a persistent connection.

Update. Twitter’s Matt Harris chimed in on the dev list:

To answer your question about the search index history, we don’t publish that information. The size of the index fluctuates based on the number of Tweets being made which means, the more Tweets there are the shorter the index period is. We’re working to improve the duration of the search index and improve the relevance of the results.

Both comments and pings are currently closed.

26 Responses to “Twitter Search History Dwindling, Now at Four Days”

August 12th, 2010
at 5:01 pm
Comment by: SearchCap: The Day In Search, August 12, 2010

[...] Twitter Search History Dwindling, Now at Four Days, blog.programmableweb.com [...]

August 12th, 2010
at 5:38 pm
Comment by: Dan Blaker

I have to admit, I hope Twitter doesn’t extend the search period. Nobody needs to know what I ate for dinner three weeks ago, much less three years ago. I can imagine a premium service where individuals can pay to archive their own tweets and/or the tweets of the people they follow, but I’d really like to see Twitter take a stand against the default “keep-it-forever” approach to web communications. I’m still a little annoyed that they’re donating my tweets to the Library of Congress (though not annoyed enough to quit Twitter, obviously).

August 12th, 2010
at 11:36 pm
Comment by: Kristof

Twitter search results are limited based on the number of results (1500) – not the date.

http://dev.twitter.com/pages/every_developer

Screenshot here:
http://images.tweetreports.com/twitter-search-results.png

August 13th, 2010
at 2:05 am
Comment by: Damon Cortesi

One of the interesting things is that Google has all of the data and it’s searchable via the web, but there’s no API access that I’m aware of.

This link, for example, shows all mentions since Google got the firehose back in February. http://www.google.com/search?q=dacort&hl=en&safe=off&prmd=u&source=lnms&ei=rd9kTM_tEYGksQPU8sjdDQ&sa=X&oi=mode_link&ct=mode&ved=0CA0Q_AU&tbs=mbl:1,mbl_hs:1262332800,mbl_he:1293868799

August 13th, 2010
at 11:41 am
Comment by: Adam DuVander

Kristof — You’re right there is an upper limit number of results, but for less common terms you’ll notice that tweets drop out after about four days.

August 13th, 2010
at 1:18 pm
Comment by: Kristof

@Adam How far back the results go is all dependent of the search term. If it’s a really popular term (such as a trending topic) results may only go back 20 minutes – sometimes much less.

August 16th, 2010
at 3:38 pm
Comment by: State of Search news roundup – episode 4: Google keywordtool, Twitter search and a non-dead girl in Streetview - State of Search

[...] Programmableweb Next Web [...]

August 18th, 2010
at 7:59 pm
Comment by: Top Topics for the Week (Aug 12-18) » Tech Media Today

[...] Networks, Status Message: Developers have been vocal about their thoughts on Twitter including reduced search history capabilities and the impending shutdown of basic authorization. That being said, readers are still engrossed in [...]

August 24th, 2010
at 2:58 pm
Comment by: Topsy Wants to Redesign Search, Starting with Twitter

[...] but the results that come back from a search aren’t very comprehensive, since the index is limited to only a few days’ worth of tweets. That’s created a market for search engines like Topsy, which says it now has the [...]

August 26th, 2010
at 2:51 pm
Comment by: Google Realtime Search Gets New Name, Its Own URL, And Kick In The Pants

[...] currently has the monopoly on serving up realtime updates, its search feature only goes back four days, whereas Google search reaches back till February of this year and is more reliable overall in my [...]

August 26th, 2010
at 2:55 pm
Comment by: Google Realtime Search Gets New Name, Its Own URL, And Kick In The Pants | Geoff Hatch Technology Resource

[...] currently has the monopoly on serving up realtime updates, its search feature only goes back four days, whereas Google search reaches back till February of this year and is more reliable overall in my [...]

August 26th, 2010
at 3:01 pm
Comment by: Google Realtime Search Gets New Name, Its Own URL, And Kick In The Pants » Tech Reviews

[...] currently has the monopoly on serving up realtime updates, its search feature only goes back four days, whereas Google search reaches back till February of this year and is more reliable overall in my [...]

August 26th, 2010
at 3:18 pm
Comment by: Recherche en temps réel de Google est rebaptisée, sa propre URL, et kick dans le pantalon | tech-news

[...] le monopole sur le support en temps réel les mises à jour, de sa fonction de recherche ne remonte Quatre jours , alors que la recherche search motors remonte à Février de cette année et est in addition fiable [...]

August 26th, 2010
at 4:03 pm
Comment by: DVRHDMI » Google Realtime Search Gets New Name, Its Own URL, And Kick In The Pants

[...] currently has the monopoly on serving up realtime updates, its search feature only goes back four days, whereas Google search reaches back till February of this year and is more reliable overall in my [...]

August 26th, 2010
at 5:58 pm
Comment by: Googleのリアルタイム検索が専用URLを得て独立したサービスに進化

[...] 競争力の核心を検索におくGoogleのような企業としては賢明な動きと評価できるだろう。リアルタイム情報の源泉としては現在Twitterが市場をほとんど独占しているものの、Twitter自身の検索は4日前までしか及ばない。Googleの検索は今年の2月まで遡れる上に、私の経験からすると全般的にTwitterより信頼性も高い。 [...]

August 27th, 2010
at 4:30 am
Comment by: Google Realtime Search Kini Punya Alamat URL Sendiri | DailySocial

[...] Seperti yang dituliskan oleh TechCrunch, hasil pencarian realtime dari Google ini merupakan langkah yang cukup baik yang dilakukan dari Google, memang kita bisa melakukan hasil pencarian dari konten yang ada di Twitter, misalnya dengan melakukan pencarian lewat web Twitter, namun salah satu kelebihan dari hasil pencarian realtime Google adalah jangka waktu konten yang didapatkan, Google akan menggali konten dari bulan Februari awal tahun 2010, sedangkan Twitter hanya dalam rentang waktu 4 hari. [...]

August 30th, 2010
at 1:37 pm
Comment by: Twitter Search Getting Shorter « Web Developers

[...] at any level. It’s a great way to connect with readers, partners, customers and so on. But as a search engine, it’s not cutting it at all. And now it seems that the rest of the world is finally beginning [...]

September 1st, 2010
at 3:01 am
Comment by: Topsy is Twitter Search Without an Expiration

[...] Twitter reducing its search history to four days,  it is a challenge for applications that are dependent on search results, that went [...]

September 1st, 2010
at 12:36 pm
Comment by: Betapond - Your Social Media Partner

[...] did a great post over on ProgrammableWeb. Hopefully I can dig up a bit more info over the weekend. // Share| [...]

September 17th, 2010
at 12:41 pm
Comment by: Kristof

At the same time, the newer streams have become the go-to API for former search use cases.

October 3rd, 2010
at 3:03 am
Comment by: Google Realtime Search Gets New Name, Its Own URL, And Kick In The Pants | TechShadez

[...] currently has the monopoly on serving up realtime updates, its search feature only goes back four days, whereas Google search reaches back till February of this year and is more reliable overall in my [...]

October 4th, 2010
at 1:27 am
Comment by: Twitter Search Getting Shorter | azwebdesignphoenix.com

[...] at any level. It’s a great way to connect with readers, partners, customers and so on. But as a search engine, it’s not cutting it at all. And now it seems that the rest of the world is finally beginning [...]

October 25th, 2010
at 11:35 am
Comment by: bubba

We’ve actually been archiving twitter and over 100 other 140 character sites. We just released our real-time livestream feed, and will be releasing our search feed shortly. we don’t limit based on time or number of records. Check out the implementation at archiver.co

October 25th, 2010
at 1:52 pm
Comment by: At Least Four Ways to Search Twitter Via API

[...] for Twitter searching, starting with Twitter’s own service. In August we reported about Twitter’s short search history. Recent improvements doubled its search index, which means more tweets, according to a Twitter [...]

April 24th, 2011
at 10:25 am
Comment by: storify. Ein (vorläufiger) Erfahrungsbericht | medienleiter

[...] auf 4 Tage beschränkte Search-History bei Twitter ist für „Storyteller“ echt eine blöde Sache, wenn sie z.B. Hintergründe einbauen [...]

October 5th, 2011
at 3:43 pm
Comment by: It’s fading fast – Metrics gain a new importance

[...] search, i.e effectively what is happening now. That's great for immediacy, rubbish for latency. Read more here > You can still read an individual users history from their profile page, starting from now, and [...]

Follow the PW team on Twitter

ProgrammableWeb
APIs, mashups and code. Because the world's your programmable oyster.

John Musser
Founder, ProgrammableWeb

Adam DuVander
Executive Editor, ProgrammableWeb. Author, Map Scripting 101. Lover, APIs.