Developers looking for trends in Twitter search are finding it more difficult now that the micro-blogging site has decreased the search history to four days. Previously going back weeks and months, the backlog has steadily decreased, now too short for some types of applications. At the same time, the newer streams have become the go-to API for former search use cases.
Back when you searched Twitter with Summize, the history went back several months. Once Twitter acquired Summize, that continued. Though, as Twitter became more popular, the history decreased. It appears to fluctuate, based on how long it takes to hit its maximum storage. Twitter stores tweets in MySQL and recently dropped a planned move to Cassandra, a system open sourced by Facebook, which some believe could improve Twitter’s search performance.
Twitter refused to comment, citing a policy against making user statistics public. However, its own documentation lists the limit at 1.5 weeks. The same page claimed a history of one month, then three weeks, in early 2009. It was last updated in March. The drop to less than one week likely happened just last month. Damon Cortesi noted the change in a tweet.
Cortesi’s company makes RowFeeder, a tool that performs social media monitoring, adding references to a spreadsheet. The service uses Twitter, among other sources, and now uses the streaming API for the bulk of its work. However, “when new customers sign up,” Cortesi told us, “they ask if we can get back data.” For that operation, RowFeeder uses Twitter search, which is subject to the what is now a four day limit. Cortesi said sometimes it goes back up to five days.
Twitter streams, which we’ve covered previously, are a “push” technology. Rather than an application polling for the latest data, it registers to receive specific searches automatically. Then, when there is new content, Twitter sends it over a persistent connection.
Update. Twitter’s Matt Harris chimed in on the dev list:
To answer your question about the search index history, we don’t publish that information. The size of the index fluctuates based on the number of Tweets being made which means, the more Tweets there are the shorter the index period is. We’re working to improve the duration of the search index and improve the relevance of the results.
Microblogging service
©ProgrammableWeb.com 2013. All rights reserved.
Terms of Service | Privacy Policy
26 Responses to “Twitter Search History Dwindling, Now at Four Days”
at 5:01 pm
[...] Twitter Search History Dwindling, Now at Four Days, blog.programmableweb.com [...]
at 5:38 pm
I have to admit, I hope Twitter doesn’t extend the search period. Nobody needs to know what I ate for dinner three weeks ago, much less three years ago. I can imagine a premium service where individuals can pay to archive their own tweets and/or the tweets of the people they follow, but I’d really like to see Twitter take a stand against the default “keep-it-forever” approach to web communications. I’m still a little annoyed that they’re donating my tweets to the Library of Congress (though not annoyed enough to quit Twitter, obviously).
at 11:36 pm
Twitter search results are limited based on the number of results (1500) – not the date.
http://dev.twitter.com/pages/every_developer
Screenshot here:
http://images.tweetreports.com/twitter-search-results.png
at 2:05 am
One of the interesting things is that Google has all of the data and it’s searchable via the web, but there’s no API access that I’m aware of.
This link, for example, shows all mentions since Google got the firehose back in February. http://www.google.com/search?q=dacort&hl=en&safe=off&prmd=u&source=lnms&ei=rd9kTM_tEYGksQPU8sjdDQ&sa=X&oi=mode_link&ct=mode&ved=0CA0Q_AU&tbs=mbl:1,mbl_hs:1262332800,mbl_he:1293868799
at 11:41 am
Kristof — You’re right there is an upper limit number of results, but for less common terms you’ll notice that tweets drop out after about four days.
at 1:18 pm
@Adam How far back the results go is all dependent of the search term. If it’s a really popular term (such as a trending topic) results may only go back 20 minutes – sometimes much less.
at 3:38 pm
[...] Programmableweb Next Web [...]
at 7:59 pm
[...] Networks, Status Message: Developers have been vocal about their thoughts on Twitter including reduced search history capabilities and the impending shutdown of basic authorization. That being said, readers are still engrossed in [...]
at 2:58 pm
[...] but the results that come back from a search aren’t very comprehensive, since the index is limited to only a few days’ worth of tweets. That’s created a market for search engines like Topsy, which says it now has the [...]
at 2:51 pm
[...] currently has the monopoly on serving up realtime updates, its search feature only goes back four days, whereas Google search reaches back till February of this year and is more reliable overall in my [...]
at 2:55 pm
[...] currently has the monopoly on serving up realtime updates, its search feature only goes back four days, whereas Google search reaches back till February of this year and is more reliable overall in my [...]
at 3:01 pm
[...] currently has the monopoly on serving up realtime updates, its search feature only goes back four days, whereas Google search reaches back till February of this year and is more reliable overall in my [...]
at 3:18 pm
[...] le monopole sur le support en temps réel les mises à jour, de sa fonction de recherche ne remonte Quatre jours , alors que la recherche search motors remonte à Février de cette année et est in addition fiable [...]
at 4:03 pm
[...] currently has the monopoly on serving up realtime updates, its search feature only goes back four days, whereas Google search reaches back till February of this year and is more reliable overall in my [...]
at 5:58 pm
[...] 競争力の核心を検索におくGoogleのような企業としては賢明な動きと評価できるだろう。リアルタイム情報の源泉としては現在Twitterが市場をほとんど独占しているものの、Twitter自身の検索は4日前までしか及ばない。Googleの検索は今年の2月まで遡れる上に、私の経験からすると全般的にTwitterより信頼性も高い。 [...]
at 4:30 am
[...] Seperti yang dituliskan oleh TechCrunch, hasil pencarian realtime dari Google ini merupakan langkah yang cukup baik yang dilakukan dari Google, memang kita bisa melakukan hasil pencarian dari konten yang ada di Twitter, misalnya dengan melakukan pencarian lewat web Twitter, namun salah satu kelebihan dari hasil pencarian realtime Google adalah jangka waktu konten yang didapatkan, Google akan menggali konten dari bulan Februari awal tahun 2010, sedangkan Twitter hanya dalam rentang waktu 4 hari. [...]
at 1:37 pm
[...] at any level. It’s a great way to connect with readers, partners, customers and so on. But as a search engine, it’s not cutting it at all. And now it seems that the rest of the world is finally beginning [...]
at 3:01 am
[...] Twitter reducing its search history to four days, it is a challenge for applications that are dependent on search results, that went [...]
at 12:36 pm
[...] did a great post over on ProgrammableWeb. Hopefully I can dig up a bit more info over the weekend. // Share| [...]
at 12:41 pm
At the same time, the newer streams have become the go-to API for former search use cases.
at 3:03 am
[...] currently has the monopoly on serving up realtime updates, its search feature only goes back four days, whereas Google search reaches back till February of this year and is more reliable overall in my [...]
at 1:27 am
[...] at any level. It’s a great way to connect with readers, partners, customers and so on. But as a search engine, it’s not cutting it at all. And now it seems that the rest of the world is finally beginning [...]
at 11:35 am
We’ve actually been archiving twitter and over 100 other 140 character sites. We just released our real-time livestream feed, and will be releasing our search feed shortly. we don’t limit based on time or number of records. Check out the implementation at archiver.co
at 1:52 pm
[...] for Twitter searching, starting with Twitter’s own service. In August we reported about Twitter’s short search history. Recent improvements doubled its search index, which means more tweets, according to a Twitter [...]
at 10:25 am
[...] auf 4 Tage beschränkte Search-History bei Twitter ist für „Storyteller“ echt eine blöde Sache, wenn sie z.B. Hintergründe einbauen [...]
at 3:43 pm
[...] search, i.e effectively what is happening now. That's great for immediacy, rubbish for latency. Read more here > You can still read an individual users history from their profile page, starting from now, and [...]