Technorati Yahoo and Google Too

In the last entry on the sub­ject, we took a look at how Tech­no­rati and Google com­pared. From there, we dis­cov­ered that Tech­no­rati was get­ting roughly a fourth of the links Google could locate. Which brought up some inter­est­ing ques­tions: could we rely on the Google num­bers? Were they so much larger than any other search engine that we were build­ing an unfair com­par­i­son? And, as some alert read­ers pointed in email, was Google under-reporting the num­ber of links to a site? In order to answer some of those ques­tions, I decided to build some more com­par­isons. So I decided to take a look at some of Google’s com­peti­tors. Today, I’ll go into how Yahoo! fared (Hint: I was sur­prised by the results).

Gath­er­ing the data

As I had done for the pre­vi­ous effort, I gath­ered data against the same list of site around the same date. This pro­vided me with some con­sis­tency in the data set that allowed for bet­ter com­par­i­son. Com­pare one or two site and you may get some false pos­i­tives. Com­pare 100 sites and things start get­ting a lit­tle more inter­est­ing. The Yahoo! data ended up look­ing at this (for peo­ple who are new to the series, I am doing the same graphs for a num­ber of search engines):

The first thing of inter­est when putting together that set of num­bers was how much larger the num­ber of links found in the Yahoo! index was, com­pared to the num­ber of links found in either Tech­no­rati or Google. The sec­ond item I found inter­est­ing was a rel­a­tive con­sis­tency in terms of Asian sites not fig­ur­ing well in the Yahoo! index com­pared to the Tech­no­rati one. It seems that Tech­no­rati is get­ting a bet­ter han­dle on the Asian blo­gos­phere than Yahoo! is, a sur­pris­ing result con­sid­er­ing how much time and effort the lat­ter has put into its Asian operations.

In order to get some real visual com­par­i­son, I decided to draw a sim­i­lar dia­gram of the link per­cent­ages dis­trib­uted across all 100 sites. It looked like this:

The inter­est­ing story, look­ing at this is that it appeared that there was much greater vari­ance from site to site in the Google index that there was in the Yahoo! one. In the Yahoo sys­tem, the vast major­ity of site fall in the below one per­cent range but what became even more inter­est­ing was that the rate of vari­ance was really not that high: when com­par­ing the median and the aver­age, it turned out to be less than .1% of difference:

While the num­ber were vastly dif­fer­ent in terms of size (it appeared Yahoo! had a lot more links), I fig­ured the pat­terns would be roughly the same in terms of cov­er­age: I expected the top sites to get bet­ter cov­er­age in a large search engine like Yahoo! than smaller sites. Imag­ine my sur­prise then when I started to do some group analysis:

Those num­bers seemed to be all over the map, a fact that became much clearer once I graphed it:

None of the nice down­grade curve I had with the Google set. Here was a much more dis­parate set, pro­vid­ing lit­tle in terms of sup­port­ing a the­ory of bias from a search engine. In fact, it worked more to poten­tially prove such the­ory wrong.

Was my data set wrong? I rechecked it and it was not. So what was hap­pen­ing here? As dreams of long tail and power law dis­tri­b­u­tions fell out, I started to won­der how Yahoo! and Google com­pared. So, of course, I decided to run the num­bers again…

Yahoo! vs. Google

This time I decided to com­pare Google and Yahoo! First, I fig­ured I would get some ref­er­ence data on the sub­ject. I was sur­prised to not find any actual side by side com­par­i­son on a large set of sites. Anec­do­tal evi­dence existed but noth­ing com­pared to the data set I had amassed so I fig­ure I would trust my own data set (note: If you have a bet­ter one, please leave a com­ment as to where it is located). The set ended up look­ing like this:

Noth­ing par­tic­u­larly impres­sive there. It seemed that Google, on aver­age, ended up with only about 3% of the links Yahoo! had in its index. How­ever, the story got more inter­est­ing when look­ing at diver­gence between the aver­age and the median, as it seemed there was a sta­tis­ti­cal diver­gence (almost half a per­cent) between the two:

But wait, for the weird­ness is only get­ting started. Next up was look­ing at the dis­tri­b­u­tions (as I’ve done for Tech­no­rati vs. each of the engines):

I looked at the num­ber and they did not seem right so I ran them again and ended up with the same results. Ran them a third time and still couldn’t make sense of it. So I graphed it:

… and to my sur­prise, it appeared that the fur­ther down the line one went, the greater the dif­fer­en­tial. In fact, sites that are in the bot­tom of the top 100 are one full per­cent more likely to get indexed in Yahoo! than in Google.

Con­clu­sions

From here, we can draw a few conclusions:

• Yahoo! gen­er­ally does a bet­ter job at index­ing the blo­gos­phere than Google does. We know they have been work­ing hard to improve their index and here’s proof that they are get­ting results
• Even if Google is the one with the motto about not doing evil, Yahoo! seems to be the one inter­ested in giv­ing equal oppor­tu­nity to the lit­tle guy: smaller blogs seem to have a bet­ter chance of being rec­og­nized by Yahoo! than they do of being rec­og­nized by Google
• While the front page of Google adver­tises they are cur­rently index­ing over 8 bil­lion pages, it is very dif­fi­cult to find ways to sup­port that claim via the link fea­ture they are offer­ing: this can be seen as con­fir­ma­tion that Google does not tell you about all the links it has in its index.
• Sure vol­ume counts but in the case of search indexes, they may count against sites: if one is less likely to appear in Google than it is to appear in Yahoo! and the Google index is much larger than the Yahoo! one, then, if Yahoo! and Google had the same amount of traf­fic, a sin­gle blog could find itself receiv­ing more traf­fic from Yahoo! than it does from Google. This would be due to the fact that each indi­vid­ual page in Yahoo! has more weight than it does in Google.
• The top 100 blogs have other 56 mil­lion links in the Yahoo!. That’s a lot of links and clearly shows that links are the cur­rency of the blog­ging world. It would be inter­ested to get data that would help ana­lyze how much inter­link­ing exists across those sites.

Up next, we’ll take a look at how MSN plays in all this game. So stay tuned!

Related Terms

More in Analysis,Business,Media,Politics,Technology (5 of 5 articles)