Faster access to pages… for Google

DNS servers, for people who don’t know the technical details, are basically the phone books of the internet. On the internet, every computer server is known by a set of digits known as its IP address. For example, when you typed tnl.net, the DNS server looked up the name and discovered that it was at 206.127.35.2 allowing your computer to connect to mine.

I’ve argued, for many years, that Google wanted to find a way to access new web pages at a much faster rate. The challenge the company has had is that it is difficult to find new pages when they appear. While traditional technology to discover what pages are available on the internet has evolved and Google has managed to coerce some site owners in providing it with a quick update when changes are available on their site, the efforts the company has pushed have also been helpful to its competitors, who could build on top of the processes and open standards Google was fostering.

With Microsoft getting some early successes in the search game with their new Bing offering, and new entrants in the real-time search business eating up some of Google’s mindshare in search, the company needs to do something radical, lest the cornerstone of its business, and the source of most of its revenue, be undermined.

Enters the DNS system. Every time a page is called, your browser makes a DNS call (several, in fact, as every web asset can result in a different one). In other words, the DNS system truly serves as the heartbeat of the internet and convincing a large swath of users could allow Google to get an idea as to what’s new on the internet.

If, for example, a user were to access a new page that’s not in the Google index, Google’s own DNS servers could be wired up to alert its search spiders to immediately pick up the page, analyse it, index it, and make it available to its search users within seconds of the page first being accessed. This could give Google a substantial advantage over Microsoft and others in indexing the web in real time.

Some potential risks

There is, however, a huge caveat in all this. For starters, Google needs to convince a large number of people to access their DNS. Providing the product for free may work for some but will not convince everyone. Another issue they may have to deal with is the perception that they might snoop on personal data (something that is already being addressed on their site). The ability to access information about everything you do on the internet, whether it is via a web browser or another application like Skype, online games, or video and music player, is granting Google some brand new capabilities and not everyone may be willing to share such information.

Google will also have to contend with large numbers of potential denial of service attacks (or worse), which have become more common of late, against those DNS servers. Such attacks could represent a substantial reputational risk to the company. If, for example, one of Google’s DNS servers could be compromised, the hacker could decide to redirect the traffic of banks or other financial institutions to their own sites. The potential financial impact of such a thing would become a legal nightmare for the company.

The Prize

All this, however, can be counter-balanced by the rich prize the company would get in being able to index every bit of internet content within seconds (or even nanoseconds) of such content being available on the internet. If that were to be achieved, the company’s perch at the top of the search heap would be guaranteed for a long long time and its continued dominance in the advertising world, based on the rich analytical data it could get from snooping on users of its services, would provide it with cashflow that other players on the internet would have a hard time to match.

Furthermore, Google could have control over where people go and could, if they decided to be evil, redirecto such traffic. That would be a tremendous amount of power.

All told, an interesting move and it will be fascinating looking at where they are planning to take this.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Google Goes Real-Time. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

Recently, though, TechMeme has gotten me thinking about the tech blogosphere conversations as a whole and their longer term relevance. To the small “web 2.0″ community, TechMeme serves as a bit of a paper of record; The subhead even claims that it represents the “Tech Web, page A1”, claiming to bring us the important stories. But how do those stories fare over time? Is today’s hot topic a step in understanding a longer term trend or is it just a temporary distraction that will be forgotten a month/3 months/6 months/a year from now.

Fortunately, Gabe Rivera, the founder of TechMeme must have anticipated such a question and provided a way to look at TechMeme as it was a particular point in its short history. Using the simple interface, it’s easy to see the page as it existed at a precise point in time. So I decided to start looking at the site at the same time in single month spaces. The middle of the night and middle of the day position ought to be good time stamps so I decided to look at the site at 12am and 12pm on the selected date. I also had to discount the fact that April 1st is April fool’s day so I could not use the first of the month as this fact could skew the data. Here are the dates and times I ended up with:

• Today: June 2nd 2008 at 12am and 12pm
• A week ago: May 26, 2008 at 12am and 12pm
• Two weeks ago: May 19, 2008 at 12am and 12pm
• One month ago: May 2, 2008 at 12am and 12pm
• Two months ago: April 2, 2008 at 12am and 12pm
• Three months ago: March 2, 2008 at 12am and 12pm
• Six months ago: December 2, 2007 at 12am and 12pm
• Nine months ago: September 2, 2007 at 12am and 12pm
• One Year ago: June 2 2007 at 12am and 12pm
• Two years ago: June 2, 2006 at 12am and 12pm

With 20 data points, here’s what I discovered.

Today

Based on today’s news at noon, it looks like the important subjects at noon in the blogosphere are Adobe’s latest move, combining Flash and Acrobat with their entry in the already crowded (Google, Microsoft, Zoho, etc..) web-based office suite market. At midnight, things were a little less exciting, with discussion around the privacy issues Google Maps is raising with their StreetView offering.

Of course, it’s still too early to tell whether those stories will have a long term impact so let’s roll the tape back a little.

One Week Ago: May 26, 2008

At noon, a week ago, the top story was about a new type of SSD, developed by Samsung. Since it’s hardware, I assume that the impact of this news can’t be felt initially but there could be longer term repercussions. Also of note on that page is a small item lower on the page about Paypal outages. An interesting trend in my research on this is that this story is slowly developing over a period of weeks and months and the noise level appears to be increasing on it.

At midnight, the discussion was around Google’s power and the needed for another organization to work as a counter balance to that powerful force in the search engine space. Coupled with the discussions last night about privacy issues relating to Google maps, it seems we are seeing an emerging pattern here.

Two Weeks Ago: May 19, 2008

Two weeks ago, at noontime, the claim that Microsoft would eventually buy Facebook and keep it close was dominating TechMeme. At this point, no announcement has been made so this is largely conjecture and, while an interesting opinion, it’s not really news. This editorial was largely in response to the news item that dominated the previous 12 hour cycle about Microsoft’s statements regarding pursuing a possible deals other than a full acquisition with Yahoo!

One Month Ago: May 2, 2008

On May 2, 2008 at noon, the big news was… that the Google RSS reader is now available for the iphone. I’m sure many people consider this event as a major turning point when… well, hmm… a big big deal. Amusingly, Adobe was also in the news that day, with news that its flash plugin would escape computers and appear in set top boxes and mobile phones.

Another big subject was Steve Ballmer’s mention that Microsoft could go it alone without Yahoo, a discussion that dominated the midnight page on that day. The Yahoo/Microsoft chat has been kind of the soap opera of our industry and this latest installment was remembered as a turning point (or not) by many.

A possibly interesting trend piece, around midnight, was also intriguing: Will Grand Theft Auto IV hurt Iron Man opening weekend sales. I haven’t seen a follow up on that piece yet, which could tell us whether video games are displacing movies as the primary form of entertainment but my guess is that the answer is no.

Two Months Ago: April 2, 2008

On April 2, 2008 at noon, the top story on techmeme was about Intel’s plan for chips that would power up more mobile devices. Interestingly, this story was largely driven by mainstream media as the lead was taken by john Markoff of the New York Times, followed by comments from Forbes magazine, and Infoworld. The other related story was the press release itself, which can be seen as bloggers pointing straight to the source of the news. I suspect that this story will probably have more legs moving forward. A cursory glance provides glances at developing stories ranging from the rumor stage (that all important Google/Skype partnership or acquisition… which didn’t happen) to the focus on process (like the approval of Office Open XML as an ISO standard).

The departure of Google’s CIO dominated the prior night’s news cycle and word of Apple’s 3G iphone started to filter through.

Three Months Ago: March 2, 2008

March 2, 2008 at noon provides us perspective on today’s news, thanks to Microsoft’s announcement of ITS entry into the web-based office suite market. When put side by side with today’s announcement by Adobe, it seems to start pointing to more of a trend. Beyond that, little news that seems to be of note from a memorable standpoint.

The interesting thing here is that the same subject was leading the previous night’s news cycle. This seems to establish a first rule for techmeme: subjects that survive on the front page more than 12 hours may be worth paying attention to.

Six Month Ago: December 2, 2007

There’s an all saying in journalism that 3 items make for a trend. In the case of this study, it looks like Web-based office suite are definitely the hottest trend around, as the top news on December 2, 2007 at noon was information about the future of Google’s offering in that space (either that or there is an unwritten rule in the technology field that information about web-based office suites MUST be introduced on the second day of the month or wait until the following month).

The subject was starting to climb the chart 12 hours earlier, even thought the discussion at the time was dominated by a Facebook misstep (remember Facebook Beacons? Well, that was around that time). From an interface standpoint, it also brings up something that I’d like to recommend to Gabe: could you add and up or down arrow to highlight if a subject is getting more play or not. On something like this, it would be nice to get an idea of the stickiness of a topic. It appears many topic appear low on the page and move up over time, the quicker and faster they move up seems to indicate the importance of the story and it would be a nice addition to have that info on the screen.

Nine Month Ago: September 2, 2007

September 2, 2007 was a quiet news day. I guess everyone was mourning the death of the newspaper, which was forced by Google on that day, according to the noon-time headlines. There doesn’t seem to have been any other major news around midnight either. This, however, could be an artifact in the data as September 2, 2007 was a Sunday, which is generally a pretty quiet news day as most people don’t work on Sunday.

Interestingly, a story that is just now starting to get more notice is the continuing brushfires around Paypal’s outages. Not that sexy a subject but one that started to be raised around that time. At the time, discussion of Google’s entry in the mobile market centered around the idea they would deliver a device instead of a platform.

Last Year and Two Years Ago

A year ago, at noon, the Techmeme conversation was around porn. During the night, though, the conversation was centering around the acquisition of Feedburner by Google. This is probably remembered by people in the industry as an important milestone and here, techmeme shines at organizing a package with the appropriate conversations.

Things do not improve much if you go further back: 2 years ago, at noon, and midnight, gives us little to mull over.

Conclusion

The data seems to point that the front page of TechMeme largely represents what’s hot right now but does not necessarily highlight stories which have a longer term type of impact. In that sense, it may also be highlighting that discussions in the tech blogosphere are largely centered on insider-type minutia while failing to put things in a larger context. This appears to present a myopic view of the tech world that leaves us with lots of data but preciously little information. So while TechMeme provides a useful tool in terms of getting an idea of the pulse of the conversation “right now,” it does little in providing data that would allow someone to understand the larger trends that are affecting our world as a result of the internet (and web 2.0 revolution).

I would argue that the answer to the question I posed in the title for that post is a resounding yes. Because it deals largely with the trivial and assess little value to longer type impact, TechMeme creates a self-imposed myopia on its readers and participants. A possible exception is when a story manages to survives through multiple 12-hour instances, providing many angles to the same events. But those events are few and far between.

Whether the lack of headlines with a major impact is a phenomenon that is unique to TechMeme or to the tech world in general is a question I’d like to leave to readers and I’d appreciate comments as to your thinking around this.

But all this comes down to a simple fact: if you’ve missed what happened on TechMeme in the last XX hours, days or weeks, you may not necessarily have missed much. so kick back, relax, step away from the computer and, if you need to catch up, you can always pick up a mainstream publication that may cover a distilled version of what happened if it’s of any particular significance.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Is Techmeme myopic?. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

Size of the market

When talking about virtual worlds, I am focusing on the new space created by the gaming industry that allows to create online avatars and interact with other players in a fully immersive environment. From an economic standpoint, estimates range from around 100 millions to a high of US$1.5 billion a year. These are not insignificant numbers and they point to an emerging phenomenon and potentially the rise a new industry, with its own set of marketplaces, gathers, owners, creators, and marketers.

Marketplaces

To understand virtual worlds marketplace, one must first understand what si going on in those virtual worlds. When a player sets up an account, he’s given a basic set of skills. As he or she progresses and interacts with the virtual world and its denizens, the player gains more and more skills and goods. However, this type of interaction requires time. Some people have figured that, because time is money, the amount of time spend in a virtual world could be converted into real hard currency. Thus was born the concept of Real Money Trading (aka RMT), whereas players go to specialized sites and buy virtual goods with real financial currency.

The action initially started on auction sites like Ebay, where characters or other virtual goods range in price from a few cents to several thousands of dollars. Because the trades were largely unregulated, some companies, like Sony, decided to set up their own exchange while others (Internet Game Exchange, Massive Online Gaming Sales, Tek Gaming Supplies, Swag Vault, and Gamers’ Loot) have created specialized marketplaces to cater to this new phenomenon. This, in turns has led to the rise of two new classes of activities: informational ones that provide analysis on the financial going ons in those worlds and arbitration, whereas companies use people in the developing world to build up assets they resell to people in the developed world. Let’s go deeper in those areas.

Information Sites

There is now a nascent information industry surrounding the costs of goods in virtual worlds. For example, Eyes on Mogs is a shopping search engine for virtual goods. All the attributes of other search engines are part of it, including comparison shopping, comparisons of the different vendors, pricing, delivery date, and buy it now info. GameUSD tracks the financial value of virtual currencies over time, providing price trends across not only the provider but also the alternative marketplaces. MMOfx claims to track “over 18,000 price quotes daily” and provide information on the fluctuation of virtual currencies.

Arbitration

Another type of economic activity to have arisen out of the marketplace phenomenon is the arbitration of virtual work. As the primary pursuit in these worlds is the acquisition of wealth, status or levels, an emerging market has arisen to give people with real money a chance to bypass the time investment required to acquire those things. For example, Chinese workers get paid between $75 and $250 a month to work in World Of Warcraft, in 12 hours shifts, “killing onscreen monsters and winning battles, harvesting artificial gold coins and other virtual goods. Affluent online gamers who lack the time and patience to work their way up to the higher levels of gamedom are willing to pay the young Chinese to play the early rounds for them.” Similarly, Romanian players can make a living wage (the ABC News story I linked to says that $200 is a good wage for Romania) on the same kind of activity.

Edward Castranova, the leading economist on the subject of money in virtual worlds has been quoted as saying that “They’re exploiting the wage difference between the U.S. and China for unskilled labor.” What is basically happening here is that these companies have found a niche on the global marketplace to accumulate goods at a low cost and resell them at a premium. This type of arbitrage has been the way a lot of developing markets have revolutionized industries, from the export of manufacturing capabilities in the 20th century to the export of some service jobs nowadays. It’s a natural phenomenon and shows that those marketplaces are starting to develop a high level of maturity, which should be noticed by a lot more people.

Virtual Goods Ownership

Beyond the buying and selling of virtual goods in virtual marketplaces, there is also an emerging trend in the real estate business, which can be broken down into three main groups: real-estate owners, creators and integrators, and marketers.

Second Life is a virtual world more focused on the social aspect of virtual environments than on the goal oriented aspect of missions and war-craft. Fortune Magazine reported last year about the interesting case of Anshe Chung, a character created by a German woman who has accumulated more than US$200,000 in virtual land holdings in Second Life. She rents the property out to other people, after having developed the property. Similarly, the BBC reports that a 23-year-old spent £13,770 in Project Entropia and recouped his investment in under a year. In fact, the land rush has been so strong that Second Life has build a model around land use fees, generating a nice chunk of income in the process.

While visiting this world, I’ve talked to people who had few problems paying $75 per month to Linden Labs for those fees. This is pretty incredible when you think that all they are buying is portion of disk space on a server. In a way, the real estate market presented by those virtual worlds can be seen as a hosting fee in a 3D environment and could represent a high growth market (in a future entry, I will look at the opportunities in the Virtual Spaces in more details).

The Integration Model

Another nascent portion of this new industry is the integration game. As with any new technology, developing and managing something in a virtual world is an endeavor that requires specialized skills. New companies like The electric sheep company and Space Think Dream have emerged as developers/integrators, offering their services to other companies. Their main business is to use the skills they’ve acquired to help existing companies experiment in these new worlds. This is, in a way, similar to the type of work that was done by early web design agencies, treating virtual worlds as a new interface either to existing systems or to create a new value proposition.

Other companies have emerged with the sole purpose of selling digital goods in those worlds. SLexchange is a virtual market where people can buy and sell such goods. Similarly, the Electric Sheep company has created SLBoutique as a competitor to SLexchange. What is interesting here is that there is a whole ecosystem building around Second Life, allowing other companies to prosper based on this new platform. This is similar to what has happened with Ebay and allows us to better understand SecondLife as a platform for e-commerce rather than just a game, a fact that Philip Rosedale, CEO of LindenLab and the power behind Second Life, likes to emphasize. This explains why the company has received investments from people like amazon.com CEO Jeff Bezos, Lotus founder Mitch Kapor, Ebay founder Pierre Omidyar, and Microsoft CTO Ray Ozzie. Those people understand that this a new emerging platform and could see potentially high return on their investment.

Bridging the gap

The development of virtual worlds as a new platform is starting to take shape. Companies and organizations like American Apparel, the BBC, Major League Baseball, NASA, The American Cancer Society, Amazon.com and Wells Fargo are starting to experiment in that space. Increasingly, virtual worlds are becoming not only a new way to market but also a new integration point for e-commerce.

Some of the virtual worlds (Project Entropia, for example) have even gone as far as issuing ATM cards that allow denizens of those worlds to take virtual money and trade it for real money that they can use for regular economic activity.

Conclusion

With large amounts of real currency already moving through virtual worlds, we are looking at a major new economic phenomenon that parallels the initial development of the commercial web and the rise of software as platform in the last few years.

With a new ecosystem forming around some of the virtual worlds, there is a fair amount of incentive for a lot of people to see this phenomenon succeed. SecondLife will probably be an early winner in this race, largely due to how quickly it has managed to get other companies to rely on it. A few more established companies are also early in staking ground in this new space and will probably reap rich rewards for their efforts, expanding their brand into those virtual spaces.

While it may appear that this is largely a subculture of gaming, the phenomenon is much more widespread. In my next entry, I will go through the demographic profile of denizens of those virtual spaces, showcasing a rich and varied texture to this phenomenon.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Economic Activity in Virtual Worlds. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

5. Data Mining/Research

A main attribute of social networks is how much data people provide to them. On top of it, this data and the interaction of users on those networks. This is rich fodder for data mining. For example, researchers recently used Where’s George, a website tracking dollar bills in the real world, to assess how disease spreads. Similarly, LinkedIn provides its users with demographic/geographic data about members of your social network.

Traditional companies spend millions of dollars trying to understand the flow of people, flow of ideas (or memes) and how to exploit them. From Milgram’s small world experiment to the success of “The Tipping Point” by Malcolm Gladwell, there has been a fairly large body of research in this area but, for what may be the first time in history, there is now a heavy trove of data that can be analyzed.

4. Problem Solving

Sites like Google Answers are working on providing better answers to questions. Add-in some social network glue and one could be able to figure whether the person is a subject matter expert in the area he/she is answering the question about. For example, you might want to trust an individual with strong network ties in technology on questions related to technology but might be a little more wary of answers that person would provide about medical care (and similarly, you might trust a doctor more about medical care than you would a computer geek). Social networks, when seen through the lens of expertise, can provide quick access to answers from subject matter experts in one area. It is impossible to know everything but you might have a friend of a friend of a friend who has the answer in a specific area you are researching.

Similarly, social networks can provide a way to get social matter experts to connect and work collectively on difficult problems. When combined with digg–like features, social networks could become a way to speed up the vetting process on scientific publications by allowing a large set of peers to review articles and rank them according to value. This, in itself, could help humanity make radical moves forward in the area of scientific research.

Take, for example, my friends at ACOR who have been thinking of developing, in partnership with the National Cancer Institute, a data-mining system that analyzes information about patients to identify potential root cause for different cancers. Here, we see social networks (in this case, via mailing lists that are finely targeted) potentially being useful to help advance science and hopefully discover some root causes for cancer. A set of tools to such granular community could help a scientist, for example, sent a questionnaire to a sub-segment of the population to test a hypothesis (eg. “let’s see if people who have skin cancer and drank more than 1 glass of milk a day are reacting better to this type of drug?”) before deciding to do a clinical trial. If a specialized social network for such community was created, there might be no end to how much data could be gathered. Thing of it as a shotgun approach to medicine.   

3. Marketing

Marketing, off course, is all about deep knowledge of the audience. The best way to market a message is to discover what motivates people and how to craft the message to match the motivations. When combined with the database of intentions, a social network can work as a set of focus groups for messages. Testing different messages on a narrow audience can allow people to better market their products.

2. Reputation Management

The old adage is that “it’s not what you know, it’s who you know“ is at the core of social networking. As more and more people are online and more and more interactions are happening between people with weak ties, assessing a person’s reputation is increasingly important. LinkedIn has keyed in on that effort by giving people the ability to “endorse” members of their social network, providing more information about how a person performed in a particular job. In a similar fashion, profiles no Ebay allow buyers and sellers to assess the track record of a buyer or seller before making a transaction. Endorsements by one’s strong ties generally reflects much higher than by someone you don’t know. Thus, social networks can work as the glue to reputation management. It is not enough for people to know that a person is seen as important by some random stranger but when one discovers that their friends or colleagues have endorsed a particular individual, they tend to trust those opinions more heavily.

Let’s take a pedestrian example: imagine you need to get some electrical work done in your house but don’t know any electricians. By looking at your social network, you could find such an expert with ease as the best electrician might be linked to your friends. In a way, social networks are just an extension of asking people for recommendations. Which brings me to the last opportunity on this list.

1. Recommendation

Recommendation is a very powerful driver to decision making: whether it is for hiring a person, picking a new product, or finding a general direction, humans tend to look to their existing network and do a subconscious “most-like” analysis of the information they receive. For example, Amazon has been very successful with the “people who bought this also bought…” and “people who looked at this also looked at…” features. As they gather more data, patterns emerge.

Similar approaches can be taken into the search space (where what people linked to or clicked on is ranked higher than other stuff) and in other areas like music (last.fm comes to mind) or other media consumption (for example, the success of aggregator like Digg, techmeme or tailrank can be attributed in large part to the need people have to know what other people think is good).

Conclusion

Social Networks should not really be a set of standalone tools but they are essential to building the next set of applications that leverage the power of the crowds. As such, social networking should be a feature and not an end-goal until itself. The companies that understand this basic rule will be the ones that succeed in that space, leveraging opportunities created by social networks in a fashion that will provide unprecedented benefits.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title 5 opportunities around social networks. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

It seems every time Apple or Google introduces a new product, the buzz is high. For example, Apple recently introduced a $350 speaker and, while the reaction was more tepid than it has been for other Apple products, no one seem to point that the emperor was looking very very naked. Yet, Microsoft keeps throwing out new products and few people seem to be very interested (no matter how Scoble tries to browbeat us into thinking of Microsoft as cool).

Similarly, today, Google introduced a finance section that mimicked much of what yahoo! finance has been doing for years. It has a couple of nice AJAX-based features but, all and all, it’s not enough of an improvement to be considered like something that could potentially dominate the tech news cycle. And yet, every major tech pub or mainstream publication has covered the release.

why?

Trying to divine the source of coolness

What Google and Apple seem to have understood is that there are ways to make oneself look cool. I’m going to try to lay out some of the things I’ve seen (and I hope that others will chime in in the comments):

Rumor Mill

First, let the rumors float or give the appearance that you don’t want rumors spreading. Google Finance has long been a rumored product (as is Google payment, for example) but no word ever came out of the company about their intentions. In fact, Google is relatively stingy in terms of providing advance information about their products. They have learned to let the rumors run wild, leaving their competitors tearing their hair out trying to divine what Google will do next.

Apple takes a different approach to this. In the past, the company has been relatively ruthless in its attempts to shut leaks down. However, it seems that, when leaks are presenting compelling products and the company doesn’t really have anything to announce, Apple is happy to let the rumor mill run wild. So, before the release of the iSpeaker, uh, iPod Hi-Fi, Apple did not crack down on rumors about a new video iPod.

The two approaches speak to two different traits: one is to be extremely secretive about your action and the other is to let rumors go wild as long as they paint a picture of your company that is far cheerier than its reality.

The one feature

When selling technology, there are two publics to serve: the early adopters, and the general public. The early adopters are a fickle bunch but they can have some influence on the general public. So giving the early adopters one feature that they will like is an important feature of creating good buzz. Similarly, when dealing with the general public, emphasize the one feature that makes your product different. It doesn’t have to be something that is actually innovative (many companies were making MP3 players years before the iPod; many companies have offered services (other than search) which did what Google did in categories like mail, web hosting, classified, news, etc..) but it has to be presented as such. The early adopters may groan but they are eventually drowned out by the masses.

Thus, Apple did not release a featureless MP3 players without a screen, they released the “Shuffle” which allowed people to get a little more randomness out of their music collection. Or Apple didn’t release a $350 speaker, they release a Hi-Fi system that will work with an iPod (iPod sold separately). Similarly, Google did not release a Geocities rethread, they released pages, a cool online web editor and page hosting service. They did not release a me-too version of finance: look at the cool graphs they have.

I may sound a little cynical in that last paragraph but I believe it is this kind of cynicism that infuses the marketing of cool products. They may not be the top technology in the market but they are different. And emphasizing that they are different gives a chance to the users to feel like they, too, are different.

Cool by association

The next item on the list, in terms of generating buzz is to create an appearance of exclusivity from the get-go. Thus Apple does not complain too much when the police report rise in theft of iPod, due to the high visibility of the white headphones (see, our product is so popular, people steal it). Similarly, Google did not offer a free web-mail service for all, you had to receive an invitation.

By creating a certain level of exclusivity or belonging to a certain tribe, Apple and Google have managed to go beyond the product. They’ve created an aura of cool in being associated with them. When a new product comes out, you have to check it out or you will be out of the loop. The trend folds on itself, ensuring that future product launches benefit from the buzz of previous product launches. Over times, the duds are forgotten, and the companies are seen as innovative.

Look! Feel!

One of the other things to consider, when creating some level of buzz is the fizz and whiz of look and feel. Apple is known for designing beautiful computers (in the mainstream PC world, only Sony puts as much thought into how their computers look). The energy they put into the design allows them to bypass some of the technology issues that other vendors would encounter.

Similarly, Google has become an expert at using AJAx for their interfaces. As a result, new products generally look more polished than the competition. In the case of Finance application, it was interesting to see comments by people in the financial space on the performance of the product in terms of delays giving stock quote prices, etc.. However, few users would drill in and discover that stock prices were about 20–25 behind, or that

What value does buzz have?

At the end of the day, though, much remains to be seen about the value of such buzz. While Apple generates a lot of buzz about its computers, it still only retains between 5 and 10 percent of the market. Similarly, while Google has generated much buzz for all its new products, its bread and butter is still revenue from advertising on the search engine. So the question that still needs to be considered is whether buzz has value beyond the introduction of a new product and what that value translates to in terms of real dollars.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Building Buzz. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

The first thing to do was to map out which of the May 19, 2005 members were still on the list. The results looked like this:

This provided me with a departure point but it wasn’t really getting at what I wanted. Obviously, a fair number of people had changed position. So I decided to take a cut of the same data on the 20th of February and start mapping out movement. It looked as follows:

This provided me with two points in time: One in May 2005 and one in February 2006, 9 months later. If the theory of gatekeepers held true, the lists should have been pretty consistent.

What the data showed, however, was that the technorati 100 list is a very dynamic one. Let’s take a look at some of the moves.

Boing Boing: King of the blogosphere

Only one blog, Boing Boing, manage to hold its position steady in the last 9 months. Sitting at the top spot, it looks like it won’t move for a long time to come.

The movers and shakers

In this new list, 9 blogs successfully moved up in the last 9 months. They are:

• EnGadget (from 6 to 2)
• Post Secret (67 to 4)
• Thought Mechanics (89 to 7)
• Google official blog (57 to
• Michelle Malkin (27 to 10)
• A list apart (24 to 17)
• Gawker (29 to 19)
• Defamer (94 to 53)
• Gothamist (88 to 85)

Those were all blogs that appeared on both lists and managed to climb up in the ranks. More surprising, however, was the fact that 65 new bloggers appeared on the list, new claimant to the title of top blogger. A quick analysis seems to point to Asian blogs becoming a major force, one that I personally have not heard much about in discussion of the evolution of the blogosphere. David Sifry’s State of the Blogosphere did not cover any of this type of movement when he did his last overview of the state of the blogosphere. I don’t know if he deliberately decided to ignore the data or whether he did not see it as that important but I consider this a pretty powerful observation. In a world where globalisation is key, the blogosphere has not yet fully grappled with the impact of the Asian Pacific region and there probably will be some interesting discussion around this in the future.

From a legacy standpoint, it also seems that upward moves are not fully distributed across the space. The following table shows how the legacy upward moves were distributed among the population:

So being in the top 50 percentile makes it easier to move up, which would give some credence to a network effect. However, because we are talking about such a small segment of the population, it is impossible to generate any meaningful conclusion from the data.

The endangered list

While 65 blogs already dropped off the list, the 25 following blogs are in danger for the next 9 months as they have suffered a drop in ranking over the last 9 months:

• Daily Kos (from 3 to 5)
• Gizmodo (4 to 9)
• Instapundit (2 to 12)
• Dooce (9 to 15)
• Kottke (14 to 21)
• Fark (5 to 23)
• Wonkette (19 to 25)
• Talking Points Memo (12 to 26)
• Powerline (21 to 33)
• Scobelizer (32 to 34)
• LGF (13 to 35)
• Eschaton (8 to 36)
• Joel On Software (36 to 39)
• Something Awful (25 to 44)
• Metafilter (16 to 47)
• Andrew Sullivan (10 to 51)
• Best Page in the Universe (11 to 52)
• Hugh Hewitt (41 to 55)
• Web Standard Project (35 to 57)
• Buzz Machine (45 to 60)
• Captain’s Quarter (63 to 70)
• The Volokh conspiracy (31 to 74)
• Doc Searls (17 to 92)
• Informed Comment (47 to 93)
• Scripting News (20 to 95)

The interesting thing, in terms of that drop is that it seems to affect members across the list as a whole in a similar fashion. A quick analysis of the drop breakdown shows no clear advantage in being near the top of the list versus being closer to the bottom:

More interesting is that this number is low compared to the blogs which disappeared completely from the top 100. That number stands at 65 and breaks down as follows:

A dynamic list

If you take those numbers, it means that a total of 90 blogs (25 dropping within the list and another 65 dropping off the list completely) ended up with a lower position in 9 months. Combined with the fact that 9 blogs moved up, this means that 99 percent of the list was dynamic.

This, to me, was a pretty stunning revelation: while there is much obsession about who is and isn’t on those lists, it seems that their nature is a lot more dynamic than expected. Going beyond that, it also look like being on top is no guarantee that you will stay there (if anything, it is a guarantee that you will not, as 9 out of 10 blogs fell and 65 percent disappeared from the list altogether).

Because the overwhelming majority of the blogs listed in May 2005 experienced a downward spin, it seems that the concept of a network effect is widely overstated. In fact, there seems to be the equivalent of a reverse pull, where being a Technorati 100 is only a short lived glory.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Technorati 100 Here Today Gone Tomorrow. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

Before I go any further in this, I want to have a huge disclaimer: I’m a pretty big fan of some Google products. One can see Google ads running on this site (I’m an AdSense user) and a portion of my traffic gets here thanks to Google’s search engine. i’m also a big user of the search engine, I have a Gmail account (although it is not my primary email system) and I use Google Maps and Google News often. I’ve played with the search API in the past and, for the most part, I’ve been happy with my overall Google experience. However, I worry that the company is heading in the wrong direction and I want to ensure they remain a viable player as they have re-ignited investments in the search space, which has benefited all users on the Internet. However, I fear that, if they are not careful, they could suffer a fate similar to that of Netscape, which popularized web browsing and ended up being gobbled up by AOL, where it is now a shadow of its former self.

That said, let’s look at some of the disturbing similarities.

Market Shares are no guarantee

In the early days of the commercial Internet (let’s say 1996), Netscape was a very successful company. It had beaten every Wall Street expectation and completed a stock offering that had captured the imagination of the general public. The Netscape management graced the covers of most magazines in America and the little browser that could (then in version 2.0) had captured an impressive 75+ percent of the market. Netscape had also introduced its own line of web servers, with a proprietary language called LiveWire, which allowed to create more dynamic applications. The company was also offering a web page development tool, and struck partnerships with many companies to integrate their audio and video components with the browser.

Microsoft had come out with Windows 95, which included a browser (Internet Explorer) which they had licensed from an outside source (NCSA, the place where Marc Andreesen had worked prior to Netscape and the browser was Mosaic, an early web browser Marc had been involved with). The world had mostly laughed at the pitiful version 1.0 offering from Redmond. It was simply a bad product, which did not get much redemption with version 2.0.

Microsoft on the Offensive

The folks at Netscape were feeling pretty smug. After all, they dominated the browser market, had managed to get a way to sell server products and comments about the upcoming irrelevance of Microsoft started making the rounds. But the giant was awake and the clouds over Redmond only covered a flurry of activity. By the time IE 3.0 was released, most people had written Microsoft off. If they couldn’t get as simple a piece of code as a browser to catch up, how could they have a chance to survive.

Netscape had come out with version 3.0 and it was good, if a little bloated from the everything but the kitchen sink approach they were taking. Netscape was now offering an Internet suite that included a browser, a mail client, a newsreader client, an IRC client, some groupware capabilities, etc, etc… There was no way Microsoft could catch up.

Netscape Navigator 4 came out and it was good. It was running Java applets, it could do DHTML, etc.. Basically people liked it and didn’t see a reason to switch…

But Microsoft released IE 4.0 and it was better than people expected. it matched the Netscape browser feature for feature and threw in a few things. One of the people in charge of that development was a guy by the name of Yusuf Medhi, who now happens to be the head of MSN.

While Microsoft had fired a major shot with that new browser, everyone expected that all that would change again when Netscape 5 would come out.

Netscape 5 never came out. In fact, Microsoft release IE 5.0 and started gaining market shares (stealing them from Netscape). Netscape seemed to be trapped in its own legacy and had problem getting a new product out. Microsoft release IE 5.5 and Netscape was working on a new rewrite of their product.

Finally, Netscape 6 came out, conveniently skipping a version. Was it the answer to Microsoft that all had hoped? Not quite and by that point it was too late.

Netscape never recovered and now lives as a shadow of its former self. Microsoft put out a 6.0 version of their browser, cleaning up some of the last parts of the markets they wanted and then went to sleep, in terms of browser, until the recent competitive threat of Firefox reared its head, eating up some of their hard earned market shares.

So what went wrong? The answer is complex but I believe that a mix of Hubris (we can beat Microsoft, we have a huge market share) combined with some sloppy releases, the development of a bit of a monoculture (we set the agenda, the industry will follow), an unwillingness to deal with massive competitive threats, a loss of focus on core assets, and a media world that loves to take down the companies they’ve built up all added up.

How does this apply to Google?

For starters, it is clear that massive market shares are no guarantee of success. Google currently holds around 60 percent of the search market, which is good but is also a reason for concern as it is more likely that this share will go down than it is that it will go up.

More worrisome, however, is the development of the Google monoculture. Much of what is going on at Google is happening with little involvement and input from the community. This is where Microsoft generally starts striking. Say what you want about the Redmond giant, they know how to listen and how to take brutal feedback and turn it into decent product. Microsoft is not known for great products but it is known for decent ones. Last week, Microsoft organized Search Champs, gathering a bunch of smart people from the industry in a room and having them talk to them. I was there and was surprised by how focused they are on winning this one. It is the kind of focus I have not seen come from them since the browser wars.

If it wants to survive, Google needs to do something similar. Throwing a product out to the world with the world beta on it is not a feedback loop. Sitting down with users, developers, thought leaders is. The feedback is not always good but it helps improve the product, which is how one wins this war. Furthermore, the goodwill generated by getting people invested in its products and their success allows a company to develop a strong following from a small group of dedicated users, who then serves as advocates in the marketplace. They can have an impact in changing opinion and not involving them can be dangerous.

Of those people, developers tend to be the more finicky. Alas, the success of many platforms on the Internet depends on developers. As developers go, so tend the marketplace because developers tend to be early adopters. Developers were the first people to switch from Yahoo to Altavista. They were the first group to switch from Altavista to Google. Where will they go next? Is it guaranteed that they will stay with Google (however, here is an interesting case, as developers tend to have a bias against Microsoft. The corollary of this is that Microsoft has to offer something that is radically better in order to make gains in the developer world). A good way for Google to mend some of the rift with the development community would be to support RSS along with ATOM as a syndication format. At the current time, Google is the only major search engine without native RSS support.

Another area to watch out for is the loss of focus. Could someone at Google please explain to me how the Google pack, Google WiFi, Google IM or the Google web accelerator fit Google’s mission (to organize the world’s information). How does owning a radio advertising business (something they acquired recently) fit in that model? It seems that Google is trying to do a lot of things in a lot of areas. I’m sure they’re all interesting things but what does that do to the core search assets on which the business was build (or is it that search is just a side business and Google’s mission is really about advertising?) There has been much discussion in the search world about the relevancy of results in the Google search engine suffering from some level of degradation. As always, expectations are high and any decrease (or lack of improvement) in the quality of the search index will be seen as a loss of focus.

Following the Netscape sloppy release, Google also has to worry about better testing before putting products out. Its recent stumbles with the release of Google NewsReader and Google Analytics showed the world products that were not fully ready for market release. The market acceptance for the word beta goes only so far and Google may suffer some reputational damage if it continues along a curve or release first and fix it later (this, however, is not necessarily a standalone cause for failure, as we’ve learned from the release of many Microsoft products that needed their own round of stabilization)

Last but not least is the burning glare of the media world and of Wall Street. As can be seen now that lofty (and, one could add, unrealistic) expectations could not be met, punishment (in the form of a declining stock price) is coming. Similarly, the press is getting more critical. This is part of a normal cycle: a company is hyped up and then taken down. These are just fads (ask your friends at Yahoo!, who have managed to go through the whole cycle and are starting to go back through a build-up phase now).

And, as a postcript, take the advice of pundits like myself with a grain of salt. There are lessons to be learned but I can’t guarantee that these are the right ones to learn. However, what is certain is that Google needs to remain a viable player in search if for no other reason than to keep companies like Microsoft honest. As we’ve seen in the browser wars, once a company wins, it tends to slow down on the innovation front and search is still so young a field that it needs major progress on the innovation front.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Googling Netscape. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

This week, Google introduced two new Firefox extensions: Google Safe Browsing and Blogger Web Comments which are providing richer integration with the desktop and a number of new features based on your surfing patterns.

But the question, when looking at those is why is Google interested in areas that don’t seem that close to search: the truth is that they are closely tied to Google’s business model, even though it’s not totally clear on a first look.

The Google business model: advertising

When you look at Google’s revenue, it becomes immediately clear that search is not really what the company is about: Google is in the business of advertising and search is the way in which it targets its advertising properly. Viewed in that prism, Google is an advertising company and advertising companies generally need a couple of things: eyeballs and data about those eyeballs.

The first part of this is easy to understand: eyeballs, to an advertising company, represent the inventory it has available for sale. However, eyeballs in and off themselves are pretty useless. The common misconceptions made by many companies in the late 90s was that eyeballs alone were important. The truth is that, without any other type of information, eyeballs are close to useless.

However, the more information you have about a set of eyeballs, the more useful it becomes. This was the realization that Google made when it moved the advertising model on its head by targeting ads based on search terms. Google then increased the amount of eyeballs it could get by offering AdSense, a program that increased inventory and provided information back to Google about what people were looking at.

With each new member of the AdSense program, Google gets more information about Internet users. The more information it has about Internet users, the better it can target its advertising.

Enter the add-ons

In May, I posited that the Google Accelerator was about distributing the indexing work. What I failed to realize at the time was that Google was also getting a lot of user information in the process: what do people look at, how long, etc… This information is extremely useful. However, the accelerator had some issues and failed to achieve high velocity.

More deals have followed, with large partnerships aimed at pushing the Google toolbar on as many desktops as possible. One could wonder why the toolbar is so important to Google. After all, they keep trying to get it bundled left and right (with Java, for example) and are pushing it very heavily in their search engine results page. The toolbar is all about getting more information about what people visit.

The new extensions are about the same thing: getting to know you better. The Google Firefox Extensions Agreement spells it out very clearly:

By using the Extensions, you acknowledge and agree that Google may access, preserve, and disclose information regarding your use of the services if required to do so by law or under other conditions set forth in the Google Privacy Policy

Digging into the privacy policy spells things out clearly (the emphasis is mine):

• Google collects personal information when you register for a Google service or otherwise voluntarily provide such information. We may combine personal information collected from you with information from other Google services or third parties to provide a better user experience, including customizing content for you.
• Google uses cookies and other technologies to enhance your online experience and to learn about how you use Google services in order to improve the quality of our services.
• Google’s servers automatically record information when you visit our website or use some of our products, including the URL, IP address, browser type and language, and the date and time of your request.

From here, we learn that Google aggregates data (no big surprise here) and can share it with third parties. Among some of the data is the URL you visited, your IP address (which can then provide some information about your physical locations), and the language you use. Those are all good attributes to narrow down information about a user. For example, if someone looks at a lot of technical web sites, Google will know that this person might respond better to a technical ad. Over time, that information can be aggregated to get a better understanding of different groups and sell very targeted advertising. Let’s look at how Google uses this information (once again, emphasis is mine and this is from their privacy policy):

• We may use personal information to provide the services you’ve requested, including services that display customized content and advertising.
• We may also use personal information for auditing, research and analysis to operate and improve Google technologies and services.
• We may share aggregated non-personal information with third parties outside of Google.
• When we use third parties to assist us in processing your personal information, we require that they comply with our Privacy Policy and any other appropriate confidentiality and security measures.
• We may also share information with third parties in limited circumstances, including when complying with legal process, preventing fraud or imminent harm, and ensuring the security of our network and services.
• Google processes personal information on our servers in the United States of America and in other countries. In some cases, we process personal information on a server outside your own country.

If you remember my first point (Google is an advertising company), it starts to click. The technology and services they provide are not necessarily to the end user; they can also be to advertisers. This is why there is little worry about Google identifying you personally but being able to provide aggregated non-personal information to a third part is what advertising is all about.

In the television world, much of that work is being done by Nielsen (the infamous Nielsen ratings) to define what the audience of a show is and target the advertising properly. This is where the ideas like “give me around 100,000 eyeballs for men 18–24 in the New York area” yields an ad on a sports show about a New York sports team.

However, Google can do that better in that they can offer advertisers something along the lines of “17,000 eyeballs of 19 year old men based in Manhattan, NY with an interest in the Knicks and the Xbox 360, who also read sport news three times last week from ESPN, like the Daily Show and bought hardware and books from Amazon.com in the last 30 days.” It may sound extreme but let me explain how it works:

• The bought hardware and books can be gathered from the fact that they looked at URLs on Amazon.com and ended up on a purchase path as a result of that session (this would all be URL info)
• The same can be true of the ESPN.com and theDailyShow sites (gathered from the URL field)
• The interests (Xbox, the Knicks) can be inferred from where they spend time in their online session or what they searched for on Google
• The location (Manhattan, NY) can be inferred from the IP address they used during their surfing session (alternately, if they use Google WiFi, it can be gathered from the info that client has reported)
• The 19 year old men can be inferred from their email (Gmail) or usage patterns (this is where the research and analysis part come in) as relating to other 19 year old men

In fact Google is so sure of their data that they will guarantee advertisers that, if their ad did not get a response, they don’t pay for it. The TV station doesn’t do that. Once you’ve run the ad and they verified the audience, if you’ve met the number, you’re good.

Let’s assume that you have a million dollars in advertising to try to sell your new widget (which targets that public): where would you put it?

The rich data set that Google is building has tremendous monetary value and that is why they keep pushing new clients that provide them with more info.

Attention as value

Because that data is very valuable, the current stream of organizations like AttentionTrust is one thing Google will have to deal with down the line. Such efforts are actually putting the power in the hands of the users and could potentially represent a threat to Google, if people refuse to start providing data to it. It will be interesting to see how Google deals with this new world.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Getting to Know You. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

Framing the debate

The discussion centered largely around the Google Print Library Project and Google’s decision to scan books without first asking for authorization from the copyright holders. They do content, however, that they will remove books from their index if the copyright holder asks them to do so. In the last few months, the Author’s Guild and the American Association of Publishers have sued Google, alleging violations of copyright law.

Meanwhile, a separate effort set up by some of Google’s competitors (notably Yahoo! and Microsoft) and called the Open Content Alliance has taken an opt-in approach to scanning copyright holdings, including only content that is no longer under copyright protection or content that has been expressly authorized by the copyright holder. This effort has not been sued by the two groups.

Private vs. Public

What is interesting here is that much of the debate really centers around an issue of public vs. private. Google is really creating a private holding out of content initially created by other people. While I initially was on the site of Google when I first heard about this debate, I am starting to wonder whether their position is correct. While it is a good thing that Google gives access to a way to search content which was not previously searchable, why is it OK for Google to not share that content with others? Why is it that they are not joining the Open Content Alliance and sharing access to content they have created? Why is it that they are creating a walled garden around content they did not create and only allow interaction with that content through Google? Those are questions that Google has not answered and need to be answered if we are to trust the company’s unofficial “Don’t be evil” motto.

However, this is an issue that goes far beyond books when you start thinking about it. Google has largely been building a reputation based on its ability to search various types of data, assuming that the copyright holders were allowing them to do so. I first looked into that issue about 5 years ago Deja News put out the “For Sale” sign, which was eventually picked up by Google. What is interesting is that Google needs data. Without it, Google is useless: the value of a search engine is related to how many assets it holds and how well it can organize them. This is why size does matter even though some now try to claim it no longer does.

I would go as far as extrapolate that this is the biggest dilemma for most web 2.0 companies: as more and more of them rely on system where the data is almost as important as how one interacts with it, they are found starving for data. However, they have to balance that with the ideal of being more transparent and share that data with other entities. The dilemma then becomes how to keep a private set of data in the public eye while keeping the public from stealing and/or misusing your private data.

I asked the panelists whether the issue was that Google was turning the author’s data into private Google property and whether Google joining the Open Content Alliance would solve the problem. David Drummond, who was there representing Google did not answer the question. Allan Adler, from the Association of American Publishers, stated that they would drop their objections (and thus potentially their lawsuit) if Google were to follow the established principles of the Open Content Alliance. In order to decipher that statement, I went back to the OCA’s website and looked for what those principles were. They are are follows:

• The OCA will encourage the greatest possible degree of access to and reuse of collections in the archive, while respecting the rights of content owners and contributors.
• Contributors will determine the terms and conditions under which their collections are distributed and how attribution should be made.
• The OCA need not be obligated to accept all content that is offered to it and may give preference to that which can be made widely accessible.
• The OCA will offer collection and item-level metadata of its hosted collections in a variety of formats.
• The OCA welcomes efforts to create and offer tools (including finding aids, catalogs, and indexes) that will enhance the usability of the materials in the archive.
• Copies of the OCA collections will reside in multiple archives internationally to ensure their long-term preservation and accessibility to all.

The last few words (“and accessibility to all”) are particularly interesting. These, I believe, may be a large part of the reason Google is not going to join the OCA.

In a way, Google is appropriating other people’s work (the actual content of the books) and creating a private property around it. Had Google created the content or provided the tools to do so, they might have a claim to being part of the creation. However, it seems that, in scanning the content, they are appropriating content which is not rightfully theirs without first asking for authority to do so. That can’t be right.

What price for those rights?

It is interesting that Google has wrapped its argument around the Fair Use doctrine as the copyright office seems to clearly state that one of the factors to consider is

the amount and substantiality of the portion used in relation to the copyrighted work as a whole

The reason that is interesting is that it points to an issue in terms of whether they are infringing or not. Considering the fact that they do have to copy the works in full in order to be successful in their undertaking, it seems that they would indeed be in infringement under a strict reading of that section of Copyright law.

One of the items that were overlooked by most of the media coverage is the question of price for the rights. Larry Lessig, during an exchange with Nick Taylor, of the Authors Guild, stated that he feared that the Author’s Guild and the Association of American Publishers would eventually settle their lawsuit with Google. This fear is well grounded when one realizes that the majority of lawsuits are settled out of court but it gains extra weight if there is a potential that Google will lose. To understand Lessig’s fears, however, one has to go one step further and start looking into the effect of such a settlement. First of all, Google is rich (as of this writing, Google had a market capitalization sitting north of $100 billion); There is nothing wrong with that, except for the fact that they can pay a lot more than other companies could. If they were to settle with the authors and publishers for a lot of money (which is what the receiving parties will be pushing for), they will create a precedent whereby rights that previously were available for free will now have a fairly hefty price tag.

This is not only bad for people trying to develop new businesses to compete with Google but has a potential for being bad for democracy in general as it might create two different groups in a society: those who can pay for access to certain content and those who can’t. In the long run, that sounds like a pretty evil thing to me and this is, once again, where the need for a system that is accessible to all and collections that reside in multiple archives are an important pre-requisite. If the authors and publishers are serious about being remunerated for their work, they are going to have to play this one for the long run. What it means is that settling is not an option! They must see this case all the way through to the Supreme Court of the United States. The reason this is necessary is that, if they settle, they change the negotiation from one where they are of equal weight to an asymmetric one where Google has all the power (because it keeps the access locked down). In the future, Google could decide what and when those authors and publishers have a say in that relationship. This is very dangerous. In a way, the relationship is one that fits a prisoner’s dilemma scheme nicely, showing that the only solution is to keep fighting:

It is interesting to see that there is really no room but to fight. In a weird way, Google has become its own anti-thesis, being evil as a direct result of its own actions. Because, in order to protect its own economic interest, it must keep a walled garden, Google is stuck in a position where it will have to negotiate rights or lose the right to go after print. From the Google standpoint, the decision is to get one party to settle and leverage that into a position of strength to force the other party to settle. Once a settlement has been accomplished with both parties, however, Google will have established a price tags on rights. Because of that price tags, many parties (whether individuals or companies) will no longer be able to play in that space. Many could debate whether this is intentionally evil or not but few can deny that it creates an evil state of affairs.

And what about CC?

A lot of this discussion, of course, cannot happen without taking into Creative Commons into account. I was surprised that Lessig was not making more of a case for the CC license to the publishers and authors. However, it was interesting to see him grilling Allan Adler on what constituted fair rights. Adler took a very evasive approach to dodge the question, leaving it absolutely unanswered. It is, however, an important question that needs to be dealt with if any resolution is to come.

One of the possible compromise would be for Google to agree they will no longer force an opt out model in exchange for a blanket endorsement of CC by the publishers and authors. Because CC licenses has a number of variables, it might allow some speeding up of the process in terms of willing to grant rights. This would also greatly benefit the Open Content Alliance project and thus ensure that content is widely shared and distributed while allowing content authors and publishers some level of control over what rights they would give away. The funny thing is that this may, in the end, be the only way out of the mess Google has created and that no one else seems to have suggested it.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Copy and Print. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

The Search Space

Google is undoubtedly the leader in search. It is what they specialized in and continues to be their most cherished asset. But does Google offer search products that fill a niche which is not covered by its competitors? Let’s take a look…

The interesting thing, when looking at this data, is that, apart from Catalog and Usenet search, Google does not offer services offered by others or currently under development. Interestingly, Google does not have any offerings in the Audio (nor a Podcast offering) and Encyclopedia space (although Wikipedia results sometimes pop-up in search results.) This seems to highlight two potential areas where Google will introduce new products: an audio search engine, which will include podcasts, and some type of partnership with Wikipedia to fill the reference space.

What is interesting here is that Google has generally been the first to market with many of the search collections listed. From this, one can deduce that Google works more as a competitive threat to its competitors, forcing them to invest more in their search product and, in the process, improving the quality and breadth of search data for every user on the Internet. This is a good thing but not revolutionary unto its own.

Search Services

The next area I decided to look into, in order to divine the whats and wheres of Google was the type of search-specific services it offered, compared to the same competitors.

This is actually interesting in that the offerings are pretty close. Of note here is a departure on the part of Microsoft, which is experimenting with clustered search. None of its competitors have show a product in that space and this may be an interesting indication of how they plan to play in that space.

Also of note is the fact that only Google offers a paid answering service (Google Answers). A question for players in that space could be whether something like the recent Mechanical Turk offering from Amazon could help a company fill that niche. This seems to be an untapped market that is only being mined by Google.

Non-search services

OK, so we can clearly see that Google has done a good job in the search space and its competitors are working hard to play catch up in that area. While they’re doing so, Google has been busy ramping up its offerings and closing some holes in terms of being an online media player. Let’s take a look at how it is fairing in the non-search space.

Of note in that area is the fact that Google has managed to revamp the email space with its Gmail offering, forcing Yahoo! and Microsoft to work on a revamp of products user-interface that had not really evolved much since their introductions. A couple of interesting holes in the Google offerings in terms of auctions and calendaring will probably be filled in the near future with online offerings closing the gap in those areas. Heck, even people working at some of their competitors are clamoring for such offerings.

More interesting, however, is the fact that Google is the only player in that space without a substantial access offering. Basically, they’ve been using the public internet as their accessibility world. This can provide some details as to the recent rumors of their developing a large scale WiFi network and some of their interest in purchasing dark fiber or other rumors about their interest in AOL.

Once again, it seems that Google has served well as spurring its competitors into action but the magic Google sauce does not seem to reside in the product offerings.

Developer Services

While all those offerings seem of interest to the general public, Google has been doing a good job in catering to early adopters, who generally impact general opinion. When doing a comparison on that space, it was fascinating to see that Google took the lead in most categories and that AOL did not even play in any of them.

That last item is one to ponder. Google is not in the hosting business yet. But it seems that there is potential for them and, once again, could play along the lines of Google trying to harvest dark fiber. They’ve revolutionized the online email space by offering a larger amount of disk space than any competitors. It seems they could be doing the same in the hosting space by offering a combination of easy to set-up and update tools (based on the blogger set of templates) with some more powerful features like Database management (the rumored GoogleBase, which now has its own URL, even though the code still seems to be sitting behind a login area.

Conclusions

Google does innovate in some spaces but has largely innovated in order to gain entry in markets that already existed. As a rule of thumb, they’ve been very smart at breathing new innovations in those markets. However, their competitors are generally quick to notice and are catching up.

In terms of future offerings, I would not be surprised to see the following products coming from Google over the next few months:

• An audio search engine, which will include a podcasting component (and possibly a podcast authoring component via blogger)
• A strategic partnership with Wikipedia or some other encyclopedia
• Some type of clustered search offering
• A calendar product, which will probably inject new life in that space
• An auction offering, tied with an internal payment system
• A web hosting service that will scale from small entities to large ones and will include Gmail as part of the email offering
• Some type of access service, probably using their WiFi solution

Whether that all happens of course is pure speculation on my part and whether it is enough to sustain their market capitalization (north of $100 billion as I write this) is something I better leave to people who know how to invest.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Reading the Google Tea Leaves. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

Grouping the metrics

In order to figure out weighting, I first started to think about how to group different metrics. For this purpose, I looked at things like the base value (which would give us a baseline as to how much a business is worth based solely on revenue and revenue growth), inventory (looking at things like traffic, reach, and output, because they all give us some data points as to the growth of monetizable assets in the future), consumer involvement (looking at info like links, subscribtions, and comments to define the value of customers), and growth potential (including some more fuzzy measure of potential growth and the advantages of the integration value).

My reasoning for grouping things in this way was that it might make it easier to figure out weighting across those large catch-all categories (and, if there is any discussion at all, I am sure that people will debate the percentage assumption against those categories). I, in no mean, try to represent those as the be-all-end-all approach to valuating a business. They are, at this time, the metrics that give me the best comparative view of a business, when I try to assess its value. However, not being much of a metrics guy to start with (my main reason for doing this series is to provoke debate among people smarter than me so there can be some consensus on metrics in this new web 1+n.x world), I hope that others will step in and show me the error of my ways along with providing some interesting information that will get all of us closer to something useful.

Base Value

The base value, as I see it, is defined by revenue and revenue growth based on historical data. The reason I would consider this to be the base value is that it is a reflection of the business as it exists today and can provide a baseline as to where the business would be headed if growth suddenly slowed or investment in the business stopped. It does not provide any information as to how to accelerate the growth of the business and does not provide more than a view into the present cash value of a business.

However, for young companies, such value does not provide much information. Start-ups, by nature, have a lower revenue and profit line than established companies because they need to recover some of their initial cost and may still be in high growth and research and development phases. As a result, to solely base one’s view of a business on its current ability to generate cash is short-sighted when it comes to start-ups.

Another question when developing the base value is how to factor in risks to the revenue base. For example, if the business relies primarily on advertising from an external network as its basis for revenue (many people have talked about businesses looking to AdSense as the primary source of revenue), one has to wonder what would happen if the dynamics of that relationship were to change.

As a whole, however, because of its overall importance in assessing the present financial value of a business, I would assume that the base value should represent about 20 to 30 percent of the overall value of a business. Initially, the value would be in the 20 percent range because potentials are higher than the current revenue line but, as the business matures, and potentials decrease, it would edge up towards 30 percent.

Inventory

Inventory would be the next potential grouping of different metrics. In it, I would include traffic (and traffic growth), as well as visitors, site counts, reach, and output. Let’s go into more details on each of those.

Traffic is important because the number of page views is something that is monetizable. However, in a web 2.0 space, pageviews are not the only traffic metric one should track. For example, RSS subscriber counts is another useful value (and controversy has already swirled around ads in RSS feeds). However, I would argue that there is one value in the inventory count that is of utmost importance: access to an API. The reason I would venture this is the most important inventory metric is that APIs, once implemented, are harder to unhook from. As such, they represent a harder type of value since they solidy a site’s reach within a particular market segment.

I believe that reach is actually going to be seen as one of the more important values in terms of inventory. The reason is that reach gives us an idea of the potential growth opportunity in a market. If a company has a high reach in an individual market, its potentials are more limited. Witness, for example, a company like Netscape, which once had a reach of 80% (ie. 80% of all internet users were using it. ) Tactically, this kind of position is one where they should have been on the defensive, the reason being that there was more potential of a drop in their reach than an expansion of it. Microsoft is now finding itself in the same position on a number of fronts: Windows, Office, Internet Explorer are all playing in a world where they will not reach a higher percentage of the market. As a result, they are forced to play defensively. One could argue that web 2.0 companies, with their reach APIs and more powerful front ends (thanks to technologies like AJAX) are representing the threat Microsoft saw coming from the Internet in the mid-90s. And one could argue that, this time, the position they’re in (ie. largest player) is endangering their future if they don’t make a radical change (because they can’t grow from the position they’re in).

Going beyond the reach, which provides some information on past growth and potentials moving forward, one has to look at output from the company. For example, in the case of a company like Ebay (arguably a web 2.x company already), the inventory is number of auctions submitted. Similarly, in the case of Craig’s List, it would be number of new ads posted, or in the case of a blog, the number of posts created. One has to be careful about ouput, however, and should measure the cost of output in order to figure out whether the output is good or not. In the case of web 1+n.x companies, output is a very good thing as it is generally created by outside parties for free. That free product is one that those companies then monetize. However, one has to be careful and evaluate if output is outpacing the company’s ability to monetize it because, if an imbalance were to start existing, the value of the output could potentially decrease.

All and all, because inventory has a measurable value and, in general, is the very thing that a company will monetize, I would guess its weight, when figuring out the value of a company would probably sit in the 10–15 percent range.

Consumer involvement

Consumer involvement, which was known in the past as stickyness, is another major group of metrics. This section would include links, subscriptions (both to RSS or API feeds and, if offered to any paid type of service), and any type of interaction a user may have with a system. For example, if you trying to get some interaction information on consumers of a blog, one could look at numbers of comments posted. Alternately, if you’re looking at a search engine, one could look at number of searches performed. Or, if you’re looking at a company which offers an API, you could look at the number of times that API has been integrated in other products and the number of times it is accessed.

I would venture to say that this metric is one of the most important ones when assessing the value of a business. The reason I would value it higher than the ones I mentioned earlier is that this is where one can see whether a business has a potential or not. The interaction with customers (either directly or via APIs) provides so much useful information that a company not looking at this metric is probably off track in terms of evaluating itself (and, generally considering the hype around new businesses, such company could fool itself into extinction as it fails to see major issues arising out of the increase or decrease in consumer involvement.)

Because it represent the value of the existing cutomer base and provides some input as to the trends surrounding that customer base, I would throw a weight of 30–35% of an overall valuation going to factors relating to consumer involvement.

Growth potentials

However, metrics in and off themselves, are pretty useless as a point in time number. As a result, one has to assume the growth potential when evaluating a business. The growth potential can be associated in a number of ways but, when it comes to web 1+n.x properties, it comes down more on the side of potential based on a number of subcomponents. In the interest of provoking more controversy, I would venture that there is a formula to calculate potential and that it is as follows:

Potential = traffic growth rate * reputation vector * brand equity vector * (integration vector (squared)) — ( Risks vector / percentage of risk that can be mitigated)

In the growth rate area, I would put an aggregate growth rate that averages out growth rates over a period of time (6 months to a year if you are computing a monthly growth rate.) The reputation vector and brand equity vector would be values based on reputational and brand equity trends, which I talked about in a previous entry. You will notice that I consider the integration vector to be of such high importance, when defining potential that I’ve decided to square its value. I will talk about integration vectors in a future entry but, put short, the integration vector is the magic glue that makes acquiring or merging a company very valuable because integrating it with another company will derive greater value for the combined entity. It is that issue generally known as synergy but trying to put a value on it would have the potential of making for better, more successful acquisitions and mergers. Last, but not least, is the rist side of the equation. Because risks have a huge impact on potentials, it is important to measure them in order to get an idea as to their potential impact. However, because some of the risks can be mitigated, it is important to capture this figure in order to assess the importance of different risks.

Ultimately, growth potentials represent the largest part of any equation when trying to value a company. Few companies are bought without an expectation of potential and this is why, in my weighting, I would assume potential to represent a substantial (30–40%) part of the equation when trying to measure a company’s value.

Conclusion: Wait, that’s more than 100 percent

If you do the math, it appears the different weight are ending up representing more than 100%. The reason is that those are ranges. However, the truth is that, in any business dealing, there is also an amount of faith and luck that comes in. For example, I sat in a meeting once where an individual was given an option to buy in whole a company which is now very successful on the Internet for around a million dollars. Looking back, it might have been worth that much at the time but I doubt that it would be worth what it is worth now (several billion dollars) had that deal being consumated. Over time, the management of that company was smart enough to mine opportunities and put people in place that helped them realize huge growth. Had that company been in the hand of more conservative (and by conservative, I mean adverse to risk) investors, it would probably not have flourished in the same way.

Having gone through a few days thinking about metrics, it is clear that there are a number of opportunities for people smarter than me to figure out some solid metrics in assessing the value of new companies. Metrics, however, should not be the sole guide when assessing a company.

Many people have asked me why I bothered to look at such boring subject (and why I’ve been blogging so incessantly about numbers lately). My main reasoning is that one of the failures in Web 1.0 (the bubble we lived through in the 90s), lack of accountability and/or expectation management lead to very inflated numbers that eventually left a lot of investors with very poor investment. Having lived (and survived) that bubble, I want people to start thinking more critically about Web 2.0 companies and, hopefully, we can all learn about the mistakes of the past and avoid over-hyping new companies into extinction… because for every bubble, there is eventually a big pop, and no one really enjoys that part.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Metrics — Weighting the Metrics. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

Google vs. Yahoo

A few weeks ago, Yahoo! claimed that its index was over 20 billion items large, broken as follows:

just over 19.2 billion web documents, 1.6 billion images, and over 50 million audio and video files

If we assume that Google believes its nearest competitor is Yahoo!, this would put the Google index at roughly 60 billion items, a fairly large number, which is probably on the high side. So we need to do more analysis in order to get closer to the truth.

Google vs. Google

As part of Google’s seventh birthday celebration, Google staffers posted an entry on the official Google blog, claiming that their index is now 1,000 times the size of their original index. If that’s the case, figuring out what the original index size was should give us a good number. Fortunately, I have a copy of John Battelle’s excellent book about the company (it’s entitled The Search, which is a must-read for anyone interested in the search space. No other book has gotten as deeply into the history of internet search and few have analyzed more keenly potential futures for Google). In the book, Battelle relays an email from Larry Page to Terry Winograd dated July 15, 1996. In order to give some context, one has to realize that Google started in March of 1996 so, in July of that year, Google was all of four months old. The email is regarding some of the growth issue that the search engine is having and reads (emphasis is mine):

I am almost out of disk space.

I have downloaded about… 24 million unique URLs

and about 100 million links… I think I will need 8 gigs more to store everything… Current retail prices are about $1000/4 gigs… I have only about 15% of the pages but it seems promising

If we take that number as a starting point, that would mean that the original index was around 24 million pages. From there, it is easy to multiply by the 1,000 factor they talk about in their blog and get a number of items in the Google index.

That number would be

24 billion items in the Google Index

, a little more than what Yahoo! has in their index.

Google vs. MSN

In November 2004, MSN was estimated to have about 5 billion pages. Ken Moss, the General Manager of MSN Search claimed that they added a lot to their index. While he’s not forthcoming with any detailed information in his post, we can still assume that the MSN search index is now larger than 5 billion.

This is interesting in itself in that it may actually help us triangulate to the right size for the Google index. If we try different growth curves against the MSN search, we could look at the following:

• Growth Curve of 50%: MSN Index is now 7.5 billion items
• Growth curve of 75%: MSN Index is now 8.75 billion items
• Growth curve of 100%: MSN index is now 10 billion items

If we take Google’s assessment that it is three times larger than its nearest competitor and assume that Google is considering MSN search to be its nearest competitor, those growth curves translate as follows:

• Growth Curve of 50% at MSN: Google index is 22.5 billion items
• Growth Curve of 75% at MSN: Google index is 26.25 billion items
• Growth Curve of 100% at MSN: Google index is 30 billion items

When one looks at those results, a pattern emerges: Let’s first remember the rough claim of 24 billion based on the Google vs. Google analysis above. On the 50% MSN growth curve, Google is at 22.5 billion items indexed. On the 75% MSN growth curve, Google is at 26.5 billion items indexed. It could then be that Google considers MSN Search, and not Yahoo! to be its nearest competitor, as the 24 billion mark seems to fall right in between.

Conclusion

While the index size is largely a game of public relations, it appears that the Google index is sitting somewhere between 22.5 and 26.5 billion items indexed and, more probably than not, at the 24 billion items indexed mark. This gives it a slight edge over the Yahoo! index and shows that the company considers Microsoft its nearest competitor. Of course, this is my own speculation so your mileage may vary.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Google has 24 billion items index, considers MSN search nearest competitor. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

As readers of this site might have noticed when using the web interface, I am using the Google Ad service called AdSense. As I am not at freedom to reveal the terms of my contract with them or discuss specific, I’ll talk in general about online advertising programs.

The first thing to take into account is that the model on using advertising in archives is one largely predicated on a long tail concept, whereas one can make more money from small increment over a long run than trying to score the big hit. In my case, this means trying to get a few good stories out on a regular basis, none of which is going to make lots of money on a single day but a few cents or a few dollars a day can add up to quite a nice payoff on a yearly basis. I believe that people who blog and develop a nice audience can see some of those results.

Let’s take a hypothetical story of 1000 words. A long-standing view in the journalism word is that a dollar per word is the standard (I’ve seen much lower rates as a result of the downturn in technology publications but more on this later). So a writer would write the standard story and get $1000 back for his or her effort. At that point, the publication would print it and/or publish it on their website and start generating revenue against it. The model here, is that the publication is taking a risk with the writer, fronting the money and will recover the money over time. The other part of the equation is that publication has a built in audience and therefore markets the writer to that audience. Because they market their own publication, there is supposed to be a halo effect that shrouds the writer into the great light of being associated with publication X.

The truth, however, is a little different. Unless you’re already an established brand or have a story that is of such import that it will rock the nation or the world, no one will care who the writer is. For a quick test, try to think of who were the writers who wrote the front page story of any given mainstream publication this week. Let’s assume you passed that test, what else have you read by them?

By contrast, blogs offer the writer a single platform. When I visit a particular blog, I have a relationship with the writer. Over time, as I visit it more, I get to know the writer’s brand. This is important because several bloggers have already made the transition from the blogging world into traditional media on the strength of their audience.

This, of course, is a role reversal as blog writers now establish their own audience, decreasing the need for the much vaunted halo effect large publications can give them.

Going further, one has to think about the long term dollar (or Yuan, or Euro) value of a story. As I mentioned above, the traditional freelance writer hands over the story and gets paid. That’s where the money stops.

However, on a blog, the story stays online. If the blog is reasonably optimized to figure prominently in search engines, this is where it starts having a second life. Long after a blog post has been made, it still gets traffic. I see this here at TNL.net on a few popular posts covering areas that no one else seems to have bothered with. And this is where the incremental revenue start to come in.

Once the entry has advertising on it, any revenue generated from that advertising goes to the blog writer. Initially, it’s not comparable to the thousand dollars the writer got from a mainstream publication but, if the entry has legs (ie, it keeps serving an audience), it continues to generate money, pretty much until one of a few possible things happen:

• The writer decides to remove ads from his/her site
• The writer decides to remove the entry from the site
• The story has run its course and is no longer useful or superceded by a better one

If one writes with such a long run view, a story can generate several times what the initial payback was from a publication.

Another important thing, when one writes a weblog, is to ensure you promote your work properly. I read a number of weblogs (over 300 at this time) and have gotten familiar with a few of the writers. If I write an entry that may be of interest to them, I have no qualms about dropping them a quick email with the link to the story and the content of the full entry in it, along with a short note as to why I thought it might interest them. This kind of self promotion generally helps in driving initial traffic to the entry.

But this does not have to be something that is limited to bloggers. Writers can also take advantage of it. As per word rates have been decreasing, it has been easier to negotiate on certain terms in a contract. One of the terms a writer should always try to negotiate is the length of exclusivity on rights. I generally negotiate for rights on a piece to revert to me after a certain period of time (for example, exclusive rights being given until 90 days after initial publication). Writers will find that most editors are more willing to negotiate such terms (as they have some leeway there) than they are to increase the per word rate. Once a story is published, I then make an entry in my calendar (or in the case of the custom publishing solution I have running this site, in the content management system itself) to publish the story on the site 90 days after the first day the story was published (I check that either by having a copy of the print publication in hand or seeing it live on their site). That content then stays online for as long as I want. In some cases, it gets interesting as stories I wrote years ago are getting a little extra traffic and making some extra money. It’s the fire and forget approach.

The result is that there is a lot of money to be made in such archival content, small dimes adding up to, hopefully, full dollars.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Money in the archives. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

At first look, it seems to work fine. Blogs are easily findable, although it’s not clear what the ranking algorithm is.

One of the thing I’d like to see, however, is for big players like Google to start offering a ping service. At the current time, every major blog search engine seems to be using Weblogs.com as their main alert mechanism. Dave Winer has been extremely generous in providing this service to the community but, now that for profit entities are starting to use it, it’s time for them to step up and provide some help here. Add to the community by offering equivalent services and making them as available as weblogs.com. Another way a company like Google (or Yahoo!, or Microsoft (hey, Scoble, can you get involved here ), or…) could help would be to offer free hosting on their server to Dave for the weblogs.com service. They have loads of bandwidth and servers and could provide it as a service to the community.

All and all, a good first start and let’s hope that we’ll see some of the improvements I suggested soon.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Google Introduces Blog Search. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

To recap, I initially took the list of Top 100 blogs listed by Technorati on May 19th, 2005 and started doing side by side comparisons. I initially looked at distribution of links among the top 100, then followed up with an analysis of Technorati against Google, this brought me to a subsequent chapter on Technorati against Google and Yahoo! (then comparing Google and Yahoo! to each other). All this created some fair amount of buzz in the search world, with people saying it was interesting to other saying I was way off the mark. Either way, it’s time to take a look at MSN, in order to complete this round-up.

So, to create some benchmarks, let’s start taking a look at distribution of Technorati links against MSN’s:

Of course, no big surprise here. This seems to be pretty consistent with what I had found in dealing with Google and Yahoo!, showing that Technorati does a good but not complete job at indexing link-backs. What’s interesting, however, is that Technorati seems to have a different pattern when dealing with MSN than it does with Yahoo or Google. Let me show you what I’m talking about. Following is the pattern of Technorati differential with MSN:

… and now is the differential between Technorati and Yahoo..

.. and finally the same graph between Technorati and Google

I’ve been trying to understand why this is and still have no clear answer, to be fully honest. Could be something, could be nothing. I’m not sure at this point and this is, in large part, one of the thing that was frustrating in working on this entry. I’m not sure there is something there, to be very honest.

Comparing the Search Engines

However, the picture gets more interesting when you get the three search engines side by side. Here’s a quick spreadsheet of the results:

The most interesting thing here is that MSN seems to prove the assertion I had made regarding Google not providing as many links as Yahoo does. The same seems to be true between MSN and Google. There were, however, a few surprises here, as far as I’m concerned:

• Sites located in the United States seem to fair better, on MSN, than other sites. Google and Yahoo seem to have a stronger indexing presence outside the US than MSN does.
• MSN spaces sites are not getting particularly great representation in MSN search, compared to its competitors. I was surprised by this since they are part of the same service

Conclusions and more!

So there you have, no great insight here apart from the fact that this linking stuff is interesting and that even small scale analysis can bring up some interesting trends. As I mentioned before, I am not an expert on this and thought to put together the numbers and start an analysis. However, I know that this series has attracted experts so here’s a deal: I’m making the spreadsheet of data I compiled available under a Creative Commons License (By Attribution, Share Alike) here on TNL.net. If you manage to do anything interesting with it, drop me a note and please make sure that you share it with the wider public. Enjoy!

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Links and Search Engines: The MSN edition. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

Some History

Before I go into details about Apple’s new offering, I want to give a little background that will clear up some of my confusions. I’ve been involved in the RSS community since 1999, way back when it was just the domain of geeks.

Back in 2000, I made a few suggestions as to how RSS could be improved. At the same, the main version of RSS was version 0.91 and there was some interest in making a new version that would be called RSS 0.92 (yes, it was the alpha days of RSS). So five years ago, I was pushing for crazy concepts like adding a date to an item or finding ways to attach sound files and video files into RSS feeds. Because of that, some people have asked me to opine on things like podcasting and my general contention is that podcasting is a good thing and that the way support for richer files is implemented in RSS is much sounder than what I had offered in the past.

Subsequent battles created a fork in the RSS movement, with one of the main issues being the use of namespaces in RSS. From there came the great split, with RSS 1.0 breaking rank with previous versions of the format, and RSS 2.0 breaking rank with RSS 1.0. Two formats, which moved in parallel. Dave Winer did a great job promoting the 2.0 format and eventually, a majority started supporting it. Since then, a third syndication format (known as ATOM) has popped up and its making its way toward a 1.0 release. With all this, we’re seeing a lot of smart people basically trying to solve some of the same problems, without really working together.

A proposal

Looking at this, I pity the fact that it took us so long to get as far as we’ve gotten. However, with large players now dancing in the syndication space, I am starting to worry that things are going to get worse before they get better. As a result, I’d like to offer a modest proposal: let’s merge all this work and come up with established data sets that are compatible. The use of namespaces for each vendor use is a great idea but shouldn’t one first think about what they are trying to accomplish and look at prior art before trying to reinvent the wheel? Let’s look at the example of today’s announcement from Apple.

RSS Does Media

So podcasting is becoming much bigger. And videocasting is coming soon. How about looking at media use in RSS. Wait, what do you know: Yahoo! has done some of the work already with the media RSS specification (I know this is the second time in as many week that I’ve pulled out the Yahoo! name but it’s because they’ve been doing good work). The specification provides a number of interesting things so I would suggest that Yahoo! and Apple developers sit down together and come up with an agreed upon set of definitions. Here are a few things that I would put on the table for discussion by both entities:

1. A common namespace: it would be nice if they both agreed to a common namespace. I’d reccommend something that does not include a version number (a mistake made in the Apple spec) but it might be nice to have it set as a DTD, which could ease validation.
2. Add media:group to the final specification, it looks like a very valuable one, especially for content that is encoded in more than one way (this will probably be something Apple does not want)
3. Retain media:category and have it replace itunes:category. Here, the Yahoo version seems to provide for more flexibility
4. Replace itunes:explicit and media:adult with media:explicit. What is defined as an adult varies from country to country whereas explicit is well, more explicit.
5. media:text should replace the itunes:subtitle and itunes:summary but it should also get something added to differentiate the two (maybe a content attribute?)
6. itunes:author could be taken care of with media:credit. Maybe this one could be required. The role of owner should be added to it and an extra attribute could be added for email which would cover the whole itunes:owner section
7. itunes:images and media:thumbnail could be merged
8. itunes:block is a good idea and could be created as a new media:block element which would also have a distributor attribute. This distributor attribute would allow to block different distributors moving forward so a creator could decide to distribute certain content only to certain channels.

If both Yahoo! and Apple were to agree to do this, they would end up with a much stronger joint specification and I believe it would also represent a show of good faith from both companies and an understanding that cooperation is good for everyone. I may dream but I hope that we will see this kind of partnership happen, which is why I’d like to ask everyone to make sure to tell their friends about this entry. Together, maybe we can get Apple and Yahoo! to work together on cleaning this stuff up (and anyone else who wants to play in that space, including Microsoft and Google). Otherwise, we will see increasing fragmentation of the markets, which will result in less content for each of the specification proponents.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title RSS and Media: Can’t we all just get along?. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

Gathering the data

As I had done for the previous effort, I gathered data against the same list of site around the same date. This provided me with some consistency in the data set that allowed for better comparison. Compare one or two site and you may get some false positives. Compare 100 sites and things start getting a little more interesting. The Yahoo! data ended up looking at this (for people who are new to the series, I am doing the same graphs for a number of search engines):

The first thing of interest when putting together that set of numbers was how much larger the number of links found in the Yahoo! index was, compared to the number of links found in either Technorati or Google. The second item I found interesting was a relative consistency in terms of Asian sites not figuring well in the Yahoo! index compared to the Technorati one. It seems that Technorati is getting a better handle on the Asian blogosphere than Yahoo! is, a surprising result considering how much time and effort the latter has put into its Asian operations.

In order to get some real visual comparison, I decided to draw a similar diagram of the link percentages distributed across all 100 sites. It looked like this:

The interesting story, looking at this is that it appeared that there was much greater variance from site to site in the Google index that there was in the Yahoo! one. In the Yahoo system, the vast majority of site fall in the below one percent range but what became even more interesting was that the rate of variance was really not that high: when comparing the median and the average, it turned out to be less than .1% of difference:

While the number were vastly different in terms of size (it appeared Yahoo! had a lot more links), I figured the patterns would be roughly the same in terms of coverage: I expected the top sites to get better coverage in a large search engine like Yahoo! than smaller sites. Imagine my surprise then when I started to do some group analysis:

Those numbers seemed to be all over the map, a fact that became much clearer once I graphed it:

None of the nice downgrade curve I had with the Google set. Here was a much more disparate set, providing little in terms of supporting a theory of bias from a search engine. In fact, it worked more to potentially prove such theory wrong.

Was my data set wrong? I rechecked it and it was not. So what was happening here? As dreams of long tail and power law distributions fell out, I started to wonder how Yahoo! and Google compared. So, of course, I decided to run the numbers again…

Yahoo! vs. Google

This time I decided to compare Google and Yahoo! First, I figured I would get some reference data on the subject. I was surprised to not find any actual side by side comparison on a large set of sites. Anecdotal evidence existed but nothing compared to the data set I had amassed so I figure I would trust my own data set (note: If you have a better one, please leave a comment as to where it is located). The set ended up looking like this:

Nothing particularly impressive there. It seemed that Google, on average, ended up with only about 3% of the links Yahoo! had in its index. However, the story got more interesting when looking at divergence between the average and the median, as it seemed there was a statistical divergence (almost half a percent) between the two:

But wait, for the weirdness is only getting started. Next up was looking at the distributions (as I’ve done for Technorati vs. each of the engines):

I looked at the number and they did not seem right so I ran them again and ended up with the same results. Ran them a third time and still couldn’t make sense of it. So I graphed it:

… and to my surprise, it appeared that the further down the line one went, the greater the differential. In fact, sites that are in the bottom of the top 100 are one full percent more likely to get indexed in Yahoo! than in Google.

Conclusions

From here, we can draw a few conclusions:

• Yahoo! generally does a better job at indexing the blogosphere than Google does. We know they have been working hard to improve their index and here’s proof that they are getting results
• Even if Google is the one with the motto about not doing evil, Yahoo! seems to be the one interested in giving equal opportunity to the little guy: smaller blogs seem to have a better chance of being recognized by Yahoo! than they do of being recognized by Google
• While the front page of Google advertises they are currently indexing over 8 billion pages, it is very difficult to find ways to support that claim via the link feature they are offering: this can be seen as confirmation that Google does not tell you about all the links it has in its index.
• Sure volume counts but in the case of search indexes, they may count against sites: if one is less likely to appear in Google than it is to appear in Yahoo! and the Google index is much larger than the Yahoo! one, then, if Yahoo! and Google had the same amount of traffic, a single blog could find itself receiving more traffic from Yahoo! than it does from Google. This would be due to the fact that each individual page in Yahoo! has more weight than it does in Google.
• The top 100 blogs have other 56 million links in the Yahoo!. That’s a lot of links and clearly shows that links are the currency of the blogging world. It would be interested to get data that would help analyze how much interlinking exists across those sites.

Up next, we’ll take a look at how MSN plays in all this game. So stay tuned!

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Technorati Yahoo and Google Too. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

Source Check

In my initial review, I noticed that Technorati was ranking sites bases on sources. However, incoming and outgoing information is not really available from the major search engines when it comes to sourcing data. So, for this particular investigation, I decided to dismiss source data and focus on link data. I decided to go and get link data from the three largest search engines: Google, Yahoo! and MSN (that last one was included at the last minute just because I knew that Robert Scoble would complain about the study being biased if I didn’t include MSN).

Picking three search engines was also interesting because it providing some sort of reference check. If one of the engines did not line up with the other two, we could point out to a potential flaw in that engine instead of trying to understand why the data was wrong.

Having picked that data set, I decided to start gathering the data. Let me say that it’s a lot of information and, should I try to do this again in the future, writing software to gather the information will probably be less time consuming that trying to get it by hand.

But enough about the process, let’s get into the numbers.

Technorati vs. Google

So the first dataset I created was a comparative index of Technorati and Google. The set was created by grabbing the number of links to a site in Google and getting the equivalent value for Technorati. The resut looked like this:

The third column in this is just a quick set of calculation providing us with some data as to what percentage of Google links was available in Technorati. From there, we’re already noticing some interesting trends. While most of the data ends up showing Google has having a larger set of links in its index than Technorati, there are 16 cases where the Technorati index of links is larger than the Google one. In any study, over 15% of a dataset is statistically significant. How Technorati ends up getting more data than Google is something that someone might want to investigate. Beyond that, it appears that Technorati gets about 30% of the links that Google get to a particular site, as illustrated in the chart below:

The next set of interesting findings is that while the linkage from Technorati is generally lower than it is in Google, it is consistently that way. A quick analysis of the data set shows that the average percentage of Technorati links compared to Google links is not that far from the average median of Technorati links compared to Google links. Confused by that last sentence? Don’t worry (I was too after I wrote it) and let me show you, by pulling out another data chart:

Doesn’t it all become clearer? On average, for the top 100 bloggers, Technorati holds 27.56% of the links that Google holds. Part of the reason behind this may be that Technorati only represents the blogs subset of the whole web while Google represents linkage for the web as a whole. From here, we could gather that for every link a blog provides, other sources on the web provide 3 links. Since blogs still represent a small portion of the web, however, the importance of links in the blog world may be outpacing the importance of links in the non-blog world. Part of the reason behind this could be that links are one of the big currency in the web space and many blogs are offering little content but are heavy on the linking. If an average blog entry is under 300 words, it often contains at least one link. This could mean that Technorati and other blog search engines are right to consider links as a strong measurement, but may show that blogs, as a medium, are not providing that much content beyond linking.

However, it gets even more interesting if you dig in. Looking at the data, these values are actually misleading. What is happening is not truly an egalitarian match. Doing a quick review of the distribution, we start seeing some interesting trends.

Let’s graph the Technorati links as percentage of Google to see a little more of what I’m inferring:

Looking at this, it seems that our friends at Technorati have a bias. On average, blogs in the top 10 are 8% more likely to get indexed by both Google and Technorati than they are to be indexed by Google only. Considering that Google already admits to some level of bias in their system (part of the foundation for PageRank is that sites with higher PageRanks get indexed more often), it is a bit worrisome, especially if the trend holds across the whole of Technorati’s universe. If Google favors indexing more popular sites more often, a clear opprtunity for world-live-web search engines like Technorati would be in the long tail of less-often-indexed sites but Technorati seems to ignore that opportunity and concentrate on the top sites. What that will translate into is a direct reproduction of the power laws when it comes to indexing of blogs.

But is that true of Google vs Technorati only? Or do the same rules apply for other search engines? We’ll look at that in the next entry.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Secrets of the A-list bloggers: Technorati vs. Google. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

Method to the madness

The first thing in preparing for this analysis was to figure out who I should pick as a subject for the study. That was the easy part, as Technorati, the blogging search engine, is generous enough to provide a Top 100 list, highlighting the superstars of the blogging world.

Sitting at the top of the list were the following blogs:

• Boing Boing, which is not written by a number of people, some of whom are are also advising the company
• InstaPundit, a libertarian weblog run by Glenn Reynolds
• Daily Kos, a left wing weblog run by Markos Moulitsas Zuniga
• Gizmodo, a weblog about gadgets
• Fark, a list of links
• Engadget, a competitor to Gizmodo for the gadget audience

I decided to eliminate Fark from my analysis as it was the outlier, generally not producing more than four or five words per link and not separating entries but rather keeping everything in one page, making it look very different from the regular blog format. Having done so, I decided to pick a 24 hour cycle and analyze data from that cycle for the remaining five blogs. The day I picked, May 19th, was a good day for gadget news and political blogs: in the gadget space, the E3 show was closing down. Meanwhile, the potential of a dramatic showdown in the American congress made for a lot of material for political bloggers.

I waited until the day was completed to start my research. Picking every entry one by one, I cut the entry out, pasted it on Microsoft Word, did a word count on the entry, and recorded the number in an Excel spreadsheet.

So let’s take a look at the numbers:

A cursory look at this shows a lot of interesting data. For starters, all the A list bloggers in that group posted at a rate of an entry per hour or more. However, looking at this, it was unclear how long each entry was.

I decided to massage the data a bit. Individual entry data did not provide much in the way of a clear view but aggregated information did give me a clearer picture. Let’s take a look:

The data became clearer. On that particular day, the top five bloggers created an average of 30 entries, with each entry being under 150 words. This reminds me of something Phillip Greenspun, another A-list blogger, had said about why he liked blogs:

It allows me to experiments with the three paragraph form

Considering the size of the average entry from this, it seems very clear that an entry should be brief.

However, going beyond that is the number of entries that come in on a day. Looking at this, the average Top 5 A-list blogger wrote an average of almost 30 entries. Think about it for a second or two. 30 entries! It’s a huge number for a single day.

From the reader standpoint

So let’s say you popped up your news aggregator of choice and have subscribed to each of those blogs. How much would you read? How much information would you get? Our little analysis shows you would have read a bit under 22,000 words. That would amount, in terms of printed pages, to 44 single spaced pages.

Your alternative? Well, on that day, you could have picked up the New York Times and read every stories on the front page. That would have netted you 12,964 words, or about 22 single spaced printed pages. You could have listened to the evening news, are about 3000 more words. Ultimately, you would have consumed more words reading blogs than going with mainstream media: 5 TV shows would have netted you about 15,000 words. 5 newspaper stories (assuming a different report on each story) would have netted you about 8,000 words. So blogs are much more prolific in terms of words.

Blogger burn out

Notice that I’ve carefully avoided the subject of quality in this particular analysis. This seems to be an increasing issue in the blogging world. Some bloggers, like Joi Ito and Fred Wilson are starting to worry about the quality of entries. Is this the onset of a rush to more substantial but less frequent posts? Only time will tell.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Secrets of the A-List Bloggers: Lots of short entries. You can follow him on twitter here or receive his weekly newsletter by subscribing here.

How Search Engines Work

Before I get into details as to why I think this web accelerator is a major search move by Google, I first need to educate some of my readers as to the basics of search and some of the issues relating to creating a good search product. If you already know about search indexing, you can skip to the next section.

Search engines are basically acting as not only giant card catalogs, similar to the ones you can find in a library but also as giant libraries in and off themselves. When you type a word in a search box, what happens next includes a number of different steps that allow to look through a giant index, which is basically an image of all the pages the search engine knows about.

The way those indexes are created is through programs that are known as spiders (sometimes also referred to as web-robots or crawlers). Those programs are independent pieces of software that go and basically surf the web at very high speed, making copies of everything they encounter and comparing what they find to what other spiders are found. That giant set of pages copied by spiders is called an index (it is also sometimes referred to as a collection). They run around the clock and their sole job is to get more pages and ensure that the pages they’ve gotten in the past still exist and that they have not changed (if they have changed, the spider will “re-index” the page, ie. delete the previous one from the index and put the new version in its place).

Size Matters

The idea is a surprisingly simple one and was first introduced in the early days of the web. At the time, creating an index of all the pages on the web was relatively easy, largely due to the fact that there were not that many pages and that not that many people were creating them (I actually enjoy surprising newbies by telling them that I once saw the whole web, every single pages on it. What I omit until later in the story is that I did this in 1993, at a time when you could count the number of web servers without hitting 100 and when you could actually see the whole web in only a few hours.)

The amazing thing is that, although the number of web sites (and hence the number of web pages) has exploded, the basic technology to build a search index has not evolved that much. The concepts are basically the same today as they were in 1994–1995 but the web is now much, much larger.

How large, you wonder? Well, a good indicator would be to take a look at the bottom of the Google home page for a number. As of this writing, that number stands at 8,058,044,651. That’s over 8 billion pages, a very large number and one that folks at Google are appropriately proud of.

There’s only one little issue with that number. It’s on the low side. In fact, it’s estimated that it represents less than one percent of the actual number of pages on the web. In 2001, that number was estimated at over 500 billion pages in what is called the Deep Web, a part of the web that has not been indexed by search engines yet. With the growth of weblogs, which are generated tons of content on a daily basis, and the connection of more systems like books, satellite maps, etc… to the web, you can only imagine that the number has grown.

Let’s pause for a moment and assume that only as many pages were created between 2001 and now as were created in the previous four years, at the high of the dotcom boom. This means that there would be over a trillion web pages on the Internet. Now that gets to be a much more interesting number.

You Call THIS Fresh?!?

So we know that Google has a problem in finding a lot of the pages that already exist on the Internet. But that’s nothing compared to the other problem Google has.

Imagine an index with 1 million pages. If you assume that a spider can index that one million pages in a day, the content on those pages is refreshed daily, meaning that the index has a new version of the pages only once a day. Now try to do the same with 8 billion pages and it becomes a pretty complicated problem. Google has solved some of that problem by basically deciding that some sites have a higher worth than others. As a results, sites which are known to refresh their content on a regular basis get more attention from Google than sites that do not.

With the explosion of weblogs, however, a new breed of sites has created a problem for Google. For starters, there are a lot of them, and most of them refresh their content regularly, in some cases more than once a day. This makes the job of producing relevant indexes almost impossible for Google, turning their search engine into something more akin to a library, the kind of place that you use when you are looking for a reference, than an up to date source.

Not only that but, if Google is to also index the deep web, keeping track of all the changes across all the web becomes impossible… Impossible, that is, if you are using crawlers.

So we now know that the crawlers are no longer the right option when it comes to keeping fresh information within a proper search engine index. Looking at this, Google needs to do something radical. On the one hand, they can try to build a system that will get the most up to date information through notification from the sites that are updating content. This is where services like Technorati and Feedster come in, getting updates from RSS feeds and thus building indexes with more recent information than Google’s.

On the other hand, they could look at increasing the number of crawlers they are using. We know that Google has a lot of machines but trying to scale to the point where they can monitor a trillion pages via crawl would require a lot more power than that.

Enters Web Accelerator!

Spreading the Load

In the late 90s, distributing computing took hold as a concept. Projects like SETI@home and Folding@Home have shown the way in terms of harnessing the power of millions of computers to solve processor-intensive kinds of problems. Google started looking at this with the roll out of their toolbar with a feature called Google Compute.

Now let’s move forward. What if you could get information as to what pages are new and what pages are changes by just observing where people are surfing? This is the space that the accelerator occupies. Sitting neatly between your web browser and the Google architecture is a mini proxy that keeps checking if it can find a way to give you pages at a faster rate from the Google index than it does from the actual existing site. Along the way, Google finds out what pages are missing from its index (and gets a chance to add them) and what pages in its index are not up to date.

Imagine a million people downloading the Google Web Accelerator and all of a sudden, you have an infrastructure that finds out about a lot of pages very quickly.

Microsoft and Yahoo! are already in competition with Google in the search space. In order to maintain its leadership, Google needs to not only provide an index that is larger than its competitors but also more up to date. With this accelerator, they can do that and only one of its competitor can ever hope to match the feature: Microsoft.

The webmaster FAQ points the accelerator does not cover pages which are secure (nicely bypassing security issues) nor large media files. I suspect that we will see that change in the future, with the addition of images coming first.

Tristan Louis is the founder and CEO of Keepskor and writes the influential tnl.net weblog, where this was initially posted under the title Google Accelerates Search. You can follow him on twitter here or receive his weekly newsletter by subscribing here.