The AltaVista Effect

(Emil's Blog)←[:BACK]

April 15, 2016
 

Note: I’ve decided to launch my personal blog on the 309th birthday of Leonhard Euler, the father of graph theory. We’ve come a long way with graphs from his first paper on The Seven Bridges of Königsberg.

I believe that we can learn a lot about the future of the enterprise by studying the history of the web.

Almost three years ago, the once-giant search engine AltaVista was shut down for good. It’s quite likely almost no one noticed as it died not with a bang but a whimper.

So how did such a startlingly successful company fade into obscurity without even making it to the 20-year mark?

The answer: big data.

The AltaVista Approach to Big Data

Back in the early days of the web, there was a plenitude of search engines all trying to serve up the most relevant results. The problem was that the web was growing faster than most of these small search engines could index it, so results pages didn’t always give you as many links as you had hoped for.

Enter AltaVista.



There was a specific moment in time when the first cohort of search engines all began to lose significant market share to this new upstart search engine. As the new kid on the block, AltaVista was far faster and wider reaching than any other search engine in existence.

Even as the web doubled in size something like every week (!), AltaVista’s approach was to download the entire web (every single document!) into their data center. When a user entered a search query, they would sift through every available file and then serve up the results.

AltaVista’s long-term plan was to complete this entire-Internet download every single day – in perpetuity. I remember as a young hacker in the late 90s being blown away by the audacity and technical ambition of their approach.

When you typed in almost any search term, you’d get millions and millions of results, and AltaVista would serve up those results faster than any other competitor on the market (and there were a lot of them). They received 300,000 hits on their first day alone, growing to 80 million hits per day in only two years.

With such a rapid (and well-retained) growth in market share, AltaVista’s web-indexing success would be its own undoing.

That’s because after you entered your query, the crush of data was overwhelming. There were plenty more results than any competitor, but there were too many results to be useful. It was the classic problem of too much quantity without substantive quality. The AltaVista Effect.

When Big Data Got Too Big

Soon enough, a new startup came along that was determined to think differently. They set out to do the same thing as AltaVista and the other search engines (i.e., a daily entire-Internet download), but on top of that, they also extracted every hyperlink they found into a connected data structure.

Their perspective was that data in isolation is valuable, but connections between data points are equally valuable. If a lot of sources linked to a particular document, they assumed that it would be more important and awarded it a higher ranking in the search results.

Cue: PageRank, one of the most powerful algorithms the web has ever known. (PageRank was not in fact named for web pages but for the algorithm’s chief architect, Larry Page.)

Of course that startup was Google, who ended up completely dominating web search and who I believe has become one of the most fundamentally transformative companies of the twenty-first century. Their dominance was all due to one shift in perspective: Reimagining their data along relationships and not in isolation.

While Google will still give you millions more results than AltaVista ever did, their emphasis on relevance through relationships in data means you hardly ever look past page one results. The Google Effect.

The AltaVista Effect and the Enterprise

Fifteen years later, we see the same phenomenon play out in the enterprise space.

The enterprise market is today at a tipping point where most companies live in the AltaVista age: Digitalization, an explosion of new devices and the gospel of “big data” have led to an overwhelming volume of information, the sheer size of which makes it impossible to find relevant insights.

But a few companies have taken it to the next level.

They find relevant results based on contextual relationships, not a myriad of unrelated data points. We see this happen amongst early adopters and visionaries in industries as diverse as insurance, transportation, telecom, retail, healthcare and more. And the impact on these companies is every bit as powerful as the impact that Google had on the web.

How did they get there? Each of these enterprises were willing to ask themselves a singular, vital question:

“What if my enterprise didn’t just collect data but examined the relationships in it and we reimagined our business along those relationships?”

I believe that tomorrow’s successful enterprises will avoid being crushed under the weight of vast volumes of data because they’ll harness the power of relationships. By making that shift early – and by continuously asking data relationship-oriented questions – these enterprises will be the first to enter the Google era.