Evolution of Internet Infrastructure

The evolution of data centers. Amazing to think that there could be an evolution in such a short period of time. Perhaps it was the bubble(s), or maybe it was the rapid pace of technology advance, but it is undeniable that the infrastructure for the Internet has changed and evolved over the last 10 years. 

I began working on systems and networks in 1997. The standard operation for most in those days was one of building your own data center. If you were a startup, you would build a closet with a rack, add a battery backup, some big fans and a T1 and poof! there was your data center. 

Ironically, this is not much different than how many of the data centers got started. When I first visited Exodus Communication's data center on Wyatt in Santa Clara, it was a mess. Racks in various positions, conduit strewn about with little planning. Our cage at Hotmail was a room inside another room. That was my real induction into the life of data centers. I can remember sitting outside the doors and getting a badge to enter thata strange new world. Looking back, the years that followed were an amazing opportunity to learn, gaining perspective on what data center's should be while witnessing various infrastructure challenges for companies such as Exodus, Level 3, Qwest, Abovenet, Equinix, Global Center, Microsoft, Yahoo, and Google. 

The changes came so quickly. One day we were building our own data closets, the next we were deploying into a room 500 square foot room in an office building that had been converted into a data center. Before I knew it, we were deploying into 20,000 sq ft of data center space in multiple data centers. The Internet boom had pushed the need for infrastructure through the roof. The pressure to keep up was nothing short of phenominal. 

The effect of the boom on the decision making process of the data centers was one of equally bold stupidity, as they invested billions of dollars into their data center and network infrastructure space. They over indulged in their development as if being the largest data center in the world would give them the upper hand selling to companies. The market reacted with the cry of commodity, and the cost of space and bandwidth plumaged. But all was not lost. Although bankruptcies ensued, the Internet continued to grow, and there was still a steady demand for infrastructure. 

However, just as companies came to the conclusion in the 1990's that building their own data centers was not their core competency and moreover that it was non-strategic and a waste of both time, resources and capital (particularly in terms of their usage over time) - a new trend that smacks as one that is parallel to the data center shift of the 1990's has emerged: Cloud Computing. 

In 1999 I had a meeting with Sherman Tuan, the Founder and CEO of Abovenet. We spoke about his plans for the company and then he began to talk about the future of data centers. He spoke about servers being rented virtually so that you would not have to build out a data center. He was ahead of his time. And although it took a few years for the market to catch on, the idea of virtual data centers with Cloud Computing has finally become mainstream. 

Amazon, Google, AppNexus, GoGrid. The names of the clouds are synonymous with scalability, flexibility, and cost savings. Pay for what you use. When I was working for Microsoft, this was our ultimate goal. Stop paying for what you are not using. Stop paying for unused overhead, empty racks, servers with no operating system installed sitting on a palette, minimum bandwidth fees. The data centers would never go for it. But now, a decade later, what seemed like a fairy tale scenario is in fact the de facto standard for cloud networks. It is so much the standard now that even investors are now assuming that their startups will use the cloud in order to stay competitive. 

I do not see this trend stopping. Clouds are the future. Cheaper, easier, faster, more reliable. It will start in the grass roots of startups and head to the mid market and the enterprise. Managing data center infrastructure is as good as managing your own PBX, or developing your own email application - it is no longer needed. Pay for the service and get what you need when you need it. Focus on your business and the value that you create. 

The evolution of the data center is not over yet. There is more to come in this saga.