Sunday, April 19, 2009

Why specialty clouds will need to hustle to stay competitive with AWS

Last week, I blogged about the advantages of specialty clouds over Amazon’s generic AWS. Now, I'll tell the other side of the story: why specialty clouds will need to hustle to stay competitive with Amazon.

I think Amazon’s competitive advantage breaks down into four buckets that are all work together to reinforce Amazon’s competitive advantage:

  1. Customer Acquisition: No matter what search queries I try, I can’t find a single keyword Amazon has bid on to promote EC2. Meanwhile, GoGrid, Rackspace, Joyent and others are forced to bid on cloud computing keywords, even on Amazon’s trademarks (e.g. check out who is bidding on “EC2”), in order to try to establish their brands. Amazon’s clear market leadership drives down their customer acquisition costs, while Amazon’s competitors must invest in online marketing to try to chip away at Amazon’s dominating market share.

  2. The new “You won’t get fired for hiring IBM”: While cloud computing is the hot topic on everyone’s mind, it is still perceived as risky, and there is a lot of fear, uncertainty and doubt when it comes to deploying applications in the cloud. This FUD and risk is compounded when you consider placing your bet on a private company startup that might have to shutter its doors in twelve months. Coghead’s unfortunate recent closing illustrates this risk. But Amazon is a $33B market cap, publicly listed company. It ain’t going anywhere anytime soon. This makes Amazon’s EC2 the safe bet, even if it’s not necessarily the best choice for a particular use case.

  3. The growing Amazon Web Service 3rd party tools ecosystem: There is a growing and vibrant list of startups emerging in the cloud computing ecosystem. Many of these aim to alleviate the management and plumbing overhead necessary to maintain an infrastructure deployed in Amazon’s EC2 cloud. These companies aren’t going to focus on integrating their technologies with other cloud computing providers for a while, much like startups that emerged in the virtualization management space haven’t focused on integrating their technologies with any hypervisor other than VMWare’s ESX. RightScale is a great example of this phenomenon. The same is true for more traditional network management companies that have historically focused on on-premise infrastructures but are now integrating with Amazon’s API to enable seamless monitoring between Amazon AMIs and captive applications hosted in internal data centers. As Amazon’s ecosystem gets richer, competitive cloud computing offerings will have a steeper wall to climb to compete.

  4. Amazon’s benefits from two network effects: Amazon benefits from what I’ll call an internal and an external network effect:
    • Internal: Once you’ve launched one application in EC2, even if you have a Ruby on Rails website that probably would be a much better fit for Engine Yard’s specialty cloud, it’s just a lot easier to have all of your applications in one place. So Amazon benefits from a sort of network effect within a customer’s application portfolio. Until companies like RightScale integrate with other cloud computing providers, Engine Yard has a slightly higher bar to meet in order to justify the additional overhead involved in logging in to another console when a company already has all of their other applications hosted in EC2.
    • External: As more and more customers move their applications and therefore their data to Amazon, other applications that use the same data source can benefit from having the data stored in S3. For example, Nasdaq now stores terabytes of Nasdaq, NYSE and Amex data in Amazon's S3, and is adding 30 to 80GB of data to S3 every workday. If I am a SF-based company that has built an application that pulls in this data, I can minimize latency by hosting my application in EC2 in the same region as Nasdaq's data. As more data piles into S3, Amazon benefits from an increasing network effect. It's worth noting that specialty clouds can benefit from the same network effect, but Amazon has the head start.
In the end, despite Amazon's strong home field advantage, I expect that a few specialty clouds will effectively carve out their own niche. What do you think? Is Amazon going to become the dominant force?

Thursday, April 16, 2009

This video made me miss my rugby days




(The "try" starts at second 40.)

I can't believe I played for four years...

Wednesday, April 15, 2009

Why Amazon’s AWS Won’t Be the Only Game in Town

The shift that is clearly taking place, however gradually, from on premise data centers to centrally-run cloud computing utilities, is enough to capture any IT geek's imagination. The trend is so massive and at the same time so logical, analogies are drawn between the shift to cloud computing and the shift to the electrical grid.

At a high level, the analogy makes sense: Before the advent of the electrical grid, most companies had to generate their own electricity onsite. This meant having employees on staff to manage the electrical generators, and because most onsite generators only produced a small amount of electricity, the cost per megawatt was very high. When the electrical grid matured, companies were suddenly able to turn off their onsite generators, fire their electrical engineers, plug into the electrical grid and benefit from vast economies of scale and skill. This drove down the effective cost of electricity, and freed companies up to focus on their core competencies (which definitely was not generating electricity).

At first glance, Amazon’s EC2 and S3 web services seem to perform the same function. Now a company can tap into the Amazon Web Services (AWS) API and launch an Amazon Machine Image (AMI) instead of having to run their own servers in house. In much the same way that companies in the late 1800’s could benefit from the economies of scale and skill thanks to ConEdison, companies can now benefit from those same economies thanks to Amazon. If this were true, then Amazon’s clear market leadership would seem to indicate that if you’re going to go to the cloud, Amazon is the only game in town worth considering. Moreover, as Amazon continues to grow, its own economies of scale and skill will grow, and the competition just won’t be able to keep up.

But computing resources, unlike electricity, are not fungible. As companies experiment with Amazon and other hosting providers, I think this will increasingly come into focus. Even with the low level building blocks Amazon provides, AWS is not a one-size-fits-all cloud. For example, some hardware intensive applications will slow down and become inefficient because of the generic algorithm EC2 uses to assign AMI processes to its underlying hardware. Just as importantly (if not more so), unlike with electricity, you can’t just “plug” your application into Amazon and have it work automagically. Instead, Amazon AWS is more analogous to renting out a piece of a ConEdison power plant, but having to constantly monitor your slice of the power plant, call into the plant to ask the operator to turn certain knobs dials, clean the pipes, adjust the gauges, etc..

When it comes to the IT plumbing (deploying new software releases, configuring your web server’s IP address, testing software patches, deploying new security patches, testing for security vulnerabilities, mounting a file system, etc), you don’t actually benefit from any of Amazon’s economies of scale or skill: you still need to do these things yourself. And that's a lot of work!

That’s why non-proprietary specialty clouds like Engine Yard, Cloudera, AppNexus and Joyent, and hosting providers like RackSpace, can still thrive despite Amazon’s massive size. Taking Engine Yard as an example, their stack is customized from the ground up for Ruby on Rails web applications, and they’ve built proprietary tools to automate any plumbing. The same goes for using Cloudera over Amazon’s Elastic MapReduce. There are many other areas that could benefit from custom-built clouds. For example, PCI or HIPAA compliant clouds, I/O intensive clouds, high-end clouds that allow for scaling up databases and servers, etc..

What do you think? Do you agree?

(P.S. fingers crossed Disqus will work on this post...)

Friday, April 3, 2009

Who do you Tweet to?

Twitter is a blunt communication tool, not just because you are limited to the number of characters you can type.  It’s blunt because, unless you want to manage multiple Twitter handles (which of course is possible and I’m sure tons of people do it), you have one voice to communicate to a random assortment of people subscribed to your feed. 


So when you tweet, who do you tweet to?  Who is your Twitter muse? 

For me, I would like to think it's a personification of something close to the average of how I subconsciously conceptualize my followers. In an ideal world, that would look something like this distribution:

In actuality (as we all know), it ends up looking more like this:

But at least when I tweet, I don’t think about the spammers who have subscribed to my Twitter feed.  They never had any intention of reading my tweets when they subscribed, so good night and good luck to them.  

And as far as I’m concerned, the few random-os I have probably aren’t much different.  Maybe they subscribed to me on a whim, or saw something I said that they liked, and now, those lucky devils, they get my tweets in their feed.  Problem is, I don’t actually know whether they are reading my tweets; there is no feedback loop in Twitter.  So as far as I’m concerned, those people aren’t, and for the minority of the random people who do: thank you and have fun. 

Then there is the other side of the curve: the people who I [would like to] think are reading my tweets.  I don’t know about you, but that’s who I think I’m writing to… to some imaginary person in the middle of that subsection of the curve.  

But this means that for everyone to the left of that point, my tweets skew towards being more intimate than they should be.  In a way, a majority of Twitter followers are free-riding off the intimacy I have with a small group of my Twitter followers.  This must be one of the things that’s so enticing about Twitter.  At the same time, I can’t help but feel a little uncomfortable every once in a while when I see someone else's tweet that’s clearly intended for people a standard deviation or so away from me!

What do you think?  Who do you Tweet to?