S3 Down for More Than 7 Hours

Amazon's S3 service went down today and more than 7 hours later, it is still down. The service initially went down around 12:00PM EST and my latest check shows troubles continuing for sites that depend on S3.

“Funny how Amazon doesn't use S3 to store any assets for amazon.com”tweet by @gruber

Smugmug, a popular photo sharing site with more than 600 TB of data stored on S3 has been accessible along with several "Web 2.0" startup sites. Because of the astounding amount of data stored by Smugmug on S3, it can definitely be considered a poster child for the service. Last time I checked Smugmug, at 11:10 PM EST, it was still inaccessible.

“As a consequence, a variety of businesses such as Twitter, digital photo sharing Web site, SmugMug and The Huffington Post all had issues. Twitterers were claiming their avatar images could not be displayed. The Huffington Post was also unable to display images to its stories, while SmugMug could not offer any service at all.
”ComputerWorld

This is not good for Amazon and for startups that are looking to count on Amazon's "redundant" S3 platform. Dr. Werner Vogels is probably pretty upset right now.

Clouds Are No Substitute for Competence

More notes from Velocity.

Next session is Clouds Are No Substitute for Competence by Javier Soltero of Hyperic.


The promise of cloud computing:

Cloud computing is the next big thing:
Because it is green, easy, scalable, available and disposable.

Cloud computing adds complexity:

• clouds allow you to run your applications, but mask the performance of the infrastructure powering them. NYT is not going to stop their own infrastructure just because they had success with one project on EC2.
• when a problem happens, where is the source of the problem? cloud or your own app.
• cloud, by definition is always available and the status is always green.
• how quickly can I provision new servers?
• what is the throughput in the regions I use?
• what latency am I getting for my messages?
  

How do you answer?

• 'is it my application, or is it the cloud?"

Hyperic is introducing cloudstatus.com which shows performance, availability and health of Amazon's Web services. On CloudStatus.com, you can monitor EC2, S3, SQS, SDB and FPS (5 most popular and critical services of AWS). You can look at performance metrics such as deployment latency. They are firing Amazon instances and monitoring response times.

Also see:

• Clouds Are No Substitute for Competence (Hyperic Blog)
  

Working the Clouds: NextGen Infrastructure for New Entrepreneurs

Now there is a panel, Working the Clouds: NextGen Infrastructure for New Entrepreneurs. Panelists include:
* Geva Perry GigaSpaces
* Jason Hoffman Joyent
* Tony Lucas XCalibre
* Lew Moorman Rackspace
* Christophe Bisciglia Google
* Joe Weinman AT&T


CB: Google App Engine can be called a cloud but it is a little different that allows you to only focus on very high levels.

Q: Are we selling our souls when moving to cloud? Or should we not worry about lock-in?
There are no standards yet. To take advantage of cloud you need to embed it into code. It does create some locking issues. Hopefully in future there will be standards. If you're going to scale on EC2 and have scripts to automatically launch instances then moving to a different cloud will create issues. A lot of clouds that are gaining traction have very custom APIs. There is going to be a proprietary stack. On the other hand we will see convergence towards standards based platform. Then we end up with choice where you will then make decision based on your strengths.


There is still value in proprietary technologies. The more value you are going to build for your customers, the more proprietary the technology behind it would be. Both open and proprietary markets have their potential.

Even though App Engine is proprietary, the documentation is very open. If you are using Big Table, you are kind of selling your soul. Until Google open sources BigTable so users can export their data, it is locked-in.

API that Google provides isn't specific to BigTable. One of the criteria is how scalable the platform is. Christophe Bisciglia thinks that Google is a step ahead of Google when it comes to BigTable and their platform.

One shouldn't be locked into proprietary data store, period.

But, CB insists that BigTable performs better. There are things for which relational databases don't perform well. There is a sacrifice but it allows you to scale. What differentiates providers is how low their latency is and how scalable the infrastructure is and how protected the data is.

LM: A lot of companies are not right away ready for cloud.

The dirty little secret model for cloud computing is that if licensing models don't catch up, everything can drop dead.

Cloud has become an overused term. duh!

Google has geographically distributed clusters.

Cloud computing is architecture 3.0.

There is a live stream going out at http://www.mogulus.com/structure08

Application development 5-10 years from now will be focused on the need to scale. The challenge is to move people up to cloud and provide them with tools to develop their applications the right way.

Thought shift is required for enterprises to move to cloud. CB gives an example of how people were hesitant to use banks when they came around as people were nervous of keeping their money with someone else. But then, as people realized the dangers of not keeping money at bank, banking became the norm.

AT&T will be spending $20 billion on their infrastructure.

Enterprises want to sign a contract. They don't want to just go and use credit card to start a relationship like this.

There is a big marketing problem. The word cloud means less and less everyday. Cloud is getting into the enterprise through the backdoor.

Werner Vogels: Keynote at Structure 08

Werner Vogels is now giving a keynote. He is the CTO of Amazon.com.

At Structure 10, the whole discussion will be different. We will be talking about different business models. This is a snapshot at the beginning of the movement. He showed an Animoto video presentation created by him.

What's so special about Animoto? They have no server infrastructure, even though what they do is very compute intensive. When they had 25,000 customers they were hovering around 50 instances. They launched a Facebook app that allowed you to import photos, create video and post it back to Facebook. At that point, they started signing up 25,000 customers per hour. They had to increase their instances to more than 5,000. Imagine Animoto going to VC and asking for money for 5k servers.

Cloud computing is moving the world from capital expenditure (CAPEX) to operating expenditure (OPEX). We are moving to a variable cost model.

Amazon is now in 7 countries with more than 79 million active customer accounts.

Bandwidth used by AWS is way higher than Amazon store.

Amazon used to be a technology consumer. Now, there is no third party software left at Amazon because of scale. There are more than 1M+ sellers. They moved from single application to a platform.

First 5-6 years, uptil 2001, Amazon was like a traditional site. The challenge was how do they keep scaling? How will we make it to the next year. Around 2001, Target came to Amazon and asked to be integrated. At the same time, a number of architecture pieces broke. Then they wanted to move to a platform while working on integrating Target.

At times, Amazon was thinking of going back to mainframe. They wanted to create a very agile environment.

They created an infrastructure where no direct database connections were allowed. Everything must go through a business logic layer.

The gateway page on Amazon can use upto 200+ services to be created.

The 70/30 switch

• Companies now have to become experts in many areas not related to their business and answer questions like, why is BGP protocol not stable?, why do datacenters go down? etc. These companies are spending upto 70% of time, energy and dollars on undifferentiated heavy lifting.
  
• Only 30% of time, energy and dollars are spent on differentiated value creation.

He is showing a photo of destroyed datacenter. If your data was in that datacenter, it is gone!. Now talking about 365 main which did 'everything right.' 6 of their 8 diesel generators failed and brought Web 2.0 down. At Amazon, the thought is to survive an entire datacenter failure.

Don't depend on Raid-5 to protect your data.

Peak capacity management is a big issue for companies such as Walmart.com and Target.com that experience seasonal spikes in traffic.

They wanted to cover three areas: compute, messaging and storage. EC2 covers compute, S3, Simple DB and EC2 PS (Persistent storage) covers storage and SQS is the fabric that holds everything together. EC2-PS still doesn't has a name.

Most data at Amazon was key-value based. There were secondary key accesses. SimpleDB was a compliment to S3.

It's easy for companies to spend as much as 70% of their intellectual capacity in scaling.

Infrastructure Services Drivers

1. Security
2. Scalablity
3. Availability
   
4. Performance
5. Cost-effective

Next he is showing example of SmugMug who relies heavily on Amazon's EC2 and S3. They currently have 600TB of pictures stored in Amazon S3. In Amazon S3 there are more than 18 billion objects as of March 2008.

SmugMug is now venturing in different businesses where they provide interface to allow you to store anything. The product is called Smugmug Vault.

Addressing Uncertainty

• Acquire resources on demand and release them
• pay for what you use
• leverage other's core competencies
• turn fixed cost into variable

What sense does it makes to order a lot of hardware when you don't even have a product? You also aren't sure how many customers you'd get.

Get everything from http://aws.amazon.com, you only need a credit card.