Josh Restivo

It occurred to me some time ago just how dramatically cloud computing stands to affect forensic computer analysis efforts. Cloud introduces a slew of quite significant gotcha’s. After the MegaUpload debacle (somewhere, right now, in a depressing windowless government forensic lab, some poor tech is still diligently duplicating original evidence from that case), the need to establish new techniques for cloud acquisition and analysis are more apparent than ever. Having recently been heavily involved in cloud computing myself, I have some insight to share.

While first contemplating the topic, I came up with a basic framework to address the essentials. That quickly ballooned well outside the scope of even the most ambitious blog entry. Then, I broke it down into a four-part series. As I walked through the framework for each part, things started getting out of hand again. In the end, the whole subject easily justifies a book. For lack of a book deal at the moment, what I’d like to do instead is provide some useful, if not totally comprehensive, technical information that I’ve not seen in other ‘cloud forensics’ narratives. Many such articles/blogs seem to have been written by individuals who have considered the subject to varying degrees but who have not actually designed, deployed, supported and/or analyzed a cloud (and its constituent components).

Step up to the diving board

Let’s jump in. Oh wait. We need to define a few things first.

Cloud
This seems like a good place to start. Trying to define this term is a slippery slope, though. It means many things to many people. For purposes here, we’ll use the broad definition of any virtualization environment where compute, memory, or storage resources are abstracted and provisioned through a pooling mechanism.

Disk Image
This term was a poor choice when it was originally used to refer to the complete capture of a disk’s contents. Ambiguity has returned to brow-beat us further in the world of virtual machines. Now it is possible, if not highly likely, to run across a dd (or raw, or ISO, or img) containing one or more ‘disk images’ in dd, vmdk, vhd or any of a handful of virtual disk formats. These images may actually be complete disk images containing multiple partitions or may simply be an individual partition.

Hypervisor
Sort of like a sun visor but then not at all. Technically, the hypervisor is the traffic-cop within a virtualized environment. It runs on the host system and manages all hardware resources for the virtualized guest machines. In practice, the term is often used to refer to the host system as a whole (eg., There are thirteen VM’s running on hypervisor #6).

Virtual Machine
Virtual Server, Virtual System, Virtual Desktop, Guest, Guest System, Virtual Host….many names for the same thing: a virtualized computing environment. This often includes, though does not require, a virtual disk image. A virtual machine may be booted from a physical (raw) hard disk or from an ISO CD/DVD image such as a ‘live’ boot disc (eg., Knoppix).

This should be enough to get us started. Let’s move on…

Acquisition

The new challenges here are as much technical as procedural. Where historically, a proper forensic acquisition involved gaining access to physical servers and imaging the physical disks, the likelihood of this happening in a cloud is greatly diminished for a variety of logistical reasons. Cloud storage tends to be expansive in size, physically distributed across multiple drive arrays and, possibly, physical locations. The underlying filesystems are clustered using various clustering protocols, both open and proprietary. It is not uncommon to find virtual disk image living on an ntfs or ext3/4 filesystem which is built upon a clustered filesystem stripe which is further built upon a a hardware or software RAID instance. If anyone believes that it is a wise expenditure of time to attempt to reassemble all of that, disk by disk, in the lab in the hopes of finding a few scraps of latent data then I offer you my most sincere condolences on your insanity.

If you think you’re going to use a linux boot disk to fire up the raid controller, identify the clustered filesystem and dd the device to a USB drive, you’ve taken a trip to Fantasy Island. If you do manage to get a valid block device assembled, WD isn’t making 20TB USB drives just yet. If they did, it would still take you 11.57 hours to acquire under unrealistically optimal conditions.

I’ve seen a couple of people suggest that it’s enough, in a cloud environment, to obtain a copy of the the virtual disk image(s) pertinent to the investigation. In a simple virtualized environment, this may be the most reasonably effective approach. In a cloud though, there are many more concerns. Typically multiple copies of a disk image are floating about (active, standby, backup). The transience of virtual machines in a cloud means that, among other things, a hypervisor other than that which is actively serving a given VM may contain live memory artifacts (from the VM) which differ from the active VM instance. Depending up on the nature of the case, it may be worthwhile to investigate the VM’s ‘location’ history within the cloud.

Further, many clouds are configured for geographic redundancy. It’s not possible to sync terabytes of storage or physical memory in real time across WAN links so synchronization occurs on a periodic basis, often daily but sometimes weekly or monthly. This means alternative instances of a given VM, possibly up to a month old or more, may exist in alternate physical locations. These VM’s may be for backup purposes but, if configured as part of a resilient cloud, are mostly definitely ‘on-line’ and are fair game for forensic investigations without triggering FRCP 26(b)(2) concerns. If anyone should wish to challenge this final point (I’m not aware of any precedent as of yet), I’m always happy to debate!

Analysis

Some of the basic items to watch for:

1. Multiple virtual disk images contained within an acquired physical disk image (if you can even get one).
2. Filesystem ‘stacks’, especially when dealing with acquisitions from physical devices.
3. Incredible amounts of redundant data as well as redundant data.
4. New types of meta-data:
   1. cloud management system audit logs (generated by the cloud software) and OS logs (generated by the OS driving the management system)
   2. hypervisor logs – these may be non-existent or a rich source of intelligence, showing the migration of virtual machines and/or backup data (eg., snapshots) within the cloud
   3. virtual switch logs – if you need to tie an IP address definitively to a virtual machine, you’ll want to look out for virtual switches
5. Cloud Billing Systems – usually separate from the cloud environment itself, tied only by software API calls, this may be the only way to establish a strong link between between a specific customer and a VM in a multi-tenancy environment
6. Multi-tenancy – Some ‘clouds’ are actually downstream abstractions of a larger upstream cloud. In such multi-tenant environments where resources are pooled amongst multiple unrelated business entities, you’re unlikely to ever get physical access to the storage subsystem. Even if you could, would you really want to? I’ll refer again to the current MegaUpload fiasco as a splendid case in point.

What I’ve presented here is just a small sample of cloud-related issues. I didn’t even touch on reporting. Unless you’re working with attorneys who can ‘read the Matrix’, you’ll need a consistent easily-understood way to refer to all of these multiple layers of abstraction. This might sound easy until you try it in a real-world scenario. Industry standards are long overdue for technology evidence naming conventions.

Also left unaddressed is the challenge of education. It’s difficult enough for the technical-types to keep on top of all of this. What happens when some technical aspect of cloud boils up in front of a jury? Their glossed eyes and slumped posture are the thing of any expert witness’ (and their clients) nightmares.

Josh Restivo

If you’re going to get in this game, you’d be well served taking a dive into the deep-end and getting acquainted with the perils first-hand. Or, you can wing it and start selling your cloud before you actually build it. The latter option does, admittedly, benefit from a few guarantees: extensive data loss, loss of business, loss of job, and loss of sanity.

Cloud is as complex as it is neat. And, it’s very neat.

Let’s break cloud down into a few key concepts and start from there.

Hardware

Big Iron vs Cheap Tin

First, be aware that many server platforms, especially in the multi-node ‘cloud server’ category are re-badged. After speaking with a customer recently who swore by Dell servers and swore at Supermicro, I was gifted the opportunity to open up one of his “rock solid” Dell servers. What to our wondering eyes did appear but a stock Supermicro motherboard. Maybe it had a real nice power supply to make up for the Dell ‘tax’.

The more important point here, above manufacturer preferences, is that cloud is built to tolerate hardware transience. Here’s a favorite quote from Jeff Dean of Google…

“Our view is it’s better to have twice as much hardware that’s not as reliable than half as much that’s more reliable. You have to provide reliability on a software level. If you’re running 10,000 machines, something is going to die every day.”

No One Ever Got Fired For Buying IBM

This cliché has seen better days. Even if its tenuous premise could be proven true, we can demonstrate affirmatively that such an approach can lose large sums of money. Just recently, Harris Corp, cut its losses on its ‘buy-big and centralize’ hosted cloud strategy. It turns out that large-scale financial and administrative efficiencies are not great selling points for government agencies who are long-accustomed to owning their own sandboxes and all the toys that go with it. Who knew? To be fair, Harris says it wants to focus on its Broadcast business but this exercise in attempting to steamroll the obvious resulted in a total stated loss of $70M. While failing to gauge their market correctly was a critical mistake, employing physically and financially heavy ‘vBlocks’ introduced a level of deployment inflexibility that must have factored heavily into their sales shortfall.

Software

The biggest lesson here is to fully quanitfy your business model and that of your intended client base. Not to belabor Harris Corp’s unfortunate story but the built a centralized hosted cloud with off-the-shelf software. This left them with no ability to seamlessly integrate on-site solutions, or sub-components thereof. Further, for many would-be clients, a forklift-migration to Harris environment would likely have been involved. Even if federal agencies could have been convinced to move over to Harris’ datacenter and virtualization environment in the long, it doesn’t appear, given the softrware that Harris employed, that there was any graceful transition route. My experience with these sorts of agencies has shown that the type of highly-capable IT admins who could successfully pull off such a feat are incredibly overburdened already. The proposition of a massive inter-datacenter migration isn’t likely to find much support. Private industry definitely has more resources at its disposal, generally speaking, but no one wants to buy into a headache.

VMWare

It’s everywhere! So was IBM, though. More poignantly, so was Novell Netware. Much like VMWare, Novell’s product did an admirable job of filling a critical role during a time of transition in computing. Also, much like VMWare, Novell adopted the ‘stick with what we know best’ approach to product development. Sometimes that approach can pay golden dividends but, in technology especially, it can be a requiem.

With solid alternatives in both open source and from Microsoft directly, I fail to see long term viability for VMWare. My purpose, here, is not eulogize VMWare though. Instead, as a warning to future cloud provider’s everywhere I offer up Novell as a lesson from the past. And, as we know: Those who cannot remember the past are condemned to repeat it. (Thank you, Santayana)

David Rokita - VP Technology Operations

The topic of piracy remains near and dear to my heart.  Of course, I am speaking of the digital version of Blackbeard patroling the high seas of the Internet to pillage defenseless multi-national corporations.  As a software provider, my interests in this topic are clear.  My team spends an inordinate amount of time building safe guards into our software to ensure a reasonable degree of protection from these rum swilling savages.  We use encryption keys on top of call-backs on top of secure registration on top of audits toprotect our software.  My life is my intellectual property.  Nothing is more important.

The Stop Online Privacy Act (SOPA) and Protect Intellectual Property Act (PIPA) aim to strike a blow to Internet piracy by fundamentally changing the way that both the Internet and our legal system work.  In general terms, these acts work to shift the burden of proof from the accuser to the accused.  As an intended side effect, it also shift the financial obligation of policing the misappropriation of content to the Internet service providers and web sites.

SOPA defines an entity called a ‘rouge site’ as any site that doesn’t operate under US law.  It further empowers the US Attorney General to obtain a court order without due process to essentially eliminate an ‘offending site’ from the Internet.

A service provider shall take technically feasible and reasonable measures designed to prevent access by its subscribers located within the United States to the foreign infringing site (or portion thereof) that is subject to the order…Such actions shall be taken as expeditiously as possible, but in any case within five days after being served with a copy of the order, or within such time as the court may order.

PIPA takes a similar position but focuses more on domain name service providers and search engines.

These bold new powers should alarm any American.  They run contradictory to basic freedoms and principals; never mind the fact that there are already laws in place to handle such things.  Don’t believe me?  File-sharing site Megaupload was brought down by the FBI on January 18th, 2012 with no help from SOPA and PIPA.  Those of us that have been in the cloud computing market have been fighting tooth and nail against fear of government.  Fear of the law and fear of government hijacking of cloud data is one of the single largest complaints about adopting cloud.  What worse, representatives in congress represent these bill as a means for protecting US jobs and innovation.  Here is Senator Roy Blunts take:

Intellectual property industries employ more than 19 million people, making it an integral part of our economy.  Rogue websites dedicated to the sale and distribution of counterfeit goods and pirated content are a direct threat to these jobs and to entrepreneurs growing and building legitimate businesses online.

(no link as this automated response was sent directly to my email).

Protecting innovation?  The cloud computing market is innovating right now.  It represents the single biggest technology opportunity since the dawn of the Internet.  In just the last year, the US government plunged a dagger into the heart of innovation by orchestrating a shutdown of wikileaks on Amazon.  Now, the US government is holding millions of legal files hostage as they shutdown Megaupload on the allegation of harboring illegal files.  More disturbing, these reckless actions display a fundamental misunderstanding of the Internet and the massive collateral damage that can occur through misguided policies and knee-jerk reactions.  Congress, its business interests and medieval bills are the biggest threat to innovation.

The charade that congress is doing this for me and my protection really pushes this over the edge.  Roy Blunt‘s (and others) automated response to me postured that intellectual piracy is a bad thing.  If you don’t agree with intellectual piracy, then you must agree with SOPA and PIPA.  You are either for us or against us.  This is a rhetorical fallacy; a phoney dillema.  Intellectual property piracy is problem.  We already have laws in place to protect intellectual property and I am reminded of them everytime I put a DVD into the player.  Draconian laws are dangerous.  SOPA and PIPA are not necessary.  Most people reject real piracy (the Captain Hook kind), but most people do not support carpet bombing the coast of Africa to prevent it.

Breaking news as of today states the SOPA/PIPA have been postponed indefinitley.  Maybe I’ll shut up before I get detained indefinitley.

Follow DaveRokita

Follow Hexagrid

Related articles

• Sopa & Pipa (kiatastic.wordpress.com)
• Megaupload sharing site shut down-SOPA and PIPA style attack (truthsupport.wordpress.com)
• SOPA and PIPA both shelved for now. (arstechnica.com)
• SOPA and PIPA = Censorship (Screed #43) (eclaireapense.wordpress.com)
• What are SOPA and PIPA? (programmingpalace.wordpress.com)
• Democrat Controlled Senate Postpones PIPA … (tarpon.wordpress.com)
• SOPA/PIPA Backers Still Don’t Get It (eliainsider.com)
• SOPA and PIPA, will they leave us speechless? (rebeccacraft.wordpress.com)
• Censored – Stop Online Piracy Act (SOPA) and Protect Intellectual Property Act (PIPA) (drayspencer.wordpress.com)

David Rokita - VP Technology Operations

Watch for Icebergs!

Of all the inflated expectations, the most dangerous is the misconception that the cloud never goes down.  Although never claimed by any cloud provider or foundational technology, this belief permeates preconceptions about cloud computing.  In the movie Titanic, passenger Joseph Ismay incredulously says, “But this ship can’t sink!”  A stoic Thomas Andrews replies, “She’s made of iron, sir!  I assure you, she can… and she will. It is a mathematical certainty.”  The beauty of that exchange arises from the self-evident clarity of Andrews response.  The truth stares the passengers in the face like a 10 ton heavy thing, yet it lies outside their ability to perceive; obscured by hopes and dreams of what they wish for the truth.  Imagine the passengers horror upon learning that there were only enough life rafts for less than 50% of the passengers.  It may be a bit of hyperbole to compare tragedy and loss of life to loss of data and loss of compute power, but the absurd consistency that belies both stands valid.  Clouds are built with computers.  It is a certainty that they will go down.  The deleterious tendency to ignore sound application design give way in favor of faith in an ethereal never-failing cloud is a recipe for disaster.

Much to the chagrin of the cloud computing media, Larry Ellison lobbed some missiles across the bow of cloud computing by explaining (paraphrase) the quantum duality of cloud computing; that it both doesn’t exist and that it pervasive.  This monologue may be the most important commentary on cloud computing since the media coined the term.  Larry explains that everything that we do is today cloud computing.  Cloud computing is not some new technology, but rather an evolution in the application of technologies that we already use.  The concept of server (mainframe) virtualization pre-dates many IT professionals career. Modern server technology simply needed to evolve to a point where more resources existed on a single chassis than any single workload could use.  The concept of accessing these workloads across the Internet falls way short of  ‘One giant leap for mankind’.  The laws of physics remain intact, icebergs exist and systems still go down.

Thanks to both the unbelievable success and the frustrating missteps of Amazon, plenty of high-profile source material exists.  Let’s start with server maintenance.  The concept shocks people, but yes major cloud providers may require reboots of serve infrastructure from time to time.  After all, the end product is simply a virtual machine running on-top of a managed hypervisor.  Why wouldn’t it need to be rebooted to pick up patches, changes or upgrades?  Lydia Leong, author of cloudpundit.com, discusses massive reboots of Amazon EC2 infrastructure.  Many make the mistake of comparing the cloud to their homegrown virtualization farm.  I use KVM all of the time.  It is quite simple for me to migrate a workload with no perceivable outage time.  While migrated, I simply patch, change or upgrade the original and move the workload back when finished.  The problem lies in that automating and managing these migration schemes is complex (not to mention extremely sensitive to error).  Complexity does not scale.  Sure, two server running VMWare may avoid reboots via this mechanism for a while.  It also a lot costs more, requires subject matter experts and isn’t delivered as a service.

Unplanned outages happen.  In April, Amazon Web Services suffered a serious outage that affected an entire region of their EC2, RDS and EBS services.  Like most outages, this situation arose from the unknown.  When designing systems, engineers labor over the obvious points of failure and harden the total system from succumbing to those weaknesses.  Engineers can never plan for the unknown.  If you are not familiar with the formal Amazon explanation, I recommend reading it to make sense of the next few sentences.  What becomes clear after reading the Amazon explanation is that at no point did Amazon consider the interconnect between availability zones to be a threat to service across availability zones.  The control plane that handles API calls across availability zones became saturated and the customer’s ability to interact with infrastructure in other availability zones was impacted.  Cloud pundits love to point this out as a failure in the Amazon design; yet, nobody complains that they can control all of their infrastructure from a single dashboard.

Clouds will go down.  This is a simple immutable fact.  That being said, companies need to understand how their partners cloud work and engineer their applications accordingly.   Unfortunately, as the Amazon outage demonstrated, even that approach falls short of ensuring success.  Clouds are designed to get people out of the business of datacenters.   In that endeavor, they succeed.  Their ability to reduce risk becomes a selling point when competing in the cloud market place.  Risk lurks in the most unlikely of places.  When computers are involved, risk can never be reduced to zero.  Don’t get caught without a lifeboat.

Follow DaveRokita

Follow Hexagrid

Related articles

• The Peak of Inflated Expectations Episode 1 – The Reign of the Evil Triangle (hexagrid.com)
• Moving to the Cloud (vendio.com)
• Cloud Computing: Predictions For 2012 (cloud-computing.learningtree.com)
• 2012: What’s Ahead for Cloud Computing? (datacenterknowledge.com)
• The great cloud computing pricing debate (infoworld.com)

David Rokita - VP Technology Operations

The Evil Triangle

Back in October I put a brief commentary on the state of cloud and its current alignment on the Gartner Hype Cycle.  At the time, the general thought was that the industry had moved past the Peak of Inflated Expectations and into the downward spiral to the Trough of Disillusionment.  Although at arms length, the Trough of Disillusionment is not yet upon us.  In that, I would like to bring to the forefront some of my favorite expectations brought back from this peak.  On the Peak of Inflated Expectations, white-board engineering and “whitepapers” reign supreme and the view is remarkable.

Simple misunderstandings of cloud computing infect the paradigm of cloud consumers, pundits and commentators.  This episode of The Peak of Inflated Expectations involves a simple yet pervasive rule that governs everything that we do in technology.  This rule states that systems that are affordable, fault-tolerant and high-performance simply do not exist.  It is entirely possible to construct a system that incorporates any two of these factors.  The Evil Triangle is no more obtainable than the impossible Penrose Triangle.  In the minds of the average cloud computing consumer and much to the surprise of the Evil Triangle, cloud computing supposedly vanquished this enemy of progress once and for all.  Taking some liberties Mark Twain’s quote, Reports of [Evil Triangle's] death are greatly exaggerated.

I am introducing this series of blogs with the Evil Triangle because it lies at the root of nearly all faulty expectation of cloud.  When confronted by the Evil Triangle, most experienced IT engineers and managers readily accept its limitations.  When obscured by clouds, the Evil Triangle creates chaos.  As you know, “The greatest trick [Evil Triangle] ever pulled was convincing the world he didn’t exist.” Amazon’s cloud services provide a perfect place to illustrate the power of the Evil Triangle.  When it comes to general purpose Infrastructure as a Service, few would argue that Amazon is a pioneer if not the world leader in offering cloud services.  For this example we will look at various service offerings and the method in which they negotiate the Evil Triangle.

Amazon S3 Storage: S3 storage, Simple Storage Service, is the central unifying fabric that connects the entire Amazon cloud services offering.  It offers a massively scalable object store for storing images, backups and general purpose unstructured data.  From a total cost of ownership perspective, $0.14/month (used) per gig for the standard replication and redundancy, S3 storage is extremely cost effective.  ”Designed to provide 99.999999999% durability and 99.99% availability of objects over a given year.”, S3 offers an unprecedented level of fault-tolerance for the price.  ”Perfect”, shouts the overly excited IT admin, “I’ll host all of my VM images there so I never have to worry about data loss”.  ”Not so fast”, replies the Evil Triangle, “S3 storage is API driven and will never support the I/O that is required.”.  S3 storage successfully fulfills the requirement for affordability and fault-tolerance, but at the cost provided is unable to meet the demand of runtime operation (for reason including but not limited to performance).  It must be mentioned that data-out charges for transfers could adversely affect this price.

Amazon EBS Storage: Amazon Elastic Block Service offers a low-latency block level storage option for facilitation higher I/O workloads.  It may be successfully used for hosting transactional database workloads.  It seems to meet the performance requirement.  What about fault tolerance?  Amazon explains that EBS volumes are replicated within an availability zone.  This means that within a closed system the EBS volume can sustain single failures.  It goes further to suggest that customer concerned about durability have “the ability to create point-in-time consistent snapshots of your volumes that are then stored in Amazon S3“, ensuring that the data is persistent.  At $0.10 per gig, EBS reigns supreme as the vanquisher of the Evil Triangle.  ”Wait” bellows the Evil Triangle as it pulls out it slide rule.  When factored in, the requirements for bandwidth and S3 storage (to meet the durability recommendations of Amazon), the picture is not so clear.  To use Amazon’s example:

As an example, a medium sized website database might be 100 GB in size and expect to average 100 I/Os per second over the course of a month. This would translate to $10 per month in storage costs (100 GB x $0.10/month), and approximately $26 per month in request costs (~2.6 million seconds/month x 100 I/O per second * $0.10 per million I/O).

Now let’s augment this estimate with S3 snapshots.  Assuming that we keep a single daily snapshot to be replicated across S3 availability zones, add $14.00 per month for a grand total of $40 dollars for 100GB ($0.40 per gig).  It makes for fault tolerant storage provided a 24 hour point of recovery time is acceptable.  It is not cheap.  ”Check and mate”, exclaims the Evil Triangle.

Amazon EC2: Amazon’s rightfully does not market Elastic Compute Cloud as a strict storage option, but since storage represents an inextricable part of compute, it will be considered here.  Not to speculate to much into the physical design of EC2, by all accounts it appears to exist on a locally available disk array using some form of software/hardware RAID.  It is easily the highest performance option for all of the Amazon offerings.  Being shared infrastructure, it has been documented to suffer at the behest of competing workloads.  Overall one can expect a higher performance profile form EC2 than any other Amazon storage offering.  Using an Extra Large Linux server as an example, the price is $0.68 per hour.  Multiplied out ($0.68 x 24 hours x 30 days), the price equals $489.60 per month.  This offering come with 1,690 GB of storage.  Using brute force, this comes to ~$0.29 per GB/month.  This in no way considers the value provided by the 8 EC2 compute units (2 cores with 2 vCPUs each) and 15GB of RAM.  EC2 presents the most cost effective way to store data.  As the Evil Triangle pulls the plug on the EC2 server it asks, “How do you like me now?”.  There is no concept of perpetual storage, durability or fault-tolerance in the EC2 offering.

A casual observer might suggest that this article intends to point out the weaknesses with the Amazon offering.  Quite to the contrary, this article intends to expatiate on the sheer brilliance of the Amazon offering.  These example illustrate a company that understands that the Evil Triangle lurks behind every corner of their datacenters.  Rather than fear the Evil Triangle, Amazon embraced it.  By doing that, they built a comprehensive approach that solves a myriad of problems using different solutions.  At no point in time does Amazon claim that any single solution is perfect for all problems.  Expectations must be level-set against the Evil Triangle.  These may be the most lessons as the cloud computing tools journeys through the Trough of Disillusionment on its way to the Plateau of Productivity.

Follow DaveRokita

Follow Hexagrid

• The pros and cons of cloud hosting (devblog.supportbee.com)
• I Wish To See Cloud As An Open Source Cloud As A Service (essayboard.com)
• Amazon Adds Object Expiration to S3 Files (readwriteweb.com)

David Rokita - VP Technology Operations

The Gartner Hype Cycle

When working with Hexagrid customers, my role is to work at the highest levels of a customer’stechnology organization to level set expectations and goals concerning the adoption of cloud computing.  What I often find uncover in these conversations are expectations that, although possible, may not be realistic. In fact, I find that the single biggest inhibitor to achieving success through cloud computing comes from expecting too much.

Following the Gartner Hype Cycle means crossing the Peak of Inflated Expectations.  At no time in the past has the Peak of Inflated Expectations towered so high.  At the peak of inflated expectations, outages will never occur again and software upgrades happen automatically and one tool to rule them all.  In this utopian place, lower CAPEX means zero CAPEX and disaster recovery  just happens.  The Peak of Expectations invariably gives way to the Trough of Disillusionment.  Here the laws of physics still apply, failures occur and sound application design still maintains a purpose.

According to Gartner’s analysis, the market is moving just beyond the Peak of Inflated Expectations.  The deafening roar of a hyperactive media coupled with under-equipped/over-hyped vendor products make slide to the Trough of Dissilutionment an ominous one.  Smart companies will start measuring their assumptions.  If they do, they can avoid becoming an unfortunate example of over-hyped expectations.

Follow DaveRokita

Follow Hexagrid

Related articles

• Entrepreneurs Challenge The Gartner Hype Cycle (startupprofessionals.com)
• Gartner Hype Cycle: Technologies 2011 (wir-sprechen-online.com)
• Silicon Alley Insider: What Is Cloud Computing? (businessinsider.com)
• Small business insurance holders ‘cannot use cloud computing’ (premierlinedirect.co.uk)

Tara Kinney, Director of Communications

After years of cloud innovation, public versus private remains the focal point of enterprise cloud evaluations despite the hybrid cloud alternative which would likely accelerate cloud adoption.   

As reported by Neal Weinberg of Network World, Daryl Plummer advised enterprises at Gartner’s IT Symposium last week to consider public cloud services first and turn to private clouds only if the public cloud fails to meet their needs.  He goes on to mention that the inability to get desired SLAs, issues with regulatory compliance, concerns about disaster recovery, and realizing that cloud might not save money are reasons to not to take the cloud route.  Couple this with his warning about the complexities of cost analysis and the potential need for a cloud broker?  Sounds complicated.  Isn’t there an easier way yet?

These are all concerns primarily associated with public cloud which is why I’m surprised by Daryl’s “public cloud first” comment when a hybrid private cloud may mitigate these concerns and retain valuable public cloud support services.  Hexagrid, like other cloud enabling companies, offers service providers a way to deliver hosted private clouds as an alternative to the traditional public cloud.  Hosted private clouds provide dedicated hardware, secured to meet enterprise specifications, and can be bundled with support services.    

According to a PricewaterhouseCoopers (PWC) study, “growing demand for Infrastructure-as-a-Service in the enterprise market is creating opportunities for a new breed of cloud-based IT infrastructure providers to deliver innovative private cloud services.”  The study also estimates that by 2014, the typical IT infrastructure for today’s customers of outsourced IT services will rely on cloud computing for more than a third of their IT resource base.”  At Hexagrid, we agree that both service providers and enterprise benefit from accelerated adoption of hybrid private cloud models to achieve the advantages of Infrastructure as a Service. 

Gartner expects the $4.2 billion Infrastructure as a Service industry to grow by over 48% in the next 5 years which parallels the PWC study findings that “77 percent of respondents already have a plan and 64 percent said some type of cloud, including private and public, would be the best way to manage IT infrastructure in three years.”  Based on my understanding of the concerns and challenges about cloud, it seems that the third option of hybrid clouds might best meet many market needs, mitigate many enterprise concerns, and open doors for increasing service provider profitability.

David Rokita - VP Technology Operations

Scalability reamins a hot topic when discussing cloud computing, yet the basis of the conversation is often misguided.  Many believe that the issues of scalability originate from limitations in technology.  In reality, inability to scale is almost always a factor of cost.  When we really understand Amazon’s cloud computing model, started with a commodity technology and then removed all of the complexity of traditional enterprise IT to create the ‘illusion’ of infinite scalability.  Through simplicity and commodity, Amazon reduced cost and therefore achieved scale.  This makes for a valuable lesson to those that have aspirations in the cloud.

Having discussed the fiscal challenge to scalability, the usual suspects (HP, IBM, VMWare, Oracle) all have something in common.  Enterprise software is bad for enterprise customers and vendors will not change for obvious reasons.  For all the money blown on AIX, Solaris, HP-UX and Windows, these products fail to chart year after year on the Top 500 Super Computer List.  Only 9% of these are non-linux (45 of 500) and the top 10 are Linux.  Some of this is likely attributed to technology, but more than anything it is a factor of cost.  Think about it, the majority of these projects are funded university projects or other pure technology-for-the-sake-of-technology endeavors.  Budgets are tight and there are no retries for overshooting your mark.  By using a completely free (or manageable cost) platform, they have removed a significant bottleneck to scale.  When applying this lesson to cloud computing, one must question enterprise software’s viability in a truly scalable cloud.  This is the same enterprise software that failed to solve any real problems other than the ones it caused in the first place.  Of the biggest cloud players (Amazon, Rackspace, GoGrid), none of these are built on these products.  My hunch is that it is purely due to cost considerations.  If the cloud doesn’t scale financially, it doesn’t scale technically.  Remember, cloud computing is a technology that if done right should become less expensive (per unit of measurement) as it grows.  Amazon’s $40 per gig of RAM is a tough number to hit using enterprise class software.

Those of that are building clouds right now will be judged not against what was done yesterday, but rather against what competitors are doing today.  It is human nature to marvel at the money saved with VMWare as compared to old-school IT discipline.  5 years ago it was about how much money I could save by ditching Windows in favor of Linux.  Then it was how money I would save ditching Weblogic in favor of JBOSS.  Likewise, there will be a day where VMWare is no longer the savior of the enterprise, but the enemy.  When that day comes Linux/KVM will be there.

*** The author is aware that GoGrid, Amazon and Rackspace use Xen, but there is reason to believe that the winds of change are blowing in KVMs favor.  KVM was not a viable production option when these services were first created.

Follow DaveRokita

Follow Hexagrid

Related articles

• Four key reasons to include cloud computing in your business strategy (zdnet.com)
• Tidemark: Enterprise disruption in the cloud (zdnet.com)
• Clouds Vs. Outsourcing: The Next Battleground (informationweek.com)
• Cloud computing: Gaps in the ‘cloud’ (physorg.com)

Get Ripped! Image via Wikipedia

David Rokita - VP Technology Operations

Several months ago, Information Week emblazoned the term “Innovation Atrophy“.  Innovation Atrophy is the grand result of too many years of trying to save money through IT.  The author, Chris Murphy, explains that year-over-year squeezing of ROI out of IT creates a negative drain on an organizations ability to innovate.  Either through fear, contempt or plain old boredom this “beat down” takes a toll.  It comes in many forms: we are a [blank] company, not a technology company, buy for parity and build for competitive advantage, no technology for the sake of technology.  At times these phrases are steadfast maintaining of maintaining the course.  At other times it is simply short-sighted.  The key to success is to know when to check the bet or raise the stakes.

Cloud computing (Iaas) offers one of those rare opportunities to move the needle back towards innovation.  In the past, innovation posed a double headed threat.  If the initiative became successful, mobilizing the human and financial resources to meet the demand posed a challenge.  If unsuccessful, invested assets were incapacitated or lost.  Business respond to this risk with analysis paralysis.  The stakes are so high, that no business is willing to take the risk required to innovate.  The physicist Joseph F. Engleberger claimed that innovation only requires three things: A recognized need, competent people with relevent technology and financial support. Unfortunately, the nature of business requires that financial support be placed before the others.

In perfect world, businesses freely create ideas and throw them against the wall to ‘see if they stick’.  Those that stick would receive financial backing and grow accordingly.  With cloud computing (IaaS), the opportunity throw ideas against the proverbial wall isn’t free, but it is considerably more cost effective.  It becomes simple to build virtual infrastructure one day, and tear it down the next.  Only for the time that the infrastructure is up does the organization incur cost.  Parallel environments can be spun up when needed and turned down when not in use.  Shelved initiatives simply sit in the bull-pen while resources are freed for the next big thing.  Fear and uncertainty will still remain, but it just may be enough for an organization to start flexing those innovation muscles again.

Related articles

• IT Spending: No Longer The First Thing Cut (informationweek.com)
• Innovation is About Execution, Despite the Myths (startupprofessionals.com)
• Innovation Atrophy: How Companies Can Fight It (informationweek.com)

Follow DaveRokita

Follow Hexagrid

David Rokita - VP Technology Operations

In addition to working technology, I dabble in real-estate.  Over the course of a few years, I have bought, sold and held onto several rental properties.  One time I purchased a condo unit from a frustrated owner that was having trouble covering the spread between the rent he could collect and the mortgage, taxes and association fees.  He was right, based on the downward trending economy, the rent he could collect was breaking even with the expenses and overall the property was operating at a bit of a loss.  I bought the property, and then proceeded to spend $700 dollars on a sofa, chair, dining room table, bed and dresser.  I advertised the property for a 50% increase as fully furnished and lived happily ever after.  I love marveling at my own ingenuity, but this was hardly genius.  People pay more when they get more, and understanding this is key when navigating changing markets.

A common myth among data-center facilities providers states that the new cloud economy will drive cloud providers to their data-centers in droves.  They need only to sit back and wait for the deluge of customers to flock to their data-centers.  To certain extent, the logic is sound.  Cloud providers will bring business into their data-centers, but a very important characteristic of cloud will prevent this from having the intended affect.  The plan to sit back, fat and happy, selling picks and shovels to the prospectors of the new gold rush is a fool-hardy strategy.  Unless these data-center providers adjust their business model, they may just find themselves irrelevant.  Smart data-centers will get out of the business of power, ping and pipe and transition to selling CPU, memory, network and disk.

The key to understanding this conundrum lies in the very same reason that end-users are evaluating cloud computing models.  Ultimately, these customers want clouds because they maximize efficiency.  This efficiency permeates through power consumption, space consumption and complexity.  Of these, data-center providers enjoy the biggest margins on the floorspace.  If their customers are building their clouds right, floorspace consumption will dissipate radically (as well as power consumption).  With the density of servers and the advent of extended memory management, it becomes possible to pack a 1.5 terabyte RAM cloud computing behemoth into 7U in a rack.  This space allocation even includes all required switching.

The last time I had this conversation with a data-center provider, it was explained to me that I didn’t understand.  These guys have been fighting Moore’s law for years, and $1250 is a good price for a rack if you can get it.  Moore’s law or not, server efficiency rarely topped 25%.  Cloud computing at scale easily achieves 80% utlization, if not more.  I realize that I am performing surgery with a meat cleaver here, but lets just take best case scenario described above.  The aforementioned 1.5 terabyte cloud, at an assumed Amazon equivalent price of ~$37.80 per gig of RAM per month for open source Linux servers.  Since the set up takes 7U, the conservative view says that we can fit 2 clouds in one rack.

(1500 GB * .80 * $37.80 * 2) = $90,000 per rack per month (approximately)

Ok, again, this was brute force.  For a variety of reasons, this is an extreme example.  Examination of a more practical example still yields an extremely compelling story.  The names have been changed to protect the innocent, but the following represents the base configuration of a cloud built by a Hexagrid partner.  This particular partner opted to use more traditional hardware as opposed to a more consolidated blade configuration.  In this configuration, 3 compute nodes with 96 gigabytes of RAM were dedicated to running VMs.  With included storage and all of the redundancy, 2 of this installation will fit in a rack.

(288 GB * .80 * $37.80 *2) = $17,418

This makes for an extremely compelling case financial case considering that IaaS services from other providers like Amazon and Rackspace are threatening to take the business and the clients that you do win will take less space this year than they did last year.

Yes, changing the data-center business model is a big scary thing.  Fortunately, the automation tools, portals and practices have already been worked out.  Sure, the workforce may need a bit of restructuring and the sales team will require some training.  You might even need to hug a hardware vendor.  You can do this and you can be very successful.  It beats selling wagon wheels in the age of the internal combustion engine.

Follow DaveRokita

Follow Hexagrid