Virtual Forecast: Increasing Cloudiness with Reduced Visibility

There are fascinating – as well as troubling – parallels between IP networks and virtualization.  The problems of network growth and complexity forecast some serious challenges ahead for virtualization.

For example, the enduring benefit of IP has been its independence from underlying devices and media.  IP is effectively abstracted away from Layer 1 and 2 so that data can move about freely, without issues of compatibility or details of hardware configuration.  Layer 3 IP packets are completely generic regardless of their origin, destination or application use.

Similarly, machine virtualization offers many of the same advantages in its abstraction away from the underlying hardware.  Running operating systems and applications become fluid and transportable.  Resources like CPU, disk, and network capacity are simply commodities that can be altered at will.  The details of physical hardware are completely hidden from the application.

Both of these technologies are nearly magical in their effectiveness.  And both suffer tremendously from lack of visibility.  Typically, you can only tell if they are working properly, or degrading in performance, by examining the symptoms of user-experience. 

Therein lies the rub. 

You have this wonderful technology that you want to deploy everywhere and yet keeping track of it can rapidly become a management headache.  Worse, it becomes a critical dependency that directly impacts application performance.  You simply have no choice – you have to monitor and assess network (and virtualization) performance on a 24/7 basis.

Like network, virtualization is becoming a victim of its own success.  As virtual sprawl begins to take over your IT infrastructure, it will get harder and harder to monitor the performance and manage resources.  Eventually it will be almost impossible to know where your application is, how much physical resource it has access to, whether the application is functioning adequately, whether the virtual machine itself has degraded your application’s performance, etc.  Management systems have been slow to catch up with the rapidly growing need – and performance measurement systems are almost non-existent.

This poses grave problems.  As has been oft repeated:  “If you can’t measure it, you can’t manage it.”

And then along comes the ever evolving  “cloud”. The word itself just reeks of poor visibility – and it is aptly chosen.  The cloud is an amorphous collection of resources that have been configured to offer bulk, undifferentiated computational functionality.  By its nature, you shouldn’t be able to figure out what individual pieces drive it.

This becomes really problematic when the cloud belongs to a 3rd party.  Just like in networking, you will have to trust that the 3rd party cloud resources are being properly managed and that your application is functioning properly.  You will have little or no way of measuring that – and your cloud provider probably won’t either!

So is the problem with virtualization itself?  No!  It’s a great technology that’s here to stay – it’s as indispensable as IP networking.  What is needed is for the industry, the engineers and experts, to start thinking hard and come up with some new solutions for measurement, management and visibility.

And this is where the parallels with networking get interesting – in fact networks have a key role to play in bringing visibility to the cloud. More on this unique possibility in my next blog: Piercing the Virtual Veil

Tags: Cloud Computing, Continuous network monitoring, IT Infrastructure, Virtualization