Building your own cloud

Let me start with a disclaimer that I do not intend to write a 'How-to' document in this post. My primary motivation behind writing this article is that I see too many overloaded and clouded use of the word 'cloud', which in my opinion is becoming serious demotivator for honest and hype-shy people to really feel interested to cloud. From that point of view my humble attempt here would be to deglamourize 'cloud' and lay it as bare as possible. Hopefully along the way we will be able to understand what it takes to build one cloud.
Conceptually cloud refers to the arrangement where application server configuration and storage space can be dynamically changed  preferably with a software interface. Essentially this means 1. application server is decoupled from physical server configuration and 2. there is software interface which allows the administrator to monitor loads and performances and add/modify server configuration most desirably remotely.  Advantages are two-fold, the application server becomes self-contained entity which can migrate from one hardware to another as needed and secondly it gives us the freedom to think in terms of our application's need alone. This simplifies a lot in application provisioning, especially if the user does not have the expertise in data-centre technologies or it does not want to incur large CapEx in data centre.
Technologically, there is one crucial element that has accelerated the shaping up of cloud and that is virtualization. Not that the concept is anything new, but the way it has been made ubiquitous has brought new value. With increasing number of faster processors and larger RAM becoming the norm for modern physical servers, it  became evident some time back that running a single OS with single server application would mean significant under-utilization of the server m/c. For example, a Dell PowerEdge 310 is a relatively low-end server m/c but it provides one Quad-core Intel® Xeon® 3400 series processor and 8GB RAM (expandable to 32GB) in one configuration. Running a single application is serious wastage of all its processing power, unless the application is heavily used all the time. In a typical case, an application server's load is input driven and only takes up compute and networking bandwidth for a fraction of the time the resources are up. 
Instead one can install VMware vSphere or Microsoft Hyper-V and have tens of VMs each with its own application server running on a single server m/c. Great thing about this VM (Virtual Machines) is that all the needed interfaces (serial, network) come bundled with it. One just have to install (provisioning is almost effortless with all commercial VMs) them and they are ready for application provisioning. Best part of it, with one time configuration of all the VMs, getting a new application up requires very small time. One can even have VM templates for different type of servers [e.g. Oracle server VM or exchange server VM or an Ubuntu based linux web server VM] and install the template once a VM is allocated.
Now a server needs storage for its data which also keeps growing. Adding a physical storage volume or a physical LUN [in SAN concept] to each VM is bound to bring under-utilization of storage bandwidth. Instead, storage vendors provide virtual volumes / LUNs which can be provisioned over a physical Volume or LUNs[which is just a bundle of disks sharing same RAID structure].
 VM and Vstorage (i.e. Virtual storage volume or virtual LUN) thus can be thought of as the unit of provisioning in an IT setup. All that one needs is a horizontal software layer that monitors and configures the VMs [with all the interfaces] and VStorages and one has basic working Cloud ready. A user of this cloud can be allocated VMs with pre-installed applications and storage and a software interface using which the user can manage and monitor his resources. When he needs more compute or network bandwidth he places the request to the Cloud administrator and from the Cloud's readily available VM pool, the administrator assigns adequate resources to the user. This generic model is used for what is  known as IAAS or Infrastructure As A Service. If the cloud needs to provide support for higher abstraction of service, it needs further sophistication at the horizontal software layer. For example, let's assume the user needs to run an application that sifts through huge data which are distributed across many compute and storage nodes. The application needs the support of a more sophisticated interface which can scale up or scale down resources with the volume of data while  providing a consistent data manipulation engine across many nodes. Yes, we are talking about Hadoop like software layer. We cannot cover Hadoop here but the point should be clear, complexity of cloud is driven by the sophistication of the application that the cloud is going to host, but at the essential level, Cloud remains a set of virtualized compute and storage resources which are governed by a single software layer.
As one can imagine, a basic Cloud setup can be built entirely using Opensource elements too. In the next post we will talk about a basic cloud setup that we built with linux KVM and Python.