I’ve been working on CloudStack since most folks knew it as ‘Cloud.com’, and it’s amazing to see the difference over the space of a couple of years. I’ve been involved with several open source projects for a number of years, and I knew of the ASF by reputation, but had no personal experience. I spent weeks reading the documentation on the website when we first began discussing the potential for a move to the ASF, and I rapidly became both impressed and afraid. Impressed, because I saw codified in front of me the most transparent and open expectations of anything I’d been involved in. I suddenly appreciated why the ASF had the reputation it did. Afraid because the magnitude of change was incredible. The transformation hasn’t been perfectly smooth, I even questioned if the dramatic change would be so much as to be overly disruptive. Many folks have written of how dramatic it is to open source a project – but to take a project that was open source but still heavily commercially governed and move that to the Apache Software Foundation, is both extremes of the open source spectrum.

Many people will write about the tremendous growth in community numbers, the more interesting story to me is the tremendous growth of community responsibility, over 50% of the Project Management Committee don’t work for Citrix. Of course committers and PMC members are expected to behave as individuals, and in the best of interest of the project, but that amount of diversity in a short time is impressive. I also find it fascinating just how many folks who are participating are CloudStack users; they have truly taken ownership and responsibility for their IaaS platform. What does graduation mean from my perspective? A number of different things, but most poignantly, it means that we have met the expectations of our mentors, the Incubator, and the Apache Software Foundation. I am excited about the future. I have no delusions that for many, or even most of our users, our graduation has little or no immediate impact. It does have an impact for the project, as we shift our focus forward, and I think that will tremendously benefit our users. I think that you’ll continue to see impressive things from Apache CloudStack, we’re really only getting started.

A couple of words of thanks to folks

I can’t express how wonderful our mentors were. They understood the process, they saw the challenges, they stepped in where appropriate and let us find the solution when we needed to. I’ve walked away thoroughly impressed at both the individuals as well as the incubation process in general, I am sure it can be improved, but struggle to think of anyone doing it better.

The folks at Apache Infrastructure – they are doing an incredible and impossible job – supporting well over 100 top level projects including such behemoths as Hadoop, Maven, and now CloudStack; dealing with plenty of inbound incubator projects, some of which, like CloudStack, have years of history, and thus plenty of baggage to bring with them. Thanks for the immense amount of help you folks have provided.

What does my personal cloud look like? It’s a small cloud of 6 physical nodes (Dell R210s) based in a datacenter somewhere in San Jose. One of the 6 nodes is a CloudStack management server running CentOS 6.3 and KVM running on the same node, and three more KVM nodes. I also have one XenServer and one XCP node.  That said, the principles behind upgrading remained the same regardless of the machine.

So why did I upgrade? Well Apache CloudStack 4.0.1-incubating, was announced today (and released late last week) What’s the difference between 4.0.0-incubating and 4.0.1-incubating? Well, there are no new features, but there are a number of bugfixes included.

CloudStack versions now adhere to Semantic Versioning, which means this is merely a maintenance update or a bugfix update. Bugfix releases are great for production deployments – the change is minimal, and it’s all focused on fixing problems. Within open source projects though, bugfix releases tend to be unsexy. Many communities skip them entirely because developers want to work on new things and don’t work on keeping up with the massive investment that is backport fixes. It is, I think, a reflection of project maturity and operator-driven community of CloudStack that a bugfix release is already out and another is on the way (4.0.2 planning emails started flowing even before 4.0.1 was announced.) Yes many developers are looking at the next feature release (4.1.0) and even the one past that (4.2.0) but that doesn’t diminish the importance of the 4.0.x install base, or the expectations that those folks have – many folks, especially large cloud deployments aren’t going to want to perform a feature upgrade every few months, but can tolerate the smaller bugfix releases.

But enough about that – if you have 4.0.0 deployed you are interested in how the upgrade went aren’t you? So I have confession to make. First, I didn’t completely follow the upgrade directions. You should. The management server upgrade was painless – I ran:

# yum -y update cloud-*

followed by

# service cloud-management restart

And all was well.

Next was my KVM agents. I shelled into these as well (yes, I know, I am awful for not leveraging configuration management, especially when I am such a CM advocate), but my environment is where I test installation instructions, and most (but not all) of the upgrade instructions) I didn’t bother to stop the agents, or even to put them into maintenance mode. I ran

# yum -y update

to get all of the operating system upgrades as well as the CloudStack upgrade, and then:

# service cloud-agent restart

and my agents reconnected to the management server without a hitch. I tested a few deploys and destroys and everything worked well. Zero VM downtime for any of my running VMs.

Again, you should follow the directions, make DB backups, etc, but you shouldn’t expect any issues on the 4.0.0->4.0.1 upgrade path.

Knowing it was modeled after Eli GoldRatt‘s “The Goal” and how it was essentially a business novel, in which Goldratt uses a story of the troubles of a manufacturing manager to lay out the tenets of Constraint Theory. I first read The Goal more than a dozen years ago, and it wasn’t necessarily the finest piece of literature, but it was a compelling book, and the book and corresponding theory has become a thing of legend.

Fast forward several years and one of my friends and DevOps guru John Willis starts talking “The Goal” in the context of DevOps, and then Ben Rockwood points back further to folks like Edward Deming. So trying to imitate the effect of books like “The Goal” is a very lofty aspiration. I honestly didn’t think that it could be done, particularly if driven around IT.

But then Tuesday, I hear via email and twitter, that the book has finally been released. I was travelling on Tuesday and ordered the e-book that afternoon so that I could perhaps finish it by the end of the week. I started reading it during dinner, and by the time I retreated to my room I was already hooked. The book was captivating. I saw myself at various stages of my life in several of the characters, I could empathize with the problems. I couldn’t stop reading, and actually finished the book that evening.

The basic synopsis of the book is that Bill gets a sudden promotion to VP of IT after two of his superiors are dismissed, and is immediately thrown into the fire with business threatening outages and a vastly overbudget and behind schedule project that is supposed to ‘save the company’. He struggles to deal with the deluge of ongoing problems and get the project back on track, while at the same time political battles rage around him. With the help of Erik, a prospective board member and ‘IT guru’ he dramatically changes the culture and method of operation for both operations and development, and saves the company.

So my thoughts – I think that the book will likely have the same effects as “The Goal” but with a focus on the IT business – or perhaps pushing folks to realize that IT really isn’t so different from manufacturing. I think it should also inspire folks to adopt the DevOps culture.
That said, I don’t expect the kind of timelines that Bill achieved to be normal, and especially without folks like the mentor Erik who has the respect and trust of the executives to shield Bill and advocate for his changes. That said, it is still a tremendous book, and I know Gene Kim and others hope that it will usher in the DevOps revolution to the masses, and I think there is a chance that it will happen.

In short, if you are remotely related to IT, or have any curiosity about DevOps you should be reading this book as soon as possible. I noted that yesterday Gene sent out an email that said that the book was #1 in the Kindle business management section on Amazon, outpacing books like “The Lean Startup” and “Good to Great”

Viva la DevOps Revolution!

Of course in addition to all of this shuffling of resources – we still have the business of running the project, producing software, and hopefully getting a release ready. Of course in getting a release ready there are plenty of new policies and guidelines to be complied with; and CloudStack is a pretty established and large codebase, so plenty of things to audit and consider and a few things needing actual change. We finally (as you hopefully have heard) had a release that met those standards and was approved by both the community and the IPMC. This is noteworthy in and of itself, but of course why many were working on getting the codebase into shape, others continued to work on their desired features – and it astounds me at the number of things that happened concurrently:

Of course this isn’t an exhaustive list – you can see that in the Release Notes.

• Inter-VLAN routing (aka VPC)
• Site-to-site VPN
• Local storage for data volumes
• Nicira NVP SDN support
• Object storage via Caringo
• CLVM support (returns)
• Ceph (RBD) support for KVM

Of course – you know you want all of this goodness –  and fortunately the 4.0.0-incubating release is available you can download it and begin using it today.

So first, lets talk about what Secondary Storage is – especially if you are not familiar with CloudStack’s hierarchy. Secondary storage is where templates (aka non-running disk images) and snapshots are stored. The  Secondary Storage VM (SSVM) handles a number of things, most importantly shuttling images and snapshots between various primary storage (where running disk images are stored) resources and the secondary storage resources. The SSVM also handles receiving uploaded disk images and aging/scavenging snapshots. The SSVM is really just some CloudStack agent software running atop a Linux VM, but bears the name SSVM because of the functionality that it is providing.

So why run this storage management service  as a stateless virtual machine instead of incorporating it into some centralized all-in-one management server? First your management server might not be network-proximate to your secondary storage resource. Particularly if you have multiple geographically disparate datacenters with resources being managed in just a few locations. Secondly, all-in-one isn’t very scalable. With the stateless ‘worker VMs’ CloudStack can dynamically scale the number of VMs up to deal with load. The minimum is that a single SSVM will exist in each zone. CloudStack uses two different configuration options to determine when to scale up and add more:

• secstorage.capacity.standby
• secstorage.session.max

secstorage.capacity.standby is the minimal number of command execution sessions that system is able to serve immediately. secstorage.session.max is the max number of command execution sessions that a SSVM can handle. By manipulating these settings (making the former higher and latter lower) you can trigger a faster automatic scale at lower load, or you can let the defaults do their job.

Of course that’s the new-fangled cloud way of doing things – just add more workers to do the work, but that’s not the only way of tuning or making your SSVM more scalable.

First, you can just make the SSVM bigger. The default service offering is for a 500Mhz, single vCPU, 256MB of RAM machine. That is pretty small, and works well for environments where change isn’t high, snapshots are infrequent, and the environment itself is small. You can of course edit that service offering or define a new one to be beefier. If you define a new offering you need to configure that with the secstorage.service.offering parameter.  Not only can you scale out, but you can also scale up.

Of course, nothing really trumps architecting things right in the first place. It’s pretty trivial to throw up a cloud computing environment with a single NIC on every host and letting management, guest, public, and storage traffic traverse a single physical network. And there isn’t inherently anything wrong with that, but if you really need to maximize efficiency, you can always have a dedicated storage network, and even better, enable jumbo frames on your networking hardware as well as your hypervisors, storage hardware – and yes your SSVM as well – you can configure that on the SSVM with the secstorage.vm.mtu.size parameter.

Even if you don’t have intense levels of snapshot work and template work, you may still want to tune your SSVM to make it more time-efficient.

Additionally the primary server that used to house CloudStack code (git.cloud.com) has been retired.

The canonical source code repo for CloudStack is (and has been for sometime):

https://git-wip-us.apache.org/repos/asf/incubator-cloudstack.git

Of course we aren’t done migrating all of the resources, more of that coming soon.

The most recent example came with some of today’s news about the 0.4.9 release of libvirt-java. libvirt-java has historically been LGPL which due to ASF licensing guidelines meant that  folks working on Apache CloudStack couldn’t include KVM support part of the default build. The reason behind this is well grounded – people have an expectation that releases from an Apache project have licenses that are more permissive than even the LGPL. 

We discussed multiple ways of getting around this, including getting approval to make convenience builds which contained a non-default build option and contained libvirt-java as a dependency. One of CloudStack’s committers, Wido den Hollander, who is also active in the libvirt community communicated the various processes and struggles that we were going through with regards to KVM support. Within a few hours Daniel Veillard sent an email to all of the libvirt-java contributors asking them to consent to a relicense to MIT. Now just over a month later that work is complete and libvirt-java is MIT licensed. 

I’ve been party to more than one relicensing effort, and they are always painful, time consuming, and some of the most boring work imaginable. To the libvirt-java community, and Daniel Veillard in particular, thanks!! I appreciate you tackling this, and getting it accomplished so rapidly.  I am sure that such responsiveness to folks who consume your work will continue to increase the number of folks who use it, and in the process you’ve made (at least) two projects better. 

I will add one caveat – there may be things that I am missing, and I reserve the right to update the post.

• Nicira NVP support
• Ceph/RBD support for KVM
• EC2/S3 API support integrated into CloudStack (formerly a separate package)
• Support for Caringo as backend storage for object storage
• Inter-VLAN Routing (aka VPC)
• Site-to-Site VPN
• Local Storage Support for Data Volumes
• Tagging virtual resources
• AWS API Changes for Tags

I honestly didn’t expect this level of new features for the first release, even my initial recounting of features was 1/2 the size of this list, so it seems it will be a feature-packed release.

That said, we are closing in on a release, and would love to welcome any testing or review. You can find some early test source builds at:

http://people.apache.org/~chipchilders/cloudstack/4.0/

You can get binaries to begin testing here:

http://jenkins.cloudstack.org/job/build-cloudstack-4.0-ubuntu12.04/

http://jenkins.cloudstack.org/job/build-cloudstack-4.0-rhel6.3/

• Ivy
• Gradle
• Maven

We also considered an interim solution of continuing to use ant, but handling dependency resolution via an ant target with get tasks.

The reality of the situation is that folks had become comfortable with ant, it was relatively widely known. We knew of the other alternatives, but few actually possessed any experience with any of the tools. Hugo Trippaers built out gradle support for us to try, but indicated he was slightly more in favor of Maven. Darren Shepherd stepped up and built out maven support, indicating that he had used Maven in Godaddy’s implementation of CloudStack. Darren went one step better though; realizing how disruptive a build system change is, Darren’s first iteration leaves ant in place working as it has been – and adds maven support in parallel. This doesn’t necessarily adopt all of the ideal maven conventions, but moves us along, and Darren indicates that next release will more full embrace the maven way.

I know build plumbing isn’t as ‘sexy’ as some new features, but it impresses me to see new people emerging in the CloudStack community as we are incubating to take on big chunks of work to improve the project.

So want to take the new build tool for a spin? It’s easy – ensure you have maven installed (maven2 and maven3 both work, but the latter preferred) and run:

 mvn -s m2-settings.xml

If you look at the existing documentation that exists, it has one other ‘flaw’, it documents almost everything. This really isn’t a flaw, you really do want documentation for everything, but if your target is new users of your software, documenting every esoteric feature they could possibly make use of is painful.

In #cloudstack on irc.freenode.net I kept hearing a common refrain, that it would take new folks 1-2 weeks to get CloudStack operating successfully, but once it was up, it ran great. The problem is that as a sysadmin, I know that being interrupt driven means that things that take a ton of time and effort are going to be dropped along the way, and perhaps never picked up again. While discussing this problem Chiradeep called my attention to RightScale’s runbooks; they don’t explore every option, instead they provide a prescriptive path to success for one niche way of doing things.

So I began to work on removing all of the choices – I picked the operating system, type of storage, network model, hypervisor, even the network addresses, and then documented the procedure for assembling those pieces into a CloudStack cloud. The first revision of which I’ve published here, with lots of help from Joe Brockmeier, Chiradeep, and Watnuss

http://people.apache.org/~ke4qqq/runbook/

A good chunk of this should be completely reusable – if people wanted to alter the network model or hypervisor, or any of the other choices, the base documentation should still be good, so feel free to fork it, or improve what is already there, you can find the source here:

https://git-wip-us.apache.org/repos/asf?p=incubator-cloudstack.git;a=tree;f=docs/runbook;h=5833cb61df2f195f8dd4e4825d5c3c8b8fc7d8ba;hb=HEAD