CloudStack system VM architecture

I had a recent discussion with some folks wondering why there was now an option for 32 or 64-bit System VMs with CloudStack 4.3. I provided an answer, and linked back to some mailing list discussions. I figured this might be of general interest, so I’d document in the short term with a blog post.

For background, system VMs provide services like dealing with snapshots and image templates, providing network services like load balancing, or proxying console access to virtual machines. They’ve historically been 32-bit. The reason for this is that the 32-bit arch has been very efficient with memory usage, and since these are horizontally scalable it’s easy to just spin up another. 

But you can have either – which do you pick?

Depending on the workload you might have a different answer. Some hypervisors work better with one arch over the other; and that might be a factor; but ignoring hypervisors lets examine the reason you’d want to use either.  32-bit:  32-bit operating systems are pretty efficient with their use of memory compared to 64-bit. (e.g. the same information typically occupies less space in memory). However there are limits on memory. (Yes, you could use PAE with a 32-bit kernel to get more addressable memory, but there is considerable CPU overhead to do so – which makes it inefficient given that all of this is virtualized) The 32-bit kernels also have a limit on how much memory is used by the kernel. This is really where the use case of 64-bit System VMs evolved from. Because one of the system VM functions is providing load balancing, the conntrack kernel module had a practical limit of ~2.5M connections – and that left precious little room for the kernel to do other things. CloudStack orchestrates HAProxy as the default virtual LB, which in turn uses conntrack. Having a heavily trafficked web property behind CloudStack’s 32-bit virtual load balancer might run into that limitation. 

64-bit: Not nearly as efficient with memory usage; however it can address more of it. You’ll actually tend to need more memory for the same level of functionality; but if you need to push the envelope further than a 32-bit machine, then at least you have an option to do so.

In short, you should probably default to 32-bit system VMs unless you envision that you might need to take advantage of the benefits a 64-bit system VM provides you.  

If you want to see the original discussion around this topic from the dev@cloudstack.apache.org mailing list, look here: 
http://markmail.org/message/i5kolazi5so52eon


Getting ready for the CloudStack Collaboration Conference Europe

I’ve been in Amsterdam two days thus far in preparation for the CloudStack Collaboration Conference. I had planned to come in and show up at the the Schuberg Philis offices, and help with getting things ready for the conference. There hasn’t been a lot for me to do though, I looked at some shiny demo racks, I helped load a monitor, and that’s about it. The folks at SBP are really very squared away in terms of conference preparation and are doing very well.

This has given me time to focus on recovering from jet lag, and getting to spend time talking with folks. I also have already started meeting folks I only know via email. Some of the pre-conference discussions are intriguing. But this is all pre-conference – tomorrow things actually start – and they start with a hackathon. The proposed hacking sessions actually leave me with multiple things that I want to work on. Top of mind is:

  • Docs – fixing 4.2.1 release notes and working on 4.3 and beyond.
  • Gluster – getting CloudStack to consume GlusterFS natively.
  • KVM Agent refactoring

We also have the space til late in the night, so this won’t be a hack for a few hours and then disappear, we can keep working well into the evening.

One of the things that I didn’t like about our last in-person hackfest, was the lack of a feedback cycle. So I want to try and encourage folks holding the hackfest sessions to report what they worked on and what actually got accomplished at least to the mailing list, but hopefully also to a blog.

Should be a fun day.


My thoughts on Apache CloudStack’s graduation

Today was a great milestone for Apache CloudStack. If you haven’t seen the news (but yet somehow have come across this blog post???) the Apache Software Foundation announced that CloudStack had graduated from the Incubator as a top-level project. While in many ways it marks the end of a number of personal and project goals, it’s also just another milepost along the journey.

I’ve been working on CloudStack since most folks knew it as ‘Cloud.com’, and it’s amazing to see the difference over the space of a couple of years. I’ve been involved with several open source projects for a number of years, and I knew of the ASF by reputation, but had no personal experience. I spent weeks reading the documentation on the website when we first began discussing the potential for a move to the ASF, and I rapidly became both impressed and afraid. Impressed, because I saw codified in front of me the most transparent and open expectations of anything I’d been involved in. I suddenly appreciated why the ASF had the reputation it did. Afraid because the magnitude of change was incredible. The transformation hasn’t been perfectly smooth, I even questioned if the dramatic change would be so much as to be overly disruptive. Many folks have written of how dramatic it is to open source a project – but to take a project that was open source but still heavily commercially governed and move that to the Apache Software Foundation, is both extremes of the open source spectrum.

Many people will write about the tremendous growth in community numbers, the more interesting story to me is the tremendous growth of community responsibility, over 50% of the Project Management Committee don’t work for Citrix. Of course committers and PMC members are expected to behave as individuals, and in the best of interest of the project, but that amount of diversity in a short time is impressive. I also find it fascinating just how many folks who are participating are CloudStack users; they have truly taken ownership and responsibility for their IaaS platform. What does graduation mean from my perspective? A number of different things, but most poignantly, it means that we have met the expectations of our mentors, the Incubator, and the Apache Software Foundation. I am excited about the future. I have no delusions that for many, or even most of our users, our graduation has little or no immediate impact. It does have an impact for the project, as we shift our focus forward, and I think that will tremendously benefit our users. I think that you’ll continue to see impressive things from Apache CloudStack, we’re really only getting started.

A couple of words of thanks to folks

I can’t express how wonderful our mentors were. They understood the process, they saw the challenges, they stepped in where appropriate and let us find the solution when we needed to. I’ve walked away thoroughly impressed at both the individuals as well as the incubation process in general, I am sure it can be improved, but struggle to think of anyone doing it better.

The folks at Apache Infrastructure – they are doing an incredible and impossible job – supporting well over 100 top level projects including such behemoths as Hadoop, Maven, and now CloudStack; dealing with plenty of inbound incubator projects, some of which, like CloudStack, have years of history, and thus plenty of baggage to bring with them. Thanks for the immense amount of help you folks have provided.


Upgrading CloudStack from 4.0.0-incubating to 4.0.1-incubating

I spent my weekend upgrading my cloud. Yes my CloudStack IaaS cloud isn’t going to threaten AWS EC2  anytime soon, but it still serves a vital purpose for me, giving me a place to provision services, provide a test bed, and many other things.

What does my personal cloud look like? It’s a small cloud of 6 physical nodes (Dell R210s) based in a datacenter somewhere in San Jose. One of the 6 nodes is a CloudStack management server running CentOS 6.3 and KVM running on the same node, and three more KVM nodes. I also have one XenServer and one XCP node.  That said, the principles behind upgrading remained the same regardless of the machine.

So why did I upgrade? Well Apache CloudStack 4.0.1-incubating, was announced today (and released late last week) What’s the difference between 4.0.0-incubating and 4.0.1-incubating? Well, there are no new features, but there are a number of bugfixes included.

CloudStack versions now adhere to Semantic Versioning, which means this is merely a maintenance update or a bugfix update. Bugfix releases are great for production deployments – the change is minimal, and it’s all focused on fixing problems. Within open source projects though, bugfix releases tend to be unsexy. Many communities skip them entirely because developers want to work on new things and don’t work on keeping up with the massive investment that is backport fixes. It is, I think, a reflection of project maturity and operator-driven community of CloudStack that a bugfix release is already out and another is on the way (4.0.2 planning emails started flowing even before 4.0.1 was announced.) Yes many developers are looking at the next feature release (4.1.0) and even the one past that (4.2.0) but that doesn’t diminish the importance of the 4.0.x install base, or the expectations that those folks have – many folks, especially large cloud deployments aren’t going to want to perform a feature upgrade every few months, but can tolerate the smaller bugfix releases.

But enough about that – if you have 4.0.0 deployed you are interested in how the upgrade went aren’t you? So I have confession to make. First, I didn’t completely follow the upgrade directions. You should. The management server upgrade was painless – I ran:

# yum -y update cloud-*

followed by

# service cloud-management restart

And all was well.

Next was my KVM agents. I shelled into these as well (yes, I know, I am awful for not leveraging configuration management, especially when I am such a CM advocate), but my environment is where I test installation instructions, and most (but not all) of the upgrade instructions) I didn’t bother to stop the agents, or even to put them into maintenance mode. I ran

# yum -y update

to get all of the operating system upgrades as well as the CloudStack upgrade, and then:

# service cloud-agent restart

and my agents reconnected to the management server without a hitch. I tested a few deploys and destroys and everything worked well. Zero VM downtime for any of my running VMs.

Again, you should follow the directions, make DB backups, etc, but you shouldn’t expect any issues on the 4.0.0->4.0.1 upgrade path.

(Note: I first published this at BuildACloud.org)

Book Review: The Phoenix Project

I’ve been hearing about the Gene Kim and company working on “The Phoenix Project” for almost a year now, most frequently at places like DevOps Days.

Knowing it was modeled after Eli GoldRatt‘s “The Goal” and how it was essentially a business novel, in which Goldratt uses a story of the troubles of a manufacturing manager to lay out the tenets of Constraint Theory. I first read The Goal more than a dozen years ago, and it wasn’t necessarily the finest piece of literature, but it was a compelling book, and the book and corresponding theory has become a thing of legend.

Fast forward several years and one of my friends and DevOps guru John Willis starts talking “The Goal” in the context of DevOps, and then Ben Rockwood points back further to folks like Edward Deming. So trying to imitate the effect of books like “The Goal” is a very lofty aspiration. I honestly didn’t think that it could be done, particularly if driven around IT.

But then Tuesday, I hear via email and twitter, that the book has finally been released. I was travelling on Tuesday and ordered the e-book that afternoon so that I could perhaps finish it by the end of the week. I started reading it during dinner, and by the time I retreated to my room I was already hooked. The book was captivating. I saw myself at various stages of my life in several of the characters, I could empathize with the problems. I couldn’t stop reading, and actually finished the book that evening.

The basic synopsis of the book is that Bill gets a sudden promotion to VP of IT after two of his superiors are dismissed, and is immediately thrown into the fire with business threatening outages and a vastly overbudget and behind schedule project that is supposed to ‘save the company’. He struggles to deal with the deluge of ongoing problems and get the project back on track, while at the same time political battles rage around him. With the help of Erik, a prospective board member and ‘IT guru’ he dramatically changes the culture and method of operation for both operations and development, and saves the company.

So my thoughts – I think that the book will likely have the same effects as “The Goal” but with a focus on the IT business – or perhaps pushing folks to realize that IT really isn’t so different from manufacturing. I think it should also inspire folks to adopt the DevOps culture.
That said, I don’t expect the kind of timelines that Bill achieved to be normal, and especially without folks like the mentor Erik who has the respect and trust of the executives to shield Bill and advocate for his changes. That said, it is still a tremendous book, and I know Gene Kim and others hope that it will usher in the DevOps revolution to the masses, and I think there is a chance that it will happen.

In short, if you are remotely related to IT, or have any curiosity about DevOps you should be reading this book as soon as possible. I noted that yesterday Gene sent out an email that said that the book was #1 in the Kindle business management section on Amazon, outpacing books like “The Lean Startup” and “Good to Great”

Viva la DevOps Revolution!


CloudStack 4.0.0 – released

Moving to the ASF has been an interesting process and quite a whirlwind. In addition to all of shuffling of infrastructure to the ASF, migration of codebases and licensing there’s been an amazing influx of participation in the project.

Of course in addition to all of this shuffling of resources – we still have the business of running the project, producing software, and hopefully getting a release ready. Of course in getting a release ready there are plenty of new policies and guidelines to be complied with; and CloudStack is a pretty established and large codebase, so plenty of things to audit and consider and a few things needing actual change. We finally (as you hopefully have heard) had a release that met those standards and was approved by both the community and the IPMC. This is noteworthy in and of itself, but of course why many were working on getting the codebase into shape, others continued to work on their desired features – and it astounds me at the number of things that happened concurrently:

Of course this isn’t an exhaustive list – you can see that in the Release Notes.

  • Inter-VLAN routing (aka VPC)
  • Site-to-site VPN
  • Local storage for data volumes
  • Nicira NVP SDN support
  • Object storage via Caringo
  • CLVM support (returns)
  • Ceph (RBD) support for KVM

Of course – you know you want all of this goodness –  and fortunately the 4.0.0-incubating release is available you can download it and begin using it today.


Secondary storage scalability configuration and tuning in CloudStack

Someone asked about this recently on the cloudstack-users mailing list, and I figure this is both an interesting design point of CloudStack as well as a specific piece of knowledge that others will find useful.

So first, lets talk about what Secondary Storage is – especially if you are not familiar with CloudStack’s hierarchy. Secondary storage is where templates (aka non-running disk images) and snapshots are stored. The  Secondary Storage VM (SSVM) handles a number of things, most importantly shuttling images and snapshots between various primary storage (where running disk images are stored) resources and the secondary storage resources. The SSVM also handles receiving uploaded disk images and aging/scavenging snapshots. The SSVM is really just some CloudStack agent software running atop a Linux VM, but bears the name SSVM because of the functionality that it is providing.

So why run this storage management service  as a stateless virtual machine instead of incorporating it into some centralized all-in-one management server? First your management server might not be network-proximate to your secondary storage resource. Particularly if you have multiple geographically disparate datacenters with resources being managed in just a few locations. Secondly, all-in-one isn’t very scalable. With the stateless ‘worker VMs’ CloudStack can dynamically scale the number of VMs up to deal with load. The minimum is that a single SSVM will exist in each zone. CloudStack uses two different configuration options to determine when to scale up and add more:

  • secstorage.capacity.standby
  • secstorage.session.max

secstorage.capacity.standby is the minimal number of command execution sessions that system is able to serve immediately. secstorage.session.max is the max number of command execution sessions that a SSVM can handle. By manipulating these settings (making the former higher and latter lower) you can trigger a faster automatic scale at lower load, or you can let the defaults do their job.

Of course that’s the new-fangled cloud way of doing things – just add more workers to do the work, but that’s not the only way of tuning or making your SSVM more scalable.

First, you can just make the SSVM bigger. The default service offering is for a 500Mhz, single vCPU, 256MB of RAM machine. That is pretty small, and works well for environments where change isn’t high, snapshots are infrequent, and the environment itself is small. You can of course edit that service offering or define a new one to be beefier. If you define a new offering you need to configure that with the secstorage.service.offering parameter.  Not only can you scale out, but you can also scale up.

Of course, nothing really trumps architecting things right in the first place. It’s pretty trivial to throw up a cloud computing environment with a single NIC on every host and letting management, guest, public, and storage traffic traverse a single physical network. And there isn’t inherently anything wrong with that, but if you really need to maximize efficiency, you can always have a dedicated storage network, and even better, enable jumbo frames on your networking hardware as well as your hypervisors, storage hardware – and yes your SSVM as well – you can configure that on the SSVM with the secstorage.vm.mtu.size parameter.

Even if you don’t have intense levels of snapshot work and template work, you may still want to tune your SSVM to make it more time-efficient.


Follow

Get every new post delivered to your Inbox.