Open Source Cooperation

I’ve been participating in various open source communities for several years, but I am still occasionally blown away at how cooperative and efficient open source can be sometimes. 

The most recent example came with some of today’s news about the 0.4.9 release of libvirt-java. libvirt-java has historically been LGPL which due to ASF licensing guidelines meant that  folks working on Apache CloudStack couldn’t include KVM support part of the default build. The reason behind this is well grounded – people have an expectation that releases from an Apache project have licenses that are more permissive than even the LGPL. 

We discussed multiple ways of getting around this, including getting approval to make convenience builds which contained a non-default build option and contained libvirt-java as a dependency. One of CloudStack’s committers, Wido den Hollander, who is also active in the libvirt community communicated the various processes and struggles that we were going through with regards to KVM support. Within a few hours Daniel Veillard sent an email to all of the libvirt-java contributors asking them to consent to a relicense to MIT. Now just over a month later that work is complete and libvirt-java is MIT licensed. 

I’ve been party to more than one relicensing effort, and they are always painful, time consuming, and some of the most boring work imaginable. To the libvirt-java community, and Daniel Veillard in particular, thanks!! I appreciate you tackling this, and getting it accomplished so rapidly.  I am sure that such responsiveness to folks who consume your work will continue to increase the number of folks who use it, and in the process you’ve made (at least) two projects better. 

New features for the upcoming Apache CloudStack (incubating) 4.0 release

CloudStack is actively working towards its first release at the Apache Software Foundation. While much of the work has been getting the tree into shape from a licensing and guideline perspective, many enterprising folks were also working on new features and functionality. Someone recently asked on list what the features were for the upcoming release – 4 people have answered and have different answers, so I figured it was time to aggregate all of that and get it into an easily consumable format  (my blog). All of these features have been discussed on the development mailing list, but cloudstack-dev tends to be a somewhat heavy volume mailing list, so it’s easy to miss things.

I will add one caveat – there may be things that I am missing, and I reserve the right to update the post.

  • Nicira NVP support
  • Ceph/RBD support for KVM
  • EC2/S3 API support integrated into CloudStack (formerly a separate package)
  • Support for Caringo as backend storage for object storage
  • Inter-VLAN Routing (aka VPC)
  • Site-to-Site VPN
  • Local Storage Support for Data Volumes
  • Tagging virtual resources
  • AWS API Changes for Tags

I honestly didn’t expect this level of new features for the first release, even my initial recounting of features was 1/2 the size of this list, so it seems it will be a feature-packed release.

That said, we are closing in on a release, and would love to welcome any testing or review. You can find some early test source builds at:

You can get binaries to begin testing here:

CloudStack moves to Maven

CloudStack has historically used ant for most of its build needs. However, we kept all of the dependencies in binary form in the repository with the code. This was expedient but a bad habit. In the process of cleaning up that source tree we needed a tool that would handle dependencies in a sane manner. The short list of tools that we considered was:

We also considered an interim solution of continuing to use ant, but handling dependency resolution via an ant target with get tasks.

The reality of the situation is that folks had become comfortable with ant, it was relatively widely known. We knew of the other alternatives, but few actually possessed any experience with any of the tools. Hugo Trippaers built out gradle support for us to try, but indicated he was slightly more in favor of Maven. Darren Shepherd stepped up and built out maven support, indicating that he had used Maven in Godaddy’s implementation of CloudStack. Darren went one step better though; realizing how disruptive a build system change is, Darren’s first iteration leaves ant in place working as it has been – and adds maven support in parallel. This doesn’t necessarily adopt all of the ideal maven conventions, but moves us along, and Darren indicates that next release will more full embrace the maven way.

I know build plumbing isn’t as ‘sexy’ as some new features, but it impresses me to see new people emerging in the CloudStack community as we are incubating to take on big chunks of work to improve the project.

So want to take the new build tool for a spin? It’s easy – ensure you have maven installed (maven2 and maven3 both work, but the latter preferred) and run:

 mvn -s m2-settings.xml

A Runbook for CloudStack

Documentation is one of those vital things that any software, but especially open source software needs to be successful. When anyone can come along and download your software and use it, you suddenly have an incredibly diverse audience for not just your software but also your documentation. My experience with CloudStack have made that all the more evident to me. CloudStack, like many projects, requires an understanding of a number of different areas of practice – namely: virtualization, networking, Linux, and storage. The number of people who are experts in all of those areas is pretty small. Someone might be an ‘expert’ in 2 or 3 of those areas, but which one is always a mystery, then of course you have folks who are experts at all. So writing for a specific audience is at best tedious.

If you look at the existing documentation that exists, it has one other ‘flaw’, it documents almost everything. This really isn’t a flaw, you really do want documentation for everything, but if your target is new users of your software, documenting every esoteric feature they could possibly make use of is painful.

In #cloudstack on I kept hearing a common refrain, that it would take new folks 1-2 weeks to get CloudStack operating successfully, but once it was up, it ran great. The problem is that as a sysadmin, I know that being interrupt driven means that things that take a ton of time and effort are going to be dropped along the way, and perhaps never picked up again. While discussing this problem Chiradeep called my attention to RightScale’s runbooks; they don’t explore every option, instead they provide a prescriptive path to success for one niche way of doing things.

So I began to work on removing all of the choices – I picked the operating system, type of storage, network model, hypervisor, even the network addresses, and then documented the procedure for assembling those pieces into a CloudStack cloud. The first revision of which I’ve published here, with lots of help from Joe Brockmeier, Chiradeep, and Watnuss

A good chunk of this should be completely reusable – if people wanted to alter the network model or hypervisor, or any of the other choices, the base documentation should still be good, so feel free to fork it, or improve what is already there, you can find the source here:;a=tree;f=docs/runbook;h=5833cb61df2f195f8dd4e4825d5c3c8b8fc7d8ba;hb=HEAD

Virtualized CloudStack development environment

Hot off the mailing lists, there is a new disk image, designed to be run in VirtualBox, that provides a complete working Apache CloudStack (incubating) instance, along with a in-house hypervisor. Edison Su, one of the CloudStack developers, built this Ubuntu-based image to make things pretty simple to stand up a development environment.

Historically, the problem with developing for CloudStack was that it was easy to stand up the management server pieces on your local machine for testing, but you still needed a hypervisor (and potentially a network). You could condense all of that down and make it a single machine with KVM, but it’s still likely going to be a different machine than your personal desktop/laptop simply due to the invasiveness of the networking changes necessary for doing so.

This also has a added bonus of giving people a relatively completely setup environment to work with, with precious little extra effort.

But given my traditional blog audience, I know there will be several questions:

Why an Ubuntu-based image? Primarily because Ubuntu has Xen+XAPI (aka Kronos) support, which is effectively a XCP or XenServer in CloudStack’s eyes. While Fedora has Xen support it doesn’t yet have all of the XAPI bits (it’s something else that’s on my terribly long todo list)

Why Xen+XAPI instead of KVM? So Xen (regardless of XAPI) can provide para-virtualization in addition to hardware virtualization, while KVM is hardware virtualization only. While some hypervisors (KVM included) can perform nested hardware virtualization, PV still tends to be a bit more performant.

And an obligatory disclaimer – this obviously isn’t an official release of any kind, just an additional tool to help people hack on Apache CloudStack (incubating).

Get the image at:

Read the documentation at:

CloudStack, FOSDEM, and other uncategorized things

I haven’t done a great job blogging since FUDcon, so this omnibus blog post should hopefully get things up to date at least.

First, I am in Brussels for FOSDEM this weekend. I know there are tons of Fedora friends in EMEA that I have only corresponded with via IRC and email, so if you are FOSDEM, I hope to meet you in person.

Of course FOSDEM is later this week, and I traveled to Brussels yesterday – and not just so I can be a tourist. Kris Buytaert (of DevOps, and Everything is a Freaking DNS problem fame) is helping CloudStack host a Build an Open Source Cloud – Day in Antwerp on Friday, the day before FOSDEM starts. We have folks speaking from Puppetlabs, Xen, Zenoss, and of course CloudStack) There are still a few seats available, so if you are around, feel free to join us.

I’ve been incredibly busy with a number of things in CloudStack-world – we are drawing closer and closer to our 3.0 release, and just kicked out our 3rd beta release – (see the announcement here). This release is pretty transformative – a completely reworked UI that is simply gorgeous, and appears a lot more intuitive. Additionally we added a ton of security enhancements, provide for storing templates and snapshots in object storage (things like Swift, GlusterFS, and Caringo), dramatically increased our Networking-as-a-Service offerings, VM Storage migration (we already support live migration within a cluster of hypervisors, but this allows us to escape the cluster level and move VMs to different storage pools. I am getting pretty excited about this.

At the same time, Eric Christensen have been working on getting the final pieces of CloudStack packaging done for Fedora- and hope to have this done in the next week or so.

Lots of other things going on- but I just realized it’s 3:12am here – so I suppose I’ll stop writing and get some sleep before it’s time to eat breakfast.

Board member goals

I posted this to the advisory-board list, but want to have this here for additional visibility

Here is what I am passionate about working on in Fedora:

The cloud.

I apologize for using a buzzword in a serious task, but I see cloud-y things becoming far more important in the free software landscape and in technology in general. In many ways it parallels the goals (albeit in different ways) free software aspires to in empowering a user, by offering on-demand self-service access to resources. I also fear what happens if free software doesn’t continue to play a leading role here, and particularly if Fedora doesn’t continue to participate.

Thusly, my goals are:
* Ensure that the major IaaS providers make Fedora available as a deployable option.
* Ensure that the major open source IaaS and PaaS platforms are packaged and available in Fedora.
* Ensure that the surrounding ecosystem is also available in Fedora, be it the API abstraction tools like jclouds and fog, or provisioning tools like puppet cloud provisioner, knife, and Boxgrinder are in Fedora and available for people to use Fedora in cloud-building activities.

In addition to this main goal I have a couple goals regarding the governance within Fedora:

* Make sure the board remains responsive and accountable – often times people are hindered from getting things done by our slow response – and I hope a more fervent attention to trac and those pressing matters makes us more responsible and accountable.

As a corollary to that I have the following goal:

* Make sure the board stays out of micromanaging those who are doing work within Fedora. I firmly believe that the government that governs least governs best, and while I don’t want a disengaged board, I do want a board that doesn’t have to bless or anoint every action within Fedora. Fedora is a community of doers, and with a few special exceptions (trademark and other legals areas jump to mind) they should always feel empowered to get things done, and not need excessive amounts of permission.

Are you an open source build engineer?

CloudStack, the fabulous open source IaaS platform,  is looking for an build engineer (core competencies would be things like ant, rpm, debs, Jenkins, experience with continuous integration in general). It’d also be nice if you knew a thing or two about open source. If you are interested, don’t hesitate to reach out david – at – cloudstack -dot -org.


FUDcon day 2 and 3

Saturday at FUDcon began with BarCamp pitches, followed by the BarCamp sessions themselves along with the workshop sessions running in parallel. Lots of awesome content made decisions on what to attend difficult. There seemed to be a ton of cloud-related stuff going on, and most of which concentrated in a single room. I pitched and was able to give a session about CloudStack – and in particular on getting it into Fedora. As evening (and FUDpub) approached things started to wind down.

FUDpub’s venue was quite nice, with TVs (for those wishing to watch the game) multiple bowling lanes, and plenty of pool tables. Plenty of food to be had. And much fun abounded.

Sunday had a slower start for me, with an early non-Fedora meeting taking place, but once  I made it over to McBryde hall there were very clearly still lots of work taking place. As normally happens on the last day of FUDcon – the crowd started thinning out after mid-day. The board had a massive session that lasted about 3 hours, and covered both business and some discussion around monies and intra-project collaboration.And thus this FUDcon drew to a close for me.

I personally find that FUDcon is great for ‘recharging’ my interest in and commitment to Fedora, and getting to meet folks I’ve previously only dealt with via IRC or mailing lists helps build relationships.

Thanks to Ben, Cathy, and Jamie for doing the tons of work necessary for pulling off a FUDcon, and to Virginia Tech for hosting us.

FUDcon Blacksburg

FUDcon never ceases to amaze me. I am fortunate (or cursed, depending on your perspective I suppose) to be able to attend lots of free software and sysadmin conferences. However, FUDcon is always unique – the people who attend are focused on getting things done, and are generally interested in helping others to get things done as well.

I started a bit late in the morning for my arrival – the winds of Appalachia emphasizing the coolness. A number of folks camped out in an empty classroom (dubbed the Nothing Room) and started working on various things. I had a chance to meet Marek Goldmann (mgoldmann) of Boxgrinder fame, who I’ve conversed with a number of times, but never met in person. Marek, Andy Grimm (mull), Bob McWhirter, and I talked about some of the struggles and challenges with getting Java-based software into Fedora – and the differences between the Java development and distro-packaging worlds.

While I was there, I was working on getting a new version of ceph packaged – and Jon Stanley (jds2001) spent some time helping me get it to build.

Lunch time rolled around and then the spins process session from Cristoph Wickert (cwickert) started and we spent some time discussing process, dates, etc.

Following that I stopped in on Mo Morsi’s (mmorsi) Aeolus/Snap session before heading to the Board Meeting at FUDcon.

One of the things that was asked was how were the goals we had set earlier in 2011 – and did we accomplish anything, and how had the board performed. One of the things that came out of this (and for which I am one of the folks at fault) is that while we had identified good strategic goals – as a board overall we had failed to champion any of them, failed to work towards getting some of the tactical actions accomplished for those goals. We talked about how, in volunteer organizations, there’s no ability to ‘direct resources’ – but instead only the ability to inspire, and excite others, and that as a board we had done a poor job of that. Out of that we agreed that in Fedora, things happen by doing, specifically leading happens by getting things done, and hopefully getting others excited in the process. We, as a Board, agreed that we needed to individually adopt some of the tactical actions aimed at furthering the strategic goals, champion those tasks, and hopefully excite others in the process. We agreed to report our individual ‘tasks’ no later than Feb 1, and to provide regular timely updates.

I hope that the preceeding paragraph doesn’t strike folks as being doom and gloom – while I tend to think that as a board we could have executed better, I am also thrilled that we had the chance to sit down, converse, acknowledge, and hopefully begin addressing the problem.

Of course 5pm doesn’t mean FUDcon stops, lots of sessions kept making progress, lots of hallway track session in the hotel lobby, and groups going off for dinner, followed by more hacking in the lobby. An awesome day at FUDcon, nice seeing Fedora friends, and meeting those I’ve only conversed with on mailing lists and IRC. Can’t wait for tomorrow.