Head in the Clouds

I had the pleasure last week of working with a Bothell-based hardware vendor, Silicon Mechanics, and a company called Private Cloud Leasing to get CloudStack up and running on what is literally a datacenter on wheels (the small unit in the picture).

We took this Cloud-on-Wheels to the LinuxFest Northwest show at Bellingham Technical College, and put it on the expo hall floor.

On Sunday, we unplugged it and rolled it out of the expo hall, across the plaza, and up to one of the classrooms for the CloudStack session. The unit itself is self-propelled, runs on UPS,  and you can basically move it around any building that is wheelchair accessible.

The Cloud-on-Wheels generated A LOT OF ATTENTION (it’s very blue) and thus was a great draw to the booth and session.  As we rolled the unit through the hallways of the college, it was a bit like the Pied Piper…we had a group of folks following us, curious to see what we were doing with this thing. 

Interested in building elastic, scalable and profitable open source clouds? Join Citrix for free day of learning, best practices and industry insights at the Build a Cloud Workshop in San Francisco, CA on May 10, 2012.  This free one-day workshop is held alongside with Citrix Synergy 2012 and includes a pass to the Synergy Keynote and Solutions Expo. This event offers a unique opportunity to learn about the cloud building as well as the challenges and successes of delivering cloud services.

• Learn how to build an open source cloud with solutions from Citrix CloudStack, Canonical, RightScale, Zenoss and Xen.org.
• Network with peers, industry experts and solution providers at the CloudStack Community Reception

The Build a Cloud Workshop is held at the Westin Market Street (Metropolitan Ballroom 1 and 2, 2nd Floor) located at 50 3rd St, San Francisco, CA 94103. Click here to learn more and register.

Build a Cloud Workshop Agenda

8:30 – 1:00 Optional            Free admission to the Synergy Keynote,Synergy Solutions Expo and lunch

1:00 – 2:00                           Introduction to Cloud Computing Mark Hinkle, Sr. Director Cloud Community

2:00 – 3:00                           Virtualization in the Cloud Lars Kurth, Xen.org Community Manager

Since we made the announcement of licensing and governance shift a few weeks back we've been inundated with requests for developer support from individuals and companies who want to develop, extend, integrate with or just hack on CloudStack. Lots of those requests were coming with requests to visit developers in person to get up to speed as rapidly as possible. After setting up several of those meetings in person, we realized it wasn't beginning to sate demand, so in reaction we decided to run a two day event decided to rapidly on-ramp folks, we put the agenda in the hands of two of CloudStack's lead developers/architects. What came out of that is the 'Developer On-ramp - two days of no-fluff CloudStack development and integration - lots of hands on work.. It's free to attend, you just need to register here.

Agenda:

Day 1: Everyone

9:30-12:00 - General CloudStack Architecture
12:00-13:00 - Lunch
13:00-14:00 - Storage deep dive
14:00-15:00 - Failure modes, H/A, maintenance planning
15:00-15:30 - Break 
15:30-17:00 - Networking deep dive
17:00-17:30 - Show and tell - see what others in the community are working on
17:30-18:30 - Beer

Day 2- Integrators

9:30-12:00 - CloudStack API deep dive.
12:00-13:00 - Lunch
13:00-14:00 - Development environment access 
14:00-17:00 - Building a sample app (bring your own ideas if you want)

Day 2 - Core developers

9:30-10:30 - Setup to contribute (git, IDE, patches, devel environment)
10:30-12:00 - Writing tests, simulator
12:00-13:00 - Lunch
13:00-14:00 - Hypervisor and bare metal management
14:00-15:00 - Scalability
15:00-15:30 - Break
15:30-17:00 - Freeform hands on hacking time - (bring your own ideas or squash bugs)

Recently, Citrix announced that it will donate CloudStack to the Apache Software Foundation (ASF).  This is a very exciting moment in the history of the CloudStack project.  The ASF is the premier open source foundation in the world and it has a track record of successful, completely open projects.  The CloudStack project has been increasingly open during its life.  But, “increasingly open” isn’t good enough.  We have to be 100% open in our technical discussions to run the project in the meritocratic fashion that successful Apache projects use.  In this blog post I’ll do a brief review of CloudStack openness and list out the changes we’ll be making over the next few months to transform CloudStack from a largely single-entity development effort to a collaborative, community project.

CloudStack 1.0 was proprietary software.  Closed specs, closed discussion, closed bug database, closed source.  The CloudStack 2.0 release in May, 2010 transitioned CloudStack to an open-core model.  About 95% of the source was available under GPLv3 and the bug database was open, but the technical discussions were closed and few technical documents were publicly available.  In August, 2011 Citrix published 100% of the CloudStack source code.  We started publishing some technical specs, but the discussions were still closed.

In late 2011, in conjunction with the Acton release, we started doing a better job publishing technical specs.   Not perfect – we weren’t iterating the designs in the open – but good progress.  We also (finally) came up with a process for accepting contributions.  In 2012 we’ve created a publicly available project dashboard , started publishing meeting minutes, and getting a larger number of non-Citrix contributions.

That’s good stuff, but there are some fundamental changes needed to make it more open.  Instead of publishing designs and discussion results after the fact, we have to publish the earliest version and then iterate in the open.  We also have to have the technical discussions on the cloudstack-dev mailing list (see below for the new one), and not on our internal engineering list.  This will take policing -- and I’m part of the problem here -- but through force of will a few folks can make a big difference here.

We also need to ask more of the community.  Our bug database is open, but, there’s no place to go with an “ask-list”.  Nowhere can you easily see important features with no owner, nor can you easily search to find the bugs with the most votes.  We need to pick a few features with a variety of technical needs and ask the community to help out.  That part is easy, BTW.  CloudStack has so many potential integrations in the datacenter that just listing those would give years of work to interested parties!

Today Citrix announced that CloudStack would become the cloud platform project in Apache Software Foundation. I’m excited not just because CloudStack will be an incredibly vibrant and successful project by itself, I also believe there is a tremendous amount of synergy between CloudStack and other cloud-related projects in Apache Software Foundation. I look forward to continuing to work with, for example, Apache Libcloud and Deltacloud projects.

I am the most excited, however, about the prospect of integrating with Apache Hadoop project. Known primarily as the technology for Big Data applications, Hadoop has gained wide-spread adoption in the industry. Similar to CloudStack which is inspired by Amazon’s EC2 service, Hadoop is modeled after Google’s MapReduce and Google File System technologies. And just like CloudStack, Hadoop is implemented in Java.

At the lowest level, Hadoop Distributed File System (HDFS) is a distributed and scalable file system. HDFS is designed to run on a large number of hosts and achieves reliability by automatically replicating data across multiple hosts. Hadoop project also includes a MapReduce engine and HBase distributed database (modeled after Google’s BigTable.) MapReduce and HBase run on top of HDFS. Highly reliable and highly efficient, Hadoop technology is being used by some of the largest cloud companies including eBay, Yahoo! and Facebook.

Today, CloudStack users already run Hadoop on CloudStack. They implement a service very similar to Amazon’s Elastic MapReduce (EMR). For cloud service providers, Hadoop represents a significant amount of workload that can be readily moved to the cloud. Enterprise deployments can achieve tremendous savings by leveraging the same CloudStack infrastructure to host Big Data workload. Users also leverage CloudStack’s bare metal provisioning capabilities to build high performance Hadoop clusters.

Working closely with Hadoop development community, we have started to explore other ways to integrate CloudStack and Hadoop. Because of its scalability, reliability, performance, and maturity, HDFS is a great object store solution for IaaS cloud. We have started the development of an S3 API front-end for HDFS. Once that work is complete, the combination of CloudStack and Hadoop will provide features equivalent to Amazon EC2 and S3 services.