DevOps engineers need a way to carefully apply their changes to production. They also need to review those changes with other stakeholders, mentor fellow colleagues, and of course remember what they did six months ago.
Tools & Process
DevOps people, sysadmins and build engineers can use Agile project management practices like Kanban through the GreenHopper Rapid Board with Jira, leverage development source control, change review tools like Crucible, and document change through Confluence. All can be integrated together to give us the information and communication we need to get our changes promoted safely from development and test to production with Puppet.
What is Puppet?
Puppet is a great open source systems management tool. It’s impressive with its ability to automate the remote management of many systems to ensure reproducibility. Puppet also ensures that each of those systems do not divert from a source of truth. We can be confident that all our systems have the same setup and if other people try to modify them, Puppet will set them straight.
At Atlassian, we use Puppet extensively with our internal systems, our Hosted products, and our build engineering infrastructure. Here’s how we do it in build engineering.
First, a stakeholder requests a change in a Jira issue to my production system, in this case Bamboo and its agents.
Our build engineering team leverages Agile Kanban to organize stand-ups and triage issues through the day. We can present a simple and efficient set of issues in a Greenhopper Rapid Board so that other groups can easily see what we are working on and as a team, see what each engineer is doing during the day.
I can drill down to only see my issues using GreenHopper Quick Filters. Quick Filters on the Rapid Board are the buttons at the top, each powered by a Jira Query Language (JQL) statement, to pin point a subset of issues – in this case, my own tasks.
I’ve now started on the production change request to add a capability to my Bamboo remote build agents.
On the Puppet side in source control here is one design on how to structure Puppet environments.
We can have more environments. Here, we have an additional Bamboo server dedicated to the main and staging branches.
To configure multiple Puppet environment branches on your puppetmaster you need to configure them in /etc/puppet/puppet.conf
In Bamboo, I set up one agent that points to my test Puppet staging tree jbac_staging. I have this agent always disabled so that no production builds can jump on and try to build. Some of my changes I want to test may or may not bring down services on the agent. No sense in risking having builds run on a changing system so the disable ability in Bamboo makes this easy to turn agents on and off for builds.
On the Bamboo remote agent, its Puppet configuration is defined again in /etc/puppet/puppet.conf and here is showing that the staging agent is pointing to the staging environment on the puppetmaster.
We can now make a change on our staging branch, restart the puppet client on the bamboo staging remote agent, and test our system change.
If the change operated normally, I can be confident that puppet will apply this change in the same manner to production. Now I am set to have my change reviewed by other members of my team and/or people from other teams. I first commit my change to SCM on the staging branch. Included in the commit message is my Jira issue number so that we can fully integrate this changeset with my entire Atlassian dev tools suite.
svn commit -m “BUILDENG-230 adding capability foobar”
Ok great! I have a review and Pete and Rene are going to look at this change. Crucible with Fisheye is great because can add comments to any part of the change and I, as the author of the change, can see the status of the review.
Pete and Rene approve of this change and have accepted the change in the Crucible review. We are now ready to push this to the production Puppet branch, of course, making sure proper scheduling and downtime of my production instance.
Again, I merge my change from the staging branch to the production branch AFTER disabling all of my Bamboo production remote agents. This will ensure that no builds jump on agents that have changes being applied.
build-engineering/puppet/jbac_main$ svn merge -c 160372 –ignore-ancestry ../jbac_staging .
— Merging r160372 into ‘.’:
U modules/bamboo/files/bin/bamboo-capabilities.properties.jbac
We have Puppet on a 30 minute cron, so to speed up the deployment of this change I can again restart the remote agent client side Puppet daemons. Once complete, I can then easily enable all of my production Bamboo remote agents on my Jira build CI server.
As a final step in the process, or a good step during the staging testing, I should also document how to reproduce the change. It’s easy in Confluence. Our team has a set of HOWTO documentation for performing the change:
Automation
As a build engineer for our Jira product, I’d like to be able to push out changes with an automated tool like Puppet efficiently to over 50 remote agents connected to Bamboo. I don’t want to push out changes by hand because that is error prone over so many machines. Because Bamboo and its agents are critical for releasing Jira, I also need to review these production changes with others. This is where Jira, Greenhopper and Crucible help me out greatly with my production changes with Puppet.
It doesn’t stop here. We can use this same design for other tools like Cucumber and Chef. If you can put changes in source control, we can manage that change properly with the help of the Atlassian tool suite.
More on Change Management
In my next blog I’m going to follow on with this story with change management and Change Review Boards leveraging the GreenHopper Rapid Board.
References:
Puppet: http://puppetlabs.com/