August isn’t officially “Agile Month”, but with so much of my attention focused on the Agile2012 conference held a couple weeks ago, that’s what it feels like. So I’ve been blowing the dust off my trusty ol’ Scrum Master hat and thinking more about team processes lately. When teams decide to go agile, they do so with visions of smoothly flowing burndown charts dancing in their heads. Not that the chart itself is important. It’s what that smooth burndown line represents: a team that is firing on all cylinders, everyone playing their positions… insert your favorite sportscaster metaphor here.
But achieving that flow is nigh on impossible if it takes hours (or days) to get new code to a testable environment and find out if it broke any existing functionality. In that situation, developers are constantly doubling-back after they’ve already moved on to the next issue on the backlog to adjust work they thought they were done with. Code issueA -> commit -> start issueB -> forget what you did with issueA -> work more on issueB -> find out that issueA needs more work -> interrupt issueB and refocus on issueA -> commit > re-focus on issueB while waiting another day for issueA to be deployed and tested -> issue overlap -> confusion -> inefficiency… Blech! That’s an agile anti-pattern for sure.
Over the next few weeks, I’ll be writing about ways to get faster feedback and break this anti-pattern. Today, I want to talk about artifact passing. It helps with fast feedback, and the benefits to agile teams in particular go way beyond that, as you’ll see.
Leave Your Shovels at Home
Most people associate “artifacts” with clipboards, carbon dating, and dust-covered grad students wearing many-pocketed vests. But in software, the word takes on a whole new meaning. To us, artifacts are files (or collections of files) produced by your some part of your build process. A .zip containing your project’s source code. A code coverage report. A .JAR, .WAR, or .EXE coming out of a compile step. All these things can be captured and passed downstream, rather than re-creating them as part of each additional step that needs to perform further operations on them. This article focuses primarily on the flavors of artifact generated by check-out and compile steps: source code trees, .JARs, .WARs, .EXEs, .TARs, etc.
The concept of artifact passing in this context applies to teams who are already automating their deployments and tests through a series of pre-production environments, or are on the path toward that. The automated pipeline itself is the biggest weapon you have in combatting the issue-overlap anti-pattern. (As well it should be… it’s a major undertaking!) Artifact passing is one of many fine-tuning techniques you can employ to take it to the next level.
Boost Velocity
It’s commonly accepted (and basically common sense) that creating an automated deploy & test pipeline within your pre-production environments tightens your feedback loop because new code is exercised faster and more thoroughly than if humans do it by hand. If your team is already rocking an auto-pipeline, then you are to be congratulated. And you can rock it even harder if you’re not checking out and building the same code multiplie times. Why? Because the act of checking out code can take many minutes depending on the size of the application and how fast your network is –I’ve talked to people who say this simple step takes over an hour. Same story with the compile step. How can you avoid this terrible fate? By building once at the top of the pipe, then passing the packaged code to downstream steps. Projects written in interpreted languages like Ruby and PHP can compress the source code in a .zip file (or similar) after check-out and pass it downstream.
When our build engineering team at Atlassian started passing artifacts instead of re-checking out and re-compiling for each environment in the pipeline, it shaved 40 minutes off Jira’s build/test/deploy duration. Wow! As you can imagine, this reduced the doubling-back factor even further and the Jira team was able to satisfy it’s definition of done faster for each issue. Also known as “increasing productivity”. Exhibit #643 in the case for investing time in automation. (Managers, directors, VPs and CTOs: you’re feelin’ me on this one, right?)
The mechanics of passing artifacts within a build Plan in Bamboo are pretty well described in the online docs, as well as this blog from a few months ago. If you want to pass an artifact between two Plans (say, a parent Plan and its child), you can do that too. Send it off to an external location like Nexus, Artifactory, Archiva or a file server on your network as one of the steps in the parent Plan. Then pull it down as the first step in the child Plan. Yes, that’s a tad clunky. Which is why the Bamboo team has a more elegant solution on the short-term roadmap. Stay tuned.
Reclaim your Peace of Mind
Beyond improving feedback times, artifact sharing also boosts confidence. The holy grail of agile development is to present your work to stakeholders on the last day of the iteration, get their thumbs-up, and buh-blam! immediately get those shiny new features & fixes out to your customers. When you pass an artifact from the build server to your test environment, then to your staging environment, then eventually to production, you know that the code going live is the exact same code that you worked so hard to vet earlier. No opportunities for “gotchas” like building from the wrong revision number or file corruption caused by a network hiccup during checkout. In other words, you can demo already-tested code on a staging environment then hit the “go” button to release that same build and head out for a celebratory team lunch.
Fight the “Agile Cowboy” Myth
Passing the same artifact through pre-production and out for release also means you have a paper trail (if you will) from source code repository all the way to your customers. Traceability is one of those “grown up” things that many nay-sayers fear will go out the window if their organization goes agile. Making that lineage part of your agile process helps skeptics warm up to the idea. (Integrating your build server with your Jira and code repo multiplies traceability by a factor of, like, a million. But that’s another blog post for another day…)
Focus on More Interesting Issues
Finally, artifact passing is a boon to the cross-functional nature of agile teams. Let’s say that your team has automated deploy jobs configured in the build server, and QA pulls new builds to test as needed (probably every few hours) rather than having new builds pushed up with each commit. Instead of combing through commit logs to find the revision they want to build and promote to the next environment, QA engineers can simply find the pre-built artifact on their CI server and press the “go” button to deploy it. The deploy automation allows QA to take that burden off of Ops, affording Ops more time to provision, monitor and generally innovate. The artifact passing allows QA to get builds up faster, affording them more time for risk assessment and exploratory testing. The risk assessment and exploratory testing means fewer production emergencies, affording developers more time to pump out new features. Win-win-win.
Artifact passing can be done with just about any continuous integration server out there. But I’d be remiss in my duties as a marketer if I didn’t mention that Bamboo makes it very easy to set up, and has more goodies for artifact passing on the roadmap. (Users of certain open-source CI tools: mention the word “fingerprinting” to your build engineer and see if they don’t go a little bit pale…) If you haven’t tried Bamboo yet, take it for a spin and start reaping all the benefits artifact passing has to offer!
Not using Bamboo? Learn about easy import from Jenkins