Database Continuous Integration with Bamboo

We were so interested in Atlassian's Bamboo, and it's role in Continuous Integration, that we wanted to find out more, such as 'why the name?'. So we sent our roving reporter, Richard Morris, out in his proverbial raincoat and trilby to find out more. Who better to ask than their marketing manager, Sarah Goff Dupont

1533-img2E.jpg

Each year, the software industry offers more and more quality control and more and more capabilities but does it with less and less time by replacing the traditional practice of applying quality control after completing all development

The magic that makes this happen is continuous integration, and it is the real driver in the software industry.

In this interview, Sarah Goff Dupont, Product Marketing Manager at Atlassian, goes positively wild about the recent iterations of Bamboo, the company’s continuous integration and delivery server.

As users who have a firm grasp of it will know, Bamboo’s latest releases enable software developers to automatically test code on development branches before merging with the main code line and automated merging for Git and Mercurial.

However, Sarah says that there is a huge amount of ideas that the development team are working on for future Bamboo releases. She also poses and answers the obvious question: why should you pay for Bamboo when Jenkins can be used for free?


RM:
Sarah, an obvious question, why is Bamboo called Bamboo?
SDG:
Atlassian purchased Bamboo from a UK company called Pols Consulting back in 2006. It was called Beetlejuice at the time, and we renamed it Bamboo. I’m not sure why that name won out, but I like it better than “Master Builder” or “Forman”, which are two other names we considered.
RM:
The team at Atlassian has been hard at work re-thinking and re-building the way developers use continuous integration especially when a team uses branches. A lot of your competitors were saying feature branching and continuous integration were incompatible not least because keeping the configuration between the main build and the branches synchronized is a lot of hard work. As you know it’s unavoidable as rain is to the British summer.
SDG:
Indeed. Working with branches is the third thing you can’t avoid in life (along with death & taxes). And recently, with the increasing adoption of distributed version control systems like Git and Mercurial, loads of short-lived feature branches is becoming the norm for many software teams.
RM:
So what made you continue with your insistence that this problem with feature branching and CI could be fixed and how does Bamboo 4.0 solve this particular problem?
SDG:
We felt that it could be fixed because it must be fixed. A team of developers, each working on their own branch without the benefit of pulling continuous integration onto those branches, is in for a world of hurt when changes are merged into the main code line. If you haven’t been running integration tests against the changes on your branch, and neither have your team-mates, untangling the bugs that are inevitably introduced can be exceedingly painful. We saw this on our own dev teams. The other thing we noticed is that if you have to painstakingly re-create a build and test pipeline so that it can run on a short-lived feature branch, people just don’t bother.

So in Bamboo 4 we introduced the idea of Plan branches. (For those not already familiar with Bamboo, “Plans” are pipelines that chain together one or more build steps.) Once Bamboo is aware of a new branch in your repo, it finds all the Plans associated with that repo, clones them, and points the clone at your new branch. Branches can be discovered automatically in Git and Mercurial repos, and our next release includes automatic branch discovery for SVN, too. P4 and CVS users simply plug the URL for the new branch into their Plan configs, and Bamboo takes it from there. Either way, Plan branches inherit their configs from the original Plan.

So if you need to update a hostname or add a step to your build, you only have to make that change in one place and it propagates out. It’s a huge time saver for build admins, and practicing CI on all active branches helps developers and QA engineers get features out the door faster as well.

RM:
Can you tell me how you went about developing Bamboo’s latest iteration. What was the reason for the upgrade? What problems were you trying to solve?
SDG:
First, I should mention that Bamboo 4 is not our latest iteration. We released Bamboo 4.1 on May 29th, and we’re already running release candidates of Bamboo 4.2 internally. Yes: we use pre-release versions of Bamboo to build and release Bamboo. I’m not sure if that’s “meta” or “recursive”, but I get a kick out of it. We also use Bamboo to build all our other products. Anyway… One of our goals with any release is to solve pain points that we are experiencing ourselves because we know other teams are experiencing them too. Maybe that’s selfish, but it seems to be a decent strategy so far! With Bamboo 4, the main thing was to making practicing CI on your branches so easy that you no longer have an excuse not to. We also introduced a one-click test quarantine that still runs and reports on your test so you can keep tabs on it, but prevents that test from failing your build.

With Bamboo 4.1, we wanted to remove the administrative overhead from fixing broken builds. So now you can create JIRA issues directly from a build result page. You can also triage responsibility for breakages to one or more team members, and let Bamboo take care of sending the nag-mails. I don’t want to reveal too much about what’s coming in Bamboo 4.2, but I will say that it includes more love for DVCS teams in addition to the SVN branch discovery feature I mentioned earlier.

Aside from solving our own problems, though, we also make a point of incorporating user feedback into our release plans. We have a public-facing JIRA instance where users come to file bug reports and enhancement requests, and we allow the entire user community to vote on them. The high-vote issues often get incorporated into our releases. And that’s true of all Atlassian products. So to all you users out there don’t be shy! Come on over and tell us what you think.

RM:
Each release has extended the functionality of Bamboo by adding valuable features. Are there a lot more ideas on the forum that haven’t been implemented yet because if you didn’t release until every feature that was requested, you would never have a release?
SDG:
Oh my gosh, there are tons of ideas we haven’t implemented yet. We try to be strategic about our releases and make sure they each have a theme. It’s more efficient to add four user stories around artifact sharing than to sprinkle them into four different releases, for example, because that lets us craft a holistic technical design. Plus, it means users keen on artifact sharing only have to upgrade one time to get that full feature set. Everybody wins. This does mean that sometimes users have to wait longer to get the features they’ve requested, which is unfortunate. But our users can identify with not wanting to do hodgepodge, piece-meal, laundry-list releases; so folks are pretty gracious about this.
RM:
When the team was working the long hours tweaking and developing the recent iterations of Bamboo did they have in mind who would use it over and above those who are already Bamboo fans?
SDG:
Bamboo 4 and 4.1 have been big hits with DVCS teams and teams that use JIRA, respectively. In those groups we find teams both from tiny start-ups and mega-enterprises, and teams in just about every geographical location. (Sorry, Antarctica…) We view DVCS as the wave of the future, and strive to build an awesome CI server for those early adopters without abandoning traditional VCS teams.
RM:
What features in the last few releases are you most proud of?
SDG:
Awww man, are you really going to make me choose? That’s just not fair!
RM:
I promise not to misspell your name if you choose one!
SDG:
To me, the coolest thing we’ve done in the last couple releases is automated merging for Git and Mercurial. That’s not a typo –I really did say “automated merging”. Let’s say you and I are collaborating on a feature, with each of us working on our own branch. I can set up my Plan to checkout not only the latest changes from my branch, but also the latest code from yours. Bamboo merges the two, runs the Plan against that merged code, and if it passes, updates my branch with it. So we don’t have to send each other patches anymore: we just get the merge for free. That works with any two branches in a repo, and you can push the merged code to either branch (or neither, if you prefer to update your branch manually).

An honorable mention goes to the JIRA issue creation feature, for two reasons. One, because flipping back and forth between applications all the time is a grossly inefficient. And two, because there are so many cool enhancements we can make to this feature. Oh, the possibilities!

RM:
The hard-bitten user will say of course that all CI servers are all pretty much similar so what makes Bamboo stand out from its competitors, especially compared with AnthillPro, Jenkins or Microsoft’s TeamBuild? Why is there no pre-tested commit feature?
SDG:
The deep integration with other Atlassian tools, especially JIRA, and the advanced support for DVCS really set Bamboo apart. Add to that our concept of Stages, which is a way to encapsulate jobs that run in parallel and give you incredible control over the flow of continuous delivery pipelines.

Pre-commit builds haven’t been added because when we’ve asked our users about it their response has been lukewarm. While it’s a cool and useful idea, most developers admitted that they probably wouldn’t use it much since Bamboo gives them such quick feedback on their committed changes anyway.

RM:
What are the major differences and plus-points of using Bamboo over and above say Jenkins for example?
SDG:
This question really boils down to “Why should I pay for Bamboo when I can use Jenkins for free?” It’s a legitimate question, and one I always enjoy answering. To start, the advanced support for practicing CI on your branches is worth paying for, particularly for DVCS teams. In addition to the Plan branches feature I talked about, Bamboo will do automated merges between any two branches in a Git or Mercurial repository. If all your build steps succeed against the merged code, Bamboo can push it to either of the branches.

This is great for longer-lived branches because you can bring over code from the main line and prevent the branch from drifting too far. And for two developers collaborating on a feature, this means they don’t have to send each other patches or even bother with pull requests. A Jenkins plugin for creating branch jobs was just released, but it has to be carefully managed or it can delete existing jobs; plus, it doesn’t do any kind of automated merging. Imitation is the sincerest form of flattery, so we take this as a signal that our Plan branches and merging features strike a chord with developers!

Teams moving toward any kind of continuous delivery scheme also find Bamboo to be a worthwhile investment. Artifact sharing between steps is dead simple –no need to mess around with fingerprinting or tweaking file names.

Same with moving back and forth between executing steps sequentially and parallelizing them: Stages make that a breeze. Jobs within a Stage will run in parallel (great for breaking up test suites into batches), but Bamboo won’t start on Jobs in the next Stage until and unless all Jobs in the current Stage complete successfully. No more worrying that a downstream deploy Job will promote a build to the next environment while tests on the current environment are still running (and potentially discovering new defects!). And test results are aggregated automatically with Bamboo, so you start with a single pass/fail number that represent all tests run throughout the Plan, then can drill into the results from individual Jobs from there. If you’ve ever struggled with aggregating test results in Jenkins, you’ll fall in love with Bamboo based on this alone.

Third, the legendary customer support that comes with all Atlassian tools is worth paying for. We’ve got trained staff here to help you understand Bamboo’s features and debug errors. And our Q&A site, Atlassian Answers provides a searchable and interactive forum for puzzling through those non-standard use cases. It’s monitored by Atlassians, including members of the Bamboo development team, as well as other people in the user community.

RM:
If we had a scenario where a developer was using a DCVS like git, they may wait until they completed one task before pushing all their local commits together up to the central repository and their changes will not be integrated until the end of the day. How does Bamboo solve this problem?
SDG:
Plan branches and automated merges address this. Now the developer can work in isolation on a feature branch, and Bamboo will make sure all the same tests and analysis that are running against master are running on the branch as well. Furthermore, developers can have Bamboo bring in the latest code from master upon each commit to the branch, and then have their branch updated with the merged code. So whether commits are pushed to the branch once a day or several times a day, developers can feel confident that they won’t be stuck in merge hell or (surprise!) stuck fixing bugs all weekend when their branch is finally pulled into master.
RM:
If I’m using Git or Mercurial, what more can Bamboo do for me than it did previously?
SDG:
Plan branches with automated branch discovery, and the automated merging have been huge wins for DVCS teams. And like I said, there is more DVCS goodness coming with Bamboo 4.2, so stay tuned.
RM:
Staying on this subject for a while, is there space in the market for both Git and Mercurial to coexist or will one eventually wither and die?
SDG:
While I do think there’s room for both to coexist (I mean, why not?), Git certainly has more momentum behind it. Rarely do I talk to other software makers and hear that they’re using Mercurial, so the Vegas odds for Git surviving and/or winning out are pretty high.
RM:
Would you say that working on a team tends to result in a better product?
SDG:
I wouldn’t say that. I would shout that from the mountain tops! There’s a saying… “Beware of the code that is written by just one person.” Software built by teams is almost universally easier to maintain, better documented and more rigorously tested than software built by a lone coder. And regardless of what build server you choose, continuous integration is the secret sauce that makes working on a team, well, work.
RM:
Is the hosted subscription model more popular than the on-premise equivalent?
SDG:
Glad you asked. Presently, there are far more Bamboo customers to run their server behind their own firewall than Bamboo OnDemand subscribers. But I expect that to change very soon, for a couple reasons.

In July we are upgrading the Bamboo OnDemand offering to be on par with Bamboo 4.1, and given that we haven’t upgraded Bamboo OnDemand in nearly a year, this brings in a TON of features that our behind-the-firewall customers rely on every day. Going forward, the hosted offering will be kept current with what is offered in the behind-the-firewall product. It was a big engineering effort to make that happen, and we’re delighted to get there because it’s going to make Bamboo OnDemand much more attractive.

The other reason is simply that we’re seeing big gains in OnDemand subscriptions in general, especially among smaller teams and startups that are extra cautious about capital expenditures like hardware. Being able to use JIRA, GreenHopper, Confluence and Bamboo for as little as $10/month for each product -without having to invest in the infrastructure or worry about doing upgrades yourself- is killer for teams rooted in the lean & agile philosophies. And you can migrate your data from OnDemand to installations on your own network, so it’s quite flexible.

RM:
What difference has Red Gate tools, in particular SQL Source Control and SQL Compare, working with Bamboo, made to database CI?”
SDG:
It’s the difference between doing long division by hand, and someone handing you a graphing calculator. DBCI has always been possible, but you had to write some pretty complicated scripts to pull it all off. RedGate’s tools do most of that work for you, and are incredibly useful whether you incorporate them into your CI scheme or use them on more of an ad-hoc basis. Once you’re up and running with SQL Source Control and SQL Compare, you can add a build task to Bamboo that calls out to them with either a single command or invokes them with a simple shell script. The pay-off in terms of product quality, DBA sanity, reducing risk around deploys, and fewer middle-of-the-night pages to Ops is well worth the investment.