07 Jan 2015 Git Flow and Immutable Build Artifacts
We love Git Flow. It’s awesome to have a standard release process where everybody is using the same terminology and tooling. It’s also great to have out-of-the-box answers to the same questions that get asked at the start of every project. For example:
- “How are we going to develop features in isolation?”
- “How are we going to separate release candidates from ongoing development?”
- “How are we going to deal with hotfixes?”
Now it’s enough to just say ‘We use Git Flow’, and everybody’s on the same page.
Well, mostly. Whilst Git Flow is terrific for managing features and separating release candidates from ongoing development, things get a little hazier when it comes time to actually release to production. This is because of a mismatch between the way that Git Flow works and another practice that is common in large development projects: immutable build artifacts.
On most enterprise projects I work on these days, it’s considered a pretty good idea to progress exactly the same build artifact through testing, pre-production and into production. It doesn’t matter whether it’s a JAR file, WAR file,
tar.gz file, or something more exotic – the key point is that you build it once, then deploy the same thing to each downstream environment. If you find a bug during this journey, you fix the bug, build a whole new artifact, and start the process again.
In the absence of any established terminology, let’s just call these things immutable build artifacts. (Note that you’ll still need a separate, external file for those things that have to be different between environments, but by keeping the amount of stuff in that file to an absolute minimum, you’ll minimise your potential exposure to variances between environments.)
Without immutable build artifacts, many development managers will start to sweat nervously. This is usually because at some stage in their careers they’ve been up at 2am in the morning prising apart build artifacts and trying to understand what changed between the pre-prod build (which worked just fine) and the current prod build (which is inexplicably failing).
A bunch of things can cause this sort of problem. For example, it could be that somebody managed to sneak some change in between the two builds, or that some downstream build dependency changed between the two builds, or even that somebody tweaked the build process between the two builds.
‘That’ll never happen to me’, I hear you say, ‘if everybody follows our development methodology correctly’. And therein lies the problem: it’s difficult to absolutely guarantee that a developer won’t at the last minute do something stupid, like slip something into the wrong branch, upload an incorrectly-versioned downstream dependency, or mess with your buildbox configuration.
My point is this: why open yourself up to the risk at all, when having a single artifact will eliminate a whole class of potential defects?
Now for the actual problem
A while back, I was working on a project using both Git Flow and immutable builds.
A junior developer came to me looking confused. He was trying to understand when it would be OK for him to finish the current Git Flow release. The testers had just signed off on the current build. But if he finished the release, Git Flow was going to merge the release branch into master – and strictly speaking, he should create a new build from that. But if he created a new build, the testers were going to want to test it.
If they found a problem, the fix was going to have to happen in a new release branch. But then when he closed that release, he was going to have to do a new build from master, which the testers would want to test again, right? And then, if they found a bug in that…
I admired his highly conscientious approach to his work, but worried that his mind was going to disintegrate as it spun around this build-management paradox.
I also had to acknowledge that he had stumbled head first into something that I had been wilfully ignoring in the hope that it would resolve itself; namely, the fundamental mismatch between Git Flow and the concept of immutable builds.
Put simply, Git Flow works on the premise that production builds will only be done from the master branch. In contrast, immutable builds work on the premise that production builds will be done from either a release branch or a hotfix branch. There is no perfect way to work around this mismatch.
The pragmatic solution
Put bluntly, if you want truly immutable builds, then you should do them from the release or hotfix branch rather than master. However, because this runs contrary to how Git Flow works, there are a couple of important consequences to keep in mind.
1. Concurrent hotfixes and releases require special handling
If you start and then finish a hotfix whilst a release branch is in process, then that hotfix’s changes won’t automatically be brought into the release branch by Git Flow. This means that a subsequent build from the release branch which goes into production won’t include the hotfix changes. Here’s a diagram illustrating the problem:
Note that the inverse problem would apply if you tried to start a release whilst a hotfix was in progress. Thankfully, Git Flow will not let you have two release branches in progress at the same time, so we don’t have to worry about that particular possibility.
(If you’re wondering how Git Flow avoids these scenarios in its regular usage, remember that it always merges back into master when a release or hotfix is finished, and that you’re supposed to always build from master. Consequently, if you’re building from master post-release or post-hotfix, you’ll always be getting the latest changes into the build.)
The problem of concurrent hotfixes and releases can be solved by merging the hotfix branch into the release branch just before the hotfix branch gets finished (or vice-versa if you started a release whilst a hotfix was in progress). Here’s what it looks like:
However, you will need to remember to do this merge manually because Git Flow won’t do it for you.
2. Version tags will be incorrect by default
When you finish a release or hotfix, Git Flow merges the corresponding branch back into master, and then tags the resultant merge commit on master with the version number. However, because your immutable artifact will have been built from the release/hotfix branch, the commit SHA for the version tag in Git will be different from the SHA that the artifact was actually built from. Continuing on from our previous example:
You may or may not care about this, for a number of reasons.
Firstly, from the perspective of Git Flow, the merge commit on master should never have any changes in it. The only scenario that might lead to it having changes is if you have run concurrent hotfix and release branches (as described in the previous section) and forgotten to merge them prior to finishing. And you’ll never forget to do that now will you? 🙂
Secondly, in the likely event that you are using some sort of continuous integration server to produce your builds, that server can probably associate its own number with each build, and stamp the resultant artifact with that number. The CI server will probably also have recorded the SHA that each build was done from. So from the artifact you could probably work backwards to the SHA that was used to produce it.
If you’re nevertheless wary of the confusion that this backtracking process might introduce when trying to debug a production issue at 2am in the morning, you might still insist on having correct version tags.
In that case the best thing I can think of is to manually create the tag on the release/hotfix branch yourself before finishing it and then, when finishing the release, run
git flow [release/hotfix] finish with the
-n option to stop it from trying to create the tag again (which would fail because the tag now already exists).
I’m on a roll with my diagrams, so what the heck, let’s do one more:
On balance, I think the benefits of having immutable builds outweigh the costs of deviating from the Git Flow way of doing things. However, you do have to be aware of a couple of problematic scenarios.
We’ll be experimenting in coming months with using Git Flow with our own immutable builds, and will let you know if anything else weird pops up. Who knows, perhaps we’ll even end up with our own version of Git Flow. Either way, I’d be interested in hearing from anybody else who has had the same problem.
David N. JohnsonPosted at 10:16h, 09 January
Does this make master redundant?
Ben TeesePosted at 13:12h, 09 January
At first I thought it might, but then I realized that Git Flow still needs to create hotfix branches from master.
However, this then raises a question mark over the validity of creating a hotfix branch from a commit that is not the same as that which the release build was created from. I don’t *think* this will be a problem as long as you ensure that your merge commits in master never actually contain any changes (which I think will be the case if you manage concurrent hotfixes and releases in the manner I described in the post). It feels a bit dicey though and probably points to another incompatibility between Git Flow and the concept of immutable builds.
Wayne NgPosted at 09:16h, 06 April
Not redundant at all, part of the gitflow process is to merge your release branch back to master. the master branch is an excellent way of keeping record of your release timelines.
Scott GoldiePosted at 11:26h, 12 January
Couldn’t you get your CI server to close the release on it’s local then roll back if the build doesn’t go into prod, or else push to remote if it does go into prod? That way your artefact would be built from master each time. You could even blow away the CI repo each time to make sure.
Ben TeesePosted at 13:46h, 14 January
That’s an interesting idea.
I’m not sure that Git really has the notion of a ‘rollback’ though. For example, once you’ve merged a release branch into develop, I can’t think of a way that you could revert that in the repository, as it has become part of the history of develop. This means that when you eventually do decide you want to push your repository, that history is going to get pushed too.
You could possibly retain old refs to ‘pre-merged’ branches, but there’d be a bit of work in storing them and then re-wiring them back into place as part of a rollback.
So I guess that leaves the other option of blowing away the whole CI repo. I guess in theory I can’t see any objections to that, although it might get kind of slow cloning a large repo repeatedly.
ClaytonPosted at 07:21h, 13 January
The original git-flow spec handles Problem 1 (hotfixes being merged into releases branches):
“The one exception to the rule here is that, when a release branch currently exists, the hotfix changes need to be merged into that release branch, instead of develop. Back-merging the bugfix into the release branch will eventually result in the bugfix being merged into develop too, when the release branch is finished. (If work in develop immediately requires this bugfix and cannot wait for the release branch to be finished, you may safely merge the bugfix into develop now already as well.)”
I’m interested in knowing what implementation of Git Flow you’re using? It should be merging hotfixes into the release branch.
Ben TeesePosted at 14:12h, 14 January
I haven’t seen it automatically merge a hotfix back into a release. I’m using the latest version of 0.4.2-pre.
To be clear, here’s the sequence of steps I’m running:
git flow release start release1
git flow hotfix start hotfix1
git add –all .
git commit -m ‘Hotfix1’
git flow hotfix finish hotfix1
git checkout release/release1
At that point, ‘hotfix.txt’ isn’t in the release/release1 branch. Are you using a version of Git Flow that works as specified? If so, what version?
clertemPosted at 22:00h, 02 March
Nice effort but to me you are trying to find a solution to a problem that you have yourself created.
First, I would not have relayed to your “junior developer” the task to finish a release nor a hot fix. This task should be handled by the technical team lead.
Second, Gitflow do not state anything about deployment. Wether it has to be done from master or a branch, it is up to you to define your ideal process. Obviously, deploying from master seems the natural way, although it is not enforced by the gitflow process.
Third, there is a major pitfall in your initial scenario. As soon as the testers sign-off the release branch, they have officially given their go on the code and it can therefore be merged into master WITHOUT the need to re-test after that. All is OK to go, no problemo.
Now, what next ?
The build server listens on master, creates an artifact ( ‘immutable’ ) that can make its way to UAT and later, Production. If once deployed to UAT some issues are found, pack them to a new ‘minor’ release, or a hotfix, and do another deployment from master. Do that again and again until you have the sign-off from the testers. A sign-off is an approval to deploy that code to production.
You have to be agile here…
Keep in mind that often testers have a tendency to raise “new issues” which are not part of the current sprint during the UAT phase. That is when they should be gently pushed back (or forward, to the next sprint). Testers are testers, not rulers. Well, if they are rulers, start a new release !
But what’s the problem with Master ?
I think the real problems comes down to what the Master branch means to you. What is it supposed to contain… To me it is a dynamic branch which life cycle is somewhat as follows :
1) Before sprint (finite list of issues) starts, master contains the code currently running in production (the current version)
2) During sprint, master can change many times, basically each time a release or hot fix is finished, until the testers are OK with the code as per the requirements defined before the sprint started
3) Once testers have approved the code, it marks the end of the sprint. The code in master has already been tested and thus can go to production. The next sprint start.
So where is the “problem” with the immutable artifact and git-flow ?
Sorry as much as I am trying to understand your point, I still can’t find one.
The ‘solution’ presented in your article over-complicates the reality while trying to find a solution to another problem, more to do with the way you handle the things.
kutziPosted at 03:02h, 02 December
I think your answer completely misses the point about what immutable artifacts are good for: to have one single artifact which goes through all stages of testing (and deployment) and not one which is only used in the UAT phase.
Also interesting that you talk about being agile and then describing a process which is not at all agile IMO – i.e. dev teams ‘throwing something over the wall’ to UAT teams which then have to approve it.
In a truly agile team the whole testing process would be in one team.
tbarbzPosted at 04:44h, 07 March
I am now facing a similar issue to this and have so far decided that it’s likely that I too will have to build from the release/hotfix branches themselves too. However, one thing i’m unsure of is how to handle code reviews for changes made directly to these branches. New feature branches are reviewed during a pull request before being merged to develop, but if making changes directly to a release or hotfix branch this wouldn’t be an option (at least not until merging to develop/master) which could be too late. Do you have any suggestions?
Ben TeesePosted at 09:12h, 07 March
For me, changes on a hotfix or release branch have only ever been bug fixes that weren’t deemed worthy of a full code review. However, I acknowledge that this mightn’t be the case for everybody.
The first option that sprang to mind is to use a PR to merge betweeen the hotfix/release branch and master, and conduct the review there. However, you can obviously only do this once per branch, whereas the changes on a hotfix/release branch may cover many bugs and fixes. It also tangles up the PR merge process with Git Flow’s process for merging hotfix and release branches into master, which strikes me as a possibly being problematic. Finally, it makes it more difficult to do a build from the hotfix/release branch.
I guess another option (although I’ve never tried this for real) could be to do bug fixes in sub-branches of hotfix/release, then use PRs to merge them back into hotfix/release. When the hotfix/release branch is finished, you can then use Git Flow as per normal to merge back into master when the hotfix/release branch is finished. The added benefit of that is that the PR merge mechanism is decoupled from the GitFlow process. You get to decide when the hotfix/release branch gets shut down, and thus have more control over when you can do the build from it.
Sorry I couldn’t be more help.
Rizwan FirdousPosted at 03:44h, 09 October
first of all, thanks for a nice article and for sharing your knowledge.
yesterday I searched “git workflow and immutable jars” and landed in your article, and after reading the article I feel that I am your junior developer.
I also get to know there is one more strategy known as “Trunk Based Development”.Can u write about that and can this strategy resolve the immutable artifacts issue.