Semantic Versioning CI

Oct 21, 2017 13:22 · 991 words · 5 minutes read Python CI

Semantic versioning is a specification for libraries to follow that lets their downstream clients know the extent of changes. This makes it much easier to know whether or not it’s safe to upgrade to a new version of a package because the version number indicates the safety, instead of the changelog. Lots of python projects do this already, and getting started with semantic versioning on any project is pretty straightforward. The pain point is that usually it has to be done manually, but using PBR and some shell scripting, you can patch together a solution that works on basically any CI server.

PBR

pbr is a python package that lets you sidestep a lot of the difficulty with setting up a standard setup.py. The configuration is straightforward and a lot of the more interesting parts of setup.py get moved out into a configuration file. pbr also adds some new features, one of which is for semantic versioning.

When a commit occurs in a project that’s using pbr, the commit automatically gets scanned for a Sem-Ver tag. The tag can be anywhere in the commit body, so commit message like:

Fixes issue #1227

* Fixes issue #1227 by replacing foo.bar with a new implementation.
* Adds unit tests to validate foo.bar and improve code coverage to 100%
  over the old implementation which acheived 85% coverage.

Sem-Ver: bugfix

Would be tagged as being a bugfix commit. When a build happens, the version for the artifact will be based off of these semantic version tags as well as how many commits happened after a release (see the official docs).

As in the documentation, the known symbols currently are

* `bugfix/depreciation` - Patch level increment
* `feature` - Minor increment
* `api-break` - Major increment

In addition to the Sem-Ver tag, PBR will automatically generate an artifact’s version based on the git tag of the commit the artifact is built from. If the commit doesn’t have a tag, a -dev postfix will be appended to the version number.

Lets take a look at an example.

Walking through developing a feature with PBR

Suppose the last release of my project was version 3.2.4. I’m using PBR to do semantic versioning, but I don’t have this process automated, so what steps would I need to take to get my project to 3.3.0?

Without getting into a discussion about branching strategies, suppose I want to develop this feature on a new branch. I check out the branch for my new feature and start working. After I’m ready to do my first commit, I can write my commit message:

Adding the basic framework for my new functionality

Sem-Ver: feature

Suppose I built the artifact right now, I would get version 3.3.0-dev1, because I declared that this new commit is a new feature level change and there has been one commit since my last release. I’m not done yet with this feature, so I add 4 more commits (testing, documentation, etc). For these commits, I don’t add the Sem-Ver tag. That’s because these commits are all going to be a part of the same branch, and the first commit already took care of bumping up the version number. After building my artifact with all the commits added, the version number will be 3.3.0-dev5 (5 commits since the last release).

I’m ready to release now, so I merge everything into master and tag the branch with 3.3.0 then generate my artifact. PBR will see the tag and automatically generate the 3.3.0 version of my artifact. From there I can push it to pypi or an internal repository.

So from this example we can see the steps to automate the process are:

  • On new branches, the first commit has to have the appropriate Sem-Ver tag
  • When a branch is merged into master, the branch has to be tagged automatically so artifacts generated from that branch will receive the correct version.

Automating your commit messages

Git has a hook for setting up commit message templates, so this is pretty straightforward. Determining the correct tag for your branch is a bit harder and depends on how you like to do branching. I tend to declare branches like:

feature/FEATURE-1287-some-description-here
bugfix/ISSUE-1399-some-other-description

So for my strategy (and in the script below), I need to make sure I name my branches with the right semantic version symbols and then just trim everything after the first / away.

#!/bin/bash
# Contents of .git/hooks/commit-msg

semantic=`git rev-parse --abbrev-ref HEAD | cut -d / -f 1`
commits_since_master=`git rev-list --count master..`

if [ "$commits_since_master" = "0" ]; then
    echo "Sem-Ver: $semantic" >> $1
fi

The script grabs the current branch name and looks at how many commits away from master the head of this branch is. If it’s 0 away (this is the first commit on this branch) then we add the Sem-Ver tag. You can find more information on the commit-msg hook (as well as all the other git hooks) from Pro Git.

NOTE: This only works if you always branch off master, if you’re branching off many different branches, you’ll need something a little more creative than this simple script. The strategy can still work, it’s just more complicated.

Automating the build

Supposing you have CI/CD set up, you’ll want to automatically generate an artifact when you merge to master. Without any more work, that artifact would have a version like 3.3.0-dev5 so we just need to chop off the -dev markers and build the artifact with the correct version. In the commands for your build, you can add in:

git tag $(python -c "import pbr.version; print(str(pbr.version.VersionInfo('<your_module_here>')))")

Which will tag your branch with the correct version without dev markers. Replace <your_module_here> with the name of your module. Also this command should only be run on master branch builds, so you’ll have to configure your CI server to not perform this step on other branches.

From there, you can have the CI/CD server automatically publish your module and/or push the generated tag.