Dependent builds in Travis

A script that uses the Travis API to trigger builds on downstream projects when a build succeeds upstream.

Background

At RightScale, we use Travis CI heavily and we love it, especially compared to our previous Jenkins CI environment. It does lose out to Jenkins in one area, though: dependent builds. (Travis has this feature on their roadmap, so hopefully that’ll be wrapped up soon and this post will be obsolete.)

We have a repo which contains Selenium and integration tests for one of our products. This repo then has four submodules, one for each repo it directly depends on. (Throughout the rest of this post, the submodules of the tests repo will be called ‘upstream repos’.) When another of the upstream repos has a change which is merged to master, we want to run a build on the tests repo (called the ‘downstream repo’ below) with the new code.

What the Travis API permits

The Travis API doesn’t currently support creating a new build, but we can be a little creative. The last build of the master branch should be the last known good state of the downstream repo, and the Travis API does allow restarting builds, so that will work: when an upstream repo’s master build succeeds, restart the last master build of the downstream repo.

The only remaining issue is that, unless we want to update all of the submodules every time, the build of the downstream repo won’t use the new code; instead, it will just use the state of the submodules in the downstream repo’s last master commit. So we need to pass an argument to the new build. Again, the Travis API doesn’t support this directly, but it does have something almost as good: environment variables!

Tying it all together

We’ll need three pieces of information in the environment variables to set this up:

  1. Is this a dependent build? (This can be inferred from the presence of the other two; we’ll pass it separately to be explicit.)
  2. Which upstream repo triggered this build.
  3. The commit SHA which triggered this build.

The trigger-dependent-build script below does this, and is explained more details in its comments. (Warning: it uses sed and grep to manipulate the Travis API’s JSON responses. This is because jq, or a similar tool, isn’t installed by default on Travis images.)

Downstream

Now that we have upstream base repos set up to do the triggering, we just need to handle those new environment variables in our downstream repo. There are a number of ways that we could do this, and it really depends on the use case. If we had a downstream repo which had tests just using the latest upstream repo, we could have skipped the environment variables completely. In our case, as we have submodules, we could add a before_script to the downstream repo that does something like this:

if [ -n "$DEPENDENT_BUILD" ]; then
  cd `echo $TRIGGER_REPO | sed 's|[^/]*/||'` # Strip the user / org from the repo slug
  git fetch
  git checkout $TRIGGER_COMMIT
fi

We also use the $TRIGGER_COMMIT and $TRIGGER_REPO variables for Slack notifications if this build fails, because Travis will only notify the original committer when a build is restarted. If they are out of the office, then we don’t want to miss a failure.

What could go wrong?

There are some limitations to this: if an upstream change is known to require changes to the downstream repo, then we may receive spurious failure notifications. (Or we can just disable the notifications until that’s done.) Similarly, if one story is spread across multiple upstream repos, we might get a failure - updating all of the submodules might solve this, but introduce other issues.

However, we find the benefits more than make up for these small inconveniences, as since we’ve been doing this, we’ve found several bugs which would have blocked a release, without needing to remember to run anything manually.

trigger-dependent-build

#!/bin/bash -x

# This script lives in each of the upstream repos. Add this to .travis.yml to
# run after each successful build (assuming that the script is in the root of
# the repo):
#   after_success:
#     - ./trigger-dependent-build
#
# There are three variables to set - `$auth_token`, `$endpoint`, and
# `$repo_id` - each is described below.
#

# An authorization token generated by running:
#   gem install travis
#   travis login
#   travis token
#
auth_token=abcdefghijklmnopqrstuv

# The Travis API endpoint. .com and .org are the commercial and free versions,
# respectively; enterprise users will have their own hostname.
#
endpoint=https://api.travis-ci.com

# The ID of the tests repo. To find the ID of a repo from its slug `$slug`, run:
#   curl -H 'Authorization: token $auth_token' https://api.travis-ci.com/repos/$slug
#
repo_id=9999999999

# Only run for master builds. Pull request builds have the branch set to master,
# so ignore those too.
#
if [ "${TRAVIS_BRANCH}" != "master" ] || [ "${TRAVIS_PULL_REQUEST}" != "false" ]; then
  exit 0
fi

# Make an API request using the auth token set above. First argument is the path
# of the API method, all later arguments are passed to curl directly.
#
function travis-api {
  curl -s $endpoint$1 \
       -H "Authorization: token $auth_token" \
       -H 'Content-Type: application/json' \
       "${@:2}"
}

# Create a new environment variable for the repo and return its ID. First
# argument is the environment variable name, and the second is the value.
#
function env-var {
  travis-api /settings/env_vars?repository_id=$repo_id \
             -d "{\"env_var\":{\"name\":\"$1\",\"value\":\"$2\",\"public\":true}}" |
    sed 's/{"env_var":{"id":"\([^"]*\)",.*/\1/'
}

# Get the build ID of the last master build.
#
last_master_build_id=`travis-api /repos/$repo_id/branches/master |
                      sed 's/{"branch":{"id":\([0-9]*\),.*/\1/'`

# Set the three environment variables needed, and capture their IDs so that they
# can be removed later.
#
env_var_ids=(`env-var DEPENDENT_BUILD true`
             `env-var TRIGGER_COMMIT $TRAVIS_COMMIT`
             `env-var TRIGGER_REPO $TRAVIS_REPO_SLUG`)

# Restart the last master build.
#
travis-api /builds/$last_master_build_id/restart -X POST

# Wait for the build to start using the new environment variables.
#
until travis-api /builds/$last_master_build_id | grep '"state":"started"'; do
  sleep 5
done

# Remove all of the environment variables set above. This does mean that if this
# script is terminated for whatever reason, these will need to be cleaned up
# manually. We can do this either through the API, or by going to Settings ->
# Environment Variables in the Travis web interface.
#
for env_var_id in "${env_var_ids[@]}"; do
  travis-api /settings/env_vars/$env_var_id?repository_id=$repo_id -X DELETE
done