The Swiss Army Knife for the Agency Project, Part 2: Release Management with GitHub

This article will examine how we handle releasing and artifact delivery with GitHub tools in our open-source project. When designing our project’s release process, we have kept it lightweight but efficient enough. One important goal has been ensuring that developers get a stable version, i.e., a working version of the code, whenever they clone the repository. Furthermore, we want to offer a ready-built deliverable whenever possible so that trying out our software stack is easy.

Branching Model

In our development model, we have three different kinds of branches:

• Feature branches, a.k.a. pull request branch: each change is first pushed to a feature branch, tested, and possibly reviewed before merging it to the dev branch.
• dev: the baseline for all changes. One merges pull requests first with this branch. If tests for dev succeed, one can merge the changes into the master branch. The dev branch test set can be more extensive than the one run for the pull request (feature) branch.
• master: the project’s main branch that contains the latest working version of the software. Changes to this branch trigger automatic deployments to our cloud environment.

The primary reason for using this branching style is to enable us to run extensive automated testing for changes introduced by one or multiple pull requests before merging them with the master branch. Therefore, this routine ensures that the master branch always has a working version. This popular branching model has other perks as well. You can read more about them here, for example.

Time for a Release

Our release bot is a custom GitHub action. Each repository that follows the branching model described above uses the bot for tagging the release. The bot’s primary duty is to check if the dev branch has code changes missing from the master branch. If the bot finds changes, it will create a version tag for the new release and update the working version of the dev branch.

The bot works night shifts and utilizes a GitHub scheduled event as an alarm clock. For each repository it is configured for, it runs the same steps:

1. Check if there are changes between the dev and master branches
2. Check if the required workflows (tests) have succeeded for the dev branch
3. Parse the current working version from the VERSION file
4. Check out the dev branch and tag the dev branch HEAD with the current version
5. Push tag to remote
6. Increase the working version and commit it to the dev-branch.

When step 5 is ready, i.e., the bot has created a new tag, another workflow will start. This workflow will handle building the project deliverables for the tagged release. After a successful release routine, the CI merges the tag to master.

Package Distribution

Each time a release is tagged, the CI builds the release deliverables for distribution. As our stack contains various-style projects built in various languages, the release artifacts depend on the project type and programming language used.

The CI stores all of the artifacts in GitHub in one way or another. Docker images and library binaries utilize different features of GitHub Packages, a GitHub offering for delivering packages to the open-source community. The Packages UI is integrated directly with the repository UI, so finding and using the packages is intuitive for the user.

Docker Containers for Backend Services

Our agency software has a microservice architecture, meaning multiple servers handle the backend functionality. To simplify cloud deployment and service orchestration in a local development environment, we build Docker images for each service release and store them in the GitHub container registry.

The GitHub Actions workflow handles the image building. We build two variants of the images. In addition to amd64, we make an arm64 version to support mainly environments running on Apple-silicon-based Macs. You can read more about utilizing GitHub Actions to create images for multiple platforms here.

The community can access the publicly stored images without authentication. The package namespace is https://ghcr.io, meaning one can refer to an image with the path ghcr.io/NAMESPACE/IMAGE_NAME:TAG, e.g., ghcr.io/findy-network/findy-agent:latest. Publishing and using images from the Docker registry is straightforward.

Libraries

We also have utility libraries that provide the common functionalities needed to build clients for our backend services. These include helpers for Go, Node.js, and Kotlin.

In the Go ecosystem, sharing modules is easy. The Go build system must find the dependency code in a publicly accessible Git repository. Thus, the build compiles the code on the fly; one does not need to distribute or download binaries. Module versions are resolved directly from the git tags found in the repository.

The story is different for Node.js and Kotlin/Java libraries. GitHub offers registries for npm, Maven, and Gradle, and one can easily integrate the library’s publishing to these registries into the project release process. However, accessing the libraries is more complicated. Even if the package is public, the user must authenticate to GitHub to download the package. This requirement adds more complexity for the library user than mainstream options, such as the default npm registry. I would avoid distributing public libraries via this feature.

Sample for publishing Node.js package to GitHub npm registry via GitHub action:

name: release
on:
  push:
    tags:
      - '*'
jobs:
  publish-github:
    # runner machine
    runs-on: ubuntu-latest
    # API permissions for the job
    permissions:
      contents: read
      packages: write
    steps:
      # checkout the repository code
      - uses: actions/checkout@v4
      # setup node environment
      - uses: actions/setup-node@v4
        with:
          node-version: '18.x'
          registry-url: 'https://npm.pkg.github.com'
          scope: 'findy-network'
      # install dependencies, build distributables and publish
      - run: npm ci
      - run: npm run build
      - run: npm publish
        env:
          NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Executables And Other Artifacts

For other artifacts, we utilize the GitHub releases feature. For instance, we use the goreleaser helper tool for our CLI, which handles the cross-compilation of the executable for several platforms. It then attaches the binary files to the release, from where each user can download them. We even have an automatically generated installation script that downloads the correct version for the platform in question.

One can also attach other files to releases. We define our backend gRPC API with an IDL file. Whenever the API changes, we release a new version of the IDL using a GitHub release. We can then automate other logic (gRPC utility helpers) to download the IDL for a specific release and easily keep track of the changes.

Summary

This post summarized how our open-source project uses different GitHub features for versioning, release management, and artifact distribution. In our last article, we will review various other GitHub tools we have utilized.

• ←Previous
• Next→