Building and testing JLLs in GitHub actions
tutorials ·Author: Oscar Dowson (@odow)
The purpose of this blog post is to document a workflow that I have found useful during the development and maintenance of solver wrappers. By the time you read this, it might be out of date, but I hope it’s a helpful prod in the right direction.
Background
Many JuMP solvers are implemented in compiled languages such as C or C++. We build and distribute open-source solvers via Julia’s excellent Yggdrasil build infrastructure.
Yggdrasil is similar to other build systems such as homebrew. Each project that we want to build gets a separate directory. For example, the conic solver embotech/ecos has a directory E/ECOS. Inside that directory is a build_tarballs.jl file, which is a Julia script that describes where to download the source code, how to compile it, what platforms it should be compiled to, and so on.
The output of running the script is a Julia package, for example, ECOS_jll.jl,
and a set of binary artifacts
with the compiled binaries for each platform. “JLL” packages are regular Julia
packages. The _jll suffix is naming convention to signify that they have been
automatically built to distribute compiled binaries.
When you make a pull request to add or edit a build_tarballs.jl file on
Yggdrasil, the CI machines start a Linux machine, run your script, and build all
variations required. When the PR is merged, these get uploaded to the associated
_jll.jl package and a new release is tagged.
An important point that will soon be relevant is that Yggdrasil cross-compiles each project from a Linux build to the target platform.
ECOS is a pretty simple project to compile, but we still cross-compile 16 different versions for a variety of different platforms. For example, there are Windows builds for 32- and 64-bit systems, Mac builds for Intel and ARM processors, and Linux builds for x64_64, i686, aarch64, armv6l, and armv7l processors, as well builds that depend on whether the user is running glibc or musl. A more compilicated project like Ipopt_jll has 85 different versions, with additional builds that depend on the C++ and Fortran versions used by the linked code.
Once things are built and running, this system works pretty smoothly. But there are two major issues.
First, upstream solvers may change their compilation workflow, or accidentally break compilation on some platforms because they do not test on the full suite of platforms that Yggdrasil supports. This means that small maintenance jobs like “hey, upstream has released a new version, let’s rebuild the JLL” can turn into long back-and-forth debugging as we attempt to compile, find a bug, develop a patch, and then either carry it in the JLL build, or upstream it and wait for a new release.
One solution is for the upstream project to add a GitHub action workflow that uses the Yggdrasil build script in their CI. This should give upstream advance notice of compilation failures. We have already added such workflows to ERGO-Code/HiGHS and cvanaret/Uno. As a side benefit, if they build a Linux binary, they can then install related Julia packages and run their tests to ensure that no breaking changes have been introduced. For example, the HiGHS script installs and tests HiGHS.jl, which indirectly provides a few thousand tests of the HiGHS C API.
The second major problem is that cross-compiling from Linux to the target platforms means that we cannot test the binaries that are built during the Yggdrasil run. We can ensure only that compilation succeeds, and thus any runtime bugs cannot be checked or tested against. As one example, SCIP.jl was unusable on Windows because of a runtime segfault. The debugging step for this build issue was painful, because it required someone to start a Linux machine, cross-compile to Windows using Yggdrasil, copy the binaries to a Windows machine, run, debug, make changes to the Linux build, and repeat. I developed the following script for use in SCIP.jl to help debug the problem.
At a high level, it starts a Linux machine, uses the Yggdrasil build script to compile a binary, saves that to GITHUB_OUTPUT, starts a new machine (in this case, Windows), downloads the compiled binary, and then patches the relevant JLL package using Julia’s Override.toml mechanism.
A workflow to test the cross-compiled binaries
To make things a bit simpler, I’ve changed it to use the relevant data for ECOS instead of SCIP.
Layout
This workflow assumes that you have a Julia package with the layout:
ECOS/
.github/
julia/
build_tarballs.jl
workflows/
cross-compile-and-test.yml
src/
ECOS.jl
test/
runtests.jl
Project.toml
Julia script
Here is the contents for .github/julia/build_tarballs.jl:
using BinaryBuilder
name = "ECOS"
version = v"2.0.8"
# Collection of sources required to build ECOSBuilder
sources = [
GitSource("https://github.com/embotech/ecos.git", "3b98fe0376ceeeb8310a06694b0a84ac59920f3f")
]
# Bash recipe for building across all platforms
script = raw"""
cd $WORKSPACE/srcdir/ecos*
make shared
mkdir -p ${libdir}
cp libecos.${dlext} ${libdir}
cp -r include ${prefix}
"""
# These are the platforms we will build for by default, unless further
# platforms are passed in on the command line
platforms = supported_platforms()
# The products that we will ensure are always built
products = [
LibraryProduct("libecos", :libecos)
]
# Dependencies that must be installed before this package can be built
dependencies = Dependency[]
build_tarballs(
ARGS,
name,
version,
sources,
script,
platforms,
products,
dependencies,
julia_compat = "1.6",
)
YAML
Here is the contents for .github/workflows/cross-compile-and-test.yml:
name: Build on Linux, Run on Windows
on:
push:
# The branch might be named something else, like `main` or `latest`
branches: [master]
pull_request:
types: [opened, synchronize, reopened]
# needed to allow julia-actions/cache to delete old caches that it has created
permissions:
actions: write
contents: read
jobs:
# The purpose of this job is to install Julia, run binary builder, and store
# the solver artifact for the next job.
build-linux:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
# Install Julia 1.7 for BinaryBuilder. Note that this is an old version of
# Julia, but it is required for compatibility with BinaryBuilder.
- uses: julia-actions/setup-julia@v2
with:
version: "1.7"
arch: x64
- uses: julia-actions/cache@v2
- run: |
# Replace as needed
PACKAGE=ECOS_jll
PLATFORM=x86_64-w64-mingw32-cxx11
julia --color=yes -e 'using Pkg; Pkg.add("BinaryBuilder")'
julia --color=yes .github/julia/build_tarballs.jl ${PLATFORM} --verbose --deploy=local
file=/home/runner/.julia/dev/${PACKAGE}/Artifacts.toml
sha1=$(grep '^git-tree-sha1' "$file" | cut -d '"' -f2)
echo "ARTIFACT_SHA=${sha1}" >> $GITHUB_ENV
- uses: actions/upload-artifact@v4
with:
name: artifacts
path: '/home/runner/.julia/artifacts/${ env.ARTIFACT_SHA }'
# The purpose of this job is to install Julia, download the artifact from
# build-linux, and use it to run the solver's tests.
run-windows:
# You could replace this `runs-on` with `macOS-latest` if desired.
runs-on: windows-latest
# Declare that we need `build-linux` to finish first.
needs: build-linux
steps:
- uses: actions/checkout@v4
- uses: julia-actions/setup-julia@v2
with:
version: "1"
arch: x64
- uses: julia-actions/cache@v2
- uses: julia-actions/julia-buildpkg@v1
# Download the artifact from the `build-linux` job, and store it in
# $/override
- uses: actions/download-artifact@v4
with:
name: artifacts
path: override
# Replace ECOS_jll with the desired JLL
- shell: julia --color=yes --project=. {0}
run: |
import ECOS_jll
artifact_dir = ECOS_jll.artifact_dir
sha = last(splitpath(artifact_dir))
dir = escape_string(joinpath(ENV["GITHUB_WORKSPACE"], "override"))
content = "$sha = \"$(dir)\"\n"
write(replace(artifact_dir, sha => "Overrides.toml"), content)
# This assumes that CI is being run from the root of a Julia package. If
# not, you could change this as desired.
- uses: julia-actions/julia-runtest@v1