Building gomobile Using Nix

Harder than I thought it would be!

This post is part of a series:
Previously: Building Mobile Nebula

When I last left off with the nebula project I wanted to nix-ify the build process for Cryptic’s mobile_nebula fork. While I’ve made progress on the overall build, one particular bit of it really held me up, so I’m writing about that part here. I’ll finish the full build at a later time.

gomobile

gomobile is a toolkit for the go programming language to allow for running go code on Android and iOS devices. mobile_nebula uses gomobile to build a simple wrapper around the nebula client that the mobile app can then hook into.

This means that in order to nix-ify the entire mobile_nebula project I first need to nix-ify gomobile, and since there isn’t (at time of writing) an existing package for gomobile in the nixpkgs repo, I had to roll my own.

I started with a simple buildGoModule nix expression:

pkgs.buildGoModule {
    pname = "gomobile";
    version = "unstable-2020-12-17";
    src = pkgs.fetchFromGitHub {
        owner = "golang";
        repo = "mobile";
        rev = "e6ae53a27f4fd7cfa2943f2ae47b96cba8eb01c9";
        sha256 = "03dzis3xkj0abcm4k95w2zd4l9ygn0rhkj56bzxbcpwa7idqhd62";
    };
    vendorSha256 = "1n1338vqkc1n8cy94501n7jn3qbr28q9d9zxnq2b4rxsqjfc9l94";
}

The basic idea here is that buildGoModule will acquire a specific revision of the gomobile source code from github, then attempt to build it. However, gomobile is a special beast in that it requires a number of C/C++ libraries in order to be built. I discovered this upon running this expression, when I received this error:

./work.h:12:10: fatal error: GLES3/gl3.h: No such file or directory
   12 | #include <GLES3/gl3.h> // install on Ubuntu with: sudo apt-get install libegl1-mesa-dev libgles2-mesa-dev libx11-dev

This stumped me for a bit, as I couldn’t figure out a) the “right” place to source the GLES3 header file from, and b) how to properly hook that into the buildGoModule expression. My initial attempts involved trying to include versions of the header file from my androidsdk nix package which I had already gotten (mostly) working, but the version which ships there appears to expect to be using clang. cgo (go’s compiler which is used for C/C++ interop) only supports gcc, so that strategy failed.

I didn’t like having to import the header file from androidsdk anyway, as it meant that my gomobile would only work within the context of the mobile_nebula project, rather than being a standalone utility.

nix-index

At this point I flailed around some more trying to figure out where to get this header file from. Eventually I stumbled on the nix-index project, which implements something similar to the locate utility on linux: you give it a file pattern, and it searches your active nix channels for any packages which provide a file matching that pattern.

Since nix is amazing it’s not actually necessary to install nix-index, I simply start up a shell with the package available using nix-shell -p nix-index. On first run I needed to populate the index by running the nix-index command, which took some time, but after that finding packages which provide the file I need is as easy as:

> nix-shell -p nix-index
[nix-shell:/tmp]$ nix-locate GLES3/gl3.h
(zulip.out)                                      82,674 r /nix/store/wbfw7w2ixdp317wip77d4ji834v1k1b9-libglvnd-1.3.2-dev/include/GLES3/gl3.h
libglvnd.dev                                     82,674 r /nix/store/pghxzmnmxdcarg5bj3js9csz0h85g08m-libglvnd-1.3.2-dev/include/GLES3/gl3.h
emscripten.out                                   82,666 r /nix/store/x3c4y2h5rn1jawybk48r6glzs1jl029s-emscripten-2.0.1/share/emscripten/system/include/GLES3/gl3.h

So my mystery file is provided by a few packages, but libglvnd.dev stood out to me as it’s also the pacman package which provides the same file in my real operating system:

> yay -Qo /usr/include/GLES3/gl3.h
/usr/include/GLES3/gl3.h is owned by libglvnd 1.3.2-1

This gave me some confidence that this was the right track.

cgo

My next fight was with cgo itself. Go’s build process provides a few different entry points for C/C++ compiler/linker flags, including both environment variables and command-line arguments. But I wasn’t using go build directly, instead I was working through nix’s buildGoModule wrapper. This added a huge layer of confusion as all of nixpkgs is pretty terribly documented, so you really have to just divine behavior from the source (good luck).

After lots of debugging (hint: NIX_DEBUG=1) I determined that all which is actually needed is to set the CGO_CFLAGS variable within the buildGoModule arguments. This would translate to the CGO_CFLAGS environment variable being set during all internal commands, and whatever go build commands get used would pick up my compiler flags from that.

My new nix expression looked like this:

pkgs.buildGoModule {
    pname = "gomobile";
    version = "unstable-2020-12-17";
    src = pkgs.fetchFromGitHub {
        owner = "golang";
        repo = "mobile";
        rev = "e6ae53a27f4fd7cfa2943f2ae47b96cba8eb01c9";
        sha256 = "03dzis3xkj0abcm4k95w2zd4l9ygn0rhkj56bzxbcpwa7idqhd62";
    };
    vendorSha256 = "1n1338vqkc1n8cy94501n7jn3qbr28q9d9zxnq2b4rxsqjfc9l94";

    CGO_CFLAGS = [
        "-I ${pkgs.libglvnd.dev}/include"
    ];
}

Running this produced a new error. Progress! The new error was:

/nix/store/p792j5f44l3f0xi7ai5jllwnxqwnka88-binutils-2.31.1/bin/ld: cannot find -lGLESv2
collect2: error: ld returned 1 exit status

So pretty similar to the previous issue, but this time the linker wasn’t finding a library file rather than the compiler not finding a header file. Once again I used nix-index’s nix-locate command to find that this library file is provided by the libglvnd package (as opposed to libglvnd.dev, which provided the header file).

Adding libglvnd to the CGO_CFLAGS did not work, as it turns out that flags for the linker cgo uses get passed in via CGO_LDFLAGS (makes sense). After adding this new variable I got yet another error; this time X11/Xlib.h was not able to be found. I repeated the process of nix-locate/add to CGO_*FLAGS a few more times until all dependencies were accounted for. The new nix expression looked like this:

pkgs.buildGoModule {
    pname = "gomobile";
    version = "unstable-2020-12-17";
    src = pkgs.fetchFromGitHub {
        owner = "golang";
        repo = "mobile";
        rev = "e6ae53a27f4fd7cfa2943f2ae47b96cba8eb01c9";
        sha256 = "03dzis3xkj0abcm4k95w2zd4l9ygn0rhkj56bzxbcpwa7idqhd62";
    };
    vendorSha256 = "1n1338vqkc1n8cy94501n7jn3qbr28q9d9zxnq2b4rxsqjfc9l94";

    CGO_CFLAGS = [
        "-I ${pkgs.libglvnd.dev}/include"
        "-I ${pkgs.xlibs.libX11.dev}/include"
        "-I ${pkgs.xlibs.xorgproto}/include"
        "-I ${pkgs.openal}/include"
    ];

    CGO_LDFLAGS = [
        "-L ${pkgs.libglvnd}/lib"
        "-L ${pkgs.xlibs.libX11}/lib"
        "-L ${pkgs.openal}/lib"
    ];
}

Tests

The CGO_*FLAGS variables took care of all compiler/linker errors, but there was one issue left: buildGoModule apparently runs the project’s tests after the build phase. gomobile’s tests were actually mostly passing, but some failed due to trying to copy files around, which nix was having none of. After some more buildGoModule source divination I found that if I passed an empty checkPhase argument it would skip the check phase, and therefore skip running these tests.

Fin!

The final nix expression looks like so:

pkgs.buildGoModule {
    pname = "gomobile";
    version = "unstable-2020-12-17";
    src = pkgs.fetchFromGitHub {
        owner = "golang";
        repo = "mobile";
        rev = "e6ae53a27f4fd7cfa2943f2ae47b96cba8eb01c9";
        sha256 = "03dzis3xkj0abcm4k95w2zd4l9ygn0rhkj56bzxbcpwa7idqhd62";
    };
    vendorSha256 = "1n1338vqkc1n8cy94501n7jn3qbr28q9d9zxnq2b4rxsqjfc9l94";

    CGO_CFLAGS = [
        "-I ${pkgs.libglvnd.dev}/include"
        "-I ${pkgs.xlibs.libX11.dev}/include"
        "-I ${pkgs.xlibs.xorgproto}/include"
        "-I ${pkgs.openal}/include"
    ];

    CGO_LDFLAGS = [
        "-L ${pkgs.libglvnd}/lib"
        "-L ${pkgs.xlibs.libX11}/lib"
        "-L ${pkgs.openal}/lib"
    ];

    checkPhase = "";
}

Once I complete the nix-ification of mobile_nebula I’ll submit a PR to the nixpkgs upstream with this, so that others can have gomobile available as well!

If you liked this post, consider checking out other posts in the series:
Previously: Building Mobile Nebula