.%*. .-.
.%@@@+. .--=%@@@-
=@@@@@@- :--+@@@@@@@@@*
*@%@@@@@%: :--=#@@@@@@@@@@@@#@@+
@@::%@@@@@@-: :::-=%@@@@@@@@@@@@@@%#*- :@*
.@@ =@@@@@@@@@+-:::::::-=*@@@@@@@@@@@@@@@@@@@%##- @@:
#@# +%@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%%#*: -@@
#@- *%@@@@@@@@@@@@@@@@@%%%#= %@=
@@: -=+++== .. @@:
@@ . -@@@ +@@
#@% =@@@@- +@@@@# *+ %@#
#@- -@@@@@@@. +@@@@@% .%@@@@= @@:
@@: +@@@@@@@@@ +@+ +@@@ %@@@@@@@@@ -@@
@@ *@%@* :@@@ *@+ -@@@ %@@@@@@@@@@@# @@@
#@% %@ @% %@@ *@* .@@@ =@@@*@@ :@@@@ @@:
%@:## *@+ :@@ :@@ @@@ %@@: @# .@@@ .@@
@@:. @@: .@@ @@ @@@ %@: @@ @@% @@%
@@ -@@. =@@ @@. @@@ %@. #@- @@# @@.
%@% +@@. @@. %@- @@@ #@ @@. :@@ -@@
%@: +@@+ .@% #@= @@@.@ *@@ +@+ @@*
@@: +@@%. .%% *@+ @@@ %@@ -@# @@
@@ :@@@@@@% :@@ @@% %@@ @# *@@
%@@ #@@@@* @@ @@% %@@. %@. @@:
@@: ===: @@. @@% %@@: =@%. .@@
@@. #@= @@% %@@%+%@#. @@%
@@ +@* @@# -@@@@@* .@@
%@@ :@@ @@# %@@#: =@@
@@: .@@ @@# : @@=
@@. @@. @@# :@@
@@ #@= @@% %@%
#@@ +@+ #@% .@@
@@: =@# +@% =@%
@@ :@% -@@ @@+
@@ .@@ -@: -@@
-@@ @% .: *@#
@@* # @@.
@@ =@@
.@@ @@*
.@@ .@@
*@@ =@#
@@* @@*
@@ -@@
.@@ +@*
.@@ @@-
-@@ =@@
%@% #@*
@@ @@:
@@ +@@
.@@ %@+
:@% @@.
:@% *@%
-@. %@=
%@ @@.
@@ *@%
@@ %@=
@% @@:
:@% +@@
:@* =+- #@+
:@: .%@@@. @@-
-@ =@@@@@# :@@
=@ *@#.=@@% #@%
*@ #@- -@@% %@-
@@ #@: :@@% @@:
@@ #% .@@% .#@@* -@@
@% #% .@@% +@@@@@- %@%
@# %@ :@@% .@@@@@@@@ %@:
@# %@ :@@% +@@% .@@@@ @@:
@# %@ :@@% #@@: :@@@ -@@
@# @@. .@@@ .@@% .@@@= %@%
@# @@. .@@@: %@@- @@@% %@:
@# @@ @@@@ %@@@ *@@@ @@:
@# .@@ @@@@@@@@@@= :@@@ @@
* @@@ :@@@@@@@@. :@@@ *@@
.@@@@@% -@@@@@. .@@@ @@=
@@@@@: .. @@@ @@.
-@%: @@@ @@.
@@@ @@
-@@ =@@
.@@ @@+
.@@:@@.
@@#@-
@@@.
+@-
-
mediocregopher's lil web corner
- With some process trees thrown in there for fun.
It's been a bit since I've written an update on the cryptic nebula project, almost 5 months (since this post, which wasn't officially part of the blog series but whatever). Since then it's switched names to "cryptic-net", and that we would likely use MinIO as our network storage service, but neither of those is the most interesting update.
The project had been stalled because of a lack of a build system which could fulfill the following requirements:
Network configuration (static IP, VPN certificates) of individual hosts is baked into the binary they run.
Binaries are completely static; no external dependencies need to exist on the host in order to run them.
Each binary runs a composition of multiple sub-services, each being a separate sub-process, and all of them having been configured to work together (with some possible glue code on our side) to provide the features we want.
The builder itself should be deterministic; no matter where it runs it should produce the same binary given the same input parameters.
Lacking such a build system we're not able to distribute cryptic-net in a way which "just works"; it would require some kind of configuration, or some kind of runtime environment to be set up, both of which would be a pain for users. And lacking a definite build system makes it difficult to move forward on any other aspect of a project, as it's not clear what may need to be redone in the future when the build system is decided upon.
My usage of nix-bundle in a previous post was an attempt at fulfilling these requirements. Nix in general does very well in fulfilling all but the second requirement, and nix-bundle was supposed to fulfill even that by packaging a nix derivation into a static binary.
And all of this it did! Except that the mechanism of nix-bundle is a bit odd.
The process of a nix-bundle'd binary jails itself within a chroot, which it then
uses to fake the /nix/store path which nix built binaries expect to exist.
This might work in a lot of cases, but it did not work in ours. For one, nebula can't create its network interface when run from inside nix-bundle's chroot. For another, being run in a chroot means there's going to be strange restrictions on what our binary is able to do and not.
What we really needed was an AppImage. AppImages are static binaries which can bundle complex applications, even those which don't expect to be bundled into single binaries. In this way the end result is the same as nix-bundle, but the mechanism AppImage uses is different and places far fewer restrictions on what we can and can't do with our program.
It's probably possible to use nix to generate an AppImage which has the
/nix/store built into it, similar to what nix-bundle does, and therefore not
worry about whether the binaries it's bundling are static or not. But if your
services are written in sane languages it's not that difficult to build them
statically and dodge the issue.
For example, here is how you build a go binary statically:
{
buildGoModule,
fetchFromGitHub,
}:
buildGoModule rec {
pname = "nebula";
version = "1.4.0";
src = fetchFromGitHub {
owner = "slackhq";
repo = pname;
rev = "v${version}";
sha256 = "lu2/rSB9cFD7VUiK+niuqCX9CI2x+k4Pi+U5yksETSU=";
};
vendorSha256 = "p1inJ9+NAb2d81cn+y+ofhxFz9ObUiLgj+9cACa6Jqg=";
doCheck = false;
subPackages = [ "cmd/nebula" "cmd/nebula-cert" ];
CGO_ENABLED=0;
tags = [ "netgo" ];
ldflags = [
"-X main.Build=${version}"
"-w"
"-extldflags=-static"
];
};
And here's how to statically build a C binary:
{
stdenv,
glibcStatic, # e.g. pkgs.glibc.static
}:
stdenv.mkDerivation rec {
pname = "dnsmasq";
version = "2.85";
src = builtins.fetchurl {
url = "https://www.thekelleys.org.uk/dnsmasq/${pname}-${version}.tar.xz";
sha256 = "sha256-rZjTgD32h+W5OAgPPSXGKP5ByHh1LQP7xhmXh/7jEvo=";
};
nativeBuildInputs = [ glibcStatic ];
makeFlags = [
"LDFLAGS=-static"
"DESTDIR="
"BINDIR=$(out)/bin"
"MANDIR=$(out)/man"
"LOCALEDIR=$(out)/share/locale"
];
};
The derivations created by either of these expressions can be plugged right into
the pkgs.buildEnv used to create the AppDir (see AppDir section below).
An important piece of the puzzle for getting cryptic-net into an AppImage was a process manager. We need something which can run multiple service processes simultaneously, restart processes which exit unexpectedly, gracefully handle shutting down all those processes, and coalesce the logs of all processes into a single stream.
There are quite a few process managers out there which could fit the bill, but
finding any which could be statically compiled ended up not being an easy task.
In the end I decided to see how long it would take me to implement such a
program in go, and hope it would be less time than it would take to get
circus, a python program, bundled into the AppImage.
2 hours later, pmux was born! Check it out. It's a go program so building it looks pretty similar to the nebula builder above, so I won't repeat it. However I will show the configuration we're using for it within the AppImage, to show how it ties all the processes together:
processes:
- name: nebula
cmd: bin/nebula
args:
- "-config"
- etc/nebula/nebula.yml
- name: dnsmasq
cmd: bin/dnsmasq
args:
- "-d"
- "-C"
- ${dnsmasq}/etc/dnsmasq/dnsmasq.conf
Generating an AppImage requires an AppDir. An AppDir is a directory which
contains all files required by a program, rooted to the AppDir. For example, if
the program expects a file to be at /etc/some/conf, then that file should be
places in the AppDir at <AppDir-path>/etc/some/conf.
These docs
were very helpful for me in figuring out how to construct the AppDir. I then
used the pkgs.buildEnv utility to create an AppDir derivation containing
everything cryptic-net needs to run:
appDir = pkgs.buildEnv {
name = "cryptic-net-AppDir";
paths = [
# real directory containing non-built files, e.g. the pmux config
./AppDir
# static binary derivations shown previously
nebula
dnsmasq
pmux
];
};
Once the AppDir is built one needs to use appimagetool to turn it into an
AppImage. There is an appimagetool build in the standard nixpkgs, but
unfortunately it doesn't seem to actually work...
Luckily nix-bundle is working on AppImage support, and includes a custom build
of appimagetool which does work!
{
fetchFromGitHub,
callPackage,
}: let
src = fetchFromGitHub {
owner = "matthewbauer";
repo = "nix-bundle";
rev = "223f4ffc4179aa318c34dc873a08cb00090db829";
sha256 = "0pqpx9vnjk9h24h9qlv4la76lh5ykljch6g487b26r1r2s9zg7kh";
};
in
callPackage "${src}/appimagetool.nix" {}
Using callPackage on this expression will give you a functional appimagetool
derivation. From there's it's a simple matter of writing a derivation which
generates the AppImage from a created AppDir:
{
appDir,
appimagetool,
}:
pkgs.stdenv.mkDerivation {
name = "cryptic-net-AppImage";
src = appDir;
buildInputs = [ appimagetool ];
ARCH = "x86_64"; # required by appimagetool
builder = builtins.toFile "build.sh" ''
source $stdenv/setup
cp -rL "$src" buildAppDir
chmod +w buildAppDir -R
mkdir $out
appimagetool cryptic-net "$out/cryptic-net-bin"
'';
}
Running that derivation deterministically spits out a binary at
result/cryptic-net-bin which can be executed and run immediately, on any
system using the x86_46 CPU architecture.
I'm extremely hyped to now have the ability to generate binaries for cryptic-net that people can just run, without them worrying about which sub-services that binary is running under-the-hood. From a usability perspective it's way nicer than having to tell people to "install docker" or "install nix", and from a dev perspective we have a really solid foundation on which to build a quite complex application.
Published 2021-09-22
This post is part of a series.
Previously: Evaluation of Network Filesystems
Next: The Cryptic Filesystem
This site can also be accessed via the gemini protocol: gemini://mediocregopher.com/