.%*. .-.
.%@@@+. .--=%@@@-
=@@@@@@- :--+@@@@@@@@@*
*@%@@@@@%: :--=#@@@@@@@@@@@@#@@+
@@::%@@@@@@-: :::-=%@@@@@@@@@@@@@@%#*- :@*
.@@ =@@@@@@@@@+-:::::::-=*@@@@@@@@@@@@@@@@@@@%##- @@:
#@# +%@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%%#*: -@@
#@- *%@@@@@@@@@@@@@@@@@%%%#= %@=
@@: -=+++== .. @@:
@@ . -@@@ +@@
#@% =@@@@- +@@@@# *+ %@#
#@- -@@@@@@@. +@@@@@% .%@@@@= @@:
@@: +@@@@@@@@@ +@+ +@@@ %@@@@@@@@@ -@@
@@ *@%@* :@@@ *@+ -@@@ %@@@@@@@@@@@# @@@
#@% %@ @% %@@ *@* .@@@ =@@@*@@ :@@@@ @@:
%@:## *@+ :@@ :@@ @@@ %@@: @# .@@@ .@@
@@:. @@: .@@ @@ @@@ %@: @@ @@% @@%
@@ -@@. =@@ @@. @@@ %@. #@- @@# @@.
%@% +@@. @@. %@- @@@ #@ @@. :@@ -@@
%@: +@@+ .@% #@= @@@.@ *@@ +@+ @@*
@@: +@@%. .%% *@+ @@@ %@@ -@# @@
@@ :@@@@@@% :@@ @@% %@@ @# *@@
%@@ #@@@@* @@ @@% %@@. %@. @@:
@@: ===: @@. @@% %@@: =@%. .@@
@@. #@= @@% %@@%+%@#. @@%
@@ +@* @@# -@@@@@* .@@
%@@ :@@ @@# %@@#: =@@
@@: .@@ @@# : @@=
@@. @@. @@# :@@
@@ #@= @@% %@%
#@@ +@+ #@% .@@
@@: =@# +@% =@%
@@ :@% -@@ @@+
@@ .@@ -@: -@@
-@@ @% .: *@#
@@* # @@.
@@ =@@
.@@ @@*
.@@ .@@
*@@ =@#
@@* @@*
@@ -@@
.@@ +@*
.@@ @@-
-@@ =@@
%@% #@*
@@ @@:
@@ +@@
.@@ %@+
:@% @@.
:@% *@%
-@. %@=
%@ @@.
@@ *@%
@@ %@=
@% @@:
:@% +@@
:@* =+- #@+
:@: .%@@@. @@-
-@ =@@@@@# :@@
=@ *@#.=@@% #@%
*@ #@- -@@% %@-
@@ #@: :@@% @@:
@@ #% .@@% .#@@* -@@
@% #% .@@% +@@@@@- %@%
@# %@ :@@% .@@@@@@@@ %@:
@# %@ :@@% +@@% .@@@@ @@:
@# %@ :@@% #@@: :@@@ -@@
@# @@. .@@@ .@@% .@@@= %@%
@# @@. .@@@: %@@- @@@% %@:
@# @@ @@@@ %@@@ *@@@ @@:
@# .@@ @@@@@@@@@@= :@@@ @@
* @@@ :@@@@@@@@. :@@@ *@@
.@@@@@% -@@@@@. .@@@ @@=
@@@@@: .. @@@ @@.
-@%: @@@ @@.
@@@ @@
-@@ =@@
.@@ @@+
.@@:@@.
@@#@-
@@@.
+@-
-
mediocregopher's lil web corner
- There can only be one.
It's been a bit since updating my progress on what I've been lately calling the "cryptic nebula" project. When I last left off I was working on building the [mobile nebula][mobile_nebula] using nix. For the moment I gave up on that dream, as flutter and nix just really don't get along and I don't want to get to distracted on problems that aren't critical to the actual goal.
Instead I'd like to pursue the next critical component of the system, and that's a shared filesystem. The use-case I'm ultimately trying to achieve is:
What I'm looking for is some kind of network filesystem, of which there are many. This document will attempt to evaluate all relevant projects and come up with the next steps. It may be that no project fits the bill perfectly, and that I'm stuck either modifying an existing project to my needs or, if things are looking really dire, starting a new project.
The ultimate use-case here is something like a self-hosted, distributed keybase filesystem; somewhere where individuals in the cluster can back up their personal projects, share files with each other, and possibly even be used as the base layer for more complex applications on top.
The individuals involved shouldn't have to deal with configuring their distributed FS, either to read from it or add storage resources to it. Ideally the FS process can be bundled together with the nebula process and run opaquely; the user is just running their "cryptic nebula" process and everything else is handled in the background.
There are some criteria for these projects that I'm not willing to compromise on; these criteria will form a low pass filter which, hopefully, will narrow our search appreciably.
The network filesystem used by the cryptic nebula must:
The last may come across as mean, but the reason for it is that I forsee the network filesystem client running on users' personal laptops, which cannot be assumed to have resources to spare.
Each criteria in the next set lies along a spectrum. Any project may meet one of thses criteria fully, partially, or not at all. For each criteria I assign a point value according to how fully a project meets the criteria, and then sum up the points to give the project a final score. The project with the highest final score is not necessarily the winner, but this system should at least give some good candidates for final consideration.
The criteria, and their associated points values, are:
Hackability: is the source-code of the project approachable?
Documentation: is the project well documented?
Transience: how does the system handle hosts appearing or disappearing?
Priority: is it possible to give certain hosts priority when choosing which will host/replicate some piece of data?
Caching: will hosts reading a file have that file cached locally for the next reading (until the file is modified)?
Conflicts: if two hosts updated the same file at the same time, how is that handled?
Consistency: how does the system handle a file being changed frequently?
POSIX: how POSIX compliant is the mounted fileystem?
Scale: how many hosts can be a part of the cluster?
Failure: how does the system handle failures (network partitions, hosts hanging, buggy client versions)?
Limitations: are there limits on how big files can be, or how big directories can be?
Encryption: how is data encrypted?
Permissions: how are modifications to data restricted?
Administration: how much administration is required for the system to function?
Simplicity: how understandable is the system as a whole?
Visibility: how much visibility is available into processes within the system?
With the rubric defined, let's start actually working through our options! There are many, many different possibilities, so this may not be an exhaustive list.
The Ceph File System, or CephFS, is a POSIX-compliant file system built on top of Ceph’s distributed object store, RADOS. CephFS endeavors to provide a state-of-the-art, multi-use, highly available, and performant file store for a variety of applications, including traditional use-cases like shared home directories, HPC scratch space, and distributed workflow shared storage.
TOTAL: 22
Ceph has been recommended to me by a few people. It is clearly a very mature project, though that maturity has brought with it a lot of complexity. A lot of the complexity of Ceph seems to be rooted in its strong consistency guarantees, which I'm confident it fulfills well, but are not really needed for the use-case I'm interested in. I'd prefer a simpler, eventually consistent, system. It's also not clear to me that Ceph would even perform very well in my use-case as it seems to want an actual datacenter deployment, with beefy hardware and hosts which are generally close together.
GlusterFS is a scalable network filesystem suitable for data-intensive tasks such as cloud storage and media streaming. GlusterFS is free and open source software and can utilize common off-the-shelf hardware.
TOTAL: 23
GlusterFS was my initial choice when I did a brief survey of DFSs for this use-case. However, after further digging into it I think it will suffer the same ultimate problem as CephFS: too much consistency for a wide-area application like I'm envisioning. The need for syncing user/groups across machines as actual system users is also cumbersome enough to make it not a great choice.
MooseFS is a Petabyte Open Source Network Distributed File System. It is easy to deploy and maintain, highly reliable, fault tolerant, highly performing, easily scalable and POSIX compliant.
MooseFS spreads data over a number of commodity servers, which are visible to the user as one resource. For standard file operations MooseFS acts like ordinary Unix-like file system.
TOTAL: 17
Overall MooseFS seems to me like a poor-developer's Ceph. It can do exactly the same things, but with less of a community around it. The sale's pitch and feature-gating also don't ingratiate it to me. The most damning "feature" is the master metadata server, which acts as a SPOF and only sort of supports replication (but not failover, unless you get Pro).
The following projects were intended to be reviewed, but didn't make the cut for various reasons.
Tahoe-LAFS: The FUSE mount (which is actually an SFTP mount) doesn't support mutable files.
HekaFS: Doesn't appear to exist anymore(?)
IPFS-cluster: Doesn't support sharding.
MinFS: Seems to only work off S3, no longer maintained anyway.
DRDB: Linux specific, no mac support.
BeeGFS: No mac support (I don't think? I couldn't find any indication it supports macs at any rate).
NFS: No support for sharding the dataset.
Going through the featuresets of all these different projects really helped me focus in on how I actually expect this system to function, and a few things stood out to me:
Perfect consistency is not a goal, and is ultimately harmful for this use-case. The FS needs to propagate changes relatively quickly, but if two different hosts are updating the same file it's not necessary to synchronize those updates like a local filesystem would; just let one changeset clobber the other and let the outer application deal with coordination.
Permissions are extremely important, and yet for all these projects are generally an afterthought. In a distributed setting we can't rely on the OS user/groups of a host to permission read/write access. Instead that must be done primarily via e2e encryption.
Transience is not something most of these project expect, but is a hard requirement of this use-case. In the long run we need something which can be run on home hardware on home ISPs, which is not reliable at all. Hosts need to be able to flit in and out of existence, and the cluster as a whole needs to self-heal through that process.
In the end, it may be necessary to roll our own project for this, as I don't think any of the existing distributed file systems are suitable for what's needed.
Published 2021-04-06
This post is part of a series.
Previously: Building gomobile Using Nix
Next: Building AppImages with Nix
This site can also be accessed via the gemini protocol: gemini://mediocregopher.com/