Earthbound

An agnostic software procurement system.

This is a very special project. It is a key component for bootstrapping nearly all other projects developed by Aquefir Consulting LLC and indeed the wider world of software using the “developmental quartet” consisting of Inbound, Outbound, Rebound, and this project: Earthbound.

Earthbound does not care what form your software comes in. It does not care where your software comes from, as long as it can find it. Its job is very simple and singular, and true to UNIX™ fashion, it is very good at it.

---

Getting started

There are two ways to get started using Earthbound: the quick way, and the right way.

The quick way

curl -fsSL tohoku.ac/eb.c | cc -oeb.bin -
./eb.bin $ini

The right way

Please remember, we do not intend to be dogmatic in calling this “the right way”. You can (and often should) do more.

Use a bootstrap script from the Releases page.

• Download an eb.sh release and place it into your repository.
• If you are using version control, check it in as-is.
• Before building, run ./eb.sh to securely download and compile Earthbound.
  • You will need an ANSI C compiler in your $PATH for this.
    • Either its name is cc, or it is provided as $CC.
  • You will also need shasum or sha256sum in your $PATH.
• Out will come an eb.bin file. Run it with your INI: ./eb.bin $ini

This approach will give your project a script that downloads a versioned variant of eb.c, which—in contrast to the plain URL downloaded above—can have its integrity verified using SHA2-256. The script also altruistically uses curl or wget, so as long as one is available, it will work. Finally, while it tries to use cc from the $PATH, it will prefer the value of $CC if it is not empty. It will also pass $CFLAGS as-is to the compiler invocation.

See also: the compilation guide

Architecture

Earthbound only concerns itself with what we will call files of interest. These are files, often source code, that may come in from anywhere over the network, that were selected by the downstream developer to contain in good faith the data they need. Earthbound keeps hashes of these files of interest and uses them to verify the files’ authenticity.

Beyond acquisition, Earthbound has a know-nothing approach, giving developers total discretion on what to do with these files after they have been downloaded and validated. This is true to its one purpose of software procurement, which exalts the UNIX™ philosophy of doing one thing and doing it well.

Earthbound INIs

Earthbound uses an INI file structure to define a collection of files of interest along with their hashes and details on where they may be found. Here is an example configuration that sources a single file from multiple potential mirrors called file1.c:

# is used for comments, not ;
# must be at the start of a logical line
	 # leading space is OK though
[file1.c]
sha2sum=70d181c0bebbac369cddb65e6304e55a8efdcef50589154faf30390fdc6a1427
# at least one is required, but multiple may be given
# if so, all hashes are checked
sha3sum=4e9e5320e30e44ff31463806d5578fbfdc9212f32d2e9dd41998d0a8aaf830c9
# all of these URLs are candidates and may be tried in any order
source=https://www.example.com/file1.c
source=https://www.example.de/file.c
source=ftp://ftp.example.ch/srv/file_1.c
# sourcing files in archives uses the # symbol like DOM IDs
source=https://cdn.example.com/package.tar.gz#/subdir/file.c

A brief list of rules observed by these INIs (consult the schema for a precise specification):

1. all key–value pairs must appear under a section heading (there is no “global section”)
2. only ASCII is permitted, except within comments, where the high bit may be set to passively allow UTF-8
3. all URIs must be valid UTF-8 and legalised into ASCII via percent-encoding
4. file names are specified as the contents of the section name in whole
5. file names cannot constitute paths; no directory component is permitted

Given an INI like the one above, Earthbound offers several benefits:

1. it will select the source URL using arbitrary algorithms which take advantage of the lack of ordering or precedence to choose the best download site
2. when multiple source files may download from the same URL sans the # suffix, the package only needs to be downloaded once
3. a project config can cause the download of only the files it needs without requiring cumbersome and error-prone post-operative scripts

URI addressing schemes

Earthbound’s agnosticism makes network diversity a central feature to offer. To this end, some of the schemes listed by IANA have been singled out for explicit support, and a few others are also specifically excluded from future support. Explicit support is (hopefully) self-explanatory, while excluded protocols come with a rationale. Schemes not mentioned may still be supported by third-party implementations and may be discussed in the course of development.

Explicit support (all implementations MUST):

1. cvs
2. ftp
3. git
4. gemini
5. gopher
6. http
7. magnet
8. news
9. nntp
10. rsync
11. ssh
12. svn
13. telnet

Excluded protocols (all implementations MUST NOT):

1. blob – no embedding payloads
2. cid – content addressing is incoherent to the tool
3. data – see rationale against blob
4. doi – no mechanism for binary interchangeability
5. file – drop the schema altogether instead
   • in any case, absolute paths are forbidden
6. ipfs – see rationale against cid
7. nfs – see rationale against file
8. urn – see rationale against cid

Features

For the sake of portability, Earthbound defers the heavy tasks of downloading files and extracting archives to the user’s system. Here is a table of supported downloader programs, the names they are expected to be found under in the $PATH, and the environment variable Earthbound will allow them to be overridden with:

Utility	Default	Override
curl	curl	$EB_CURL
wget	wget	$EB_WGET
CVS	cvs	$EB_CVS
Git	git	$EB_GIT
aria2	aria2c	$EB_ARIA2
rsync	rsync	$EB_RSYNC
OpenSSH	ssh	$EB_SSH
SVN	svn	$EB_SVN
Telnet	telnet	$EB_TELNET

Here is a table of supported archive extraction tools and decompression tools, the names they are expected to be found under in the $PATH, and the environment variable Earthbound will allow them to be overridden with:

Utility	Default	Override
7zip	7z	$EB_7Z
GNU tar	tar1	$EB_TAR
Unzip	unzip	$EB_UNZIP
Gzip	gzip	$EB_GZIP
Bzip2	bzip2	$EB_BZIP2
Lzip	lzip2	$EB_LZIP
Lrzip	lrzip	$EB_LRZIP
Rzip	rzip	$EB_RZIP
Xz	xz	$EB_XZ
LZ4	lz4	$EB_LZ4
LZO	lzop	$EB_LZOP
Zstd	zstd	$EB_ZSTD

Earthbound expects to be compiled and run on an operating system that complies with POSIX.1-2001. It will use methods made available by such compliance to invoke external programs, get environment variable data and perform file system operations.

Algorithms and dependencies

In general, Earthbound implementations will need to facilitate both SHA-2 256-bit and SHA-3 256-bit for the purposes of validating hashes of incoming source files. In the interest of security, it is strongly recommended that implementations have these utilities securely integrated into the program, either directly as the reference implementation does here, or via a package manager with the usual strong security guarantees. That said, it is possible to outsource it to the wider system as is done with the archive extraction utilities and the resource fetcher utilities.

The spirit of Earthbound is one of absolutely minimalist dependency usage, since its job is to bootstrap dependencies themselves and it is not designed to rely on itself in doing so. Implementations should not rely on anything not totally ubiquitous for a given target platform, although judgement of that is necessarily left up to the implementor.

Footnotes

1. will look for gtar on macOS hosts, falling back to BSD tar. ↩

2. will prefer plzip with -n $(nproc) if available. ↩