etcfs-showcase-webapp

A wiki that runs on every node of an EtcFS cluster at the same time, so the cluster can be demonstrated rather than described.

It is stock Gollum — a git-backed wiki — pointed at a single git repository on the shared EBS io2 Multi-Attach volume. There is no replication and no primary. Every node opens the same repository, an Application Load Balancer round-robins across them with stickiness off, and the wiki is seeded with the EtcFS documentation, so the pages describing the filesystem are stored on the filesystem they describe.

Why Gollum

Git's concurrency safety rests on two filesystem primitives: lock files created with O_CREAT|O_EXCL (index.lock, refs/heads/master.lock) and rename() to publish a result. It does not use flock or fcntl locks — which matters, because cross-node advisory locking is the one POSIX surface EtcFS accepts but does not yet enforce.

In EtcFS both of those primitives are single etcd transactions: create is an atomic compare-and-swap that returns EEXIST to the loser, and rename commits as one transaction. Both are ordered by Raft, so they hold across nodes rather than merely across processes on one node. Git's existing lock discipline becomes cluster-wide without git knowing that a cluster exists — which is exactly the claim the showcase needs to make, demonstrated by software that was not written to make it.

The same workload on ext4 over a Multi-Attach volume mounted twice destroys the filesystem, which is what the EBS documentation warns about and the reason EtcFS exists.

What it adds to Gollum

Everything is loaded through gollum --config config.rb; no fork, no patches.

• Author attribution. A Rack middleware sets the commit author to the id of the node that accepted the edit, so git log is the write-propagation ledger, produced by git rather than by this application.
• A served-by bar. Every response carries x-etcfs-served-by and a footer naming the serving node. Reloading a page flips it while the content stays identical: that is write propagation in one line.
• Filesystem-level provenance. A post_commit hook stamps user.etcfs.writer_node onto each page file. Git would carry the same fact on any filesystem; the extended attribute is stored and read back through EtcFS, so a page edited on one node and read on another proves that EtcFS propagated the metadata.
• A health check that knows it is leaving. /showcase/health reports unhealthy once the Auto Scaling Group targets the node for termination, so the load balancer stops sending it traffic before the on-node leave watcher removes its etcd member, and unhealthy when the shared filesystem is not readable. etcd being unreachable is deliberately not a failure: such a node still serves every page, which is the behaviour the showcase exists to show.
• Live cluster macros. <<ClusterStatus()>>, <<WriteLedger(20)>> and <<PageProvenance()>> render etcd membership, the per-node write history and the recorded writer inside ordinary wiki pages.

Layout

config.rb                       Gollum configuration: middleware, hook, macros
lib/etcfs_showcase/             the application
bin/import-docs                 mkdocs tree -> Gollum pages
bin/showcase-packages.sh        OS packages (before the join)
bin/showcase-gems.sh            gollum and its gems (across the join)
bin/showcase-bootstrap.sh       wiki seed and systemd unit (after the join)
deploy/cloudformation/          the showcase stack
deploy/systemd/                 the unit file
wiki/                           seed pages: Home, the demo, the guestbook
test/                           unit tests

Deploying

deploy/cloudformation/etcfs-showcase-template.yml is a self-contained stack: etcfs-asg-template.yml from etcfs-cloudformation-template copied verbatim, with a load balancer and this application added. That repository's template is the one to deploy for anything real and is left untouched; this copy exists so the showcase owns its own deployment.

The join protocol is not duplicated by either. The copied user-data still fetches node-bootstrap.sh from etcfs-terraform-modules exactly as the cluster template does, and the showcase layer is appended after it — nothing about how a node joins is altered.

Boot ordering

Installing gollum compiles rugged, which builds libgit2 from source, and that is the longest step in a node's boot. It needs neither etcd nor the shared mount, so it runs in the background across the election stagger and the cluster join rather than after them.

The OS packages it needs are installed before that and synchronously, which is load-bearing: dnf holds an RPM lock, node-bootstrap.sh runs dnf three times of its own, and a compile-toolchain install racing those could outlast dnf's lock timeout and fail the join outright. Only the gem build, which takes no lock, is safe to overlap.

Being a copy, it goes stale. The template header records the upstream commit it was taken from; when the cluster template changes, re-copy it and re-apply the showcase additions (the parameters, the load balancer, the user-data layer and the WikiUrl output) rather than editing the cluster parts in place.

aws cloudformation deploy \
  --template-file deploy/cloudformation/etcfs-showcase-template.yml \
  --stack-name etcfs-showcase \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
      pVpcId=vpc-... pSubnetId=subnet-... \
      pLoadBalancerSecondSubnetId=subnet-... \
      pAvailabilityZone=eu-west-1a

The stack's WikiUrl output is the wiki. Note that it is open by default: anyone who can reach the load balancer can edit it, which is the point of the guestbook and a reason to narrow pWikiIngressCidr for anything but a demo.

Seeding

The wiki is seeded once per cluster, not once per node. Every node races to create /mnt/etcfuse/wiki/.seeded with O_CREAT|O_EXCL; through EtcFS that is a single etcd transaction, so exactly one node wins cluster-wide and the rest wait for its first commit. The election needs no coordinator beyond the filesystem — a small demonstration of the same property the wiki exists to show.

Running it locally

Gollum on a normal filesystem works fine for developing the app; only the cluster claims need a cluster.

gem install gollum
bin/import-docs ../etcfs-docs/docs /tmp/wiki
cp -r wiki/. /tmp/wiki/
git -C /tmp/wiki init -b master && git -C /tmp/wiki add -A && git -C /tmp/wiki commit -m seed
ETCFS_NODE_ID=dev gollum --config config.rb --ref master /tmp/wiki

Tests

ruby test/importer_test.rb
ruby test/cluster_test.rb