Bind Mount On Ephemeral Sometimes Causes Mount To Be Ignored On Reboot

[XaverStiensmeier]

Currently, when you bind mount the ephemeral this rarely leads to the ephemeral not being mounted on reboots. This happens because of a circular dependency problem:

May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: mnt.mount: Job dev-vdb.device/start deleted to break ordering cycle starting with mnt.mount/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: mnt.mount: Found dependency on mnt.mount/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: mnt.mount: Found dependency on vol-scratch.mount/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: mnt.mount: Found dependency on local-fs.target/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: mnt.mount: Found dependency on systemd-tmpfiles-setup.service/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: mnt.mount: Found dependency on vgauth.service/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: mnt.mount: Found dependency on open-vm-tools.service/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: mnt.mount: Found dependency on cloud-init-local.service/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: mnt.mount: Found dependency on cloud-init.service/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: mnt.mount: Found dependency on network-online.target/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: mnt.mount: Found ordering cycle on dev-vdb.device/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: remote-fs-pre.target: Job open-iscsi.service/start deleted to break ordering cycle starting with remote-fs-pre.target/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: remote-fs-pre.target: Found dependency on remote-fs-pre.target/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: remote-fs-pre.target: Found dependency on mnt.mount/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: remote-fs-pre.target: Found dependency on vol-scratch.mount/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: remote-fs-pre.target: Found dependency on local-fs.target/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: remote-fs-pre.target: Found dependency on systemd-tmpfiles-setup.service/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: remote-fs-pre.target: Found dependency on vgauth.service/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: remote-fs-pre.target: Found dependency on open-vm-tools.service/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: remote-fs-pre.target: Found dependency on cloud-init-local.service/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: remote-fs-pre.target: Found dependency on cloud-init.service/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: remote-fs-pre.target: Found dependency on network-online.target/start
May 07 14:22:41 bibigrid-worker-wuan1ty32nih3ju-0 systemd[1]: remote-fs-pre.target: Found ordering cycle on open-iscsi.service/start

Systemd solves this cycle by ignoring one of the cycling services. This can result in the mount not being executed. It does not happen 100% of the time. Here yo usee how sometimes vdb is not mounted correctly on reboot:

ubuntu@bibigrid-master-wuan1ty32nih3ju:~$ for i in $(seq 1 20); do echo "Iteration $i"; ssh $(bibiname 0) "df /mnt/ && sudo reboot"; sleep 10; until ssh $(bibiname 0) "echo up" 2>/dev/null; do sleep 5; done; done
Iteration 1
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/vdb        51290592   336  48652432   1% /mnt
up
Iteration 2
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/vdb        51290592   336  48652432   1% /mnt
up
Iteration 3
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vda1       49691512 4466816  45208312   9% /
up
Iteration 4
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/vdb        51290592   336  48652432   1% /mnt
up
Iteration 5
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/vdb        51290592   336  48652432   1% /mnt
up
Iteration 6
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/vdb        51290592   336  48652432   1% /mnt
up
Iteration 7
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vda1       49691512 4516024  45159104  10% /
up
Iteration 8
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vda1       49691512 4528288  45146840  10% /
up
Iteration 9
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/vdb        51290592   336  48652432   1% /mnt
up
Iteration 10
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/vdb        51290592   336  48652432   1% /mnt
sudo: unable to resolve host bibigrid-worker-wuan1ty32nih3ju-0: Temporary failure in name resolution
up
Iteration 11
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/vdb        51290592   336  48652432   1% /mnt
up
Iteration 12
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/vdb        51290592   336  48652432   1% /mnt
up
Iteration 13
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/vdb        51290592   336  48652432   1% /mnt
up
Iteration 14
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vda1       49691512 4601992  45073136  10% /
up
Iteration 15
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vda1       49691512 4614288  45060840  10% /

This dependency problem can be solved by:

• deactivating _netdev (waiting for network) for the ephemeral mount
• adding _netdev to the bind mount (not 100% sure)
• add a script at the end of the boot that executes mount -a to ensure mounting
• removing the bind mount

This is not a BiBiGrid specific bug, but BiBiGrid needs to decide how to handle it.