1329738 - /repo/hg/chroot_mozbuild/repo/hg/mozilla bind mount not mounted on startup

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Gregory Szorc [:gps]]

The hgweb machines have a bind mount of /repo/hg/mozilla into /repo/hg/chroot_mozbuild/repo/hg/mozilla. The mount is currently defined in Ansible as part of the hg-web role via the "shell" module invoking `mount`. As a result, the mount point isn't remounted after rebooting. This means the moz.build chroot environment requires an Ansible deploy after reboot before it starts working.

We should get this bind mount added to system startup. However, as the comment in the Ansible code explains, this may be more complicated than just an fstab entry:

# In order to get a read-only bind mount, we have to first do a regular
# bind mount then do a remount. We can't work this magic with the
# "mount" ansible module, so do it by hand.
- name: bind mount repositories into mozbuild chroot (servers only)
  shell: mount -o bind /repo/hg/mozilla /repo/hg/chroot_mozbuild/repo/hg/mozilla && mount -o remount,ro,bind /repo/hg/mozilla /repo/hg/chroot_mozbuild/repo/hg/mozilla creates=/repo/hg/chroot_mozbuild/repo/hg/mozilla/mozilla-central

needinfo fubar to see if he has any ideas.

Comment 1

[Gregory Szorc [:gps]]

I confirmed that executing just these `mount` commands after a reboot is sufficient for the chroot execution environment to work. So AFAICT this is the only thing preventing an hgweb machine from coming back from reboot and "just working."

Comment 2

[Kendall Libby [:fubar] (he/him)]

http://unix.stackexchange.com/questions/128336/why-doesnt-mount-respect-the-read-only-option-for-bind-mounts has a great rundown on this. It looks like we might actually be able to get away with `mount --bind,ro foo foo`; it's in the mount man page which corresponds to the fix in util-linux (https://git.kernel.org/cgit/utils/util-linux/util-linux.git/commit/?id=9ac77b8a78452eab0612523d27fee52159f5016a). 

It might also be possible to stack two fstab entries, though I'm wary of that:
/source/dir            /destination/dir    none  bind            0 0
/source/dir            /destination/dir    none  remount,bind,ro 0 0


If that doesn't work, would suggest going with a cheapo init.d (err, systemctl) script.

Comment 3

[Gregory Szorc [:gps]]

I added the dual fstab entry to hgweb11 before a reboot. While the bound mount on reboot worked, the remount as read-only failed: the mount was mounted rw after reboot.

So it looks like we may need a systemctl hack here :/

Comment 4

[Gregory Szorc [:gps]]

It turns out that systemd parses fstab and turns it into mounts automagically. See https://www.freedesktop.org/software/systemd/man/systemd.mount.html.

Further sleuthing of the internets reveals that systemd's fstab parser doesn't like the multiple fstab entries. So the hack to do a bind + remount,bind,ro only works without systemd (at least with the systemd currently installed on CentOS).

Furthermore, I couldn't find any evidence that systemd's mount units support a read-only bind mount. I did find evidence suggesting that invoking `mount` twice is the only way to do this.

Comment 5

[Gregory Szorc [:gps]]

https://blog.iwakd.de/systemd-fstab-and-bind-mounts-with-options has a write-up of this with long history. I'm going to implement it because this blocks me writing Ansible playbooks for doing a rolling reboot of the hg.mo servers.

Comment 6

[Gregory Szorc [:gps]]

Attachment: ansible/hg-web: systemd units to manage repos bind mount (bug 1329738);

Currently, we had to run an Ansible playbook after rebooting an hgweb
machine in order to mount the hg repos bind mount, which is used by
moz.build sandbox evaluation.

This is because we never took the time to figure out how to get a
read-only bind mount to work.

Getting a read-only bind mount to work is surprisingly difficult. Until
very recently, `mount` didn't support -o bind,ro: you had to do a bind
mount then `mount -o remount,bind,ro` to get a read-only bind mount.
Unfortunately, the version of mount on CentOS 7 doesn't yet support this
syntax.

A workaround is to have 2 lines in fstab, one with -o bind and a
subsequent with -o remount,bind,ro. Unfortunately, this doesn't
work with systemd! History of this is documented at
https://blog.iwakd.de/systemd-fstab-and-bind-mounts-with-options.

So, we have to resort to hacks.

We introduce a systemd mount unit that establishes a bind mount. That
is simple enough. But, the mount is read-write.

We also introduce a oneshot service unit that depends on the mount
unit. This service invokes `mount -o remount,bind,ro` to remount the
bind mount.

While we're here, we also configure the new service unit to create the
tmpfs mount for /dev inside the mount and to create a urandom character
file. Previously, this was done in Ansible and required manual action
after reboots.

With this change, the mount should come fully online after a reboot
without any Ansible invocation needed.

Review commit: https://reviewboard.mozilla.org/r/107240/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107240/

Comment 7

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8830481 [details]
ansible/hg-web: systemd units to manage repos bind mount (bug 1329738);

https://reviewboard.mozilla.org/r/107240/#review108404

Nice archaeology on that; it's a shame that we have to go this route, but nicely done.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e7c5874b3f21
ansible/hg-web: systemd units to manage repos bind mount ; r=fubar

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/516025e2ab43
ansible/hg-web: move dev tmpfs mount into own systemd unit file