1283212 - Need NFS mount for Mercurial bundles

Status: RESOLVED FIXED

(Infrastructure & Operations :: Storage, task)

Description

[Gregory Szorc [:gps]]

The new hgssh machines being installed in bug 1167935 don't have enough local disk to hold the "Mercurial bundle files" we generate via CRON once a day. (A list of the files we generate is available at https://hg.cdn.mozilla.net/).

We previously had the bundles stored on the /hg/hg_qtree volume on the Netapp. But this caused chaos with snapshotting which we worked around by moving the bundles to local disk. See bug 1167935.

Since our local disk isn't large enough to hold the bundles, we need a way to use remote storage (I assume Netapp+NFS).

Perhaps we could get a new volume that isn't snapshotted (or backed up) and doesn't suffer from the issues in bug 1167935?

A 128 GB volume should be sufficient to hold the hg bundles. It should be mountable from hgssh[45].dmz.scl3.mozilla.com.

Comment 1

[Greg Cox [:gcox]]

Per an irc conversation, we're doing this as a junction mount on the filer.  The normal mount is /hg/hg_qtree, but now there's a subvolume /hg/hg_qtree/bundles which is without snapshots.

Lemme know if there's anything else needed here.

Comment 2

[Gregory Szorc [:gps]]

I'm already using the new subvolume from hgssh4 and it seems to be working great! I'll let you know if I encounter any issues.

I think the only thing left to do is ensure /etc/fstab on hgssh[12345] have this new mount. Then we can close.

Comment 3

[Greg Cox [:gcox]]

It's a submount on the filer side, and it doesn't need an fstab entry.  "For all you know" it's just a directory.

Comment 4

[Gregory Szorc [:gps]]

Oh, I was surprised to see the mount show up as a different entity in `df` and `mount` without requiring an explicit fstab entry! But I confirmed from rebooting hgssh5 that the mount magically appears without the fstab entry. So I think we're all good here!

Thanks for addressing this promptly: it allowed us to transition some services to the new hgssh hardware today!

Comment 5

[Rob Tucker [:rtucker]]

(In reply to Gregory Szorc [:gps] (conference/holiday until October 17) from comment #4)
> Oh, I was surprised to see the mount show up as a different entity in `df`
> and `mount` without requiring an explicit fstab entry! But I confirmed from
> rebooting hgssh5 that the mount magically appears without the fstab entry.
> So I think we're all good here!
> 
> Thanks for addressing this promptly: it allowed us to transition some
> services to the new hgssh hardware today!

Per comment #1, :gcox states that there are no snapshots, I also wanted to confirm that I'm not backing these up via bacula. Unless there is a really strong reason to backup these bundles, I'd prefer to not burn ~90GB of storage per night.

Comment 6

[Kendall Libby [:fubar] (he/him)]

(In reply to Rob Tucker [:rtucker] from comment #5)
> 
> Per comment #1, :gcox states that there are no snapshots, I also wanted to
> confirm that I'm not backing these up via bacula. Unless there is a really
> strong reason to backup these bundles, I'd prefer to not burn ~90GB of
> storage per night.

thanks for the explicit call out. confirming that no snapshots and no backups of the bundle mount is fine; the bundles are generated nightly via cron, so there's little point in either.