1133638 - Modify upload[12].dmz.scl3 tmp disk to avoid snapshot issues

Status: RESOLVED FIXED

(Infrastructure & Operations :: Virtualization, task, P4)

Description

[Greg Cox [:gcox]]

What:
upload1.dmz.scl3 threw a minor alert tonight, needing disk consolidation in vmware.  This happens when data changes so rapidly that some of our backup processes can't heal themselves.  I suspect with the tree closures/impending release/stackup of builds we tripped over a high-work period.

When we virtualized the upload servers back in bug 1061879, I broke /tmp out into a separate disk because we were flirting with the idea of /tmp filling up with upload data and threatening to clog/kill the root disk.  I didn't think about this being transient data, so I made it as a normal disk, which is getting snapshotted.

There's no point in snapshotting the /tmp disk (and multiple reasons not to) but to change this we'll need to reboot the VM and change /tmp to an independent/not-nightly-snapshotting disk, which we could do with failover to upload2 if this were a big deal, but that's a burp so better to drift this to a quiet time.

To be clear, I'm ONLY talking about changing this on /tmp.  The root/OS disk will still get backups.


When:          next TCW, 30 mins.
Affected:      Builds, upload[12].dmz.scl3
Impact:        Disruption in uploading completed builds
Rollback plan: Reboot, undo change
Notif:         Usual TCW notifs.
Point:         someone from virtualization

Comment 1

[Greg Cox [:gcox]]

Both upload boxes rebooted, /tmp set to independent+persistent, and booted back cleanly.