859900 - FTP server NFS mount changes

Status: RESOLVED FIXED

(Infrastructure & Operations :: Change Requests, task)

Description

[Jake Maul [:jakem]]

4 directories on the FTP cluster will be replaced by independent NetApp NFS mount points (see bug 858609 for details).

Affected directories are:
thunderbird
seamonkey
xulrunner
b2g

We'll do these one at a time, over the course of a few days/weeks.

These do have the potential to burn tests for the affected platforms/branches while the change is in progress. For example over the weekend we changed "mobile", and this burned some Android 2.2 tests.

This affects ftp[1-7].dmz.scl3, upload1.dmz.scl3, and upload-cron.private.scl3.

There is a small likelihood that some users will experience 404's when downloading an update or installer in these directories during each of these changeovers. This should be a very small % of overall traffic, given the usage and typical traffic patterns (Play Store, Zeus/CDN caching, silent update mechanisms).

precise date/time TBD... night, preferably weekend.

:jakem is point, :gcox and SREs to assist if needed. Will coordinate with sheriff and/or buildduty folks in #infra and #releng.

Etherpad started here https://infra.etherpad.mozilla.org/ftp-cluster-nfs-upgrades to plan/track the work.


We have to go on this for at least 1 or 2 of the volumes listed above *before* the b2g work week next week, else we risk running out of space during it.

Comment 1

[Shyam Mani [:fox2mike]]

Jake,

Please fill out additional details as requested in : https://wiki.mozilla.org/IT/ChangeControl#Submitting_a_Change_Request (since the etherpad is empty)

Thanks!

Also, if this needs to be done _before_ the b2g work week (which as I understand is next week) I hope you can make this week's CAB meeting, which is today at 0900 PDT, mrz's Vidyoroom)

Comment 2

[Shyam Mani [:fox2mike]]

Approved for urgent work for the following mounts to be moved *before* the b2g work week on a time/date most suitable for everyone involved :

1) seamonkey
2) xulrunner

This will most probably be complete by Friday, 12 April at which point the CAB can review for changes still left to do. Therefore, leaving the flag at ? as this is ongoing work.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Shyam Mani [:fox2mike] from comment #2)
> Approved for urgent work for the following mounts to be moved *before* the
> b2g work week on a time/date most suitable for everyone involved :
> 
> 1) seamonkey
> 2) xulrunner
> 
> This will most probably be complete by Friday, 12 April at which point the
> CAB can review for changes still left to do. Therefore, leaving the flag at
> ? as this is ongoing work.

cc-ing callek on behalf of seamonkey. Who should be cc'd on behalf of xulrunner?

Comment 4

[Shyam Mani [:fox2mike]]

(In reply to John O'Duinn [:joduinn] from comment #3)

> cc-ing callek on behalf of seamonkey. Who should be cc'd on behalf of
> xulrunner?

Was hoping you/hal might be able to help with that. I'll try to search Bugzilla as well.

Comment 5

[Justin Wood (:Callek)]

Per IRC convo and e-mail ben (and by extension me if ben takes off) are on the hook for being around for xulrunner on friday.

Comment 6

[Jake Maul [:jakem]]

Not going to get to xulrunner today. Still working on Seamonkey, delayed in part due to my own stupidity, and in part due to a very slow copy.

However, it won't matter... we found a good chunk of space tied up in old b2g tinderbox-build directories that needed to be purged. We're currently sitting on 3.3TB of available space. :)

Still hoping to get Seamonkey done today, which is another ~2.3TB or so.

Comment 7

[Jake Maul [:jakem]]

Seamonkey is done. Putting off xulrunner and the others until after b2g work week. We're sitting at 3.7TB available right now, and I'm expecting to get at least another TB free soon... it's still deleting the old copy of the seamonkey directory.

Comment 8

[[e-mail] (:bhearsum)]

(In reply to Justin Wood (:Callek) from comment #5)
> Per IRC convo and e-mail ben (and by extension me if ben takes off) are on
> the hook for being around for xulrunner on friday.

I won't be around on Friday for XULRunner, but I also told Jake that nobody needs to be around for it. My understanding is that there will be ~10min of unavailability, and XULRunner builds only run early in the day, so they won't be affected by this. If something goes terribly wrong I can be called.

Comment 9

[Shyam Mani [:fox2mike]]

Jake is point on any further changes needed here. The CAB will be happy to help, if need be, but I think Jake's got this™ ;)

Comment 10

[Jake Maul [:jakem]]

Closing this bug out. Work is being tracked in other bugs, this one was just to alert the CAB.

Additional changes are still on the table, but have been back-burnered due to lack of need. We found out where the space was going and cleaned some stuff up. Between that and moving a couple sites, we're much better off now than before.