707843 - [Tracking bug] integrate s3/glacier with ftp/stage as reliable and expandable storage for RelEng build artifacts

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Aki Sasaki (not active)]

Once we have a good description written.

Comment 1

[Aki Sasaki (not active)]

Current usage + free space:

10.253.0.10:/vol/ftp_stage
                      6.1T  5.2T  826G  87% /mnt/netapp/stage
10.253.0.11:/vol/stage
                      3.9T  3.7T  223G  95% /mnt/netapp/stage/archive.mozilla.org/pub/firefox
/mnt/cm-ixstore01/tinderbox-builds/tinderbox-builds
                       17T  3.4T   14T  21% /mnt/netapp/stage/archive.mozilla.org/pub/firefox/tinderbox-builds/old

Comment 2

[Aki Sasaki (not active)]

John wanted this.
I'm willing to help with the wording / request if wanted.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

followup from discussions w/jake, cshields, catlee (and some others)

1) capacity requirements increasing because of v.large ASAN builds (>300MB per), and new requests to increase retention of per-checkin builds from 30days to 

2) Instead of having all builds stored on an increasingly-large-and-hard-to-upgrade netapp, we discussed some alternatives to dynamically grow as needed. We believe it should be possible to:
a) have newest builds on netapp, for fastest access during builds/tests
b) age older builds to s3, for less frequent access 
c) age oldest builds to glacier, for long term storage.

...but thats in theory. We need to try this to find out!

This bug to track this setup and experiment, and if successful, to track automation changes needed for age-based migrations.

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

assigning to coop after quick chat in person.

Comment 5

[Chris Cooper [:coop] (he/him)]

I'm on buildduty this week, but I'll plan to sync up with nthomas early next week about this.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Current status - working on bug 765258 (keeping a whole cycle of dep builds)
* upload is progressing, we'll have the whole of the Firefox 24 cycle. 12 days done so far, about a month's worth still to upload (ETA a few days)
* investigating ways to serve that which are no-op changes for existing scripts (is or looks like apache dir listings)
 * s3fs, s3fuse - both seem too slow when apache is in front of them
 * static index.html files
 * write a wsgi app, which uses boto to get key lists and create listings on the fly

Comment 7

[(not currently active) Ted Mielczarek]

Low-tech static index.html files sound pretty compelling, given that this data won't actually change once archived...

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

The painful bit is that when directories get expired off the index.html in some dirs will get out of date, so lets try the wsgi I wrote and see how the perf is.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Bug 765258 tracks keeping important mozilla-inbound dep builds for 24 weeks (a full development cycle), and is the extent of this bug covered by a Q3 goal

Comment 10

[Chris AtLee [:catlee]]

We've got this up and running at the following URL. If there are any issues, please file bugs blocking this one.

http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/

Comment 11

[[e-mail] (:bhearsum)]

Seems like this is in progress finally :)

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Between bug 765258 and the recent project to move ftp.m.o to use S3, this is superceded/done.