1233623 - Tear down inbound-archive

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We have a nice archive going back to mid-march 2015 at http://inbound-archive.pub.build.mozilla.org, which could be integrated with https://archive.mozilla.org. I'm calculating how much space that is now, for oremj's approval.

If we go ahead we'd copy the data over using S3 API calls, then make inbound-archive a CNAME that points to archive.m.o, and tear down the S3 bucket (firefox-mozilla-inbound-usw2) and Elastic Beanstalk environment.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

After trimming it down to 24 weeks (4 release cycles at 6 weeks each) we would have about 115TB of extra data, spread over 14M objects. That includes a discount for the 31 days of overlap leading up to the migration on ~ Oct 20.

If that's OK with oremj then catlee could organize access to the firefox-mozilla-inbound-usw2 bucket to start the copy. Please note that mappings would go like this:
* anything matching *android*
   firefox-mozilla-inbound-usw2:<dir>  -->  net-mozaws-prod-delivery-archive:pub/mobile/tinderbox-builds/<dir>
* everything else
   firefox-mozilla-inbound-usw2:<dir>  -->  net-mozaws-prod-delivery-firefox:pub/firefox/tinderbox-builds/<dir>

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

There are also objects for directories in that bucket which should be ignored, eg  alder-linux-debug/1439500790/. Maybe a test like size > 0 to exclude them.

Comment 3

[Jeremy Orem [:oremj]]

Travis is going to follow up in e-mail.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I think we shouldn't bother with this now.

Inbound-archive was keeping 168 days of builds, and currently has files from 03-Jul-2015 to 20-Oct-2015 (old files have been expiring, and we stopped syncing in when we switched to S3 in October).

Ideally archive.m.o would have the same retention, but has files from 20-Sep-2015 to today (some of these files have modification times of ~ 20-Oct-2015 due to syncing to S3).

So there's only a three week gap to backfill, and any lost-productivity is already done.

I'm going to tear down inbound-archive at the end of the month.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

CNAME changed from serves3-mozilla-inbound.elasticbeanstalk.com to archive.mozilla.org

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Turns out the bucketlister app behind archive.m.o doesn't work like this, and I don't think it's worth asking CloudOps to fix, so I've removed inbound-archive.pub.build.mozilla.org from DNS.

Cleaned up the Beanstalk env, removed s3serve from IAM. Still need to untangle the buckets and delete 'em.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Removed old and empty bucket from dev of the archive - mozilla-releng-tinderbox-builds
Removed logging bucket - firefox-s3-logging
Removed data bucket - firefox-mozilla-inbound-usw2

RIP.