Set up cron jobs to copy tinderbox-builds content to S3

RESOLVED WONTFIX

Status

Release Engineering
Other
P2
normal
RESOLVED WONTFIX
4 years ago
a year ago

People

(Reporter: nthomas, Assigned: nthomas)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Comment hidden (empty)
QA Contact: john+bugzilla
(Assignee)

Comment 1

3 years ago
yum says python-boto.noarch v2.9.5 is available, so if we install that with package, and a copy of the modified boto-rsync, then we can just write normal cron jobs.
Summary: Set up cron jobs to copy mozilla-inbound dep builds to S3 → Set up cron jobs to copy tinderbox-builds content to S3
(Assignee)

Comment 2

3 years ago
Currently using
while [ true ]; do (time for d in `ls -d /pub/mozilla.org/firefox/tinderbox-builds/* | egrep -v 'l10n|esr10|esr17|mozilla-b2g18_v1_0|mozilla-b2g18_v1_1|mozilla-b2g|old|projects|releases'` ;       do echo `date` $d; time boto-rsync-local -x '*/robocop.apk' -x '*/gecko-unsigned-unaligned.apk' -x '*/logs.tar.gz' -x '*/mar*' -x '*/mbsdiff*' -x '*/*.xpi' - -g public-read $d/ s3://firefox-mozilla-inbound-usw2/`basename $d`/; done; date) 2>&1 | tee -a ~/s3-logs/sync-firefox-`date +%Y%m%d-%H%M`.log;sleep 7200; done

while [ true ]; do (time for d in `ls -d /pub/mozilla.org/mobile/tinderbox-builds/* | egrep -v 'l10n|esr10|esr17|mozilla-b2g18_v1_0|mozilla-b2g18_v1_1|mozilla-b2g2old|projects|releases'` ;       do echo `date` $d; time boto-rsync-local -x '*/robocop.apk' -x '*/gecko-unsigned-unaligned.apk' -x '*/logs.tar.gz' -x '*/mar*' -x '*/mbsdiff*' -x '*/*.xpi' -x-g public-read $d/ s3://firefox-mozilla-inbound-usw2/`basename $d`/; done; date) 2>&1 | tee -a ~/s3-logs/sync-mobile-`date +%Y%m%d-%H%M`.log;sleep 7200; done

Will also need the python-argparse.noarch package, and to figure out how to handle the access key which is stashed at ~/.boto.
(Assignee)

Comment 3

3 years ago
Fixed commands:

while [ true ]; do (time for d in `ls -d /pub/mozilla.org/firefox/tinderbox-builds/* | egrep -v 'l10n|esr10|esr17|mozilla-b2g18_v1_0|mozilla-b2g18_v1_1|mozilla-b2g|old|projects|releases'` ;       do echo `date` $d; time boto-rsync-local -x '*/robocop.apk' -x '*/gecko-unsigned-unaligned.apk' -x '*/logs.tar.gz' -x '*/mar*' -x '*/mbsdiff*' -x '*/*.xpi' -g public-read $d/ s3://firefox-mozilla-inbound-usw2/`basename $d`/; done; date) 2>&1 | tee -a ~/s3-logs/sync-firefox-`date +%Y%m%d-%H%M`.log;sleep 7200; done

while [ true ]; do (time for d in `ls -d /pub/mozilla.org/mobile/tinderbox-builds/* | egrep -v 'l10n|esr10|esr17|mozilla-b2g18_v1_0|mozilla-b2g18_v1_1|mozilla-b2g2old|projects|releases'` ;       do echo `date` $d; time boto-rsync-local -x '*/robocop.apk' -x '*/gecko-unsigned-unaligned.apk' -x '*/logs.tar.gz' -x '*/mar*' -x '*/mbsdiff*' -x '*/*.xpi' -g public-read $d/ s3://firefox-mozilla-inbound-usw2/`basename $d`/; done; date) 2>&1 | tee -a ~/s3-logs/sync-mobile-`date +%Y%m%d-%H%M`.log;sleep 7200; done

They currently depend on a virtualenv at ~nthomas/venv/s3 being activated first (or replace boto-rsync-local with an abspath to that).
(Assignee)

Comment 4

a year ago
inbound-archive is defunct since the S3 migration, and the screen session has been disabled.
Status: ASSIGNED → RESOLVED
Last Resolved: a year ago
QA Contact: mshal
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.