920885 - Set up cron jobs to copy tinderbox-builds content to S3

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P2)

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

yum says python-boto.noarch v2.9.5 is available, so if we install that with package, and a copy of the modified boto-rsync, then we can just write normal cron jobs.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Currently using
while [ true ]; do (time for d in `ls -d /pub/mozilla.org/firefox/tinderbox-builds/* | egrep -v 'l10n|esr10|esr17|mozilla-b2g18_v1_0|mozilla-b2g18_v1_1|mozilla-b2g|old|projects|releases'` ;       do echo `date` $d; time boto-rsync-local -x '*/robocop.apk' -x '*/gecko-unsigned-unaligned.apk' -x '*/logs.tar.gz' -x '*/mar*' -x '*/mbsdiff*' -x '*/*.xpi' - -g public-read $d/ s3://firefox-mozilla-inbound-usw2/`basename $d`/; done; date) 2>&1 | tee -a ~/s3-logs/sync-firefox-`date +%Y%m%d-%H%M`.log;sleep 7200; done

while [ true ]; do (time for d in `ls -d /pub/mozilla.org/mobile/tinderbox-builds/* | egrep -v 'l10n|esr10|esr17|mozilla-b2g18_v1_0|mozilla-b2g18_v1_1|mozilla-b2g2old|projects|releases'` ;       do echo `date` $d; time boto-rsync-local -x '*/robocop.apk' -x '*/gecko-unsigned-unaligned.apk' -x '*/logs.tar.gz' -x '*/mar*' -x '*/mbsdiff*' -x '*/*.xpi' -x-g public-read $d/ s3://firefox-mozilla-inbound-usw2/`basename $d`/; done; date) 2>&1 | tee -a ~/s3-logs/sync-mobile-`date +%Y%m%d-%H%M`.log;sleep 7200; done

Will also need the python-argparse.noarch package, and to figure out how to handle the access key which is stashed at ~/.boto.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Fixed commands:

while [ true ]; do (time for d in `ls -d /pub/mozilla.org/firefox/tinderbox-builds/* | egrep -v 'l10n|esr10|esr17|mozilla-b2g18_v1_0|mozilla-b2g18_v1_1|mozilla-b2g|old|projects|releases'` ;       do echo `date` $d; time boto-rsync-local -x '*/robocop.apk' -x '*/gecko-unsigned-unaligned.apk' -x '*/logs.tar.gz' -x '*/mar*' -x '*/mbsdiff*' -x '*/*.xpi' -g public-read $d/ s3://firefox-mozilla-inbound-usw2/`basename $d`/; done; date) 2>&1 | tee -a ~/s3-logs/sync-firefox-`date +%Y%m%d-%H%M`.log;sleep 7200; done

while [ true ]; do (time for d in `ls -d /pub/mozilla.org/mobile/tinderbox-builds/* | egrep -v 'l10n|esr10|esr17|mozilla-b2g18_v1_0|mozilla-b2g18_v1_1|mozilla-b2g2old|projects|releases'` ;       do echo `date` $d; time boto-rsync-local -x '*/robocop.apk' -x '*/gecko-unsigned-unaligned.apk' -x '*/logs.tar.gz' -x '*/mar*' -x '*/mbsdiff*' -x '*/*.xpi' -g public-read $d/ s3://firefox-mozilla-inbound-usw2/`basename $d`/; done; date) 2>&1 | tee -a ~/s3-logs/sync-mobile-`date +%Y%m%d-%H%M`.log;sleep 7200; done

They currently depend on a virtualenv at ~nthomas/venv/s3 being activated first (or replace boto-rsync-local with an abspath to that).

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

inbound-archive is defunct since the S3 migration, and the screen session has been disabled.