Created attachment 8672551 [details] [diff] [review] [puppet] New hosts & tidy up * swap to new upload hosts * drops dead gaia-try and shadow-central, xulrunner * for thunderbird try, makes a guess which way bug 1213723 will go Holding on review until we know more.
Comment on attachment 8672551 [details] [diff] [review] [puppet] New hosts & tidy up Bug 1213723 is still pending, but this looks good in staging for everything except try-comm-central. Deployment of this might be an interesting problem. It's going to go something like this * oremj deploys cloudfront change, directing requests for firefox, mobile etc to S3 instead of SCL3. Propagation time is ~20 min * we land this patch in puppet, and masters pick it up within 30 minutes (unless we force). All logs from that point will be uploaded to S3 instead of SCL3 * logs that get uploaded to stage will report locations with ftp.m.o in them, which will point to SCL3 before the transition and S3 afterwards. After we swap they'll be archive.mozilla.org * we're keep syncing SCL3 -> S3 until all the jobs with old config finish So there's a couple of things that will go wrong here * upload to stage after cloudfront switches - log is missing until sync picks it up * upload to S3 before cloudfront switches - log is missing until propagation is done We might just have to put up with a period where treeherder can't get logs. emorley, what's the best bet here ?
Attachment #8672551 - Flags: review?(rail)
I'm not sure I understand what "uploaded to stage" means? Do you meaning a staging instance or is that the name of another downloads area? Treeherder doesn't automatically retry 404 or 403 logs (https://github.com/mozilla/treeherder/blob/master/treeherder/log_parser/utils.py#L101) since it commonly happens (at least in the past, not sure now) with TaskCluster. However when users click the job in the UI it should initiate a repeat parse, so the impact shouldn't be too bad.
Sorry, I meant stage.mozilla.org, the soon-to-be-deprecated upload host. Sounds like repeat parse will save the day, thanks.
Attachment #8672551 - Flags: review?(rail) → review+
Comment on attachment 8672551 [details] [diff] [review] [puppet] New hosts & tidy up https://hg.mozilla.org/build/puppet/rev/0a5caeb217b5 https://hg.mozilla.org/build/puppet/rev/cc97b416245f
Attachment #8672551 - Flags: checked-in+
Created attachment 8676505 [details] [diff] [review] [puppet] Fix b2g We still need to upload b2g logs via stage.mozilla.org. It will transition later.
Attachment #8676505 - Flags: review?(hwine)
Comment on attachment 8676505 [details] [diff] [review] [puppet] Fix b2g lgtm!
Attachment #8676505 - Flags: review?(hwine) → review+
Comment on attachment 8676505 [details] [diff] [review] [puppet] Fix b2g https://hg.mozilla.org/build/puppet/rev/ec151da42a58 https://hg.mozilla.org/build/puppet/rev/07c547efcba4
Attachment #8676505 - Flags: checked-in+
We also had some problems uploading logs for mobile builds on try, when s3://net-mozaws-prod-delivery-archive/pub/mobile/try-builds/ wasn't writeable from upload.trybld.productdelivery.prod.mozaws.net. oremj added new policy to allow that, but we'll have to loop around and figure out the proper fix. On stage.mozilla.org we had a symlink from mobile/trybuilds to ../firefox/try-builds.
Bug 1217742 and bug 1217223 were also relevant here.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.