Upload buildbot logs via the new upload hosts

RESOLVED FIXED

Status

RESOLVED FIXED
3 years ago
7 months ago

People

(Reporter: nthomas, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

Comment hidden (empty)
(Reporter)

Comment 1

3 years ago
Created attachment 8672551 [details] [diff] [review]
[puppet] New hosts & tidy up

* swap to new upload hosts
* drops dead gaia-try and shadow-central, xulrunner
* for thunderbird try, makes a guess which way bug 1213723 will go

Holding on review until we know more.
(Reporter)

Comment 2

3 years ago
Comment on attachment 8672551 [details] [diff] [review]
[puppet] New hosts & tidy up

Bug 1213723 is still pending, but this looks good in staging for everything except try-comm-central.

Deployment of this might be an interesting problem. It's going to go something like this
* oremj deploys cloudfront change, directing requests for firefox, mobile etc to S3 instead of SCL3. Propagation time is ~20 min
* we land this patch in puppet, and masters pick it up within 30 minutes (unless we force). All logs from that point will be uploaded to S3 instead of SCL3
* logs that get uploaded to stage will report locations with ftp.m.o in them, which will point to SCL3 before the transition and S3 afterwards. After we swap they'll be archive.mozilla.org
* we're keep syncing SCL3 -> S3 until all the jobs with old config finish

So there's a couple of things that will go wrong here
* upload to stage after cloudfront switches - log is missing until sync picks it up
* upload to S3 before cloudfront switches - log is missing until propagation is done

We might just have to put up with a period where treeherder can't get logs. emorley, what's the best bet here ?
Flags: needinfo?(emorley)
Attachment #8672551 - Flags: review?(rail)
I'm not sure I understand what "uploaded to stage" means? Do you meaning a staging instance or is that the name of another downloads area?

Treeherder doesn't automatically retry 404 or 403 logs (https://github.com/mozilla/treeherder/blob/master/treeherder/log_parser/utils.py#L101) since it commonly happens (at least in the past, not sure now) with TaskCluster. However when users click the job in the UI it should initiate a repeat parse, so the impact shouldn't be too bad.
Flags: needinfo?(emorley)
(Reporter)

Comment 4

3 years ago
Sorry, I meant stage.mozilla.org, the soon-to-be-deprecated upload host. Sounds like repeat parse will save the day, thanks.

Updated

3 years ago
Attachment #8672551 - Flags: review?(rail) → review+
(Reporter)

Comment 6

3 years ago
Created attachment 8676505 [details] [diff] [review]
[puppet] Fix b2g

We still need to upload b2g logs via stage.mozilla.org. It will transition later.
Attachment #8676505 - Flags: review?(hwine)
Comment on attachment 8676505 [details] [diff] [review]
[puppet] Fix b2g

lgtm!
Attachment #8676505 - Flags: review?(hwine) → review+
(Reporter)

Comment 9

3 years ago
We also had some problems uploading logs for mobile builds on try, when s3://net-mozaws-prod-delivery-archive/pub/mobile/try-builds/ wasn't writeable from upload.trybld.productdelivery.prod.mozaws.net. oremj added new policy to allow that, but we'll have to loop around and figure out the proper fix. On stage.mozilla.org we had a symlink from mobile/trybuilds to ../firefox/try-builds.
(Reporter)

Updated

3 years ago
Depends on: 1216788
(Reporter)

Comment 10

3 years ago
Bug 1217742 and bug 1217223 were also relevant here.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Assignee)

Updated

7 months ago
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.