Closed Bug 1213782 Opened 9 years ago Closed 8 years ago

Upload buildbot logs via the new upload hosts

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nthomas, Unassigned)

References

Details

Attachments

(2 files)

      No description provided.
* swap to new upload hosts
* drops dead gaia-try and shadow-central, xulrunner
* for thunderbird try, makes a guess which way bug 1213723 will go

Holding on review until we know more.
Comment on attachment 8672551 [details] [diff] [review]
[puppet] New hosts & tidy up

Bug 1213723 is still pending, but this looks good in staging for everything except try-comm-central.

Deployment of this might be an interesting problem. It's going to go something like this
* oremj deploys cloudfront change, directing requests for firefox, mobile etc to S3 instead of SCL3. Propagation time is ~20 min
* we land this patch in puppet, and masters pick it up within 30 minutes (unless we force). All logs from that point will be uploaded to S3 instead of SCL3
* logs that get uploaded to stage will report locations with ftp.m.o in them, which will point to SCL3 before the transition and S3 afterwards. After we swap they'll be archive.mozilla.org
* we're keep syncing SCL3 -> S3 until all the jobs with old config finish

So there's a couple of things that will go wrong here
* upload to stage after cloudfront switches - log is missing until sync picks it up
* upload to S3 before cloudfront switches - log is missing until propagation is done

We might just have to put up with a period where treeherder can't get logs. emorley, what's the best bet here ?
Flags: needinfo?(emorley)
Attachment #8672551 - Flags: review?(rail)
I'm not sure I understand what "uploaded to stage" means? Do you meaning a staging instance or is that the name of another downloads area?

Treeherder doesn't automatically retry 404 or 403 logs (https://github.com/mozilla/treeherder/blob/master/treeherder/log_parser/utils.py#L101) since it commonly happens (at least in the past, not sure now) with TaskCluster. However when users click the job in the UI it should initiate a repeat parse, so the impact shouldn't be too bad.
Flags: needinfo?(emorley)
Sorry, I meant stage.mozilla.org, the soon-to-be-deprecated upload host. Sounds like repeat parse will save the day, thanks.
Attachment #8672551 - Flags: review?(rail) → review+
Attached patch [puppet] Fix b2gSplinter Review
We still need to upload b2g logs via stage.mozilla.org. It will transition later.
Attachment #8676505 - Flags: review?(hwine)
Comment on attachment 8676505 [details] [diff] [review]
[puppet] Fix b2g

lgtm!
Attachment #8676505 - Flags: review?(hwine) → review+
We also had some problems uploading logs for mobile builds on try, when s3://net-mozaws-prod-delivery-archive/pub/mobile/try-builds/ wasn't writeable from upload.trybld.productdelivery.prod.mozaws.net. oremj added new policy to allow that, but we'll have to loop around and figure out the proper fix. On stage.mozilla.org we had a symlink from mobile/trybuilds to ../firefox/try-builds.
Depends on: 1216788
Bug 1217742 and bug 1217223 were also relevant here.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: