Closed Bug 1117960 Opened 5 years ago Closed 5 years ago

Switch testers to download from s3 instead of ftp.

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(firefox39 fixed)

RESOLVED FIXED
Tracking Status
firefox39 --- fixed

People

(Reporter: mshal, Assigned: mshal)

References

Details

Attachments

(2 files, 1 obsolete file)

Once build artifacts are in s3 (bug 1100624), we'll want to have testers pull data from s3 instead of ftp. The initial target is to support trunk & try builds.
Depends on: 1100624
Attached patch s3-testsSplinter Review
This still needs the dependent bug to land and stick in mozharness before it can land, but it seems to work fine on try.

I've just duplicated the annoying property_conditions for now. This set will be able to replace the ones in m-c/build/gen_mach_buildprops.py. The other sets in mozharness (MakeUploadOutputParser) and buildbotcustom (parse_make_upload) can hopefully be removed as things are converted mach+mozharness and take advantage of the S3 uploading.

I think I would prefer avoiding this silly pattern matching to determine the property type though, and somehow expose that from the build system into mach_build_properties.json - if you have any thoughts on that let me know.

In the try push I've just launched a single test and talos job to verify that the files are pulled from S3 instead of ftp.m.o:
try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8c0735a40931
Attachment #8560689 - Flags: review?(jlund)
Comment on attachment 8560689 [details] [diff] [review]
s3-tests

Review of attachment 8560689 [details] [diff] [review]:
-----------------------------------------------------------------

this is awesome. I see you took my painful logic in MakeUploadOutputParser and made it better! Though I agree it is still a pain. maybe you would be interested in talking with myself and nick next week about improving that: https://bugzilla.mozilla.org/show_bug.cgi?id=1122509#c3
Attachment #8560689 - Flags: review?(jlund) → review+
See Also: → 1122509
(In reply to Jordan Lund (:jlund) from comment #2)
> this is awesome. I see you took my painful logic in MakeUploadOutputParser
> and made it better! Though I agree it is still a pain. maybe you would be
> interested in talking with myself and nick next week about improving that:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1122509#c3

Definitely - let me know when you guys are meeting and I'll join in.
Depends on: 1121293
:jonasfj recommended waiting until the TC queue rewrite is complete (bug 1121293) before starting to use the files from s3 for testers & other things.
Depends on: 1133876, 1133856
We'll also want to spruce up the python client error handling before switching testers over. I hit an issue where a tests.zip file wasn't uploaded correctly. We suspect that the upload died part way through, and then retries failed since the file-handle wasn't reset (bug 1133876). Even if a retry did succeed, it looks like it would have retried several times unnecessarily (bug 1133856).
Depends on: 1135681
Depends on: 1135756
bump mozharness rev, this time with a production revision.

Note I haven't tried this one on try, but the previous default rev looks good: https://treeherder.mozilla.org/#/jobs?repo=try&revision=53caf322caad
Attachment #8581782 - Attachment is obsolete: true
Attachment #8581782 - Flags: review?(jlund)
Attachment #8581914 - Flags: review?(jlund)
Comment on attachment 8581914 [details] [diff] [review]
0001-Bug-1117960-bump-mozharness.json-to-rev-93f17b170f43.patch

Review of attachment 8581914 [details] [diff] [review]:
-----------------------------------------------------------------

thanks for changing this to a prod rev :)
Attachment #8581914 - Flags: review?(jlund) → review+
https://hg.mozilla.org/mozilla-central/rev/50d681158e9f
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.