Improvements to upload make target

RESOLVED FIXED in Firefox 61

Status

enhancement
RESOLVED FIXED
Last year
8 months ago

People

(Reporter: gps, Assigned: gps)

Tracking

(Blocks 2 bugs)

3 Branch
mozilla61
Dependency tree / graph

Firefox Tracking Flags

(firefox61 fixed)

Details

Attachments

(11 attachments)

59 bytes, text/x-review-board-request
ted
: review+
Details
59 bytes, text/x-review-board-request
ted
: review+
Details
59 bytes, text/x-review-board-request
ted
: review+
Details
59 bytes, text/x-review-board-request
ted
: review+
Details
59 bytes, text/x-review-board-request
ted
: review+
Details
59 bytes, text/x-review-board-request
ted
: review+
Details
59 bytes, text/x-review-board-request
ted
: review+
Details
59 bytes, text/x-review-board-request
ted
: review+
Details
59 bytes, text/x-review-board-request
ted
: review+
Details
59 bytes, text/x-review-board-request
ted
: review+
Details
59 bytes, text/x-review-board-request
gps
: review+
mtabara
: review+
Details
The "upload" make target is taking a few minutes on Windows. We can do better.
Attachment #8969112 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969112 [details]
Bug 1455143 - Make hashlib required;

https://reviewboard.mozilla.org/r/237810/#review243576
Attachment #8969112 - Flags: review?(ted) → review+
Attachment #8969113 - Flags: review?(core-build-config-reviews)
Attachment #8969114 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969114 [details]
Bug 1455143 - Use a global logger instance;

https://reviewboard.mozilla.org/r/237814/#review243920
Attachment #8969114 - Flags: review?(ted) → review+
Attachment #8969116 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969116 [details]
Bug 1455143 - Remove code for failing to obtain a hash;

https://reviewboard.mozilla.org/r/237818/#review243964
Attachment #8969116 - Flags: review?(ted) → review+
Attachment #8969117 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969117 [details]
Bug 1455143 - Use a reasonable buffer size for reading files;

https://reviewboard.mozilla.org/r/237820/#review243966
Attachment #8969117 - Flags: review?(ted) → review+
Attachment #8969118 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969118 [details]
Bug 1455143 - Use .write() instead of print >>;

https://reviewboard.mozilla.org/r/237822/#review243968
Attachment #8969118 - Flags: review?(ted) → review+
Attachment #8969119 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969119 [details]
Bug 1455143 - Refactor checksumming to occur after upload.py;

https://reviewboard.mozilla.org/r/237824/#review243972
Attachment #8969119 - Flags: review?(ted) → review+
Attachment #8969115 - Flags: review?(core-build-config-reviews)
Attachment #8969113 - Flags: review?(core-build-config-reviews)
Attachment #8969114 - Flags: review?(core-build-config-reviews)
Attachment #8969112 - Flags: review?(core-build-config-reviews)
Attachment #8969440 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969440 [details]
Bug 1455143 - Don't remove common test archive;

https://reviewboard.mozilla.org/r/238194/#review243988
Attachment #8969440 - Flags: review?(ted) → review+
Attachment #8969441 - Flags: review?(core-build-config-reviews) → review?(ted)
Comment on attachment 8969441 [details]
Bug 1455143 - Generate test archives into final path;

https://reviewboard.mozilla.org/r/238196/#review243990

File a followup on fixing this for everything else? Aside from setup.exe, everything we pass in `UPLOAD_FILES` gets generated after the build.
Attachment #8969441 - Flags: review?(ted) → review+
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4c987b999ff1
Make hashlib required; r=ted
https://hg.mozilla.org/integration/autoland/rev/37fec6042438
Remove hash name validation; r=ted
https://hg.mozilla.org/integration/autoland/rev/e6d309fde362
Use a global logger instance; r=ted
https://hg.mozilla.org/integration/autoland/rev/2bbe78c011d3
Handle directory inputs earlier; r=ted
https://hg.mozilla.org/integration/autoland/rev/c9cbbb881c78
Remove code for failing to obtain a hash; r=ted
https://hg.mozilla.org/integration/autoland/rev/1f1186400490
Use a reasonable buffer size for reading files; r=ted
https://hg.mozilla.org/integration/autoland/rev/1fa5254b9a69
Use .write() instead of print >>; r=ted
https://hg.mozilla.org/integration/autoland/rev/12cfbcd2ccf4
Refactor checksumming to occur after upload.py; r=ted
https://hg.mozilla.org/integration/autoland/rev/b604acdb1c78
Don't remove common test archive; r=ted
https://hg.mozilla.org/integration/autoland/rev/7f69dd1ba3cb
Generate test archives into final path; r=ted
These changes broke the beetmover jobs[1] in the latest nightly. The reason is, the artifacts aren't produced anymore in the build task under the "multi locale" folder. Therefore, ChainOfTrust bails out as they don't exist.

My patch removes the artifacts offending artifacts. :gps, could you check I didn't miss one? :mtabara, this also removes the artifacts from the "ship" graphs, which is expected, I think. 

[1] Like https://tools.taskcluster.net/groups/FJ_74wd1Sl6jUpkObCBPGQ/tasks/Bph5uS4BQlinnQQ6snCcvg/runs/0/logs/public%2Flogs%2Fchain_of_trust.log#L219
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment on attachment 8969656 [details]
Bug 1455143 - Stop beetmoving deleted testing artifacts

https://reviewboard.mozilla.org/r/238450/#review244216

LGTM. I was gonna suggest a cleanup in beetmoverscript as well but I guess we still need these values within the templates mappings for the en-US counterparts.
Attachment #8969656 - Flags: review?(mtabara) → review+
Comment on attachment 8969656 [details]
Bug 1455143 - Stop beetmoving deleted testing artifacts

https://reviewboard.mozilla.org/r/238450/#review244282

Aside from robocop.apk removal, this seems reasonable.

FWIW, we plan to follow up and generate most artifacts in their final location. We'll try to keep this file in mind when we do that.

::: taskcluster/taskgraph/transforms/beetmover.py
(Diff revision 1)
>      "target.txt",
> -    "target.web-platform.tests.tar.gz",
> -    "target.xpcshell.tests.zip",
>      "target_info.txt",
> -    "mozharness.zip",
> -    "robocop.apk",

My changes shouldn't have touched robocop.apk.
Attachment #8969656 - Flags: review?(gps) → review+
Comment on attachment 8969656 [details]
Bug 1455143 - Stop beetmoving deleted testing artifacts

https://reviewboard.mozilla.org/r/238450/#review244540

::: taskcluster/taskgraph/transforms/beetmover.py
(Diff revision 1)
>      "target.txt",
> -    "target.web-platform.tests.tar.gz",
> -    "target.xpcshell.tests.zip",
>      "target_info.txt",
> -    "mozharness.zip",
> -    "robocop.apk",

Good catch. Fixed.
Pushed by jlorenzo@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d615164b2f6f
Stop beetmoving deleted testing artifacts r=gps,mtabara
https://hg.mozilla.org/mozilla-central/rev/d615164b2f6f
Status: REOPENED → RESOLVED
Closed: Last yearLast year
Resolution: --- → FIXED
Version: Version 3 → 3 Branch
You need to log in before you can comment on or make changes to this bug.