Closed
Bug 1442793
Opened 6 years ago
Closed 6 years ago
support private artifacts for partner repacks
Categories
(Release Engineering :: Release Automation: Other, enhancement)
Release Engineering
Release Automation: Other
Tracking
(firefox61 fixed)
RESOLVED
FIXED
Tracking | Status | |
---|---|---|
firefox61 | --- | fixed |
People
(Reporter: mozilla, Assigned: mozilla)
References
Details
Attachments
(21 files)
14.51 KB,
patch
|
Details | Diff | Splinter Review | |
55 bytes,
text/x-github-pull-request
|
jlorenzo
:
review+
|
Details | Review |
678 bytes,
text/plain
|
Details | |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
59 bytes,
text/x-review-board-request
|
bhearsum
:
review+
|
Details |
No description provided.
Assignee | ||
Comment 1•6 years ago
|
||
Switch to signing-linux-dev on try, test artifact expiration: https://treeherder.mozilla.org/#/jobs?repo=try&revision=863c51eacf3c16c2bee8fe306f2fbf8ac076773c&selectedJob=165665179 Try to publish and sign private artifacts: https://treeherder.mozilla.org/#/jobs?repo=try&revision=43471c4d162970a19e8e7b207fd695d5baef132d
Assignee | ||
Comment 2•6 years ago
|
||
15:55 <aki> woo, try build publishing to aki-test/... https://tools.taskcluster.net/groups/fyuV8WLnTmqNQHTEodo2sQ/tasks/fdAiEEzlSFyfcfL4rnpAfA/runs/0/artifacts 16:00 <aki> woo! signed https://tools.taskcluster.net/groups/fyuV8WLnTmqNQHTEodo2sQ/tasks/Fi181XtXQuyKjzZk38lNiQ/runs/0/artifacts
Assignee | ||
Comment 3•6 years ago
|
||
Hopefully we don't have to do all of this. But I do think we should make public/build a task attribute default, and allow for overriding.
Assignee | ||
Comment 4•6 years ago
|
||
Thanks Johan! I think this will become scriptworker 10.0.0.
Attachment #8955731 -
Flags: review?(jlorenzo)
Updated•6 years ago
|
Attachment #8955731 -
Flags: review?(jlorenzo) → review+
Assignee | ||
Comment 5•6 years ago
|
||
Rolled out with https://hg.mozilla.org/build/puppet/rev/b9dd0bf8f210 . Next: docker-worker private artifact downloads, which may involve taskcluster-proxy.
Assignee | ||
Comment 6•6 years ago
|
||
works! https://tools.taskcluster.net/groups/K4OTA4xFQHWz4aHGEZjEFA/tasks/K4OTA4xFQHWz4aHGEZjEFA/runs/0/logs/public%2Flogs%2Flive.log the main thing for in-tree is `taskcluster-proxy: true`; then it's http://taskcluster/queue/...
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 17•6 years ago
|
||
Fixed the graph-generation exceptions. Still need to make sure the generated graph is correct, and that setting a non-public artifact_prefix works.
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 29•6 years ago
|
||
(In reply to Aki Sasaki [:aki] from comment #17) > Fixed the graph-generation exceptions. Still need to make sure the generated > graph is correct, and that setting a non-public artifact_prefix works. And finish writing the patch set.
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 73•6 years ago
|
||
Patch set is written to add the artifact_prefix attribute in the graph, and now the graph doesn't have a ton of other unexpected side effects... the only thing changed is toolchains, and not in their artifact_prefix. I now need to test to make sure that a) this is passed down appropriately when we set a non-public/build artifact_prefix, and b) whether we need any build system changes.
Assignee | ||
Comment 74•6 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b3c1bcda7f3d43168c703d92d0dce6e456caa016
Assignee | ||
Comment 75•6 years ago
|
||
The task definitions for the try push look good: aki-test is propagated in the build, build-signing, and xpcshell. Let's see how it goes.
Assignee | ||
Comment 76•6 years ago
|
||
setup-stub broke for some reason [!]. Also, the tests now need the queue:get-artifact:aki-test/* scope. Neither of these are blockers for partners, I don't think - we can keep the build publishing to public/build, and change the partner repack task to publish to releng/partners/.... Landed on maple; I'll see if partner private artifacts work there.
Assignee | ||
Comment 77•6 years ago
|
||
Maple: Landed some partner fixes here: https://hg.mozilla.org/projects/maple/pushloghtml?changeset=50f625050323 Ben previously landed https://hg.mozilla.org/projects/maple/rev/778a96269e846794df5439da14fa3658ef976d29 and https://hg.mozilla.org/projects/maple/rev/93de33f82693903769cc0b65b589ac08e996211b to fix partners after I landed my central-based artifact_prefix patch on maple.
Assignee | ||
Comment 78•6 years ago
|
||
https://hg.mozilla.org/projects/maple/rev/3da127c845b5694d8ccbe394d9feb3185de19d89 to allow repackage-partner to download private artifacts.
Assignee | ||
Comment 79•6 years ago
|
||
https://hg.mozilla.org/projects/maple/rev/ed81fa0029fa5a027b3d8caac1d19f204a31c225 to add taskcluster-proxy support to generic-worker task transforms. The docs say it's a thing: https://docs.taskcluster.net/reference/workers/generic-worker/docs/features#feature-taskclusterproxy
Assignee | ||
Comment 80•6 years ago
|
||
(In reply to Aki Sasaki [:aki] from comment #79) > https://hg.mozilla.org/projects/maple/rev/ > ed81fa0029fa5a027b3d8caac1d19f204a31c225 to add taskcluster-proxy support to > generic-worker task transforms. The docs say it's a thing: > https://docs.taskcluster.net/reference/workers/generic-worker/docs/ > features#feature-taskclusterproxy Pete - is taskcluster-proxy rolled out to generic workers? getting `[taskcluster:error] Could not start taskcluster proxy: exec: "taskcluster-proxy": executable file not found in %PATH%` https://tools.taskcluster.net/groups/a2x2127ZQt6tZYgbRkr2sA/tasks/HyLv2HeIT6-cunfK46duPg/details
Flags: needinfo?(pmoore)
Comment 81•6 years ago
|
||
(In reply to Aki Sasaki [:aki] from comment #80) > (In reply to Aki Sasaki [:aki] from comment #79) > > https://hg.mozilla.org/projects/maple/rev/ > > ed81fa0029fa5a027b3d8caac1d19f204a31c225 to add taskcluster-proxy support to > > generic-worker task transforms. The docs say it's a thing: > > https://docs.taskcluster.net/reference/workers/generic-worker/docs/ > > features#feature-taskclusterproxy > > Pete - is taskcluster-proxy rolled out to generic workers? getting > `[taskcluster:error] Could not start taskcluster proxy: exec: > "taskcluster-proxy": executable file not found in %PATH%` > https://tools.taskcluster.net/groups/a2x2127ZQt6tZYgbRkr2sA/tasks/HyLv2HeIT6- > cunfK46duPg/details Indeed taskcluster-proxy will need to be installed on gecko-3-b-win2012. I've updated bug 1449981 with details.
Flags: needinfo?(pmoore)
Comment 82•6 years ago
|
||
Ah wait - if you just need to download private artifacts and mount them into the task directory, you can do that without taskcluster-proxy. You just need to use the mounts feature of generic-worker. It will take care of downloading a private artifact for you (if the task has the required scopes) and will mount it into a path of your choosing inside the task directory. See the "mounts" property of the task payload.
Comment 83•6 years ago
|
||
If it is an archive (zip, tar.gz, etc) you have the choice whether to mount it as a file (and unpack it yourself) or to mount it as a read only directory (i.e. let generic-worker do the extraction for you). Both ways should work.
Comment 84•6 years ago
|
||
Of course you can use taskcluster-proxy if you want to, e.g. if you already have scripts to do so that you use with docker-worker - but if this is greenfield development, I'd recommend mounting the artifact using the mounts feature, as it is more explicit than downloading at runtime.
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment 103•6 years ago
|
||
mozreview-review |
Comment on attachment 8958297 [details] bug 1442793 - add artifact_prefix attribute. https://reviewboard.mozilla.org/r/227228/#review242570
Attachment #8958297 -
Flags: review?(bhearsum) → review+
Comment 104•6 years ago
|
||
mozreview-review |
Comment on attachment 8958298 [details] bug 1442793 - get_artifact_prefix and get_artifact_path. https://reviewboard.mozilla.org/r/227230/#review242572
Attachment #8958298 -
Flags: review?(bhearsum) → review+
Comment 105•6 years ago
|
||
mozreview-review |
Comment on attachment 8958299 [details] bug 1442793 - repackage_signing artifact_prefix. https://reviewboard.mozilla.org/r/227232/#review242574 ::: taskcluster/taskgraph/transforms/repackage_signing.py:100 (Diff revision 5) > > upstream_artifacts = [{ > "taskId": {"task-reference": "<repackage>"}, > "taskType": "repackage", > "paths": [ > - "public/build/{}target.complete.mar".format(locale_str), > + get_artifact_path(dep_job, "{}target.complete.mar".format(locale_str)), This shouldn't block anything, but I see changes like this to practically every transform that deals with artifacts. It makes me wonder if this should be handled in taskgraph/transforms/task.py instead - what do you think?
Attachment #8958299 -
Flags: review?(bhearsum) → review+
Comment 106•6 years ago
|
||
mozreview-review |
Comment on attachment 8958300 [details] bug 1442793 - tests artifact_prefix. https://reviewboard.mozilla.org/r/227234/#review242576
Attachment #8958300 -
Flags: review?(bhearsum) → review+
Comment 107•6 years ago
|
||
mozreview-review |
Comment on attachment 8958301 [details] bug 1442793 - diffoscope artifact_prefix. https://reviewboard.mozilla.org/r/227236/#review242578
Attachment #8958301 -
Flags: review?(bhearsum) → review+
Attachment #8958302 -
Flags: review?(bhearsum) → review+
Comment 108•6 years ago
|
||
mozreview-review |
Comment on attachment 8958302 [details] bug 1442793 - job public_artifacts -> artifacts. https://reviewboard.mozilla.org/r/227238/#review242580
Comment 109•6 years ago
|
||
mozreview-review |
Comment on attachment 8958303 [details] bug 1442793 - generated sources artifact_prefix. https://reviewboard.mozilla.org/r/227240/#review242602
Attachment #8958303 -
Flags: review?(bhearsum) → review+
Attachment #8958304 -
Flags: review?(bhearsum) → review+
Comment 110•6 years ago
|
||
mozreview-review |
Comment on attachment 8958304 [details] bug 1442793 - signed_artifacts artifact_prefix. https://reviewboard.mozilla.org/r/227242/#review242606
Comment 111•6 years ago
|
||
mozreview-review |
Comment on attachment 8958305 [details] bug 1442793 - beetmover artifact_prefix. https://reviewboard.mozilla.org/r/227244/#review242608
Attachment #8958305 -
Flags: review?(bhearsum) → review+
Comment 112•6 years ago
|
||
mozreview-review |
Comment on attachment 8958306 [details] bug 1442793 - mozharness test artifact_prefix. https://reviewboard.mozilla.org/r/227246/#review242610
Attachment #8958306 -
Flags: review?(bhearsum) → review+
Attachment #8961097 -
Flags: review?(bhearsum) → review+
Comment 113•6 years ago
|
||
mozreview-review |
Comment on attachment 8961097 [details] bug 1442793 - release_generate_checksums_beetmover artifact_prefix https://reviewboard.mozilla.org/r/229912/#review242612
Comment 114•6 years ago
|
||
mozreview-review |
Comment on attachment 8961098 [details] bug 1442793 - release_generate_checksums_signing artifact_prefix https://reviewboard.mozilla.org/r/229914/#review242616
Attachment #8961098 -
Flags: review?(bhearsum) → review+
Attachment #8961099 -
Flags: review?(bhearsum) → review+
Comment 115•6 years ago
|
||
mozreview-review |
Comment on attachment 8961099 [details] bug 1442793 - generate_taskcluster_artifact_prefix. https://reviewboard.mozilla.org/r/229916/#review242618
Comment 116•6 years ago
|
||
mozreview-review |
Comment on attachment 8961100 [details] bug 1442793 - repackage artifact_prefix. https://reviewboard.mozilla.org/r/229918/#review242620
Attachment #8961100 -
Flags: review?(bhearsum) → review+
Comment 117•6 years ago
|
||
mozreview-review |
Comment on attachment 8961223 [details] bug 1442793 - partials signing artifact_prefix. https://reviewboard.mozilla.org/r/230000/#review242622
Attachment #8961223 -
Flags: review?(bhearsum) → review+
Comment 118•6 years ago
|
||
mozreview-review |
Comment on attachment 8961224 [details] bug 1442793 - partials artifact_prefix. https://reviewboard.mozilla.org/r/230002/#review242624
Attachment #8961224 -
Flags: review?(bhearsum) → review+
Attachment #8961225 -
Flags: review?(bhearsum) → review+
Comment 119•6 years ago
|
||
mozreview-review |
Comment on attachment 8961225 [details] bug 1442793 - pushapk artifact_prefix https://reviewboard.mozilla.org/r/230004/#review242626
Comment 120•6 years ago
|
||
mozreview-review |
Comment on attachment 8960852 [details] bug 1442793 - nightly-l10n artifact_prefix. https://reviewboard.mozilla.org/r/229612/#review242628
Attachment #8960852 -
Flags: review?(bhearsum) → review+
Comment 121•6 years ago
|
||
mozreview-review |
Comment on attachment 8958306 [details] bug 1442793 - mozharness test artifact_prefix. https://reviewboard.mozilla.org/r/227246/#review242610
Comment 122•6 years ago
|
||
I think all of these patches look good. I was grepping around maple and found a few places that I thought might also need similar changes: taskgraph/transforms/upload_generated_sources.py taskgraph/transforms/release_sign_and_push_langpacks.py taskgraph/transforms/repackage_partner.py taskgraph/transforms/release_snap_push.py taskgraph/transforms/job/debian_package.py taskgraph/transforms/job/toolchain.py taskgraph/transforms/job/spidermonkey.py taskgraph/transforms/source_test.py taskgraph/transforms/release_beetmover_signed_addons.py All still contain 'public/build' references. Quite a few of them are only on maple code, so changes might belong in a different patch or bug.
Assignee | ||
Comment 123•6 years ago
|
||
(In reply to Ben Hearsum (:bhearsum) from comment #122) > I think all of these patches look good. I was grepping around maple and > found a few places that I thought might also need similar changes: > taskgraph/transforms/upload_generated_sources.py > taskgraph/transforms/release_sign_and_push_langpacks.py > taskgraph/transforms/repackage_partner.py > taskgraph/transforms/release_snap_push.py > taskgraph/transforms/job/debian_package.py > taskgraph/transforms/job/toolchain.py > taskgraph/transforms/job/spidermonkey.py > taskgraph/transforms/source_test.py > taskgraph/transforms/release_beetmover_signed_addons.py > > All still contain 'public/build' references. Quite a few of them are only on > maple code, so changes might belong in a different patch or bug. Thanks! Yeah, I'm leaving some of those alone because they're - maple only (e.g. partners hasn't landed on central yet), or a combination of - difficult to change and - unlikely to require private artifacts. I knew going in that this patchset would lay the groundwork for private artifacts, but would only be supported for real in the partner repack critical path.
Assignee | ||
Comment 124•6 years ago
|
||
mozreview-review-reply |
Comment on attachment 8958299 [details] bug 1442793 - repackage_signing artifact_prefix. https://reviewboard.mozilla.org/r/227232/#review242574 > This shouldn't block anything, but I see changes like this to practically every transform that deals with artifacts. It makes me wonder if this should be handled in taskgraph/transforms/task.py instead - what do you think? That probably makes sense, if and when artifact_prefix is supported through the entire graph. I left things like toolchains alone because I didn't see a straightforward way to pass the prefix down to the task.
Comment 125•6 years ago
|
||
Pushed by asasaki@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/9f8c088dce9f add artifact_prefix attribute. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/0ffeb06f9541 get_artifact_prefix and get_artifact_path. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/d8d0e4665d0b repackage_signing artifact_prefix. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/4055c1ca987b tests artifact_prefix. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/0b45c230926f diffoscope artifact_prefix. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/e7821c5c236a job public_artifacts -> artifacts. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/25ea434bd55e generated sources artifact_prefix. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/298f38a58ec7 signed_artifacts artifact_prefix. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/1a93b230ff0d beetmover artifact_prefix. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/b1ce919d3f34 mozharness test artifact_prefix. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/b0d243813a06 release_generate_checksums_beetmover artifact_prefix r=bhearsum https://hg.mozilla.org/integration/autoland/rev/d1c361a5ddc7 release_generate_checksums_signing artifact_prefix r=bhearsum https://hg.mozilla.org/integration/autoland/rev/58b09a0406d5 generate_taskcluster_artifact_prefix. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/d552d6d9ac7f repackage artifact_prefix. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/dac61b3288af partials signing artifact_prefix. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/fcf2f5931ff8 partials artifact_prefix. r=bhearsum https://hg.mozilla.org/integration/autoland/rev/bd8ad86b57d5 pushapk artifact_prefix r=bhearsum https://hg.mozilla.org/integration/autoland/rev/55ca7aeeeccf nightly-l10n artifact_prefix. r=bhearsum
Comment 126•6 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/9f8c088dce9f https://hg.mozilla.org/mozilla-central/rev/0ffeb06f9541 https://hg.mozilla.org/mozilla-central/rev/d8d0e4665d0b https://hg.mozilla.org/mozilla-central/rev/4055c1ca987b https://hg.mozilla.org/mozilla-central/rev/0b45c230926f https://hg.mozilla.org/mozilla-central/rev/e7821c5c236a https://hg.mozilla.org/mozilla-central/rev/25ea434bd55e https://hg.mozilla.org/mozilla-central/rev/298f38a58ec7 https://hg.mozilla.org/mozilla-central/rev/1a93b230ff0d https://hg.mozilla.org/mozilla-central/rev/b1ce919d3f34 https://hg.mozilla.org/mozilla-central/rev/b0d243813a06 https://hg.mozilla.org/mozilla-central/rev/d1c361a5ddc7 https://hg.mozilla.org/mozilla-central/rev/58b09a0406d5 https://hg.mozilla.org/mozilla-central/rev/d552d6d9ac7f https://hg.mozilla.org/mozilla-central/rev/dac61b3288af https://hg.mozilla.org/mozilla-central/rev/fcf2f5931ff8 https://hg.mozilla.org/mozilla-central/rev/bd8ad86b57d5 https://hg.mozilla.org/mozilla-central/rev/55ca7aeeeccf
You need to log in
before you can comment on or make changes to this bug.
Description
•