1839891 - Replace `multi_dep` loaders with `from_deps` transforms

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Andrew Halberstadt [:ahal]]

The multi_dep loader has been subsumed by the from_deps transforms in upstream Taskgraph:
https://taskcluster-taskgraph.readthedocs.io/en/latest/reference/transforms/from_deps.html

Let's replace this loader with those transforms. As we work on projects like the Firefox Android repo migration, the more we can share code across projects, the easier stuff like that will become.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - Update vendored taskcluster-taskgraph to 5.7.0, r?#taskgraph-reviewers!

Depends on D181804

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert signing tasks to 'from_deps' transforms, r?gbrown!,jcristau!

This results in an identical diff after running:

$ ./mach taskgraph full -p taskcluster/test/params --diff .~1 -J --fast --exclude-key attributes.primary-kind-dependency

Depends on D181899

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert beetmover tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D181900

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7a0483c6c4c0
Update vendored taskcluster-taskgraph to 5.6.1, r=taskgraph-reviewers,gbrown

Comment 5

[Cristina Horotan [:chorotan]]

Backed out 2 changesets (bug 1843902, bug 1839891) for causing pyton related build bustages

Backout: https://hg.mozilla.org/integration/autoland/rev/ea2b10290674fae9ab7b34cfbd3e69991b3eb32a

Failure push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=SCVtq7-EQc6pIKhSS421jw.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=bafc35ec5ecd3c25f43751a9d9cac15a3c9ea377

Failure logs:
https://treeherder.mozilla.org/logviewer?job_id=422901330&repo=autoland&lineNumber=429
https://treeherder.mozilla.org/logviewer?job_id=422901283&repo=autoland&lineNumber=142

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/16dd34d01d61
Update vendored taskcluster-taskgraph to 5.6.1, r=taskgraph-reviewers,gbrown
https://hg.mozilla.org/integration/autoland/rev/ba91e51b4501
[ci] Convert signing tasks to 'from_deps' transforms, r=gbrown,jcristau,taskgraph-reviewers,releng-reviewers
https://hg.mozilla.org/integration/autoland/rev/06748e1bdfcb
[ci] Convert beetmover tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum

Comment 7

[Serban Stanca [:SerbanS]]

Backed out for causing py3 failures.

• Backout link
• Push with failures
• Failure Log
• Failure line: taskcluster/test/test_generate_params.py::test_generate_graphs[mr-ship-geckoview] TEST-UNEXPECTED-FAIL

Comment 8

[Natalia Csoregi [:nataliaCs]]

Backout merged to mozilla-central: https://hg.mozilla.org/mozilla-central/rev/79649c51aca313eafafcd0a69b5fc8e8298402f3

Comment 9

[Andrew Halberstadt [:ahal]]

I have a fix, but it will need a new Taskgraph change + release.

Comment 10

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Default `enable_always_target` parameter to docker-image tasks, r?jcristau!

Upstream taskgraph removed the special casing of docker-image tasks w.r.t
implicitly adding them to the graph, and instead switched over to relying on
the always_target feature to accomplish this.

However, in Gecko we usually disable the always_target feature entirely (only
enabling it on autoland and try). As a compromise, we implemented a feature in
Taskgraph to allow selectively enabling always_target to a specific list of
kinds.

This patch gets Gecko to start using this new feature.

Depends on D181899

Comment 11

[Phabricator Automation]

Comment on attachment 9340741 [details]
Bug 1839891 - Update vendored taskcluster-taskgraph to 5.7.0, r?#taskgraph-reviewers!

Revision D181899 was moved to bug 1844592. Setting attachment 9340741 [details] to obsolete.

Comment 12

[Phabricator Automation]

Comment on attachment 9344841 [details]
Bug 1839891 - [ci] Default enable_always_target parameter to docker-image tasks, r?jcristau!

Revision D184107 was moved to bug 1844592. Setting attachment 9344841 [details] to obsolete.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8fa9011e9da0
[ci] Convert signing tasks to 'from_deps' transforms, r=gbrown,jcristau,taskgraph-reviewers,releng-reviewers
https://hg.mozilla.org/integration/autoland/rev/c2750a529212
[ci] Convert beetmover tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum

Comment 14

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/8fa9011e9da0
https://hg.mozilla.org/mozilla-central/rev/c2750a529212

Comment 15

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert repackage tasks to 'from_deps' transforms, r?#releng-reviewers!

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a249a76cfbdf
[ci] Convert repackage tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum

Comment 17

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/a249a76cfbdf

Comment 18

[Andrew Halberstadt [:ahal]]

Apologies for the Thunderbird bustage.. there will be quite a few more similar commits here before this is done. I can try to give you a heads up before landing in the future.

Comment 19

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert l10n tasks to 'from_deps' transforms, r?#releng-reviewers!

Comment 20

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert upload-symbols tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D184849

Comment 21

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert system-symbols-reprocess tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D184850

Comment 22

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Remove 'multi_dep' loader and move remaining logic to util/dependencies.py, r?#releng-reviewers!

Depends on D184856

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e2f9742243f9
[ci] Convert l10n tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/a7125d17ab8a
[ci] Convert upload-symbols tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/9032e862dd83
[ci] Convert system-symbols-reprocess tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/5d2471f18196
[ci] Remove 'multi_dep' loader and move remaining logic to util/dependencies.py, r=releng-reviewers,taskgraph-reviewers,bhearsum

Comment 24

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert release-*-repack-repackage-signing tasks to 'from_deps' transforms, r?#releng-reviewers!

Comment 25

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert release-*-repack-beetmover tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D185082

Comment 26

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert geckodriver-signing tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D185083

Comment 27

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert release-beetmover-signed-langpacks-checksums tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D185084

Comment 28

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert repackage-signing tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D185085

Comment 29

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert partials tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D185086

Comment 30

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert mar-signing tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D185123

Comment 31

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/e2f9742243f9
https://hg.mozilla.org/mozilla-central/rev/a7125d17ab8a
https://hg.mozilla.org/mozilla-central/rev/9032e862dd83
https://hg.mozilla.org/mozilla-central/rev/5d2471f18196

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/26dceb1c8969
[ci] Convert release-*-repack-repackage-signing tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/812a516dc9c0
[ci] Convert release-*-repack-beetmover tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/ddb697bc134f
[ci] Convert geckodriver-signing tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/dcea1fb8f726
[ci] Convert release-beetmover-signed-langpacks-checksums tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/0b8c791fbc39
[ci] Convert repackage-signing tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/05541d66861f
[ci] Convert partials tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/39eeca96f07d
[ci] Convert mar-signing tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum

Comment 33

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/26dceb1c8969
https://hg.mozilla.org/mozilla-central/rev/812a516dc9c0
https://hg.mozilla.org/mozilla-central/rev/ddb697bc134f
https://hg.mozilla.org/mozilla-central/rev/dcea1fb8f726
https://hg.mozilla.org/mozilla-central/rev/0b8c791fbc39
https://hg.mozilla.org/mozilla-central/rev/05541d66861f
https://hg.mozilla.org/mozilla-central/rev/39eeca96f07d

Comment 34

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert release-partner-repack-repackage tasks to 'from_deps' transforms, r?#releng-reviewers!

This is a bustage fix because I accidentally converted the
partner_repackage.py transforms to the new format without
converting a kind that used them. Normally I've been trying
to convert all kinds that use a given transform file at
once but I neglected this one.

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a5c57fffac4b
[ci] Convert release-partner-repack-repackage tasks to 'from_deps' transforms, r=releng-reviewers,gabriel

Comment 36

[Andrew Halberstadt [:ahal]]

Comment on attachment 9348013 [details]
Bug 1839891 - [ci] Convert release-partner-repack-repackage tasks to 'from_deps' transforms, r?#releng-reviewers!

Beta/Release Uplift Approval Request

• User impact if declined: Bustage fix for release-partner-repack-repackage tasks
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky):
• String changes made/needed:
• Is Android affected?: No

Comment 37

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/cfdaf87ed056

Comment 38

[Dianna Smith [:diannaS]]

Comment on attachment 9348013 [details]
Bug 1839891 - [ci] Convert release-partner-repack-repackage tasks to 'from_deps' transforms, r?#releng-reviewers!

Approved for 117.0b5

Comment 39

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/a5c57fffac4b

Comment 40

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9348013 [details]
Bug 1839891 - [ci] Convert release-partner-repack-repackage tasks to 'from_deps' transforms, r?#releng-reviewers!

Removing the beta approval here to get this off the needs-landing radar.

Comment 41

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert *-dummy tasks to 'from_deps' transforms, r?#releng-reviewers!

Comment 42

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert release-generate-checksums-signing tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D186348

Comment 43

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert fxrecord tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D186351

Comment 44

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert release-partner-repack-chunking-dummy tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D186352

Comment 45

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6f3fe563c4b3
[ci] Convert *-dummy tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,gbrown
https://hg.mozilla.org/integration/autoland/rev/889eee3a1ad2
[ci] Convert release-generate-checksums-signing tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,hneiva
https://hg.mozilla.org/integration/autoland/rev/1cc19a613941
[ci] Convert fxrecord tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,hneiva
https://hg.mozilla.org/integration/autoland/rev/7453fae110a5
[ci] Convert release-partner-repack-chunking-dummy tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,hneiva
https://hg.mozilla.org/integration/autoland/rev/12465a0d94a5
apply code formatting via Lando

Comment 46

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/6f3fe563c4b3
https://hg.mozilla.org/mozilla-central/rev/889eee3a1ad2
https://hg.mozilla.org/mozilla-central/rev/1cc19a613941
https://hg.mozilla.org/mozilla-central/rev/7453fae110a5
https://hg.mozilla.org/mozilla-central/rev/12465a0d94a5

Comment 47

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert balrog tasks to 'from_deps' transforms, r?#releng-reviewers!

Comment 48

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert upload-generated-sources tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D187311

Comment 49

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e574eedd0375
[ci] Convert balrog tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/170d2b5b7c48
[ci] Convert upload-generated-sources tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum

Comment 50

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/e574eedd0375
https://hg.mozilla.org/mozilla-central/rev/170d2b5b7c48

Comment 51

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert release-eme-free-repack-beetmover-checksums tasks to 'from_deps' transforms, r?#releng-reviewers!

Comment 52

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert release-partner-attribution-beetmover tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D192497

Comment 53

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert 'release-source-checksums-signing' tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D192498

Comment 54

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert 'release-beetmover-source-checksums' tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D193072

Comment 55

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert 'release-generate-checksums-beetmover' tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D193073

Comment 56

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert 'geckodriver-mac-notarization' tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D193074

Comment 57

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert 'release-push-langpacks' tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D193075

Comment 58

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert 'openh264-signing' tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D193076

Comment 59

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Convert 'release-beetmover-signed-langpacks' tasks to 'from_deps' transforms, r?#releng-reviewers!

Depends on D193077

Comment 60

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1839891 - [ci] Remove the 'single_dep' loader, r?#releng-reviewers!

Depends on D193078

Comment 61

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/926172bb8c1b
[ci] Convert release-eme-free-repack-beetmover-checksums tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/2f1f570f63be
[ci] Convert release-partner-attribution-beetmover tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/adc35946027b
[ci] Convert 'release-source-checksums-signing' tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/a67c296da97d
[ci] Convert 'release-beetmover-source-checksums' tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/959d3f16353f
[ci] Convert 'release-generate-checksums-beetmover' tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/0aff44a0f638
[ci] Convert 'geckodriver-mac-notarization' tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/ec256ad095cd
[ci] Convert 'release-push-langpacks' tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/af2ca83ceae6
[ci] Convert 'openh264-signing' tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/985f16feb042
[ci] Convert 'release-beetmover-signed-langpacks' tasks to 'from_deps' transforms, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/29f3612d4cef
[ci] Remove the 'single_dep' loader, r=releng-reviewers,taskgraph-reviewers,hneiva,bhearsum

Comment 62

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/926172bb8c1b
https://hg.mozilla.org/mozilla-central/rev/2f1f570f63be
https://hg.mozilla.org/mozilla-central/rev/adc35946027b
https://hg.mozilla.org/mozilla-central/rev/a67c296da97d
https://hg.mozilla.org/mozilla-central/rev/959d3f16353f
https://hg.mozilla.org/mozilla-central/rev/0aff44a0f638
https://hg.mozilla.org/mozilla-central/rev/ec256ad095cd
https://hg.mozilla.org/mozilla-central/rev/af2ca83ceae6
https://hg.mozilla.org/mozilla-central/rev/985f16feb042
https://hg.mozilla.org/mozilla-central/rev/29f3612d4cef