1607382 - no linux32 partial gen tasks for beta

Status: VERIFIED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Jordan Lund (:jlund)]

we only had a linux64 partial generation task. no linux32 equivalent of: https://firefox-ci-tc.services.mozilla.com/tasks/dCLNypu9RCGxuLw3TglaNw
6:06 PM and for devedition, we did have a partial gen task for linux32: https://firefox-ci-tc.services.mozilla.com/tasks/Yll4fhZ1TPmuL-6yCyjRFg

I do see linux32 in the platform list for partial gen: https://hg.mozilla.org/releases/mozilla-beta/file/tip/taskcluster/ci/partials/kind.yml#l33

this bug was discovered after bouncer check and UV tasks started to fail:

UV: https://firefox-ci-tc.services.mozilla.com/tasks/EBXDIv5yTv6TCY2If1cCLg/runs/2/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FEBXDIv5yTv6TCY2If1cCLg%2Fruns%2F2%2Fartifacts%2Fpublic%2Flogs%2Flive.log

bouncer: https://firefox-ci-tc.services.mozilla.com/tasks/eM2UD0mmRnOi2qAi1Ab3WQ/runs/1/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FeM2UD0mmRnOi2qAi1Ab3WQ%2Fruns%2F1%2Fartifacts%2Fpublic%2Flogs%2Flive.log

Comment 1

[Jordan Lund (:jlund)]

notably, linux32 was demoted to tier 2 but only on central: Bug 1599197 - demote linux32 to tier-2, run builds not on autoland and try by default, only m-c

Comment 2

[Jordan Lund (:jlund)]

we can ship without linux32 partials for b1. However, it was pointed out that that may cause errors in our update verify checks for many betas throughout this current cycle

Comment 3

[Edwin Takahashi (:egao | infrequent contributor)]

I suspect it might be my changes in https://phabricator.services.mozilla.com/D56605, as one of the first steps to put ubuntu1804-test docker image into production use?

Comment 4

[Jordan Lund (:jlund)]

the decision task log is not showing much of a difference when grepping for partial and diffing beta vs devedition

https://firefox-ci-tc.services.mozilla.com/tasks/HMjOS8DGQny4lPhqM_nRDw/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FHMjOS8DGQny4lPhqM_nRDw%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log

[task 2020-01-06T20:53:26.885Z] loading kind `partials` from `taskcluster/ci/partials`
[task 2020-01-06T20:53:27.165Z] loading kind `partials-signing` from `taskcluster/ci/partials-signing`
[task 2020-01-06T20:53:27.638Z] No partials defined! Bouncer submission task won't send any partial-related entry for "firefox"
[task 2020-01-06T20:53:27.639Z] No partials defined! Bouncer submission task won't send any partial-related entry for "fennec"
[task 2020-01-06T20:53:27.639Z] No partials defined! Bouncer submission task won't send any partial-related entry for "devedition"
[task 2020-01-06T20:53:33.087Z] Loading tasks for kind partials
[task 2020-01-06T20:53:33.458Z] Generated 1151 tasks for kind partials
[task 2020-01-06T20:53:33.523Z] Loading tasks for kind partials-signing
[task 2020-01-06T20:53:33.797Z] Generated 1151 tasks for kind partials-signing
[task 2020-01-06T20:54:24.641Z] Creating task with taskId HzO9C0wKSZ2sjbL9RWHWYw for partials-win32-shippable/opt
[task 2020-01-06T20:54:24.671Z] Creating task with taskId cyWFA2F6SLao3sHsRbd9EQ for partials-win64-shippable/opt
[task 2020-01-06T20:54:25.690Z] Creating task with taskId NitpFBh2QNi6FieaEGhuOw for partials-macosx64-shippable/opt
[task 2020-01-06T20:54:25.743Z] Creating task with taskId dCLNypu9RCGxuLw3TglaNw for partials-linux64-shippable/opt
[task 2020-01-06T20:54:25.931Z] Creating task with taskId SrvuplmoQZubjNslQ-w30A for partials-win64-aarch64-shippable/opt
[task 2020-01-06T20:54:26.734Z] Creating task with taskId P6xH2-qVQJOefI1orEcf2w for partials-signing-macosx64-shippable/opt
[task 2020-01-06T20:54:27.171Z] Creating task with taskId RPvkI19MTxm7RJv1rO-Aew for partials-signing-win64-shippable/opt
.... etc

https://firefox-ci-tc.services.mozilla.com/tasks/bW4FLWf8SiegMcYt3iP-tw/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FbW4FLWf8SiegMcYt3iP-tw%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log

[task 2020-01-06T20:53:20.327Z] loading kind `partials` from `taskcluster/ci/partials`
[task 2020-01-06T20:53:20.927Z] loading kind `partials-signing` from `taskcluster/ci/partials-signing`
[task 2020-01-06T20:53:21.310Z] No partials defined! Bouncer submission task won't send any partial-related entry for "firefox"
[task 2020-01-06T20:53:21.311Z] No partials defined! Bouncer submission task won't send any partial-related entry for "fennec"
[task 2020-01-06T20:53:21.311Z] No partials defined! Bouncer submission task won't send any partial-related entry for "devedition"
[task 2020-01-06T20:53:27.122Z] Loading tasks for kind partials
[task 2020-01-06T20:53:27.521Z] Generated 1151 tasks for kind partials
[task 2020-01-06T20:53:27.593Z] Loading tasks for kind partials-signing
[task 2020-01-06T20:53:27.888Z] Generated 1151 tasks for kind partials-signing
[task 2020-01-06T20:54:21.882Z] Creating task with taskId Yll4fhZ1TPmuL-6yCyjRFg for partials-linux-devedition-nightly/opt
[task 2020-01-06T20:54:22.416Z] Creating task with taskId aTeNiSeWQXORuDuEd5DYGQ for partials-macosx64-devedition-nightly/opt
[task 2020-01-06T20:54:22.459Z] Creating task with taskId VQgvWR3aTluY_DwOsM_DXQ for partials-win32-devedition-nightly/opt
[task 2020-01-06T20:54:22.621Z] Creating task with taskId QipWqY-VT5OsfQGpcNd8Dg for partials-win64-aarch64-devedition-nightly/opt
[task 2020-01-06T20:54:23.016Z] Creating task with taskId KPVK2_uPTCa1TCpYbmpvkw for partials-linux64-devedition-nightly/opt
[task 2020-01-06T20:54:23.045Z] Creating task with taskId RoQ_p_7OTdGgP9WlRF7u4A for partials-win64-devedition-nightly/opt
[task 2020-01-06T20:54:23.171Z] Creating task with taskId LLvDdsI-RHufvxcjZNYrAA for partials-signing-linux-devedition-nightly/opt
[task 2020-01-06T20:54:23.488Z] Creating task with taskId JgC7rDrATRClRetcby20TQ for partials-signing-win64-devedition-nightly/opt
[task 2020-01-06T20:54:23.970Z] Creating task with taskId RIkou1pTSSu8kc7XTubKuQ for partials-signing-macosx64-devedition-nightly/opt
.... etc

Comment 5

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1607382 - revert changing of treeherder name of linux-shippable/opt build to avoid breaking partials

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4cf1f16e4763
revert changing of treeherder name of linux-shippable/opt build to avoid breaking partials r=jlund

Comment 7

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/releases/mozilla-beta/rev/e6d0d1a9b6db

Comment 8

[Jordan Lund (:jlund)]

backout in comment 6 was needed because it turns out we may actually depend on treeherder names.

https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/partials.py#71

Which is something we should postmortem on. Ideally we don't depend on Treeherder labels to determine what release tasks we trigger.

Comment 9

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/4cf1f16e4763

Comment 10

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #8)

backout in comment 6 was needed because it turns out we may actually depend on treeherder names.

https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/partials.py#71

Which is something we should postmortem on. Ideally we don't depend on Treeherder labels to determine what release tasks we trigger.

@sfraser - do you have any context or opinion about depending on treeherder labels for determining which release partial tasks to run?

Comment 11

[Simon Fraser [:sfraser] ⌚️GMT]

(In reply to Jordan Lund (:jlund) from comment #10)

Which is something we should postmortem on. Ideally we don't depend on Treeherder labels to determine what release tasks we trigger.

@sfraser - do you have any context or opinion about depending on treeherder labels for determining which release partial tasks to run?

We don't currently have a nicely parseable target platform in the task metadata, and we probably should. The current tags.os is the workerType's operating system, which may cause confusion. I'd be in favour of adding the target platform and other fields to another area, such as metadata, tags or extra. I imagine we'd want to encode 'opt', 'debug', 'asan' and so on in there as a different key, rather than all mashed into one pseudoplatform name.

Comment 12

[Tom Prince [:tomprince]]

We actually do have the build platform in an attribute, and I have a patch switching to use it mostly ready. (There is some dead code in funsize that passes platform around, but doesn't consume it that I need to still adjust).

Comment 13

[Simon Fraser [:sfraser] ⌚️GMT]

(In reply to Tom Prince [:tomprince] from comment #12)

We actually do have the build platform in an attribute, and I have a patch switching to use it mostly ready. (There is some dead code in funsize that passes platform around, but doesn't consume it that I need to still adjust).

There are around ~129 unique values for build_platform, and it encodes information that's a bit too specific for the partial generation's needs, as we need to map it to what Balrog thinks of as a platform, which would mean reliably removing things like '-devedition-nightly', '-shippable', '-ccov' and so on. Removing text based on '-' seems like it would work at first glance, but it also makes me wonder why we're not encoding this in a machine readable way to start with, and building the more complex strings when we need them.

Comment 14

[Aki Sasaki (not active)]

(In reply to Simon Fraser [:sfraser] ⌚️GMT from comment #13)

There are around ~129 unique values for build_platform

I'm guessing once we strip this down to platforms we actually create partials for (shippable and nightly only, desktop only), that number will be significantly lower.

Comment 15

[Firefox Bug Husbandry Bot]

Please specify a root cause for this bug. See :tmaity for more information.

Comment 16

[Edwin Takahashi (:egao | infrequent contributor)]

For this bug the issue was my changes to infrastructure configuration. I selected Testing Error but now I see there is an entry for Infrastructure, so I'd like the classification to be changed.

Comment 17

[Tania Maity (:tmaity)]

(In reply to Edwin Takahashi (:egao) from comment #16)

For this bug the issue was my changes to infrastructure configuration. I selected Testing Error but now I see there is an entry for Infrastructure, so I'd like the classification to be changed.

Hi Edwin,
You should be able to change it now.

Comment 18

[Tania Maity (:tmaity)]

Updated RCA as Infrastructure as per comment#16