1604749 - daily cron job won't start

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect, P1)

Description

[Magnus Melin [:mkmelin]]

https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=bc449b862ba612f5ce0346c7d32264c67a231233

HTTPError: 403 Client Error: Forbidden for url: http://taskcluster/queue/v1/task/aYXgzetSQDuMoVBGO025iA

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=281718102&repo=comm-central&lineNumber=1451

[task 2019-12-18T11:05:37.552Z] * errorCode: InsufficientScopes

Comment 1

[Rob Lemley [:rjl]]

Tom, Sorry to bother you with this. Any thoughts? I'll note that I did recently update our .taskcluster.yml file.

Comment 2

[Rob Lemley [:rjl]]

Notes:

My .taskcluster.yml change landed on Mon 2019-12-09. (bug 1504004)
Which broke actions in treeherder. Daily cron continued to work through Fri 12-12. IIRC that is due to bustage within Thunderbird. There was a Daily build on 2019-12-16.

It's noted in bug 1504004 comment 9 that actions are still busted. But Tom landed some ci-configuration changes at approx the same time which I believe fixed the actions because they work now. But that appears to have broken the Daily cron.

But oddly enough Periodic file updates cron worked just fine this morning.

Comment 4

[Rob Lemley [:rjl]]

More notes:

The issue appears to be specific to creating the "index" tasks.

index-task-build-linux64-shippable/opt
index-task-build-win64-shippable/opt
index-task-build-macosx64-shippable/opt
(possibly others)

Comment 5

[Rob Lemley [:rjl]]

Attachment: bug1604749.patch

I'm basing this patch on suspicion that the "old" index routes are
being referenced when the cron decision task runs causing the failure,
and I should have kept them in Bug 1504004.

Comment 6

[Geoff Lankow (:darktrojan)]

Those routes weren't removed in bug 1504004. They never existed. :-/

Comment 7

[Rob Lemley [:rjl]]

The index-build-* tasks get defined here: https://searchfox.org/mozilla-central/rev/c79fc6e32a5b561205a71fcfef9112d34b4037ae/taskcluster/taskgraph/morph.py#51

Always on a gecko-t/misc worker.

Looks like this removed access for comm repos to those workers (bug 1596177):
https://hg.mozilla.org/ci/ci-configuration/rev/f4f25ddf50561cc7c0e4ec117ea463fa2873f809

Index tasks pretty much only get generated for Thunderbird on release promotion and for Nightly builds. We will not be able to release beta or esr68 with this bug, so setting it as a blocker.

I see two potential fixes:

• Create a comm-t/misc worker pool, grant access in ci-configuration, and then update morph.py to use it for Thunderbird builds. Then do the required uplift(s) to M-esr68 so releases work.
• Grant access to gecko-t/misc to Thunderbird builds in ci-configuration/grants.yml

Patch coming for option 2 in the interest of expediency. If it's necessary I'll be happy to work on option 1.

Comment 8

[Rob Lemley [:rjl]]

Attachment: Bug 1604749 - Grant access to gecko-t/misc workers to Thunderbird. r?Callek

Since 2019-Dec-17 Thunderbird Nightly builds fail during the Cron Decision
task when creating the index-build tasks.
Those tasks run on gecko-t-misc workers. There is no equivalent comm-t-misc worker.

Comment 9

[Tom Prince [:tomprince]]

Attachment: Bug 1604749: Get misc worker-type from graph config; r?Callek

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f6cf3ca2de50
Get misc worker-type from graph config; r=Callek

Comment 11

[Rob Lemley [:rjl]]

Attachment: bug1604749_config-yml.patch

Looks like this will be needed after the next M-C merge,

Comment 13

[Tom Prince [:tomprince]]

Attachment: Bug 1604749: [taskgraph] Fix tests for index morph; r?Callek

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/66a1c07a8f48
[taskgraph] Fix tests for index morph; r=Callek

Comment 15

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/f6cf3ca2de50
https://hg.mozilla.org/mozilla-central/rev/66a1c07a8f48

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/489349930561
Add misc worker alias to config.yml. r=darktrojan

Comment 17

[Tom Prince [:tomprince]]

Attachment: Bug 1604749: Get misc worker-type from graph config; r?Callek

The original patch calculated the worker type, but didn't use it in the
generated task.

Comment 18

[Rob Lemley [:rjl]]

Is the plan to move Thunderbird to a 'comm-t/misc' worker for these?

(In reply to Tom Prince [:tomprince] from comment #17)

Created attachment 9124397 [details]
Bug 1604749: Get misc worker-type from graph config; r?Callek

The original patch calculated the worker type, but didn't use it in the
generated task.

Comment 19

[Rob Lemley [:rjl]]

Attachment: Bug 1604749 - Create comm-t/misc workers for Thunderbird. r?tomprince

Comment 20

[Rob Lemley [:rjl]]

Attachment: thunderbird_use_comm-t.patch

Once D61780 lands in ci-configuration, we need to switch over to use comm-t
workers for indexing tasks.

This will need to be uplifted to comm-beta, and something will need to change
on comm-esr68 as well. This patch won't apply to c-esr68 though.

Comment 21

[Rob Lemley [:rjl]]

For esr68, "gecko-t" was hardcoded as the worker type:

https://searchfox.org/mozilla-esr68/source/taskcluster/taskgraph/morph.py#63

    task_def = {
        'provisionerId': 'gecko-t',
        'workerType': 'misc',
        'dependencies': [task.task_id, image_taskid],
        'created': {'relative-datestamp': '0 seconds'},

Are there uplift plans for the original work that allows setting the worker type in config.yml? We do have THUNDERBIRD_68_VERBRANCH on mozilla-esr68 that we build against with some Thunderbird-only changes already, so this could be one more of those if necessary.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/c25b9b3f598a
Use comm-t/misc workers for index tasks. r=darktrojan

Comment 23

[Magnus Melin [:mkmelin]]

Dailies didn't run now, https://treeherder.mozilla.org/#/jobs?repo=comm-central&selectedJob=287910869

Comment 24

[Rob Lemley [:rjl]]

(In reply to Magnus Melin [:mkmelin] from comment #23)

Dailies didn't run now, https://treeherder.mozilla.org/#/jobs?repo=comm-central&selectedJob=287910869

This isn't what broke it though.

Comment 25

[Intermittent Failures Robot]

1 failures in 3364 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 1

Platform breakdown:

• gecko-decision: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1604749&startday=2020-02-03&endday=2020-02-09&tree=all

Comment 26

[Wayne Mery (:wsmwk)]

Rob, thunderbird_esr68 is still set as affected

Comment 27

[Rob Lemley [:rjl]]

It is and it isn't... nothing is actually broken on esr68 from our perspective. This bug was driven by a need to separate Thunderbird from Firefox a bit more in CI however, the code that we rely on in M-C to apply this patch is not on M-ESR68. I will check with them, but since we're rather late in ESR68's lifetime, I suspect that it won't happen and we can change that to a 'wontfix'.

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d154bbd8a45b
Get misc worker-type from graph config; r=Callek

Comment 29

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/d154bbd8a45b

Comment 30

[Rob Lemley [:rjl]]

Tom indicated that we should be okay on esr68 as the necessary scopes will be applied in ci-configuration.

Comment 31

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/57824aa9e179d1c66d2059758c3381f7cf6f8e7c
https://hg.mozilla.org/releases/mozilla-beta/rev/4ab9d8beb09df90e472f42ea0d40c02342effe44

Comment 32

[Tom Prince [:tomprince]]

Attachment: Bug 1604749: Limit gecko-t/misc on comm branches to comm-esr68; r?Callek

Now that we have landed https://phabricator.services.mozilla.com/D61643
on central, beta and esr78, the only active comm-* branch that doesn't
have the fix is comm-esr8. Restict the scope to that branch, and we can
let that scope live out the life of that branch.

Comment 33

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/10c72b3b4ce8
Limit gecko-t/misc on comm branches to comm-esr68; r=Callek