1625168 - Decision task frequently fails with mach try auto

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Simon Giesecke [:sg] [he/him]]

In the last days, I experienced Decision Task failures/timeouts on try pushes frequently. Today, it only worked on the fourth attempt.

https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=294851879&revision=17bdf54eb556907310e1194393a0de6490a2daa4

https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=294855005&revision=beec17c2c8722532518477fc6e1faf3e1646f030

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a381ecb762e2fd51d9b2f38f0ebbcbe650f2b2a0&selectedJob=294858656

all failed, then finally

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1fc665bd3fd471d955dd7c816b077c9165685970

succeeded (I changed the commit message on the last one, that's why it shows a different revision, but the content was exactly the same).

This has cost me a lot of time, not sure if others are affected as well.

Comment 1

[Armen [:armenzg]]

I've asked the Taskcluster team to look into this.
Treeherder mainly displays what happens on Taskcluster.

Comment 2

[Armen [:armenzg]]

Hi marco, ahal,
This seems to be an issue with mach try auto.

Is there a way to make it more obvious under which component/repo should issues be filed against?

Comment 3

[Andrew Halberstadt [:ahal]]

Fyi ./mach try auto is very experimental atm (we haven't announced it anywhere yet), so expect issues.

I think what's happening here is that for some reason the bugbug service is failing to compute the results for this push, then the taskgraph isn't propagating the error properly. It would also help if ./mach try auto enabled verbose logging in the Decision task to help see what's going on.

Comment 4

[Simon Giesecke [:sg] [he/him]]

Oh, interesting. Sorry I didn't mention that these used mach try auto. Since it didn't fail deterministically, I thought it were an infrastructure issue. (mach try auto is incredibly useful, so it would be really great if it worked reliably)

Comment 5

[Andrew Halberstadt [:ahal]]

Yikes.. I forgot to increment i in the timeout code:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/optimize/bugbug.py#46

So my guess was correct. I'll fix the timeout so that this doesn't wait 30 minutes to fail. Though the underlying cause seems to be that the service just isn't processing this push (it presumably keeps returning 202).

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1625168 - [taskgraph] Raise exception when timing out waiting for bugbug service, r?marco

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cd0c3c759c83
[taskgraph] Raise exception when timing out waiting for bugbug service, r=marco

Comment 8

[Marco Castelluccio [:marco]]

(In reply to Simon Giesecke [:sg] [he/him] from comment #4)

Oh, interesting. Sorry I didn't mention that these used mach try auto. Since it didn't fail deterministically, I thought it were an infrastructure issue. (mach try auto is incredibly useful, so it would be really great if it worked reliably)

Have you seen failures with specific patches, or generically? I'm going to add more logging in the bugbug service so I can more easily find out what happens when things go wrong.

Comment 9

[Marco Castelluccio [:marco]]

Just a suggestion, while this is still experimental, instead of pushing again you could retrigger the decision task.

Comment 10

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/cd0c3c759c83

Comment 11

[Simon Giesecke [:sg] [he/him]]

(In reply to Marco Castelluccio [:marco] from comment #8)

(In reply to Simon Giesecke [:sg] [he/him] from comment #4)

Oh, interesting. Sorry I didn't mention that these used mach try auto. Since it didn't fail deterministically, I thought it were an infrastructure issue. (mach try auto is incredibly useful, so it would be really great if it worked reliably)

Have you seen failures with specific patches, or generically? I'm going to add more logging in the bugbug service so I can more easily find out what happens when things go wrong.

I am not completely sure, but I guess the failed attempt were all changing quite basic things in mfbt or xpcom/ds.

(In reply to Marco Castelluccio [:marco] from comment #9)

Just a suggestion, while this is still experimental, instead of pushing again you could retrigger the decision task.

Unfortunately, due to an issue with my account, I can't retrigger any tasks at the moment. Hope this will be resolved soon.

Comment 12

[Marco Castelluccio [:marco]]

I made quite a few improvements in the bugbug HTTP service, so this should be fixed.