1596201 - MozRegression unable to find any builds on TaskCluster

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[Molly Howell (she/her) (no longer active)]

After MozRegression switches from mozilla-central builds to looking up individual pushes on TaskCluster, it doesn't find any build info there. Here's the CLI output from that point:

 1:28.69 INFO: Switching bisection method to taskcluster
 1:28.69 INFO: Getting mozilla-central builds between 9e3f44e87a1a2be927c7fab135653c6b86b982c9 and 2f19e7b646e0a52fa855b75c868f0a3f3a990ad3
 1:34.91 WARNING: Skipping build 2f19e7b646e0: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.2f19e7b646e0a52fa855b75c868f0a3f3a990ad3.firefox.win64-opt'
 1:35.08 WARNING: Skipping build 9e3f44e87a1a: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.9e3f44e87a1a2be927c7fab135653c6b86b982c9.firefox.win64-opt'
 1:41.22 WARNING: Skipping build 5f0b392beadb: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.5f0b392beadb7300abdaa3e5e1cc1c0d5a9f0791.firefox.win64-opt'
 1:41.55 WARNING: Skipping build d609eb2a8fe2: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.d609eb2a8fe2930c6c10dab295eb5106f5ba0bc1.firefox.win64-opt'
 1:41.94 WARNING: Skipping build 49320c7fe8b3: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.49320c7fe8b3a547856dc67e7d57a64db6b56ffd.firefox.win64-opt'
 1:47.47 WARNING: Skipping build 72c52c0101cf: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.72c52c0101cfd102b207224646c3007d6872e65c.firefox.win64-opt'
 1:47.58 WARNING: Skipping build 33f64c1ef3e4: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.33f64c1ef3e40dc619c5eb8e43c5967959f57b29.firefox.win64-opt'
 1:47.86 WARNING: Skipping build 7a75cab3dd55: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.7a75cab3dd550b77ba4f748b31c1b45ce2784c84.firefox.win64-opt'
 1:53.83 WARNING: Skipping build 641fe2e15d6f: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.641fe2e15d6f1e046f4064af76d1b7c701baa469.firefox.win64-opt'
 1:53.89 WARNING: Skipping build 4314b9f7e992: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.4314b9f7e9923820472d33436673407eb3348242.firefox.win64-opt'
 1:53.98 WARNING: Skipping build 8373434efcdb: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.8373434efcdb93aa2fef0459d357a120aa020e6e.firefox.win64-opt'
 1:56.12 INFO: Expanding lower limit of the range to caf55914ccdd
 2:02.17 WARNING: Skipping build 7a55cb6e7aec: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.7a55cb6e7aecdd0d3fb786fefe8cc165558b78e0.firefox.win64-opt'
 2:02.19 CRITICAL: Last build 2f19e7b646e0 is missing, but mozregression can't find a build after - so it is excluded, but it could contain the regression!
 2:07.64 WARNING: Skipping build 48a708577712: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.48a708577712a2eb1b19aafcda25b2aa0e607629.firefox.win64-opt'
 2:07.65 INFO: No more inbound revisions, bisection finished.
 2:07.65 INFO: Last good revision: caf55914ccddba34d462a1206530d7868b6c4992
 2:07.65 INFO: First bad revision: caf55914ccddba34d462a1206530d7868b6c4992
 2:07.65 INFO: Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=caf55914ccddba34d462a1206530d7868b6c4992

 2:07.65 WARNING: It seems that you used two changesets that are in the same push. Check the pushlog url.

I'm guessing the problem is that the TaskCluster URL is out of date, because changing the rootUrl to https://firefox-ci-tc.services.mozilla.com from https://taskcluster.net seems to have made it work for me.

Comment 1

[William Lachance (:wlach)]

(In reply to Molly Howell (she/her) [:mhowell] from comment #0)

After MozRegression switches from mozilla-central builds to looking up individual pushes on TaskCluster, it doesn't find any build info there. Here's the CLI output from that point:

...
I'm guessing the problem is that the TaskCluster URL is out of date, because changing the rootUrl to https://firefox-ci-tc.services.mozilla.com from https://taskcluster.net seems to have made it work for me.

Hi Dustin, can you shed some light on what might have happened here? I'm guessing we just need to bump the taskcluster python dependency or something?

Source code for mozregression is here: https://github.com/mozilla/mozregression

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

The older builds and newer builds are, unfortunately, in different places. Given Molly's observation that changing the rootUrl fixes the issue, I'd suspect (without checking) that some logic to use the new rootUrl for pushes after November 9 and the old rootUrl for pushes before then would do the trick.

Comment 3

[Molly Howell (she/her) (no longer active)]

That might be tricky; I don't see any natural place to insert that kind of logic, and I'm not sure it's even possible to make that kind of decision in general without just trying both locations, which there also isn't any structure to facilitate that concept right now.

It would be lovely if the taskcluster Python module could handle this, but since it's a Firefox-specific problem I assume that isn't in the cards.

Comment 4

[William Lachance (:wlach)]

[ when in doubt, needinfo more people ]

(In reply to Molly Howell (she/her) [:mhowell] from comment #3)

That might be tricky; I don't see any natural place to insert that kind of logic, and I'm not sure it's even possible to make that kind of decision in general without just trying both locations, which there also isn't any structure to facilitate that concept right now.

It would be lovely if the taskcluster Python module could handle this, but since it's a Firefox-specific problem I assume that isn't in the cards.

It probably wouldn't be too terribly hard to add a parameter to this function and make callers use it:

https://github.com/mozilla/mozregression/blob/e94d7d2f9362c68371382d1dd6de993d980c2562/mozregression/fetch_configs.py#L376

Chris, is this something your team has had to deal with yet?

If all else fails I can probably muddle through this, but I'm really trying to keep my cycles spent on mozregression to a minimum these days.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Trying both seems like a good solution too.

Comment 7

[William Lachance (:wlach)]

Looking into this now.

Comment 9

[William Lachance (:wlach)]

Have a PR up, just going to have dustin give it a once over and then I'll land: https://github.com/mozilla/mozregression/pull/559

Comment 11

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub master push: https://github.com/mozilla/mozregression/commit/f5f7466dd14e4a4e863cbc061a38025d152f7d99

Commit merged into master by GitHub
Authored by William Lachance (wlach)

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:wlach, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Comment 14

[William Lachance (:wlach)]

This is fixed! New versions of mozregression have been released

Comment 15

[Chris Peterson [:cpeterson]]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #14)

This is fixed! New versions of mozregression have been released

William, what is the version number of the fixed mozregression? The latest version I see on GitHub is gui-0.9.43 (from 2019-11-18), which still has this bug.

https://github.com/mozilla/mozregression/releases

Comment 16

[William Lachance (:wlach)]

(In reply to Chris Peterson [:cpeterson] from comment #15)

(In reply to William Lachance (:wlach) (use needinfo!) from comment #14)

This is fixed! New versions of mozregression have been released

William, what is the version number of the fixed mozregression? The latest version I see on GitHub is gui-0.9.43 (from 2019-11-18), which still has this bug.

https://github.com/mozilla/mozregression/releases

Yup, that version has the fix, which I've verified locally. If you're still seeing problems, could you file a new bug with STR and link here?

Comment 17

[Chris Peterson [:cpeterson]]

Yup, that version has the fix, which I've verified locally. If you're still seeing problems, could you file a new bug with STR and link here?

Clearing needinfo. I think was experiencing a user error.

Comment 18

[Jeremy Karlsson]

I just installed mozregression-gui 0.9.43 and I'm seeing this problem. Found an issue between two "mozilla central builds" and after a while of spamming

2020-01-02T11:35:04: WARNING : Skipping build 736e6ca6a7e1: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.shippable.revision.736e6ca6a7e18ad7f7fa4acbb59d2a5a197dd63d.firefox.win64-opt'
2020-01-02T11:35:04: DEBUG : nothing found via route 'gecko.v2.mozilla-central.revision.0132b59bb0937c3bc2e8ee7c04b8127726c0b768.firefox.win64-pgo'
2020-01-02T11:35:04: WARNING : Skipping build 0132b59bb093: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.shippable.revision.0132b59bb0937c3bc2e8ee7c04b8127726c0b768.firefox.win64-opt'
2020-01-02T11:35:04: DEBUG : nothing found via route 'gecko.v2.mozilla-central.revision.7ea008f8701b6f95320a16d78ed6ed56e22235c6.firefox.win64-opt'
2020-01-02T11:35:04: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.7ea008f8701b6f95320a16d78ed6ed56e22235c6.firefox.win64-pgo'
2020-01-02T11:35:04: DEBUG : nothing found via route 'gecko.v2.mozilla-central.revision.7ea008f8701b6f95320a16d78ed6ed56e22235c6.firefox.win64-pgo'
2020-01-02T11:35:04: WARNING : Skipping build 7ea008f8701b: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.shippable.revision.7ea008f8701b6f95320a16d78ed6ed56e22235c6.firefox.win64-opt'

and similar it just ends with a warning pop up saying "Unable to find enough data to bisect". http://prntscr.com/qif1no

Comment 19

[William Lachance (:wlach)]

(In reply to Jeremy Karlsson from comment #18)

I just installed mozregression-gui 0.9.43 and I'm seeing this problem. Found an issue between two "mozilla central builds" and after a while of spamming

2020-01-02T11:35:04: WARNING : Skipping build 736e6ca6a7e1: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.shippable.revision.736e6ca6a7e18ad7f7fa4acbb59d2a5a197dd63d.firefox.win64-opt'
2020-01-02T11:35:04: DEBUG : nothing found via route 'gecko.v2.mozilla-central.revision.0132b59bb0937c3bc2e8ee7c04b8127726c0b768.firefox.win64-pgo'
2020-01-02T11:35:04: WARNING : Skipping build 0132b59bb093: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.shippable.revision.0132b59bb0937c3bc2e8ee7c04b8127726c0b768.firefox.win64-opt'
2020-01-02T11:35:04: DEBUG : nothing found via route 'gecko.v2.mozilla-central.revision.7ea008f8701b6f95320a16d78ed6ed56e22235c6.firefox.win64-opt'
2020-01-02T11:35:04: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.7ea008f8701b6f95320a16d78ed6ed56e22235c6.firefox.win64-pgo'
2020-01-02T11:35:04: DEBUG : nothing found via route 'gecko.v2.mozilla-central.revision.7ea008f8701b6f95320a16d78ed6ed56e22235c6.firefox.win64-pgo'
2020-01-02T11:35:04: WARNING : Skipping build 7ea008f8701b: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.shippable.revision.7ea008f8701b6f95320a16d78ed6ed56e22235c6.firefox.win64-opt'

and similar it just ends with a warning pop up saying "Unable to find enough data to bisect". http://prntscr.com/qif1no

Hmm, that's unfortunate. Could you please file a new issue for this? This approach definitely mostly works but I suppose it's possible mozregression might get confused about cases which are right on the edge of the flag date. I probably won't be able to fix this myself but it would be good to get it on the books.