1532412 - Adjust for disabled opt builds in bug 1522111

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Bug 1522111 disabled some opt builds on inbound and autoland, in favour of pgo. If mozregression supports bisecting into on-push builds on those branches it may need to make some changes.

Comment 1

[William Lachance (:wlach)]

From the code, I am fairly certain this should still work as expected. Is this change landed and ready for testing? i.e. if I bisect into inbound/autoland on March 3rd/4th, will I only get pgo builds?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Yes, I believe it's all landed. There's a slight regression in that linux64 opt builds are still running.

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Try to bisect bug 1532110 and switching to pgo will be necessary.

Comment 4

[William Lachance (:wlach)]

Aryx, is mozregression not working for you? As far as I can tell from the source it should be able to fetch pgo builds from integration branches in its current form, at least on Windows and Linux. I think we might need to make it search for pgo builds on Mac though.

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

It's more about switching from default 'opt' to 'pgo' as default if available. Else it will work fine as long as one bisects on central but fail once it tries to switch to autoland.

Comment 6

[William Lachance (:wlach)]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #5)

It's more about switching from default 'opt' to 'pgo' as default if available. Else it will work fine as long as one bisects on central but fail once it tries to switch to autoland.

My impression was that mozregression would actually search for builds of both types for every revision, at least if specified:

https://github.com/mozilla/mozregression/blob/df17ccdb445a9ebb665e695d4f77d8bdae36938f/mozregression/fetch_configs.py#L431

Is that not the case?

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

No, it will fail with messages similar to this:
WARNING : Skipping build 1bc807fb4c89: Unable to find build info using the taskcluster route 'gecko.v2.autoland.revision.1bc807fb4c895d326aafc4a5bc5c9239c093b432.firefox.win64-opt'

Comment 8

[William Lachance (:wlach)]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #7)

No, it will fail with messages similar to this:
WARNING : Skipping build 1bc807fb4c89: Unable to find build info using the taskcluster route 'gecko.v2.autoland.revision.1bc807fb4c895d326aafc4a5bc5c9239c093b432.firefox.win64-opt'

Hi :coop, could someone on your team with taskcluster python client library experience help debug this? We really need mozregression to work for tracking down the root causes of bugs and not being able to bisect inbound/autoland is quite a deal-breaker.

The problem is probably either in taskcluster itself / the client library we're using (the copy used by mozregression is very out of date -- you can see an effort to update it here-- https://github.com/mozilla/mozregression/pull/513/commits/b962c5e23a07f55761106d3c7dee2e389446b080) or in the file I link to in comment 6. I am very happy to provide mozregression help/guidance/reviews as needed, either here or on irc/slack.

Comment 9

[Chris Cooper [:coop] (he/him)]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #8)

The problem is probably either in taskcluster itself / the client library we're using (the copy used by mozregression is very out of date -- you can see an effort to update it here-- https://github.com/mozilla/mozregression/pull/513/commits/b962c5e23a07f55761106d3c7dee2e389446b080) or in the file I link to in comment 6. I am very happy to provide mozregression help/guidance/reviews as needed, either here or on irc/slack.

I've cc-ed Pete here for further guidance. Updating the client library here is a good first step.

Comment 10

[Aki Sasaki (not active)]

https://tools.taskcluster.net/index/gecko.v2.autoland.revision.1bc807fb4c895d326aafc4a5bc5c9239c093b432.firefox.win64-opt doesn't exist. I'm guessing someone has changed which indexes we populate.

https://tools.taskcluster.net/index/gecko.v2.autoland.revision.1bc807fb4c895d326aafc4a5bc5c9239c093b432.firefox/win64-pgo does exist, though :Callek may be changing these when we move to shippable builds in bug 1352113. You can look at https://tools.taskcluster.net/index/gecko.v2.autoland.revision.1bc807fb4c895d326aafc4a5bc5c9239c093b432.firefox to see which platforms are indexed under that revision.

Does mozregression find the -pgo build? (Comment 7 says no)

Comment 11

[William Lachance (:wlach)]

(In reply to Aki Sasaki [:aki] from comment #10)

Does mozregression find the -pgo build? (Comment 7 says no)

I don't presently have a win64 environment to test this case, but that is my read of what :Aryx in comment 7, yes.

I never really had any doubt that the builds were being produced, my question is why isn't mozregression able to find them? As mentioned in comment 6, I thought it was configured to do so.

Comment 12

[Aki Sasaki (not active)]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #11)

(In reply to Aki Sasaki [:aki] from comment #10)

Does mozregression find the -pgo build? (Comment 7 says no)

I don't presently have a win64 environment to test this case, but that is my read of what :Aryx in comment 7, yes.

I never really had any doubt that the builds were being produced, my question is why isn't mozregression able to find them? As mentioned in comment 6, I thought it was configured to do so.

I'm guessing it's a mozregression bug in this block. Most likely we want to try multiple indexes and only fail out if we don't find any of them.

Comment 13

[William Lachance (:wlach)]

(In reply to Aki Sasaki [:aki] from comment #12)

I'm guessing it's a mozregression bug in this block. Most likely we want to try multiple indexes and only fail out if we don't find any of them.

Ah yes, that sounds like a plausible theory. Is this something you or someone else on the taskcluster team might be able to do? Not meaning to pass the buck, but I don't really have a lot of time to maintain mozregression these days, and don't understand the taskcluster all that well.

Comment 14

[Aki Sasaki (not active)]

:Callek is willing to take a look.

Comment 15

[Ian Moody [:Kwan] (UTC+1)]

Is it just me, or has multiple build types on the command line not actually worked since, well, ever as far as I can tell. Not to mention asan-debug not working at all because it has a hyphen in it.

Edit:
Though you can trick asan-debug into working by replacing the hyphen with a comma. And optionally reversing the words, it isn't fussed.

Comment 16

[Ian Moody [:Kwan] (UTC+1)]

Anyway, here's a rough WIP that seems to actually work. Even passes the (current) tests. Need to write new ones.

Though frankly I'm tempted to rip out any support for users passing in multiple builds (since clearly no-one uses it...), and only keep the internal support for use for the shippable->opt->pgo fallbacking.

Comment 17

[William Lachance (:wlach)]

(In reply to Ian Moody [:Kwan] (UTC+0) from comment #16)

Anyway, here's a rough WIP that seems to actually work. Even passes the (current) tests. Need to write new ones.

This is looking great, thank you!

Though frankly I'm tempted to rip out any support for users passing in multiple builds (since clearly no-one uses it...), and only keep the internal support for use for the shippable->opt->pgo fallbacking.

It could be that things used to work, but broke when we moved over to taskcluster. I wouldn't be too sad if we lost this feature -- if someone really wants it, they can always re-add it.

Comment 18

[Ian Moody [:Kwan] (UTC+1)]

(In reply to Ian Moody [:Kwan] (UTC+0) from comment #15)

Is it just me, or has multiple build types on the command line not actually worked since, well, ever as far as I can tell. Not to mention asan-debug not working at all because it has a hyphen in it.

Edit:
Though you can trick asan-debug into working by replacing the hyphen with a comma. And optionally reversing the words, it isn't fussed.

(In reply to William Lachance (:wlach) (use needinfo!) from comment #17)

(In reply to Ian Moody [:Kwan] (UTC+0) from comment #16)

Anyway, here's a rough WIP that seems to actually work. Even passes the (current) tests. Need to write new ones.

This is looking great, thank you!

Though frankly I'm tempted to rip out any support for users passing in multiple builds (since clearly no-one uses it...), and only keep the internal support for use for the shippable->opt->pgo fallbacking.

It could be that things used to work, but broke when we moved over to taskcluster. I wouldn't be too sad if we lost this feature -- if someone really wants it, they can always re-add it.

Ah, actually part of the problem is that I misread the code/help text and misunderstood what it is for. It was introduced for FxOS which more often had two-part (or more) build types, like "jb-opt", "kk-opt", "1-opt", or even "kk-spark-eng-opt" ("asan-debug" is the only such one these days), and I guess the comma use was intended to save the user worrying about getting the parts in the right order. Given that FxOS has gone away and asan-debug is the only such left, I don't know if there is any value to keeping it around just so a user can do asan,debug (or swapped) instead of asan-debug.

Comment 19

[Ian Moody [:Kwan] (UTC+1)]

Attachment: [GitHub PR] Support shippable builds in mozregression

Well I think I've got this into a usable state, but I think it needs a little more testing before review.

Windows GUI build for the brave (direct exe link) (with bonus bug 1536013 included as well if you happen to be on win64-aarch64)

Comment 20

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Thank you. It works as expected. Tested with bisections for 2018-08, 2019-02 and the last days.

Comment 21

[Ian Moody [:Kwan] (UTC+1)]

Thank you for the testing Aryx :)

Comment 22

[William Lachance (:wlach)]

This is merged and new versions of mozregression have been released! Thank you :Kwan and :Callek!