Closed Bug 1532412 Opened 8 months ago Closed 6 months ago

Adjust for disabled opt builds in bug 1522111

Categories

(Testing :: mozregression, defect)

Version 3
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nthomas, Assigned: Kwan)

References

Details

Attachments

(1 file)

Bug 1522111 disabled some opt builds on inbound and autoland, in favour of pgo. If mozregression supports bisecting into on-push builds on those branches it may need to make some changes.

From the code, I am fairly certain this should still work as expected. Is this change landed and ready for testing? i.e. if I bisect into inbound/autoland on March 3rd/4th, will I only get pgo builds?

Flags: needinfo?(nthomas)

Yes, I believe it's all landed. There's a slight regression in that linux64 opt builds are still running.

Flags: needinfo?(nthomas)

Try to bisect bug 1532110 and switching to pgo will be necessary.

Aryx, is mozregression not working for you? As far as I can tell from the source it should be able to fetch pgo builds from integration branches in its current form, at least on Windows and Linux. I think we might need to make it search for pgo builds on Mac though.

Flags: needinfo?(aryx.bugmail)

It's more about switching from default 'opt' to 'pgo' as default if available. Else it will work fine as long as one bisects on central but fail once it tries to switch to autoland.

Flags: needinfo?(aryx.bugmail)

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #5)

It's more about switching from default 'opt' to 'pgo' as default if available. Else it will work fine as long as one bisects on central but fail once it tries to switch to autoland.

My impression was that mozregression would actually search for builds of both types for every revision, at least if specified:

https://github.com/mozilla/mozregression/blob/df17ccdb445a9ebb665e695d4f77d8bdae36938f/mozregression/fetch_configs.py#L431

Is that not the case?

No, it will fail with messages similar to this:
WARNING : Skipping build 1bc807fb4c89: Unable to find build info using the taskcluster route 'gecko.v2.autoland.revision.1bc807fb4c895d326aafc4a5bc5c9239c093b432.firefox.win64-opt'

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #7)

No, it will fail with messages similar to this:
WARNING : Skipping build 1bc807fb4c89: Unable to find build info using the taskcluster route 'gecko.v2.autoland.revision.1bc807fb4c895d326aafc4a5bc5c9239c093b432.firefox.win64-opt'

Hi :coop, could someone on your team with taskcluster python client library experience help debug this? We really need mozregression to work for tracking down the root causes of bugs and not being able to bisect inbound/autoland is quite a deal-breaker.

The problem is probably either in taskcluster itself / the client library we're using (the copy used by mozregression is very out of date -- you can see an effort to update it here-- https://github.com/mozilla/mozregression/pull/513/commits/b962c5e23a07f55761106d3c7dee2e389446b080) or in the file I link to in comment 6. I am very happy to provide mozregression help/guidance/reviews as needed, either here or on irc/slack.

Flags: needinfo?(coop)

(In reply to William Lachance (:wlach) (use needinfo!) from comment #8)

The problem is probably either in taskcluster itself / the client library we're using (the copy used by mozregression is very out of date -- you can see an effort to update it here-- https://github.com/mozilla/mozregression/pull/513/commits/b962c5e23a07f55761106d3c7dee2e389446b080) or in the file I link to in comment 6. I am very happy to provide mozregression help/guidance/reviews as needed, either here or on irc/slack.

I've cc-ed Pete here for further guidance. Updating the client library here is a good first step.

Flags: needinfo?(coop) → needinfo?(pmoore)
Flags: needinfo?(pmoore) → needinfo?(aki)

https://tools.taskcluster.net/index/gecko.v2.autoland.revision.1bc807fb4c895d326aafc4a5bc5c9239c093b432.firefox.win64-opt doesn't exist. I'm guessing someone has changed which indexes we populate.

https://tools.taskcluster.net/index/gecko.v2.autoland.revision.1bc807fb4c895d326aafc4a5bc5c9239c093b432.firefox/win64-pgo does exist, though :Callek may be changing these when we move to shippable builds in bug 1352113. You can look at https://tools.taskcluster.net/index/gecko.v2.autoland.revision.1bc807fb4c895d326aafc4a5bc5c9239c093b432.firefox to see which platforms are indexed under that revision.

Does mozregression find the -pgo build? (Comment 7 says no)

Flags: needinfo?(aki)

(In reply to Aki Sasaki [:aki] from comment #10)

Does mozregression find the -pgo build? (Comment 7 says no)

I don't presently have a win64 environment to test this case, but that is my read of what :Aryx in comment 7, yes.

I never really had any doubt that the builds were being produced, my question is why isn't mozregression able to find them? As mentioned in comment 6, I thought it was configured to do so.

(In reply to William Lachance (:wlach) (use needinfo!) from comment #11)

(In reply to Aki Sasaki [:aki] from comment #10)

Does mozregression find the -pgo build? (Comment 7 says no)

I don't presently have a win64 environment to test this case, but that is my read of what :Aryx in comment 7, yes.

I never really had any doubt that the builds were being produced, my question is why isn't mozregression able to find them? As mentioned in comment 6, I thought it was configured to do so.

I'm guessing it's a mozregression bug in this block. Most likely we want to try multiple indexes and only fail out if we don't find any of them.

(In reply to Aki Sasaki [:aki] from comment #12)

I'm guessing it's a mozregression bug in this block. Most likely we want to try multiple indexes and only fail out if we don't find any of them.

Ah yes, that sounds like a plausible theory. Is this something you or someone else on the taskcluster team might be able to do? Not meaning to pass the buck, but I don't really have a lot of time to maintain mozregression these days, and don't understand the taskcluster all that well.

:Callek is willing to take a look.

Assignee: nobody → bugspam.Callek

Is it just me, or has multiple build types on the command line not actually worked since, well, ever as far as I can tell. Not to mention asan-debug not working at all because it has a hyphen in it.

Edit:
Though you can trick asan-debug into working by replacing the hyphen with a comma. And optionally reversing the words, it isn't fussed.

Anyway, here's a rough WIP that seems to actually work. Even passes the (current) tests. Need to write new ones.

Though frankly I'm tempted to rip out any support for users passing in multiple builds (since clearly no-one uses it...), and only keep the internal support for use for the shippable->opt->pgo fallbacking.

(In reply to Ian Moody [:Kwan] (UTC+0) from comment #16)

Anyway, here's a rough WIP that seems to actually work. Even passes the (current) tests. Need to write new ones.

This is looking great, thank you!

Though frankly I'm tempted to rip out any support for users passing in multiple builds (since clearly no-one uses it...), and only keep the internal support for use for the shippable->opt->pgo fallbacking.

It could be that things used to work, but broke when we moved over to taskcluster. I wouldn't be too sad if we lost this feature -- if someone really wants it, they can always re-add it.

(In reply to Ian Moody [:Kwan] (UTC+0) from comment #15)

Is it just me, or has multiple build types on the command line not actually worked since, well, ever as far as I can tell. Not to mention asan-debug not working at all because it has a hyphen in it.

Edit:
Though you can trick asan-debug into working by replacing the hyphen with a comma. And optionally reversing the words, it isn't fussed.

(In reply to William Lachance (:wlach) (use needinfo!) from comment #17)

(In reply to Ian Moody [:Kwan] (UTC+0) from comment #16)

Anyway, here's a rough WIP that seems to actually work. Even passes the (current) tests. Need to write new ones.

This is looking great, thank you!

Though frankly I'm tempted to rip out any support for users passing in multiple builds (since clearly no-one uses it...), and only keep the internal support for use for the shippable->opt->pgo fallbacking.

It could be that things used to work, but broke when we moved over to taskcluster. I wouldn't be too sad if we lost this feature -- if someone really wants it, they can always re-add it.

Ah, actually part of the problem is that I misread the code/help text and misunderstood what it is for. It was introduced for FxOS which more often had two-part (or more) build types, like "jb-opt", "kk-opt", "1-opt", or even "kk-spark-eng-opt" ("asan-debug" is the only such one these days), and I guess the comma use was intended to save the user worrying about getting the parts in the right order. Given that FxOS has gone away and asan-debug is the only such left, I don't know if there is any value to keeping it around just so a user can do asan,debug (or swapped) instead of asan-debug.

Well I think I've got this into a usable state, but I think it needs a little more testing before review.

Windows GUI build for the brave (direct exe link) (with bonus bug 1536013 included as well if you happen to be on win64-aarch64)

Assignee: bugspam.Callek → moz-ian
Status: NEW → ASSIGNED

Thank you. It works as expected. Tested with bisections for 2018-08, 2019-02 and the last days.

Attachment #9055358 - Attachment description: [GitHub PR] WIP → [GitHub PR] Support shippable builds in mozregression
Attachment #9055358 - Flags: checked-in+

Thank you for the testing Aryx :)

This is merged and new versions of mozregression have been released! Thank you :Kwan and :Callek!

Status: ASSIGNED → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.