1382214 - Stop running Talos on opt builds when we have PGO builds for that platform

Status: RESOLVED WONTFIX

(Release Engineering :: General, enhancement)

Description

[Chris AtLee [:catlee]]

Some platforms (win32, win64, linux64) run PGO builds per push in Taskcluster. That means we'll also get Talos tests scheduled on the PGO builds per push.

We currently will also get Talos tests scheduled on the opt builds.

This is a bit of a waste of machine time, since we're much more interested in the PGO results as those represent what we ship to users.

We should be able to make PGO builds & talos easy to do on Try now.

Given that, can we stop running Talos on opt builds?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I like removing the duplication and focusing on what we ship.  A few concerns (which might not be a reason to avoid this):

1) many times PGO is noisy or changes values when opt doesn't, it is nice to compare
2) developers locally test with opt builds, not pgo
3) pushing to try will require pgo, right now there is not a clear way to do this consistently for all platforms
4) I don't believe pgo works with artifact builds, our end to end time will be much longer for try pushes
5) we run linux64 as opt on beta, but it is with a pgo build- we should make sure the data we run on trunk == data on 'beta'

There could be other concerns- I would like to think on this more.

:rwood, do you have other concerns, thoughts, opinions on this topic?

Comment 2

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher) (UTC-8) from comment #1)
> 
> :rwood, do you have other concerns, thoughts, opinions on this topic?

Also may impact adding new tests, if new tests need to be tested locally first with PGO builds.

Comment 3

[Chris AtLee [:catlee]]

Local tests with opt builds aren't really comparable to results from automation are they? What's the particular concern about local testing with opt builds?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

pgo has changes in values that are not seen in opt- I see this at least once every other week- given this we could be trying to inveestigate a regression before/after a  change and not see it locally with opt.

Comment 5

[Johan Lorenzo [:jlorenzo]]

(In reply to Joel Maher ( :jmaher) (UTC-8) from comment #1)
> 3) pushing to try will require pgo, right now there is not a clear way to do this consistently for all platforms
* Windows works out of the box with Taskcluster windows \o/
* Mac hasn't had any pgo builds, if I understand correctly
* Linux64 will be handled in 1382725

Once Windows is tier-1, I'll make the pgo-jobs visible on trychooser via 691673.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

:jlorenzo, we are now at tier-1, is there a timeline for hacking on bug 691673 ?

Comment 7

[Johan Lorenzo [:jlorenzo]]

Thank you Joel for reminding me! I had the PR ready, I just haven't pushed it to GitHub. This is now done and in review, in bug 691673 comment 35.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

do we have a uniform method for running pgo on try, possibly:

./mach try -b o -p linux64-pgo,macosx64,win32-pgo,win64-pgo -u none -t all


if we can get to that state, then I would be fine moving over to pgo only.  What is confusing is mozilla-beta shows everything as opt, when it is really pgo- that is confusing both for comparing inside of perfherder, as well as examining inside of treeherder.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

we have had a few instances this quarter where pgo only regressions show up and they appear to be artifacts of pgo only, opt shows no change.