Closed Bug 691673 Opened 9 years ago Closed 3 years ago

give trychooser the option of triggering (or not) a PGO build for platforms that support PGO

Categories

(Release Engineering :: General, defect, P3)

x86
macOS
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jhford, Assigned: u504868)

References

Details

(Keywords: sheriffing-P2, trychooser, Whiteboard: [pgo] [workaround: http://mzl.la/13H6QiJ ])

Attachments

(2 files)

Probably, the easiest way is to do this is to:
1) for try branches, always add PGO builders.  This will create the linux-pgo, linux64-pgo and win32-pgo platforms
2) add the PGO builders to the normal dep scheduler
3) don't create a Nightly scheduler for 4-hourly builds

This will result in someone asking for PGO by asking for any of the linux-pgo, linux64-pgo and win32-pgo platforms.
Whoever takes this should consult compare-talos devs.
Whiteboard: [trychooser][pgo]
Priority: -- → P3
Duplicate of this bug: 588099
Depends on: 701559
Is there something in particular holding this up? I could really use the ability to run both pgo and non-pgo builds in one try push.
Keywords: trychooser
Whiteboard: [trychooser][pgo] → [pgo]
Keywords: sheriffing-P2
We could now do #1 but make them nondefault on try.
I attempted to do a PGO build as per: https://wiki.mozilla.org/ReleaseEngineering/TryChooser#What_if_I_want_PGO_for_my_build, but it didn't seem to work, specifically I didn't see the pgo/profilserver.py run as part of the build process.
(In reply to Joel Maher (:jmaher) from comment #5)
> specifically I didn't see the pgo/profilserver.py run as part of the build
> process.

Me neither, but I did see pgo/profileserver.py with an e run in https://tbpl.mozilla.org/php/getParsedLog.php?id=20694829&tree=Try
Product: mozilla.org → Release Engineering
What is the status of this bug?
(In reply to :Ehsan Akhgari (needinfo? me!) from comment #7)
> What is the status of this bug?

I saw this bug listed as a 'blocker' to the australis work -> whilst this would be good to see fixed, don't forget:
https://wiki.mozilla.org/ReleaseEngineering/TryChooser#What_if_I_want_PGO_for_my_build
Oh though just read comment 5, is that still broken?
(In reply to Ed Morley [:edmorley UTC+1] from comment #9)
> Oh though just read comment 5, is that still broken?

I'm not sure, this try run seems like is PGOed? <https://tbpl.mozilla.org/?tree=Try&rev=195c4d142edf>

I basically asked because this is really annoying and it seems like it has falled off everybody's radar...
(In reply to :Ehsan Akhgari (needinfo? me!) from comment #10)
> (In reply to Ed Morley [:edmorley UTC+1] from comment #9)
> > Oh though just read comment 5, is that still broken?
> 
> I'm not sure, this try run seems like is PGOed?
> <https://tbpl.mozilla.org/?tree=Try&rev=195c4d142edf>

Does this mean yes or no?

PGOMGR : warning PG0188: No .PGC files matching 'xul!*.pgc' were found.

   Creating library xul.lib and object xul.exp

LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs; use /NODEFAULTLIB:library

Generating code

4738 of 153786 (  3.08%) profiled functions will be compiled for speed

153786 of 153786 functions (100.0%) were optimized using profile data

2576771502 of 2576771502 instructions (100.0%) were optimized using profile data
Yes, that shows PGO as working.
The warning about the missing pgc file is due to it having already been merged earlier in the process. It's expected.
Yeah, so it's good that the instructions here are good.  But note that it's very easy to make a mistake in following them, and then you may end up drawing the wrong conclusions, etc.
Comment 5 implying the workaround wasn't working was the reason this got brought up as blocking Australis. I can confirm that the workaround in comment 5 does work now.
Whiteboard: [pgo] → [pgo] [workaround: http://mzl.la/13H6QiJ ]
Found in triage.

(In reply to Matthew N. [:MattN] from comment #15)
> Comment 5 implying the workaround wasn't working was the reason this got
> brought up as blocking Australis. I can confirm that the workaround in
> comment 5 does work now.

ok, good to know the workaround does work and this is no longer blocking Australis. Thanks MattN.
Component: Other → Tools
OK, I just wasted another try run because I messed up how I defined MOZ_PGO.  Can we please prioritize this?  We don't need PGO builds but every now and then we keep getting bitten by not having this as trychooser syntax.  Thanks!
I wanted to do some try pushes on beta to bisect a failure and I didn't get PGO builds.  I followed this:
https://wiki.mozilla.org/ReleaseEngineering/TryChooser#What_if_I_want_PGO_for_my_build

by adding "mk_add_options MOZ_PGO=1", and it failed.

Does this not work on pushes for beta? or for windows only pushes?
(In reply to Joel Maher (:jmaher) from comment #18)
> Does this not work on pushes for beta? or for windows only pushes?

I don't know about Beta but I just recently debugged an intermittent orange on Linux PGO builds, so that definitely works.
(In reply to Joel Maher (:jmaher) from comment #18)
> Does this not work on pushes for beta? or for windows only pushes?

Anything other than Linux32/Linux64/Windows opt will fail. We have bugs on file for doing saner things when PGO is specified on unsupported platforms.
I did windows opt with no luck.  I am trying to track down a performance regression on beta, here is a link to try:
https://tbpl.mozilla.org/?tree=Try&startdate=2014-04-16&enddate=2014-04-19&rev=21fa8f07e167

and the specific revision:
https://hg.mozilla.org/try/rev/21fa8f07e167
That's a PGO build, see the lines like "1663 of 43071 (  3.86%) profiled functions will be compiled for speed" in the log. It doesn't say on tbpl that it is, because saying so there comes from buildbot knowing that it is, which comes from buildbot being the one to decide it should be MOZ_PGO=1, rather than the mozconfig, but it's still PGO.
thanks philor, user error!
Until this bug is fixed, can we add a link to the trychooser page that says something like "Want a PGO build? See [here]" with a link to the wiki page?
This is a temporary thing only and should be reverted once we actually have proper PGO support in trychooser.
Attachment #8506961 - Flags: review?(jhford)
Comment on attachment 8506961 [details] [diff] [review]
Add a link to the wiki page on trychooser

Review of attachment 8506961 [details] [diff] [review]:
-----------------------------------------------------------------

While this looks OK to me, I don't own that particular tool or repo.  I think Lukas will know more about this.
Attachment #8506961 - Flags: review?(jhford) → review?(lsblakk)
Comment on attachment 8506961 [details] [diff] [review]
Add a link to the wiki page on trychooser

Ed Morley would probably be the right one to ask for this. Except there's no need, I am comfortable saying this is totally fine. (And a good idea, too.)
Attachment #8506961 - Flags: review?(lsblakk) → review+
QA Contact: hwine
:glandium and :gps indicated in bug 1322735 that you can do this via treeherder now. I assume there's some magic incantation to make it work via try params (or could be), but I don't know what that is nor who would know.
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(gps)
TaskCluster automatically schedules PGO builds with "-p all -b do". I /think/ the magic incantation to schedule just PGO is e.g. "-p win32-pgo". If that doesn't work, we should get a TaskCluster :: Task Configuration bug (this one?) tracking better support for triggering PGO builds. dustin can assist with that.

Buildbot builds use their own Try syntax parser. I'm pretty sure it doesn't have support for PGO and you need to go through the "trigger job" API (likely via Treeherder) to schedule the PGO jobs. TaskCluster is the future, so hopefully we can just ignore Buildbot's shortcomings.
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(gps)
for buildbot to do pgo, you need to hack your mozconfig prior to pushing.  It would be nice if we can clarify this so that we use -b or -p most effectively- and I agree, sticking to taskcluster as the primary system will be ideal :)
And given that what you wanted wasn't builds but instead "talos run on Windows PGO builds" and neither Taskcluster (which doesn't run any tests at all on its Windows PGO, and which only runs talos by telling buildbot to run talos, and cannot tell buildbot to run a talos job that buildbot doesn't already know about), nor the buildbot builds as triggered by Add new jobs (which are just a weird artifact of how things are set up, not intentional and supported jobs, and which run absolutely no test suites at all) will run your talos, what they actually meant to tell you in bug 1322735 as it applied to your problem was "".
Component: Tools → General
Depends on: 1382725
See Also: → 1382214
You can do this right now with e.g. '-p linux64-pgo,win64-pgo'
those are not documented on http://trychooser.pub.build.mozilla.org/ (not that that page is the best thing in the world)
Assignee: nobody → jlorenzo
Status: NEW → ASSIGNED
Attachment #8893370 - Flags: review?(rgarbas)
Comment on attachment 8893370 [details] [review]
mozilla-releng/services PR

PR is merged, release (to production) is going to happen on Tue/Wed
Attachment #8893370 - Flags: review?(rgarbas) → review+
Is this done? I think try chooser is dead and/or dying because of "./mach try fuzzy".
Flags: needinfo?(jlorenzo)
Comment on attachment 8893370 [details] [review]
mozilla-releng/services PR

Was landed and deployed in August 2017[1]. Thank you for spotting this unclosed bug!

[1] https://github.com/mozilla-releng/services/pull/548#issuecomment-320203915
Flags: needinfo?(jlorenzo)
Attachment #8893370 - Flags: checked-in+
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.