709192 - Make all builds on mozilla-inbound PGO

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

We are continually getting inbound into states (including now) where we have PGO bustage, despite the assurances before we went to periodic PGO that we never have PGO bugs.

Once we do, it's massively annoying to go back and trigger/retrigger, since the only option is to trigger a new set of every flavor of PGO build on a given revision.

It's only a matter of time before we will merge something that looks green, only because it hasn't yet had a PGO build on it or a rev above it.

The only people who should be waiting on inbound results are inbound sheriffs, all of whom are sick of periodic PGO.

Please switch us back to all-PGO.

Comment 1

[Phil Ringnalda (:philor)]

http://groups.google.com/group/mozilla.dev.tree-management/browse_thread/thread/784bf66882738e01

Comment 2

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

An alternative idea, we could identify pushes where PGO would not make a difference, for example we could do PGO only when the patch includes changes to cpp/h files or the build system files?

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

Filed bug 710048 to increase the frequency of PGO builds.  I am also writing up a post to the newsgroup from comment 1

Comment 4

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to John Ford [:jhford] from comment #3)
> Filed bug 710048 to increase the frequency of PGO builds.  I am also writing
> up a post to the newsgroup from comment 1

woukd it be hard to make the hg hook I suggest in comment 2?

Comment 5

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

For example, just looking at today, all pushes were java and js, so ideally there was no need to pgo at all, apart nightlies. That may save similar resources to doing a pgo every X hours, without the risk to not doing it often enough.

Comment 6

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

and should be feasible with a changegroup hook, using hg log -r $HG_NODE tip --template '{files}' in the hook script to get the list of the changed files

Comment 7

[Chris AtLee [:catlee]]

I wouldn't do it as an hghook, but rather in the buildbot scheduler.

Still, a list of files to trigger or ignore for PGO would be handy.

Comment 8

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

well whatever is better for buildbot, I was playing with a sip_pgo hook with something like this (it's probably safer to always-pgo-but-when-files-are than the opposite)

hg log -r $HG_NODE: --template '{files}' | tr ' ' '\n' | grep -v "\.js$
\.jsm$
\.sjs$
\.java$
\.xul$
\.xml$
\.xhtml$
\.htm?$
\.woff$
\.png$
\.jpg$
\.gif$
\.ico$
\.py$
\.dtd$
\.properties$
\.txt$
\.ini$
\.rdf$
\.css$"

On the other side, we should pgo for .c,.h,.cpp,.idl,.mm,configure,.in, ... but may be less safe, the advantage with the other approach is that the list can be improved by hitting false positives, without big risks.

Comment 9

[Phil Ringnalda (:philor)]

I wanted to solve two problems with this bug:

* Ensure that we don't merge anything to m-c which has not built and tested PGO

I don't know how careful about that the regular mergers are, since I'm not one of them. I do know that non-regular mergers fairly frequently merge things in a hurry, when they have barely tested out on non-PGO, much less PGO.

Switching to 3 hour periods would cut that risk in half, but not eliminate it.

* When faced with a PGO failure, and a PGO success before it, know what push caused the problem *then*, not in four to twelve hours.

Switching to 3 hour periods will cut the number of suspect pushes in half, but not eliminate them. Backing everything out blindly is not the solution, see the newsgroup. It's entirely possible that I'll do it the next few times, just to accentuate the problem, but that is not something that I will ask other people to do, so I guess when I see bustage while I'm at work and don't have a tree, I'll just ignore it.

Comment 10

[Phil Ringnalda (:philor)]

Anyway, we're not going to do this, and doing other things should go elsewhere.

Comment 11

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to Phil Ringnalda (:philor) from comment #9)
> I wanted to solve two problems with this bug:
> 
> * Ensure that we don't merge anything to m-c which has not built and tested
> PGO

We can add this rule to the inbound rules.
I'll post new rules to the ng and get agreement.