1276384 - Since enabling PGO on TC build times have become bimodal

Status: NEW

(Firefox Build System :: General, defect)

Description

[David Burns :automatedtester]

...And we should see why

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=2e29695fee8730d8dae52960dc98ff7ade38aacb&tochange=3e069aea556fa27693c7ea0660f98107148143af is the range that perfherder suggests and the top item looks like the cause.

Comment 1

[Gregory Szorc [:gps]]

IMO we should change the Perfherder metrics to take into account buildbot vs TC so the numbers are part of separate metrics. Pretty sure this would fall under the domain of mozharness since it is what's outputting the TinderboxPrint messages that get turned into Perfherder data.

FWIW, we have similar problems elsewhere. e.g. "clobber" builds have completely different times from non-clobber builds. For any given job, you end up with N variations. e.g. "clobber+treeherder" versus "non-clobber+buildbot." Trying to shoehorn everything into a string gets difficult because order matters. So what you really want is the ability to associate N tags/annotations against a metric so you group/filter/distinguish data accordingly. That would certainly require changes on Perfherder to support, however.

Comment 2

[William Lachance (:wlach)]

(In reply to Gregory Szorc [:gps] from comment #1)
> IMO we should change the Perfherder metrics to take into account buildbot vs
> TC so the numbers are part of separate metrics. Pretty sure this would fall
> under the domain of mozharness since it is what's outputting the
> TinderboxPrint messages that get turned into Perfherder data.
> 
> FWIW, we have similar problems elsewhere. e.g. "clobber" builds have
> completely different times from non-clobber builds. For any given job, you
> end up with N variations. e.g. "clobber+treeherder" versus
> "non-clobber+buildbot." Trying to shoehorn everything into a string gets
> difficult because order matters. So what you really want is the ability to
> associate N tags/annotations against a metric so you
> group/filter/distinguish data accordingly. That would certainly require
> changes on Perfherder to support, however.

You can specify extraOptions as part of a perfherder series as of bug 1261922, which we could use to specify things like this:

https://github.com/mozilla/treeherder/blob/master/schemas/performance-artifact.json#L66

:rwood recently made talos jobs generate 'e10s' options. Search for PERFHERDER_DATA in e.g. http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win32/1464444695/mozilla-inbound_win7_ix_test-g2-e10s-bm111-tests1-windows-build109.txt.gz