901715 - PGO talos jobs were reporting as Non-PGO after switchover to talos mozharness

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Justin Dolske [:Dolske]]

My team's looking at a regression between the UX and m-c branches, and have noticed a recent oddity with the Talos data.

Here's a link for the results of the Windows XP "Paint" test, for UX/m-c, PGO/non-PGO:

http://graphs.mozilla.org/graph.html#tests=[[82,1,37],[82,59,37],[82,94,37],[82,137,37]]&sel=1373147453313,1375739453313&displayrange=30&datatype=running

Note that no PGO data is reported after July 30th.

Also note that after July 30th, the PGO data has become bimodal.

Perhaps the PGO results are being reported as not-PGO? The data sets do suspiciously line up.

Perhaps complicating things is that I additionally saw that this test on OS X 10.8 is also currently bimodal, but it's done that on-and-off over the last few months. But the XP thing is definitely new.

Comment 1

[Matthew N. [:MattN]]

(In reply to Justin Dolske [:Dolske] from comment #0)
> Perhaps the PGO results are being reported as not-PGO? The data sets do
> suspiciously line up.

If this is the case, I would like for us to drop the bad data so it doesn't mess with tools like talos' compare.py which looks at the minimum and maximum value for a period of many days. The PGO numbers will lower the lower bound and mess up comparisons.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

this is true, since our switch to mozharness, all of our pgo builds are being reported as Non-PGO!

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

on desktop ubuntu pgo from m-c (https://tbpl.mozilla.org/php/getParsedLog.php?id=26244855&tree=Mozilla-Central&full=1), we run this command:

/home/cltbld/talos-slave/test/build/venv/bin/talos --noisy --debug -v --executablePath /builds/slave/talos-slave/test-pgo/build/application/firefox/firefox --title talos-linux32-ix-001 --symbolsPath http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-pgo/1375837202/firefox-26.0a1.en-US.linux-i686.crashreporter-symbols.zip --activeTests tresize --results_url http://graphs.mozilla.org/server/collect.cgi --output talos.yml --branchName Firefox-Non-PGO --datazilla-url https://datazilla.mozilla.org/talos --authfile /builds/slave/talos-slave/test-pgo/oauth.txt --mozAfterPaint --filter ignore_first:5 --filter median --webServer localhost


Notice the '--branchName Firefox-Non-PGO', this should be '--branchName Firefox'.

Comment 4

[Armen [:armenzg]]

Fallout from bug 900545.

Comment 5

[Armen [:armenzg]]

Attachment: talos_branch.diff

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 787124 [details] [diff] [review]
talos_branch.diff

>+                            args = _makeGenerateMozharnessTalosBuilderArgs(suite, talosBranch, platform,
>                                                                           factory_kwargs, branch_config, platform_config)

nit: line up the factory_kwargs (additional space)

Comment 7

[Armen [:armenzg]]

Comment on attachment 787124 [details] [diff] [review]
talos_branch.diff

https://hg.mozilla.org/build/buildbotcustom/rev/0e472dbf2bf0

Comment 8

[Armen [:armenzg]]

This will be live after the next reconfig.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

In production

Comment 10

[Armen [:armenzg]]

I see PGO reporting resuming:
http://graphs.mozilla.org/graph.html#tests=[[82,59,37],[82,1,37],[82,63,1]]&sel=none&displayrange=30&datatype=running

I think it has stopped being bi-modal:
http://graphs.mozilla.org/graph.html#tests=[[82,137,37],[82,94,37],[82,131,37]]&sel=1375723973547.1123,1375967101637,55.26315789473685,181.57894736842104&displayrange=7&datatype=running

I don't have a clear understanding on how to move the misreported data points.

Any suggestions? Owners?

Comment 11

[Matthew N. [:MattN]]

(In reply to Armen Zambrano G. [:armenzg] (Release Enginerring) (EDT/UTC-4) from comment #10)
> I don't have a clear understanding on how to move the misreported data
> points.
> 
> Any suggestions? Owners?

TBPL has the data from each job and know whether it's PGO so we can use its sources.

Comment 12

[Armen [:armenzg]]

If I understand correctly, once the misreporting stops, the new baseline is what is used to catch regressions (only the last N numbers are taken into consideration).