1487316 - 6.57 - 10.78% tresize / tscrollx (windows10-64, windows7-32) regression on push b8fdb9372c775f81cbe2f5da4e4c2257433980a2 (Tue Aug 28 2018)

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=b8fdb9372c775f81cbe2f5da4e4c2257433980a2

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 11%  tresize windows10-64 opt e10s stylo     6.62 -> 7.34
  7%  tscrollx windows7-32 opt e10s stylo     0.68 -> 0.73

Improvements:

 15%  tsvgx windows7-32 opt e10s stylo     168.11 -> 142.96


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=15385

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

These regressions seems to have been caused by infra changes. I retriggered old changesets and their new jobs results are similar to those of the regressions. Out of the message above, only the timestamp could point to when the causing change happened.

Comment 2

[Ionuț Goldan [:igoldan]]

:gps do you have any clue on this issue?

Comment 3

[Gregory Szorc [:gps]]

You can always ping one of the Taskcluster Windows people about the Windows machines to see if a deploy occurred around the time of the regression. Start with :grenade.

It is possible this is due to rolling out of patches to mitigate Intel CPU vulnerabilities. I know Intel recently released a new microcode. I anticipate that its rollout will cause performance regressions across all of CI. Something similar happened when AWS rolled out the Spectre/Meltdown mitigations earlier this year. Whether this has happened on our infrastructure in the past few days, I'm not sure.

Comment 4

[Ionuț Goldan [:igoldan]]

:grenade can you confirm :gps's explanation?

Comment 5

[Rob Thijssen [:grenade (EET/UTC+0300)]]

infra changes to windows taskcluster test workers this week were limited to Windows 10 only. for context, see bug 1485757.

the changes were to:
- install the windows performance toolkit (wpt):
  https://github.com/mozilla-releng/OpenCloudConfig/commit/929d0c425ebabd5f00963a882dbe033b2f38b170
- run mofcomp mozprofilerprobe:
  https://github.com/mozilla-releng/OpenCloudConfig/commit/20dd55f14327742c322e9f0988e9405eaedd3602

i know nothing about what xperf/wpt or mozprofilerprobe does or if it could be linked to the performance regression described here. my involvement was only due to my familiarity with occ and being able to get these changes applied to the windows 10 machines. :jmaher may have the knowledge to understand if there is a relationship between the changes and the regressions.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I am not sure we have an explanation here and I believe we need to accept this as infrastructure.