1494882 - [meta] 2.73 - 54% sessionrestore / sessionrestore_no_auto_restore / tp5n nonmain_startup_fileio / tp5o responsiveness / tp5o_webext responsiveness / tps (linux64-qr, windows) regression on push 126409bdf326 (Wed Sep 26 2018)

Status: RESOLVED DUPLICATE of bug 1495065

(Testing :: Talos, defect)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=1b268215517c906a22e5341b3978f5b551e92328&tochange=126409bdf326645e735ee147bd70bd7d09759165

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 54%  tp5o responsiveness windows10-64 pgo e10s stylo                0.49 -> 0.76
 49%  tp5o responsiveness windows7-32 pgo e10s stylo                 0.47 -> 0.70
 45%  tp5o responsiveness windows7-32 opt e10s stylo                 0.52 -> 0.74
 42%  tp5o responsiveness windows10-64 opt e10s stylo                0.55 -> 0.79
 38%  tp5o responsiveness windows10-64-qr opt e10s stylo             0.57 -> 0.78
 28%  tp5o_webext responsiveness windows7-32 pgo e10s stylo          0.95 -> 1.21
 21%  tp5o_webext responsiveness windows10-64 opt e10s stylo         1.09 -> 1.33
 21%  tp5o_webext responsiveness windows10-64 pgo e10s stylo         1.01 -> 1.23
 17%  tp5n nonmain_startup_fileio windows7-32 opt e10s stylo         2,382,029.08 -> 2,795,380.50
 16%  tp5o responsiveness linux64-qr opt e10s stylo                  1.16 -> 1.35
 14%  tp5o_webext responsiveness linux64-qr opt e10s stylo           1.73 -> 1.96
 13%  tps windows7-32 opt e10s stylo                                 11.46 -> 12.98
 11%  tps windows10-64 pgo e10s stylo                                11.23 -> 12.44
 10%  tps windows7-32 pgo e10s stylo                                 10.22 -> 11.24
  6%  sessionrestore windows10-64 opt e10s stylo                     288.50 -> 305.58
  5%  sessionrestore_no_auto_restore windows10-64 pgo e10s stylo     287.88 -> 301.42
  5%  sessionrestore_no_auto_restore windows10-64 opt e10s stylo     314.12 -> 328.67
  5%  sessionrestore_no_auto_restore windows7-32 opt e10s stylo      318.92 -> 333.42
  4%  sessionrestore windows7-32 opt e10s stylo                      300.42 -> 313.67
  4%  sessionrestore windows10-64 pgo e10s stylo                     268.42 -> 278.92
  4%  sessionrestore windows7-32 pgo e10s stylo                      277.00 -> 287.58
  3%  sessionrestore_no_auto_restore windows7-32 pgo e10s stylo      297.71 -> 305.83


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=16204

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

Unless there's something strange with these figures, I'll close this as WONTFIX, as it's just a harness update.

Comment 2

[Andrew Swan [:aswan]]

I agree with comment 1 in principle but some of these changes are quite large, I think follow-ups to look into these and understand them would be good.  I will file those...

Comment 3

[Ionuț Goldan [:igoldan]]

Any updates here?

Comment 4

[Andrew Swan [:aswan]]

Ah, I spun off the 3 smaller bugs.  I think this bug can just be closed or labeled meta or something, any real investigation will take place on those other bugs.  Since we're approaching the 64 cutoff and this is clearly a regression in the test infrastructure rather than in the product, I don't think that looking into it has been a priority for anybody.

Comment 5

[Ionuț Goldan [:igoldan]]

(In reply to Andrew Swan [:aswan] from comment #4)
> Ah, I spun off the 3 smaller bugs.  I think this bug can just be closed or
> labeled meta or something, any real investigation will take place on those
> other bugs.  Since we're approaching the 64 cutoff and this is clearly a
> regression in the test infrastructure rather than in the product, I don't
> think that looking into it has been a priority for anybody.

Closing this given comment above.