1570893 - 15.63% tp5n nonmain_startup_fileio (windows10-64-shippable) regression

Status: RESOLVED WONTFIX

(Testing :: General, defect, P3)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression around this push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=b1d382dd18da4460ac88849c12049fa6d1a4cee2

Regressions:

16% tp5n nonmain_startup_fileio windows10-64-shippable opt e10s stylo 1,238,800.65 -> 1,432,454.33

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=21373

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Performance_sheriffing/Talos/Running

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Performance_sheriffing/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

This is a really tricky situation. Had 2 explanations for this so far, but none held.

It's possible this is some sort of performance intermittent regression.
My new explanation now: between May 29 and June 1 a Windows infra change happened. Now, why am I not able to reproduce this on older commits? It's possible the infra change problem kicks in because of a Bugzilla bug that's also related to Windows.

:aklotz could you look into this weird situation? I'm pretty much clueless at this point. The test outputs bimodal data at this moment, which is not ideal.

Comment 2

[Liz Henry (:lizzard) (relman/hg->git project)]

Following up on n-i in email.

Comment 3

[(No longer employed by Mozilla) Aaron Klotz]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriff from comment #1)

This is a really tricky situation. Had 2 explanations for this so far, but none held.

It's possible this is some sort of performance intermittent regression.
My new explanation now: between May 29 and June 1 a Windows infra change happened. Now, why am I not able to reproduce this on older commits? It's possible the infra change problem kicks in because of a Bugzilla bug that's also related to Windows.

:aklotz could you look into this weird situation? I'm pretty much clueless at this point. The test outputs bimodal data at this moment, which is not ideal.

Unfortunately I don't really have any ideas as to what is going on. The good news is that, even though that alert shows a spike in startup I/O, it is not on the main thread. I'd be more concerned if the I/O was for the main thread, but since it isn't, and in the absence of any other corroborating evidence of negative impact, I'm inclined to say that we should just eat this regression.