1566744 - 2.26 - 14.81% raptor-tp6-facebook-firefox-cold / raptor-tp6-google-firefox-cold / (windows10-64-shippable, windows10-64-shippable-qr, windows7-32-shippable) regression on push 9559ef8f347dc0e4e92546954472d4153283d5f2 (Thu July 11 2019)

Status: RESOLVED FIXED

(Core :: Security: Process Sandboxing, defect, P1)

Description

[Florin Strugariu [:Bebe]]

Raptor has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=9559ef8f347dc0e4e92546954472d4153283d5f2

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

15% raptor-tp6-amazon-firefox-cold fcp windows7-32-shippable opt 428.17 -> 491.58
15% raptor-tp6-amazon-firefox-cold windows7-32-shippable opt 508.92 -> 583.35
14% raptor-tp6-amazon-firefox-cold windows10-64-shippable opt 502.21 -> 571.00
13% raptor-tp6-amazon-firefox-cold fcp windows10-64-shippable opt 421.50 -> 474.42
11% raptor-tp6-google-firefox-cold fcp windows7-32-shippable opt 322.17 -> 357.08
11% raptor-tp6-google-firefox-cold windows7-32-shippable opt 332.54 -> 368.57
11% raptor-tp6-google-firefox-cold windows7-32-shippable opt 333.48 -> 369.07
10% raptor-tp6-amazon-firefox-cold fcp windows10-64-shippable-qr opt 441.04 -> 485.67
9% raptor-tp6-google-firefox-cold windows10-64-shippable opt 328.78 -> 358.68
9% raptor-tp6-google-firefox-cold fcp windows10-64-shippable opt 319.17 -> 348.00
8% raptor-tp6-google-firefox-cold loadtime windows10-64-shippable-qr opt 480.00 -> 516.33
8% raptor-tp6-google-firefox-cold loadtime windows7-32-shippable opt 513.67 -> 552.33
7% raptor-tp6-google-firefox-cold windows10-64-shippable-qr opt 334.85 -> 357.33
7% raptor-tp6-google-firefox-cold loadtime windows10-64-shippable opt 509.08 -> 542.25
4% raptor-tp6-facebook-firefox-cold windows7-32-shippable opt 429.46 -> 446.65
3% raptor-tp6-facebook-firefox-cold loadtime windows10-64-shippable-qr opt 827.92 -> 854.58
3% raptor-tp6-facebook-firefox-cold loadtime windows7-32-shippable opt 840.17 -> 863.75
2% raptor-tp6-facebook-firefox-cold windows10-64-shippable-qr opt 418.23 -> 427.70

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=21930

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a Treeherder page showing the Raptor jobs in a pushlog format.

To learn more about the regressing test(s) or reproducing them, please see: https://wiki.mozilla.org/Performance_sheriffing/Raptor

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Performance_sheriffing/Talos/RegressionBugsHandling

Comment 1

[Bob Owen (:bobowen)]

I'm a little sceptical about this, because of the windows 7 differences.
This policy change excludes windows versions earlier than 10, because it broke audio, so "in theory" this wouldn't have affected windows 7 runs. Unless somehow just the changes to the binary have, which seems unlikely.

I'll do some separate performance try runs.

Comment 2

[Bob Owen (:bobowen)]

Grrr, it turns out that our windows 7 performance jobs are not windows 7 at all, they are win32 builds running on windows 10 [1]
This is very confusing and frustrating, but at least now I can stop trying to work out how the impossible is happening and try and find what causes the actual regression.

[1] https://wiki.mozilla.org/Performance_sheriffing/Talos/Misc#linux64.2C_win7x32.2C_win10x64

Comment 3

[Bob Owen (:bobowen)]

I'm dropping the sandbox level until I can investigate, because we also have a profiler regression (bug 1567236).

Comment 4

[Bob Owen (:bobowen)]

These were resolved by disabling the policy feature see bug 1567236 comment 11.