1604410 - 3.97 - 4.56% tsvgx (linux64-shippable-qr) regression on push 3badff2b37246ec2e3347a99556f1e9f0bf1277d (Thu December 12 2019)

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect, P3)

Description

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=3badff2b37246ec2e3347a99556f1e9f0bf1277d

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

5% tsvgx linux64-shippable-qr opt e10s stylo 242.57 -> 253.63
4% tsvgx linux64-shippable-qr opt e10s stylo 242.77 -> 252.42

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=24497

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/TestEngineering/Performance/Talos

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/TestEngineering/Performance/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/TestEngineering/Performance/Talos/RegressionBugsHandling

Comment 1

[Jamie Nicol [:jnicol]]

Is this an expected/acceptable outcome of moving some things to low priority threads, or does this need further investigation?

Comment 2

[Jeff Muizelaar [:jrmuizel]]

I think so yes. We're explicitly trading throughput for responsiveness here.

Comment 3

[Bert Peers [:bpeers]]

It seems a reasonable trade-off, I agree.

I'm slightly worried about two things though.

One, we didn't see a similar regression when the windows code went in. Are the Linux and Windows tests using different CPUs?

Two, Linux doesn't actually use the LP pools yet (gfx.webrender.enable-low-priority-pool false in StaticPrefList.yaml) so this would have to be entirely due to assigning core affinity. I guess it's possible that the regression is from having > 8 cores and the affinity causes scheduling conflicts by not allowing the workers to float to idle cores?

I guess I'll need to enable this by default and then watch for any further regressions. On the off chance that windows wasn't flagged for regression because actually enabling the feature offset any losses... hmm.

Comment 4

[Bert Peers [:bpeers]]

It looks like enabling this new pool by default has added another 20% regression on 2 of these 3 tests which is way too much (and also ~2.5% here https://bugzilla.mozilla.org/show_bug.cgi?id=1605084).
I can only speculate that locking the pool to cores has increased contention, maybe with some other processes/threads that are also locked, or maybe within the pool(s) via hyperthreading.
I'm no Linux expert so for now we'll have to back it out :(

@alexandrui Can we have these backed out please? Thanks!

commit c1b100a07224371a7a85ba6bbe2f642dd6d0260b
Author: Bert Peers <[e-mail]>
Date:   Wed Dec 18 00:06:40 2019 +0000

    Bug 1604570 - Enable low priority thread pool on Linux r=jrmuizel

    Differential Revision: https://phabricator.services.mozilla.com/D57504

commit c14794eba3ab27aa401207e22ac44a8c9b402362
Author: Bert Peers <[e-mail]>
Date:   Thu Dec 12 16:22:44 2019 +0000

    Bug 1602907 - Add thread priority and affinity code for non-Windows r=jrmuizel

    pthread priority and affinity for Linux; moved to a helper function

    Differential Revision: https://phabricator.services.mozilla.com/D56782

Comment 5

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Done. Thanks!

Comment 6

[Marian Raiciof [:marauder]]

These 2 backouts fixed the regression:
https://bugzilla.mozilla.org/show_bug.cgi?id=1602907#c6
https://bugzilla.mozilla.org/show_bug.cgi?id=1604570#c5

== Change summary for alert #24580 (as of Mon, 23 Dec 2019 10:51:57 GMT) ==

Improvements:

18% tsvgx linux64-shippable-qr opt e10s stylo 293.88 -> 241.97
17% tsvgx linux64-shippable-qr opt e10s stylo 289.82 -> 241.57
7% tsvg_static linux64-shippable-qr opt e10s stylo 48.20 -> 44.66

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24580