1657769 - Start running talos tests with software webrender

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task, P2)

Description

[Matt Woodrow (:mattwoodrow)]

Similar to bug 1656811, it would be really good to enable some performance tests for software webrender, so that we can get an idea of performance.

I did an initial run forcing it to be enabled to get some initial numbers:

https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=0b7747f42985e593f79b1af9731c2ee44d7fd871&framework=1&selectedTimeRange=172800

From that, I think the tests that show useful data (where SW WR is slower than BasicCompositor, and it's worse than any existing WR regressions) are:

basic_compositor_video - bcv
pdfpaint - o
rasterflood_gradient - g4
rasterflood_svg - g4
TART - s
tp5o_scroll - g1
tscrollx - s
tsvgx - s

I think just enabling s,g1,g4,o and bcv would be sufficient to get useful coverage for now.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: Bug 1657769 - Run some Talos tests with software webrender on linux64-qr, r=jmaher

Comment 2

[Andrew Halberstadt [:ahal]]

Looks like this patch was WIP and wasn't ready to land yet. Things are a bit hectic atm, but I'll try to get to this soonish. Feel free to needinfo me if it's taking too long.

Comment 3

[Andrew Halberstadt [:ahal]]

Looks like there are some broken tasks on Windows and Mac:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d774af14e44a8be35a2eabbd6b125ec04b5bca9e

Want me to adjust to only run the tasks that are passing? Or want me to just run them all on Linux for now and wait for you or someone on the perftest team to fix the broken tasks, then enable the platform wholesale?

Comment 4

[Andrew Halberstadt [:ahal]]

Also do we want to run these on -qr platforms, not -qr platforms or both?

Comment 5

[Matt Woodrow (:mattwoodrow)]

I think we only need this to run on linux64-qr for now, and we can add other platforms once it becomes more stable there.

It looks like the bcv job isn't currently enabled for linux64-qr though, is it possible to get that added?

Comment 6

[Andrew Halberstadt [:ahal]]

(In reply to Matt Woodrow (:mattwoodrow) (PTO until 24th August) from comment #5)

It looks like the bcv job isn't currently enabled for linux64-qr though, is it possible to get that added?

It looks like talos-bcv is currently not configured to run on any -qr platforms. Does that change your mind? There's no simple way of only turning it on there for the "software-webrender" variant, we'd likely need to turn it on for normal -qr platforms as well. I'd be tempted to handle this in a follow-up bug and get feedback from the perftest team.

Comment 7

[Andrew Halberstadt [:ahal]]

(In reply to Andrew Halberstadt [:ahal] from comment #6)

There's no simple way of only turning it on there for the "software-webrender" variant, we'd likely need to turn it on for normal -qr platforms as well.

On second thought, this isn't true.. I can get it running only with software-webrender. I guess the question is whether you still wanted to if there's no baseline to compare against.

Comment 8

[Andrew Halberstadt [:ahal]]

Looks like talos-svgr is also just disabled right across the board:
https://searchfox.org/mozilla-central/source/taskcluster/ci/test/talos.yml#446

I'll post a patch soon that at least gets g1, g4 and o running on linux64-qr platforms.

Comment 9

[Andrew Halberstadt [:ahal]]

I decided to go ahead and land the above as any further work needed here can be done incrementally on top of that patch.

Just to clarify, the landed patch will not schedule any T(s) or T(bcv) tasks, as both of those are already skipped more generally (the former everywhere, the latter on all -qr platforms).

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/01cf90b1b5db
Run some Talos tests with software webrender on linux64-qr, r=jmaher

Comment 11

[Matt Woodrow (:mattwoodrow)]

bcv is 'basic compositor video', which explicitly disables acceleration using a pref and then runs some video tests. For normal -qr platforms this isn't useful, since the pref would disable WebRender, and it'd be an identical configuation to the non-qr variant.

When software webrender is enabled, disabling acceleration won't disable WR, and test remains useful (testing video performance of the software webrender configuration).

The baseline I want to compare against here is the non-qr bcv, to see if webrender-software is worse than layers-software.

Thanks!

Comment 12

[Andrew Halberstadt [:ahal]]

Ok, Ieave-open to make that change. I think it'll end up being tricky after all, but I'll figure something out.

Comment 13

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/01cf90b1b5db

Comment 14

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1657769 - [taskgraph] Add ability to 'defer' some keys in resolve_keyed_by, r?#taskgraph-reviewers

It's often useful to nest 'by-*' statements in the CI configs. However, calling
'resolve_keyed_by' will keep drilling down and resolving these statements all
at once. Normally this is fine, but sometimes we may want to use, e.g 'by-foo'
when 'foo' hasn't been resolved yet (i.e, it is being set by a later transform
or something like that).

Currently, this means that we'll hit the 'default' case as 'foo' doesn't exist.
This revision adds the ability to 'defer' evaluation of 'foo' until a later
time.

While running into this scenario might be a code smell, re-organizing
transforms to untangle these kinds of ordering dependencies will be a major
undertaking. Plus, there are legitimate reasons we might want one kind of
ordering with certain tasks, then another ordering for other tasks.

Comment 15

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1657769 - [ci] Run talos bcv on -qr platforms when software webrender is enabled, r?jmaher

From mwoodrow:

bcv is 'basic compositor video', which explicitly disables acceleration using a
pref and then runs some video tests. For normal -qr platforms this isn't
useful, since the pref would disable WebRender, and it'd be an identical
configuation to the non-qr variant.

When software webrender is enabled, disabling acceleration won't disable WR,
and test remains useful (testing video performance of the software webrender
configuration).

The baseline I want to compare against here is the non-qr bcv, to see if
webrender-software is worse than layers-software.

Depends on D88191

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/485e155cf1d6
[taskgraph] Add ability to 'defer' some keys in resolve_keyed_by, r=taskgraph-reviewers,aki
https://hg.mozilla.org/integration/autoland/rev/89adcd190276
[ci] Run talos bcv on -qr platforms when software webrender is enabled, r=jmaher

Comment 17

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/485e155cf1d6
https://hg.mozilla.org/mozilla-central/rev/89adcd190276

Comment 18

[Mike Hommey [:glandium]]

This made decision tasks take 10 more minutes than they used to.

Comment 19

[Andrew Halberstadt [:ahal]]

Whaaa? Ok I'll investigate or back out tomorrow (if this is causing sheriff issues feel free to back out now).