1672740 - Software WebRender results are mixing with regular WebRender results

Status: RESOLVED FIXED

(Testing :: Talos, defect, P2)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Perfherder graph showing mixed results appearing as bimodal

Bug 1657769 started running a number of Talos tests with software WebRender enabled, however there's no way in Perfherder to distinguish these results from the 'regular' WebRender results.

In Raptor we're adding 'webrender' to extraOptions as a way to distinguish these results (in addition to the -qr platform suffix). Perhaps we should also include this in Talos, with a variant of 'webrender-sw'? I'm open to other suggestions, but I am keen to avoid overloading the test name with these characteristics.

Example results in Perfherder: https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=autoland,1925804,1,1&timerange=5184000

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

It looks like adding a 'webrender' extra option should be simple based on what we're doing for Fission:

# differentiate fission vs non-fission results in perfherder
if browser_config.get('enable_fission', False):
    talos_results.add_extra_option('fission')

We should be able to use enable_webrender for 'regular webrender, but I'm less sure how we can detect software enabled WebRender. :ahal do you have any suggestions? Can we check the gfx.webrender.software pref, or the presence of this on the command line?

Comment 2

[Andrew Halberstadt [:ahal]]

Yes, here's the webrender-sw definition:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/tests.py#256

So either that pref, or you could check for -swr in the label, or we could add a tag or something to the tags attribute. Personally I feel like checking for the pref is the safest because then even if a developer manually uses --setpref gfx.webrender.software=1 on the command line, then webrender-sw will still be detected.

Comment 3

[Andrew Halberstadt [:ahal]]

Note, we do similar things in the mochitest harness:
https://searchfox.org/mozilla-central/source/testing/mochitest/runtests.py#2671

Comment 4

[Matt Woodrow (:mattwoodrow)]

I believe T-gli is affected by this issue too.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Matt Woodrow (:mattwoodrow) from comment #4)

I believe T-gli is affected by this issue too.

As far as I can tell, the tests in T-gli are unique to that group so I don't think we're mixing these results with others.

Comment 6

[Matt Woodrow (:mattwoodrow)]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #5)

(In reply to Matt Woodrow (:mattwoodrow) from comment #4)

I believe T-gli is affected by this issue too.

As far as I can tell, the tests in T-gli are unique to that group so I don't think we're mixing these results with others.

OSX 10.14 Shippable opt - T(webgl) and T-gli(webgl) had combined results on my try run. The numbers are pretty similar though, so it's not obviously bimodal.

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Matt Woodrow (:mattwoodrow) from comment #6)

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #5)

(In reply to Matt Woodrow (:mattwoodrow) from comment #4)

I believe T-gli is affected by this issue too.

As far as I can tell, the tests in T-gli are unique to that group so I don't think we're mixing these results with others.

OSX 10.14 Shippable opt - T(webgl) and T-gli(webgl) had combined results on my try run. The numbers are pretty similar though, so it's not obviously bimodal.

You're correct, here's a mix for T(webgl) and T-gli(webgl). I'll clone this bug, and we should add an extra option to distinguish webgl-ipc.

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1657769

Comment 9

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1672740 - Add missing extraOptions for some test variants in Talos. r?#perftest

This patch adds extra options to the perfherder data for webrender, webrender-sw (webrender-software), and webgl-ipc (to distinguish from webgl results).

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ce2c3809f07f
Add missing extraOptions for some test variants in Talos. r=perftest-reviewers,davehunt

Comment 11

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/ce2c3809f07f