create -noqr variant to be used to test fallback for platforms/suites where -qr and -swr are not appropriate
Categories
(Testing :: General, task)
Tracking
(Not tracked)
People
(Reporter: jmaher, Unassigned)
References
(Depends on 1 open bug)
Details
as we transition to -qr by default and -swr as a fallback, the need for no webrender is falling away.
for a while we want to test a few things out to ensure users are covered- it is recommended we test on:
- reftest
- mochitest-media
These can be per platform, ideally on a single config- ideally on shippable (so we have coverage on beta/release).
one consideration will be perf- specifically (graphics and perf should comment):
- do we care about -noqr vs -qr perf?
- do we care about -noqr vs -swr perf?
- will the perf team need to adjust anything for -noqr in order to track historical data?
Another consideration is what do we do with ccov/asan/tsan builds. I think these are shifting to be -qr by default, questions are:
- do we want -swr on asan/tsan?
- do we want -noqr on asan/tsan?
- is -qr ok for ccov builds?
|
Reporter | |
Comment 1•3 years ago
|
||
:davehunt- can you help with some of the perf questions- if there are goals around -qr vs -noqr? any other considerations from scheduling, reporting, sheriffing?
:aosmond- can you confirm this looks right and add answers to any questions from a graphics perspective?
|
|
Reporter | |
Comment 2•3 years ago
|
||
ideally we would like to make -qr the default and get rid of it in the platform/config title. If we did that it could have other issues with perf reporting. So I think the scope now is just to get rid of any platform that is not -qr and for the test suites we want to run, have them as a variant of -noqr (which wouldn't be -fis-noqr, -swr-noqr, etc.)
|
||
Comment 3•3 years ago
•
|
||
Any change to the platform names will cause a new performance signature and disrupt continuity of our data series. We also use the extraOptions to distinguish signatures, so as long as we continue to include webrender in extraOptions then we'll avoid non-webrender and webrender results becoming muddled if/when the platform name changes.
For software WebRender we use the webrender-sw in extraOptions rather than a platform suffix, so I don't think that will be an issue.
We won't be able to combine the current non-WebRender with a new platform suffix of -noqr as these will be distinct signatures. That said, I'm not aware of a performance need to track non-WebRender. Do we know how many users will use this configuration?
|
|
Reporter | |
Comment 4•3 years ago
|
||
I might add some hacks to preserve the graphs, but simplify our treeherder UI- not sure, but that would help. as for the -noqr, it should be the same, just ensure that we don't have an entry in extraOptions
If there are no specific criteria for -qr vs -noqr, maybe we can reduce our perf needs on -noqr in the coming months?
I thought the -swr variant was to account for the <10% of users that -qr doesn't apply to, I will let :aosmond comment with better data.
|
|
||
Comment 5•3 years ago
|
||
(In reply to Joel Maher ( :jmaher ) (UTC -0800) from comment #4)
I might add some hacks to preserve the graphs, but simplify our treeherder UI- not sure, but that would help. as for the -noqr, it should be the same, just ensure that we don't have an entry in
extraOptionsIf there are no specific criteria for
-qrvs-noqr, maybe we can reduce our perf needs on-noqrin the coming months?
I support reducing -noqr to just essential tests, and reviewing again in a few months.
I thought the
-swrvariant was to account for the <10% of users that-qrdoesn't apply to, I will let :aosmond comment with better data.
We are running some Talos tests against software webrender, but this is using the -qr platform. For example https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=talos%2Cswr&revision=2d291a99004dd0aea5304bb8cae306a47083ff94
|
||
Comment 6•3 years ago
|
||
(In reply to Joel Maher ( :jmaher ) (UTC -0800) from comment #0)
as we transition to -qr by default and -swr as a fallback, the need for no webrender is falling away.
for a while we want to test a few things out to ensure users are covered- it is recommended we test on:
- reftest
- mochitest-media
Yes, that is perfect. I assume this includes crashtests too.
These can be per platform, ideally on a single config- ideally on shippable (so we have coverage on beta/release).
one consideration will be perf- specifically (graphics and perf should comment):
- do we care about -noqr vs -qr perf?
No.
- do we care about -noqr vs -swr perf?
No.
- will the perf team need to adjust anything for -noqr in order to track historical data?
No.
Another consideration is what do we do with ccov/asan/tsan builds. I think these are shifting to be -qr by default, questions are:
- do we want -swr on asan/tsan?
Yes.
- do we want -noqr on asan/tsan?
Nice to have.
- is -qr ok for ccov builds?
Yes. They should share most of the pipeline with -swr. It would be nice to have it for -swr but I don't see it as a priority.
|
|
||
Comment 7•3 years ago
•
|
||
I mentioned it before, but it would be good to have similar smoketests for Software WebRender + OpenGL compositing on Android (any device, the simulator should be adequate here; there is no pure Software WebRender on Android) and Software WebRender + D3D11 on Windows (ideally both 7 and 10). Reftests and mochitest-media. But this is a secondary priority, since we don't have this test coverage today.
Description
•