Closed Bug 1621095 Opened 4 years ago Closed 4 years ago

split reftests into general reftests and webrender reftests

Categories

(Testing :: Reftest, task, P3)

Product:
Testing
Component:
Reftest
Version:
Version 3
Type:
task

Tracking

(firefox77 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla77
Tracking Flags:
Tracking Status
firefox77 --- fixed

People

(Reporter: jmaher, Assigned: jmaher)

References

Details

(Whiteboard: [ci-costs-2020:done])

Attachments

(2 files)

list of test related files found on webrender patches
70.94 KB, text/plain
Details
Bug 1621095 - add reftest-qr to run focused webrender reftests on android phones. r=bc
47 bytes, text/x-phabricator-request
Details | Review

It is my understanding that many of the reftests provide little value for webrender specifically- so if we are running tests on limited hardware (like android) in a webrender config, we should run the specific tests that provide the most value.

I assume once identified we could run those tests in a R(qr) job that is controlled by command line flags to the reftest harness.

I am not sure if a directory (full manifest) or specific tests will be specified, until we have a reasonable list, we shouldn't be planning out implementation.

Jeff, could you help point us to tests, docs, queries, or give some guidance on how to identify webrender specific reftests?

Flags: needinfo?(jmuizelaar)

I don't really have much to suggest here. The best starting point would perhaps be tests that were added that are associated with bugs in the "Graphics: WebRender" component.

We could potentially move those to 'layout/reftests/webrender/' and then add other tests there as appropriate.

Flags: needinfo?(jmuizelaar)

I feel like there's a bunch of tests that often need their fuzz values tweaked as a result of webrender changes. Those should probably also be included. In both cases it's a matter of trawling through bugs in the webrender component, looking at commits associated with those bugs, and looking at files touched by those commits. I can try and script some of that next week if you don't get to it before me.

any values we should look for in the fuzzy-if, or any fuzzy-if(webrender) ? This needs a script as there are many reftests.

I don't think you need to look for fuzzy-if or fuzzy-if(webrender) specifically, but look at lines in reftest.list files that were touched as a result of WR bug commits. I guess a lot of those will have fuzzy-if(webrender) but you don't need to specifically look for those, they will turn up automatically.

There's probably some bugs that we'll want to exclude (e.g. the ones that turned on WR tests for new platforms) because those touch a ton of reftest.list files and are probably not super useful to include. So some manual curating will be needed as well, and it would be worthwhile keeping all the intermediate results (list of bug numbers, list of commits per bug, files touched per commit) of this process so that we can tune and iterate it.

has the component always been "Graphics: Webrender"? Are there other components as well?

We got the bug component created pretty early on so yeah I think everything interesting should be in there. I don't think there are other components that are relevant for this.

I have a script running collecting data about which files were changed in the 4400+ webrender bugs- I will have a list of reftest/crashtests in due time.

looking at the webrender bugs, I found 472 that have test files, 1909 total test files, 1335 files with 'reftest' in the name, and 32 with 'crashtest' in the name. Now to find actual tests in the list.

ok, I had duplicate files, here is what we have:
total test files: 1327
total 'reftest' files: 990
total 'crashtest' files: 30

reftest has this breakdown:
gfx/wr/wrench: 820 (runs as the wrench job)
gfx/tests/reftest: 32 (basically all tests in here - a single manifest)
layout/reftests: 117
other: 21 (harness, etc.)

looking at the non reftest/crashtest files, they fall into:

  • web platform tests
  • third_party code with 'test' in the filename
  • gfx/wr/* files with 'test' somewhere in the name
  • test harness files to support wr changes
  • gtest
  • mochitest

:kats, should we look at moving all the reftest/crashtest files into their own .list manifests and setup special QR jobs? Do we care about the other mochitest/gtest/wpt tests?

Flags: needinfo?(kats)

To make things simpler, let's focus on reftests first, and deal with the other test suites afterwards. Your list shows that WR patches have concentrated activity in:

gfx/tests/reftest/
layout/reftest/async-scrolling/
layout/reftests/bugs/
layout/reftests/css-animations/
layout/reftests/invalidation/
layout/reftests/position-sticky/
layout/reftests/svg/
layout/reftests/transform-3d/

which (apart from /invalidation/) agrees with my gut feeling about which tests would be the most impacted by changes in WR. So can we just create a separate top-level reftest.list file (e.g. layout/reftests/reftest-qr-subset.list) that includes those folders, and run those in a job instead of the full layout/reftest/reftest.list set? That way we minimize churn with moving files around and whatnot.

Flags: needinfo?(kats)

awesome; makes sense to do that; and not splitting up existing manifests is a good idea.

the layout/reftests/bugs/ dir is a big set of tests, but there are 16 modifications to test files there, and this balances out some risk.

one other thought- we would keep all of these tests in the regular reftest.list, but create reftest-qr.list for -qr variants and only run that on -qr variants, correct?

Flags: needinfo?(kats)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #14)

one other thought- we would keep all of these tests in the regular reftest.list, but create reftest-qr.list for -qr variants and only run that on -qr variants, correct?

Depends on what exactly you mean by the -qr variants. We already have some -qr test sets, e.g. here but in that case (which is run on Linux) we still want to run all the reftests. It's only specific resource-constrained platforms (e.g. Android) where we want to run this reftest-qr.list subset.

So the way I would do it is create a new reftest-qr-subset job in reftest.yml which uses the reftest-qr.list. And then for example here I'd add reftest-qr-subset. And then adjust the scheduling so that reftest runs very rarely (~once/day) and reftest-qr-subset runs a little more frequently (for test platform android-hw-aarch64-qr). If adjusting the scheduling is not possible then we can just drop reftest from that test-set.

Does that sound reasonable?

Flags: needinfo?(kats)

yes, this works out well. I just wanted to remove ambiguity from the bug and comments and comment 15 did that perfectly. I assume we can make that work smoothly and reduce from 8 chunks to 1 or 2 :)

Whiteboard: [ci-costs-2020:todo]
Assignee: nobody → bob
Status: NEW → ASSIGNED

add reftest-qr to run focused webrender reftests on android phones.

:kats, could you take a look at the above patch and see if this is reasonable for what your needs are?

Flags: needinfo?(kats)

Left comments on the patch. Overall looks good, r+ from me but would be nice to leave the crashtests in for now, or make a reduced set of them to keep running.

Flags: needinfo?(kats)
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/97280448a042
add reftest-qr to run focused webrender reftests on android phones. r=bc
Blocks: 1632822
Assignee: bob → jmaher

Backed out changeset 97280448a042 (bug 1621095) for test_ext_test.html failures

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=android%2C7.0%2Cx86-64%2Copt%2Cmochitests%2Ctest-android-em-7.0-x86_64%2Fopt-geckoview-mochitest-plain-e10s-4%2Cm%284%29&fromchange=cc2df60c66b9e153a92661b464ca2c8783666295&tochange=d6257d6b60a844e526525fa47c0ef720733a2421&selectedJob=299240618

Backout link: https://hg.mozilla.org/integration/autoland/rev/d6257d6b60a844e526525fa47c0ef720733a2421

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=299240618&repo=autoland&lineNumber=6102

...
[task 2020-04-24T13:43:15.354Z] 13:43:15     INFO -  2732 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script ([object HTMLHtmlElement])
[task 2020-04-24T13:43:15.354Z] 13:43:15     INFO -  2733 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Object falsey)
[task 2020-04-24T13:43:15.354Z] 13:43:15     INFO -  2734 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Array falsey)
[task 2020-04-24T13:43:15.354Z] 13:43:15     INFO -  2735 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Element falsey)
[task 2020-04-24T13:43:15.354Z] 13:43:15     INFO -  2736 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (True falsey)
[task 2020-04-24T13:43:15.354Z] 13:43:15     INFO -  2737 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (False falsey)
[task 2020-04-24T13:43:15.354Z] 13:43:15     INFO -  2738 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Null falsey)
[task 2020-04-24T13:43:15.355Z] 13:43:15     INFO -  2739 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Void falsey)
[task 2020-04-24T13:43:15.355Z] 13:43:15     INFO -  2740 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script ([object HTMLHeadElement])
[task 2020-04-24T13:43:15.355Z] 13:43:15     INFO -  2741 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Object equality - [object Object] - [object Object])
[task 2020-04-24T13:43:15.355Z] 13:43:15     INFO -  2742 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Array equality -  - )
[task 2020-04-24T13:43:15.355Z] 13:43:15     INFO -  2743 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Element equality - [object HTMLBodyElement] - [object HTMLBodyElement])
[task 2020-04-24T13:43:15.355Z] 13:43:15     INFO -  2744 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Null equality - null - null)
[task 2020-04-24T13:43:15.355Z] 13:43:15     INFO -  2745 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Void equality - undefined - undefined)
[task 2020-04-24T13:43:15.355Z] 13:43:15     INFO -  2746 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Object reference ineqality - [object Object] - [object Object] (different))
[task 2020-04-24T13:43:15.356Z] 13:43:15     INFO -  2747 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Array reference ineqality -  -  (different))
[task 2020-04-24T13:43:15.356Z] 13:43:15     INFO -  2748 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Element ineqality - [object HTMLBodyElement] - [object HTMLBodyElement] (different))
[task 2020-04-24T13:43:15.356Z] 13:43:15     INFO -  2749 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (Null and void ineqality - null - undefined)
[task 2020-04-24T13:43:15.356Z] 13:43:15     INFO -  2750 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script ([object HTMLDivElement] - true - false)
[task 2020-04-24T13:43:15.356Z] 13:43:15     INFO -  2751 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (obj with dynamic toString() - [object Object] - [object Object])
[task 2020-04-24T13:43:15.356Z] 13:43:15     INFO -  2752 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script (http:)
[task 2020-04-24T13:43:15.357Z] 13:43:15     INFO -  2753 INFO TEST-PASS | toolkit/components/extensions/test/mochitest/test_ext_test.html | content script ()
[task 2020-04-24T13:43:15.357Z] 13:43:15     INFO -  Buffered messages finished
[task 2020-04-24T13:43:15.357Z] 13:43:15  WARNING -  2754 INFO TEST-UNEXPECTED-FAIL | toolkit/components/extensions/test/mochitest/test_ext_test.html | No more results - got ["test-done", true, "dot notifyPass"], expected undefined
[task 2020-04-24T13:43:15.357Z] 13:43:15     INFO -      SimpleTest.is@SimpleTest/SimpleTest.js:383:14
[task 2020-04-24T13:43:15.357Z] 13:43:15     INFO -      verifyTestResults@toolkit/components/extensions/test/mochitest/test_ext_test.html:152:5
[task 2020-04-24T13:43:15.357Z] 13:43:15     INFO -      test_test_in_content_script@toolkit/components/extensions/test/mochitest/test_ext_test.html:185:20
[task 2020-04-24T13:43:15.357Z] 13:43:15     INFO -  2755 INFO Received test result: ["test-result",false,"Object falsey"]
[task 2020-04-24T13:43:15.357Z] 13:43:15     INFO -  2756 INFO Received test result: ["test-result",false,"Array falsey"]
[task 2020-04-24T13:43:15.358Z] 13:43:15     INFO -  2757 INFO Received test result: ["test-result",false,"Element falsey"]
[task 2020-04-24T13:43:15.358Z] 13:43:15     INFO -  2758 INFO Received test result: ["test-result",false,"True falsey"]
[task 2020-04-24T13:43:15.358Z] 13:43:15     INFO -  2759 INFO Received test result: ["test-result",true,"False falsey"]
[task 2020-04-24T13:43:15.358Z] 13:43:15     INFO -  2760 INFO Received test result: ["test-result",true,"Null falsey"]
[task 2020-04-24T13:43:15.358Z] 13:43:15     INFO -  2761 INFO Received test result: ["test-result",true,"Void falsey"]
[task 2020-04-24T13:43:15.358Z] 13:43:15     INFO -  2762 INFO Received test result: ["test-result",false,"[object HTMLHeadElement]"]
[task 2020-04-24T13:43:15.358Z] 13:43:15     INFO -  2763 INFO Received test result: ["test-eq",true,"Object equality","[object Object]",
...
Flags: needinfo?(jmaher)
Flags: needinfo?(jmaher)
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e940d4a30344
add reftest-qr to run focused webrender reftests on android phones. r=bc

https://hg.mozilla.org/mozilla-central/rev/e940d4a30344

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
status-firefox77: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla77
Whiteboard: [ci-costs-2020:todo] → [ci-costs-2020:done]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: