Closed Bug 1483292 Opened 4 years ago Closed 3 years ago

TV-bf sometimes fails to run a test

Categories

(Testing :: General, defect, P2)

Version 3
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gbrown, Assigned: gbrown)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

:jmaher has run TV-bf on several tests and found some of them failed to run the requested test. Let's investigate.

https://bugzilla.mozilla.org/buglist.cgi?list_id=14280605&resolution=---&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=TV-bf%20notfound
For bug 1475601, the reported failure is 

TEST-UNEXPECTED-TIMEOUT | /WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html?1001-2000 | Derived key of type name: (anything) hash: (anything) length: (anything) using (anything), with (anything) - Test timed out

TV-bf with testPath: testing/web-platform/tests/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html fails:

https://treeherder.mozilla.org/logviewer.html#?job_id=193790258&repo=try&lineNumber=515

Per-test run could not find requested web-platform test 'testing/web-platform/tests/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html'


There is no such file as 'testing/web-platform/tests/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.html' in the tree, so this isn't an issue for regular TVw. I think the html is generated from testing/web-platform/tests/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.js.

It is possible to run TV-bf on testing/web-platform/tests/WebCryptoAPI/derive_bits_keys/pbkdf2.https.any.js (although it times out!):

https://treeherder.mozilla.org/logviewer.html#?job_id=193790163&repo=try


The js and html files are related in MANIFEST.json...I wonder if that could be used?

   "WebCryptoAPI/derive_bits_keys/hkdf.https.any.js": [
    [
     "/WebCryptoAPI/derive_bits_keys/hkdf.https.any.html?1-1000",
     {}
    ],
    [
     "/WebCryptoAPI/derive_bits_keys/hkdf.https.any.html?1001-2000",
     {}
    ],
    [
     "/WebCryptoAPI/derive_bits_keys/hkdf.https.any.html?2001-3000",
     {}
    ],
    [
     "/WebCryptoAPI/derive_bits_keys/hkdf.https.any.html?3001-last",
     {}
    ],
    [
     "/WebCryptoAPI/derive_bits_keys/hkdf.https.any.worker.html?1-1000",
     {}
    ],
    [
     "/WebCryptoAPI/derive_bits_keys/hkdf.https.any.worker.html?1001-2000",
     {}
    ],
interesting idea for using manifest.json; can we access that from the decision task?
We use it in mozharness currently at:

https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/testing/mozharness/mozharness/mozilla/testing/per_test_base.py#149

but we are missing out on this relationship.

I think there would be some difficulties in accessing this from the decision task: want to keep the decision task quite lightweight, etc.
Several TV-bf:notfound bugs are for desktop dom/canvas tests; those have likely been addressed by bug 1476318.

I verified these can now be run in TV-bf:

bug 1474113 dom/canvas/test/webgl-conf/generated/test_2_conformance__ogles__GL__swizzlers__swizzlers_089_to_096.html
https://treeherder.mozilla.org/logviewer.html#?job_id=193938141&repo=try

bug 1474415 dom/canvas/test/webgl-conf/generated/test_2_conformance__ogles__GL__swizzlers__swizzlers_105_to_112.html
https://treeherder.mozilla.org/logviewer.html#?job_id=193950722&repo=try

bug 1475296 dom/canvas/test/webgl-conf/generated/test_2_conformance__ogles__GL__abs__abs_001_to_006.html
https://treeherder.mozilla.org/logviewer.html#?job_id=193950806&repo=try

bug 1475812 dom/canvas/test/webgl-conf/generated/test_conformance__glsl__bugs__conditional-discard-in-loop.html
https://treeherder.mozilla.org/logviewer.html#?job_id=193951206&repo=try

bug 1481750 dom/canvas/test/webgl-conf/generated/test_2_conformance__ogles__GL__exp2__exp2_009_to_012.html
https://treeherder.mozilla.org/logviewer.html#?job_id=193950895&repo=try

:jmaher -- What, if anything, should I do about the "TV-bf notfound" whiteboard entries for these bugs? TV-bf passes on all of them now, against a current m-c revision.
Flags: needinfo?(jmaher)
I suggest [tv-bf nowfound], then we can convert those, or if we want, just remove the [tv-bf notfound] whiteboard tag entirely.
Flags: needinfo?(jmaher)
OK - will update such bugs with "nowfound".
Similarly, this Android test is okay now, perhaps fixed in bug 1474393:

bug 1474230 mobile/android/tests/browser/chrome/test_media_playback.html
https://treeherder.mozilla.org/logviewer.html#?job_id=193961149&repo=try
Depends on: 1483421
Android jsreftest and mochitest-media were not being run -- addressed in bug 1483421.
Priority: -- → P2
Consider dom/canvas/crashtests/896047-1.html on win7/debug, from bug 1474751.

Here we see that it can be verified in TV, but not TVg:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=83a38a93900d222607434edc30ece4b89809ff21

but backfill schedules only TVg-bf, where it does not run:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b0b1a746c8149afe5867a1152f1bfea36ba54897


I suppose that's because of:

https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/taskcluster/taskgraph/actions/backfill.py#125-131

:jmaher - Would you happen to know what to do here?
Flags: needinfo?(jmaher)
This is interesting and showing cracks in our loose file matching we do to determine gpu or regular test type.

We match files for test-verify proper here:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/util/perfile.py#33

that looks like this:
    elif type.startswith('test-verify-gpu') or type.startswith('test-coverage-gpu'):
        file_patterns = ['**/*webgl*/**/test_*',
                         '**/dom/canvas/**/test_*',
                         '**/gfx/tests/**/test_*',
                         '**/devtools/canvasdebugger/**/browser_*',
                         '**/reftest*/**']
    elif type.startswith('test-verify') or type.startswith('test-coverage'):
        file_patterns = ['**/test_*',
                         '**/browser_*',
                         '**/crashtest*/**',
                         'js/src/tests/test/**',
                         'js/src/tests/non262/**',
                         'js/src/tests/test262/**']


which is different than what we use for backfill:
                    if (not is_wpttest) and \
                       ('gpu' in task.task['metadata']['name'] or
                        'webgl' in task.task['metadata']['name'] or
                        'canvas' in tp or
                        'gfx/tests' in tp or
                        ('reftest' in tp and 'jsreftest' not in tp)):
                        gpu_required = True


I think we should make the file patterns the same and do a slightly more expensive matching.  We use mozpackmatch (from mozpack.path) in perfile.py, I assume we could do the same in backfill.py since both have code in taskcluster/taskgraph.

Ideally we could find a way to refactor the code to do a single call to determine if it is a test and what type of job to run in.  We might want to wait until more edge cases are resolved for that, or take a stab at it now.
Flags: needinfo?(jmaher)
Depends on: 1484225
See Also: → 1492632
there are only 3 bugs left in the query of tests that were not found:
https://bugzilla.mozilla.org/buglist.cgi?resolution=---&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=DUPLICATE&resolution=WORKSFORME&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=%5BTV-bf%20notfound&list_id=14366405

one is a crashtest, and the other 2 are the same test with different urlparams for a wpt test.

:gbrown, are these ok to run, or do you think that these are not worth the time to fix them and we can resolve this bug:)
Flags: needinfo?(gbrown)
Blocks: 1499208
No longer blocks: 1499208
Depends on: 1499208
I found several wpt bugs had been marked notfound but without any obvious cause. I suspect the entered testPath may have been incorrect. It usually needs to be "testing/web-platform/tests" + the path reported in the bug, trimmed of any ?... parameters. Perhaps we can make the backfill action smarter to avoid this sort of problem. Until then, I verify the path with 'mach test-info --show-info <file-name>'.
Most of the remaining known problems are in toolkit/components/extensions/test. I wonder if this issue is related to the TestResolver (or manifestparser?) problem seen in bug 1498636.
Blocks: test-verify

(In reply to Geoff Brown [:gbrown] from comment #14)

Most of the remaining known problems are in
toolkit/components/extensions/test. I wonder if this issue is related to the
TestResolver (or manifestparser?) problem seen in bug 1498636.

Bug 1498636 is resolved now and has had a positive effect, but those tests still fail:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&selectedJob=227006937&revision=703d4508966f56d3ad444badc46df138bdcafc52

Now the reason is apparent: They are found in manifests, but not run because they are skipped. Actually, they are all skipped in one manifest, but not necessarily in another. This is working as designed, but not working optimally.

https://searchfox.org/mozilla-central/rev/03ebbdab952409640c6857d835d3040bf6f9e2db/testing/mozharness/mozharness/mozilla/testing/per_test_base.py#60-64

I think we've done all that's reasonable to do here.

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.