1476318 - test-verify-backfill doesn't seem to run webgl tests

Status: RESOLVED FIXED

(Testing :: General, defect, P1)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

when running tv-bf on new intermittent failures filed by the sheriffs, we find a common pattern where about 25% of the tests we try to reproduce fail to run any test.  These are marked with the whiteboard [TV-bf notfound]:
https://bugzilla.mozilla.org/buglist.cgi?resolution=---&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=DUPLICATE&resolution=WORKSFORME&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=[TV-bf%20notfound&list_id=14219633

22 of those bugs fall into the pattern of: dom/canvas/test/webgl-conf/*:
https://bugzilla.mozilla.org/buglist.cgi?list_id=14237945&status_whiteboard_type=allwordssubstr&status_whiteboard=%5BTV-bf%20notfound%5D&short_desc_type=allwordssubstr&short_desc=dom%2Fcanvas%2Ftest%2Fwebgl-conf&resolution=---&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=INACTIVE&resolution=DUPLICATE&query_format=advanced

I think that is actionable to fix.

Comment 1

[Geoff Brown [:gbrown]]

Some examples from that list:

  dom/canvas/test/webgl-conf/generated/test_conformance__glsl__bugs__compare-loop-index-to-uniform.html
  dom/canvas/test/webgl-conf/generated/test_conformance__ogles__GL__swizzlers__swizzlers_097_to_104.html
  dom/canvas/test/webgl-conf/generated/test_2_conformance__context__context-release-upon-reload.html

Comment 2

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9d28e387128a71c6733532f337683b2a0bc10376

reproduces the backfill failure, for test_conformance__glsl__bugs__compare-loop-index-to-uniform.html specifically.

Extra logging shows that the test path is recognized as a test but is discarded because it is a gpu task, but not a gpu test.

The test is considered non-gpu because the suite is webgl1-ext, while the code is checking for "webgl".

Comment 3

[Geoff Brown [:gbrown]]

This also explains

https://treeherder.mozilla.org/#/jobs?repo=try&revision=efa3f057bc1137a68d790d8975d05dce45f9aea4

in which non-gpu TV ran webgl tests which it should have skipped.

Comment 4

[Geoff Brown [:gbrown]]

Attachment: use a more generic test to identify webgl tests for test-verify variants

This generalizes the webgl comparison, to allow for "webgl1-ext", etc.

I added more logging, to help us diagnose similar problems in future.


With this patch, I can backfill a variety of webgl tests, and non-webgl backfill seems to keep working:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d6729e85e5164d8de1819507dcff02956fb8f534

Good old TV/TVg keeps working too:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1f2fecfa30b8a8ac0b0476423c7286ccfe8f8507

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8992786 [details] [diff] [review]
use a more generic test to identify webgl tests for test-verify variants

Review of attachment 8992786 [details] [diff] [review]:
-----------------------------------------------------------------

this makes a lot of sense, thanks for doing this.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c451bf86addf
Use a more generic test for webgl tests in per_test_base; r=jmaher

Comment 7

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/c451bf86addf