1708338 - TVg on Windows 7 is flaky

Status: RESOLVED WONTFIX

(Testing :: General, defect, P3)

Description

[Mats Palmgren (inactive)]

Here's a Try push containing nothing but a few white-space changes that should have no effect at all:
https://treeherder.mozilla.org/jobs?repo=try&author=mpalmgren%40mozilla.com&fromchange=5e7ba3184229f9c3b5e37aa973f253a714c15ba7&selectedTaskRun=AiTfYD1bTgGVfMcdT_3R5Q.0

It makes the TVg test fail on Win7. Here's the relevant part of the log:

   INFO - REFTEST TEST-START | layout/reftests/bugs/428810-1b-ltr.html == layout/reftests/bugs/428810-1-ltr-ref.html
   INFO - REFTEST TEST-LOAD | file:///Z:/task_1619660843/build/tests/reftest/tests/layout/reftests/bugs/428810-1b-ltr.html | 0 / 1 (0%)
   INFO - REFTEST INFO | drawWindow flags = DRAWWINDOW_DRAW_CARET | DRAWWINDOW_DRAW_VIEW | DRAWWINDOW_USE_WIDGET_LAYERS; window size = 816,1038; test browser size = 800,1000
   INFO - REFTEST TEST-LOAD | file:///Z:/task_1619660843/build/tests/reftest/tests/layout/reftests/bugs/428810-1-ltr-ref.html | 0 / 1 (0%)
   INFO - REFTEST TEST-PASS | layout/reftests/bugs/428810-1b-ltr.html == layout/reftests/bugs/428810-1-ltr-ref.html | image comparison, max difference: 0, number of differing pixels: 0
   INFO - REFTEST TEST-END | layout/reftests/bugs/428810-1b-ltr.html == layout/reftests/bugs/428810-1-ltr-ref.html
   INFO - REFTEST INFO | Slowest test took 49ms (file:///Z:/task_1619660843/build/tests/reftest/tests/layout/reftests/bugs/428810-1b-ltr.html)
   INFO - REFTEST INFO | Total canvas count = 2
   INFO - 1619661910977	Marionette	TRACE	Received observer notification xpcom-will-shutdown
   INFO - 1619661910977	Marionette	INFO	Stopped listening on port 2828
   INFO - 1619661910977	Marionette	DEBUG	Marionette stopped listening
   INFO - REFTEST INFO | Process mode: e10s
WARNING - leakcheck | refcount logging is off, so leaks can't be detected!
   INFO - REFTEST INFO | Result summary:
   INFO - REFTEST INFO | Successful: 1 (1 pass, 0 load only)
   INFO - REFTEST INFO | Unexpected: 0 (0 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 failed load, 0 exception)
   INFO - REFTEST INFO | Known problems: 0 (0 known fail, 0 known asserts, 0 random, 0 skipped, 0 slow)
   INFO - REFTEST SUITE-END | Shutdown
   INFO - REFTEST INFO | :::
   INFO - REFTEST INFO | ::: Test verification summary for:
   INFO - REFTEST INFO | :::
   INFO - REFTEST INFO | ::: Z:/task_1619660843/build/tests/reftest/tests/layout/reftests/bugs/428810-1b-ltr.html
   INFO - REFTEST INFO | :::
   INFO - REFTEST INFO | ::: 1. Run each test 10 times in one browser. : Pass
   INFO - REFTEST INFO | ::: 2. Run each test 5 times in a new browser each time. : Pass
   INFO - REFTEST INFO | ::: 3. Run each test 10 times in one browser, in chaos mode. : Pass
   INFO - REFTEST INFO | ::: 4. Run each test 5 times in a new browser each time, in chaos mode. : Pass
   INFO - REFTEST INFO | :::
   INFO - REFTEST INFO | ::: Test verification PASSED
   INFO - REFTEST INFO | :::
   INFO -  0
  ERROR - Return code: 1
   INFO - TinderboxPrint: reftest-reftest<br/>64/0/0
  ERROR - # TBPL FAILURE #
WARNING - setting return code to 2
  ERROR - TinderboxPrint: Per-test run of ...\reftests\bugs\428810-1b-ltr.html<br/>: FAILURE

It appears to me that the test actually passed, but that some process returned a non-zero exit code that the test framework is picking up as a test failure.
I suspect that this is a bug in the CI or test framework and not in Firefox.

This failure is bad for a couple of reasons:

1. there's no clear indication of what has failed so it's hard to investigate what might have gone wrong
2. this test doesn't always run, or at least only runs on a subset of files, so the error only occurs if you change certain files which (from a Firefox engineers POV) makes it appear as if they have introduced a regression and then proceed spending hours trying to figure out what they did wrong

We used to have a policy (idk if it still is) that flaky tests should be turned off. I think that would be justified in this case unless a solution can be found quickly.

If indeed it is a bug in Firefox then please try to capture a stack for the process with the non-zero exit code and reassign to the appropriate component. Thanks.

Comment 1

[Mats Palmgren (inactive)]

FYI, we also have reports of intermittent TVg failures such as bug 1706530, bug 1706574 etc that look very similar (the test appears to have passed but there's a non-zero exit code after the test has run). Is suspect those are caused by the same underlying bug.

Comment 2

[Sarah Clements [:sclements]]

Treeherder is only the consumer of pushes and tasks and parses the raw logs according to regex logic. So unless you need Treeherder to parse the log differently (ignore certain log lines) it looks like this bug needs to be filed with the maintainers of this particular test.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

thanks for filing this :mats- I am working with :ahal to review when/where to run test-verify and where we get the most value.

Comment 4

[Chris Peterson [:cpeterson]]

Resolving this bug as WONTFIX because Firefox versions >= 116 no longer support Windows versions < 10.