Open Bug 1565286 Opened 5 months ago Updated 5 months ago

Improve "lastTestSeen" string in Android reftests to avoid misattributing shutdown hangs/failures to the unlucky last test

Categories

(Testing :: Reftest, enhancement, P3)

Version 3
enhancement

Tracking

(Not tracked)

People

(Reporter: dholbert, Unassigned)

References

Details

+++ This bug was initially created as a clone of Bug #1475141 +++

Per bug 1475141 comment 4, we changed the reftest harness to avoid misattributing shutdown hangs/failures to the "unlucky" last test.

However, I think crashtest runs still suffer from this problem -- shutdown failures seem to be attributed to the last test that was run. At least, this seems to be what's going on in bug 1553971. (Here's a sample failure log).

Could we extend/port bug 1475141's fix to benefit crashtest runs as well?

Flags: needinfo?(ahal)
Summary: Improve "lastTestSeen" string in crashtests → Improve "lastTestSeen" string in crashtests to avoid misattributing shutdown hangs/failures to the unlucky last test

(As I recall, crashtests largely use the same harness as reftests, so I'm hoping this isn't too much trouble to fix. Maybe this is already supposed to work? If so, I wonder what's going on in bug 1553971 and if I'm merely misunderstanding what's going on there.)

Yeah, I'm not terribly surprised it doesn't work there, though I would have expected it to. I'll leave the needinfo to remind myself to take a look when I get a chance (won't be until mid/late next week probably).

ahal provided this link for a place where this is working correctly (i.e. where Last test finished shows up instead of the test name), in a Windows reftest log:
https://treeherder.mozilla.org/logviewer.html#?job_id=218864193&repo=autoland

That log ends like this:

19:59:33     INFO - REFTEST TEST-PASS(EXPECTED RANDOM) | file:///Z:/task_1545853338/build/tests/reftest/tests/layout/reftests/svg/smil/sort/sort-additive-1.svg == file:///Z:/task_1545853338/build/tests/reftest/tests/layout/reftests/svg/smil/sort/sort-additive-1-ref.svg | image comparison, max difference: 0, number of differing pixels: 0
19:59:33     INFO - REFTEST TEST-END | file:///Z:/task_1545853338/build/tests/reftest/tests/layout/reftests/svg/smil/sort/sort-additive-1.svg == file:///Z:/task_1545853338/build/tests/reftest/tests/layout/reftests/svg/smil/sort/sort-additive-1-ref.svg
19:59:33     INFO - REFTEST INFO | Slowest test took 71ms (file:///Z:/task_1545853338/build/tests/reftest/tests/layout/reftests/svg/smil/sort/sort-startSame-1b.svg)
19:59:33     INFO - REFTEST INFO | Total canvas count = 2
[...]
19:59:34     INFO - 1545854374695	Marionette	DEBUG	Init aborted (running=false, enabled=true, finalUIStartup=false)
20:05:44    ERROR - REFTEST ERROR | Last test finished | application timed out after 370 seconds with no output
20:05:44    ERROR - REFTEST ERROR | Force-terminating active process(es).
20:05:44     INFO - REFTEST TEST-INFO | started process screenshot
20:05:44     INFO - REFTEST TEST-INFO | screenshot: exit 0

In contrast, here's an Android crashtest log where this isn't working properly:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=255173994&repo=autoland&lineNumber=14569

That log ends like this:

17:59:21     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/layout/style/crashtests/1546255.html | 3176 / 3775 (84%)
17:59:21     INFO -  REFTEST TEST-PASS | http://10.0.2.2:8854/tests/layout/style/crashtests/1546255.html | (LOAD ONLY)
17:59:21     INFO -  REFTEST TEST-END | http://10.0.2.2:8854/tests/layout/style/crashtests/1546255.html
18:05:52     INFO -  wait for org.mozilla.geckoview.test complete; top activity=org.mozilla.geckoview.test
18:05:53     INFO -  org.mozilla.geckoview.test unexpectedly found running. Killing...
[...]
18:06:07  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/style/crashtests/1546255.html | application timed out after 370 seconds with no output

(Some unrelated-looking logspam replaced with [...] for brevity.)

Oh, if you're only seeing this with Android crashtests, then that makes a bit more sense. I'd wager the difference between working and not is "android" rather than "crashtest".

Sorry for the late reply.

This feature doesn't exist in Android, so I'm pretty sure that is why you aren't seeing it (and not due to crashtests). I suspect this feature is working on desktop crashtests.

Flags: needinfo?(ahal)
Priority: -- → P3
Summary: Improve "lastTestSeen" string in crashtests to avoid misattributing shutdown hangs/failures to the unlucky last test → Improve "lastTestSeen" string in Android reftests to avoid misattributing shutdown hangs/failures to the unlucky last test
You need to log in before you can comment on or make changes to this bug.