1565286 - Improve "lastTestSeen" string in Android reftests to avoid misattributing shutdown hangs/failures to the unlucky last test

Status: RESOLVED FIXED

(Testing :: Reftest, enhancement, P3)

Description

[Daniel Holbert [:dholbert]]

+++ This bug was initially created as a clone of Bug #1475141 +++

Per bug 1475141 comment 4, we changed the reftest harness to avoid misattributing shutdown hangs/failures to the "unlucky" last test.

However, I think crashtest runs still suffer from this problem -- shutdown failures seem to be attributed to the last test that was run. At least, this seems to be what's going on in bug 1553971. (Here's a sample failure log).

Could we extend/port bug 1475141's fix to benefit crashtest runs as well?

Comment 1

[Daniel Holbert [:dholbert]]

(As I recall, crashtests largely use the same harness as reftests, so I'm hoping this isn't too much trouble to fix. Maybe this is already supposed to work? If so, I wonder what's going on in bug 1553971 and if I'm merely misunderstanding what's going on there.)

Comment 2

[Andrew Halberstadt [:ahal]]

Yeah, I'm not terribly surprised it doesn't work there, though I would have expected it to. I'll leave the needinfo to remind myself to take a look when I get a chance (won't be until mid/late next week probably).

Comment 3

[Daniel Holbert [:dholbert]]

ahal provided this link for a place where this is working correctly (i.e. where Last test finished shows up instead of the test name), in a Windows reftest log:
https://treeherder.mozilla.org/logviewer.html#?job_id=218864193&repo=autoland

That log ends like this:

19:59:33     INFO - REFTEST TEST-PASS(EXPECTED RANDOM) | file:///Z:/task_1545853338/build/tests/reftest/tests/layout/reftests/svg/smil/sort/sort-additive-1.svg == file:///Z:/task_1545853338/build/tests/reftest/tests/layout/reftests/svg/smil/sort/sort-additive-1-ref.svg | image comparison, max difference: 0, number of differing pixels: 0
19:59:33     INFO - REFTEST TEST-END | file:///Z:/task_1545853338/build/tests/reftest/tests/layout/reftests/svg/smil/sort/sort-additive-1.svg == file:///Z:/task_1545853338/build/tests/reftest/tests/layout/reftests/svg/smil/sort/sort-additive-1-ref.svg
19:59:33     INFO - REFTEST INFO | Slowest test took 71ms (file:///Z:/task_1545853338/build/tests/reftest/tests/layout/reftests/svg/smil/sort/sort-startSame-1b.svg)
19:59:33     INFO - REFTEST INFO | Total canvas count = 2
[...]
19:59:34     INFO - 1545854374695	Marionette	DEBUG	Init aborted (running=false, enabled=true, finalUIStartup=false)
20:05:44    ERROR - REFTEST ERROR | Last test finished | application timed out after 370 seconds with no output
20:05:44    ERROR - REFTEST ERROR | Force-terminating active process(es).
20:05:44     INFO - REFTEST TEST-INFO | started process screenshot
20:05:44     INFO - REFTEST TEST-INFO | screenshot: exit 0

In contrast, here's an Android crashtest log where this isn't working properly:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=255173994&repo=autoland&lineNumber=14569

That log ends like this:

17:59:21     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/layout/style/crashtests/1546255.html | 3176 / 3775 (84%)
17:59:21     INFO -  REFTEST TEST-PASS | http://10.0.2.2:8854/tests/layout/style/crashtests/1546255.html | (LOAD ONLY)
17:59:21     INFO -  REFTEST TEST-END | http://10.0.2.2:8854/tests/layout/style/crashtests/1546255.html
18:05:52     INFO -  wait for org.mozilla.geckoview.test complete; top activity=org.mozilla.geckoview.test
18:05:53     INFO -  org.mozilla.geckoview.test unexpectedly found running. Killing...
[...]
18:06:07  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/style/crashtests/1546255.html | application timed out after 370 seconds with no output

(Some unrelated-looking logspam replaced with [...] for brevity.)

Comment 4

[Andrew Halberstadt [:ahal]]

Oh, if you're only seeing this with Android crashtests, then that makes a bit more sense. I'd wager the difference between working and not is "android" rather than "crashtest".

Comment 5

[Andrew Halberstadt [:ahal]]

Sorry for the late reply.

This feature doesn't exist in Android, so I'm pretty sure that is why you aren't seeing it (and not due to crashtests). I suspect this feature is working on desktop crashtests.

Comment 6

[Geoff Brown [:gbrown]]

Attachment: Bug 1565286 - Ensure Android lastTestSeen displays whether the test was in-progress or finished; r=

Extends the desktop changes made in bug 1475141 to Android, to avoid confusion when the
browser hangs between tests, or during shutdown.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/34ab926c9b85
Ensure Android lastTestSeen displays whether the test was in-progress or finished; r=jmaher

Comment 8

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/34ab926c9b85