1371347 - it looks as if we can sometimes get a screenshot of the 'next test' instead of the failure when it happens (in browser chrome at least)

Status: NEW

(Testing :: Mochitest, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

I was pointed to an interesting problem, given this failure:
https://treeherder.mozilla.org/logviewer.html#?job_id=105189307&repo=try&lineNumber=3378

we see this screenshot:
https://public-artifacts.taskcluster.net/Qx2DVtFjSHCbTwlO97yYgQ/0/public/test_info//mozilla-test-fail-screenshot_ayDru2.png

the screenshot is for printpreview which is the test immediately after the failing test.

I don't see any other reason why that would happen other than we preload or do something in parallel/out of sequence.

We should look in more detail at the screenshot on failure code for browser-chrome and ensure that it is taking a screenshot before cleaning up and before moving onto the next test.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:ahal, is this something you could pick up?

Comment 2

[Andrew Halberstadt [:ahal]]

Yep! I'll take a look.

Comment 3

[Andrew Halberstadt [:ahal]]

The screenshot handling code is all in python, which means that there's no way it can block the JS side from moving on to the next test. I think it's a race condition where the screenshot may or may not be captured before the next test is loaded. As far as I can tell, this has always happened. I guess it is mostly useful when the browser hangs and doesn't progress to the next test (or gets killed immediately after).

I also think the log is a bit confusing, some things may be out of order due to buffering. Yesterday I discovered that mochitest does a whole bunch of wrong things related to structured logging (bug 1372567) which contributes further to the log confusion.