736923 - mochitest-robocop logs are very frequently missing parts, including tests

Status: RESOLVED INCOMPLETE

(Testing :: Mochitest, defect)

Description

[Phil Ringnalda (:philor)]

If you look at the pass/fail counts on tbpl, you'll see that green mochitest-robocop runs vary between around 200 passes and 206. Looking at the logs, say comparing https://tbpl.mozilla.org/php/getParsedLog.php?id=10161889&tree=Mozilla-Inbound (200 passes) with https://tbpl.mozilla.org/php/getParsedLog.php?id=10161953&tree=Mozilla-Inbound (206 passes), it looks like a pretty hefty percentage of the individual test runs have their logs cut off before the finish/pass/fail part, and quite often something is cut off without showing all of the test bits.

That 200 pass is missing 3 things from testWebContentContextMenu, and I'm not sure what else - probably bits of testBrowserProvider, but so far I haven't seen a single log that shows the end of it.

I'm assuming just logs getting cut off, but I guess it's equally likely that it's actually something worse, like a crash not getting logged and not marking the run as a failure.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

For testBrowserProvider, it looks like we are crashing.  The reason why we don't report it is right now we don't have a solid solution for reporting.  I haven't ran into a case where a log is missing lines that exist on the tegra.  

My bugzilla skills are not very good today, but there is a bug which has some information about this general problem with some proposed solutions.

Comment 2

[Phil Ringnalda (:philor)]

Wonder whether that bug ever fixed it?