Closed
Bug 949221
Opened 12 years ago
Closed 9 years ago
Robocop test fails with Automation Error but tbpl shows green
Categories
(Testing :: General, defect)
Tracking
(Not tracked)
RESOLVED
WORKSFORME
People
(Reporter: gbrown, Assigned: gbrown)
References
(Blocks 1 open bug)
Details
Android 2.2: https://tbpl.mozilla.org/php/getParsedLog.php?id=31821963&tree=Mozilla-Central&full=1#error0
Android 4.0: https://tbpl.mozilla.org/php/getParsedLog.php?id=31821413&tree=Mozilla-Central&full=1#error0
These fail with:
Mochi-Remote ERROR | Automation Error: Missing end of test marker (process crashed?)
but tbpl shows a green run. (This seems to happen on all of our robocop runs currently.)
I note there are other bugs open for related topics, some with wip patches:
Bug 898165 - Panda Android jobs show green on TBPL even if they were aborted due to Python errors
Bug 895966 - tbpl shows green for Android 4.0 rc2 job that failed with DMError
Bug 677964 - Some mochitest errors are not counted in the totals, resulting in green results in TBPL for failing runs
![]() |
Assignee | |
Comment 1•12 years ago
|
||
I don't understand how the "oranging" works. Do we need a new regex in global_errors? tegra_errors??
http://hg.mozilla.org/build/buildbotcustom/annotate/9c71ca2b0e0b/status/errors.py#l8
Comment 2•12 years ago
|
||
Neither one - that's used to overrule the buildstep's own evaluateCommand, so that we don't have to teach each separate step of each separate job that if its log included "No space left on device" then it should set RETRY no matter what it otherwise thought it did.
Since it looks like buildbot thinks of robocop as a mochitest, and it looks like robocop isn't running under mozharness, I *think* you're looking at http://mxr.mozilla.org/build/source/buildbotcustom/steps/unittest.py#256, which of course doesn't even remotely work for robocop, since it was written for test suites which run their tests, then summarize them. Once. Since robocop runs as though the number of tests equals the number of separate suites each job runs, using an evaluateCommand which doesn't know that probably means that if a single robocop test passes, so there's a single instance of "INFO Failed: 0" in the log, and no instances of "TEST-UNEXPECTED-", then the run passed, end of story. It's not going to care whether every single one of the other hundred tests crashed, since they didn't TEST-UNEXPECTED-FAIL crash.
![]() |
Assignee | |
Comment 3•9 years ago
|
||
I haven't seen anything like this recently.
Assignee: nobody → gbrown
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
You need to log in
before you can comment on or make changes to this bug.
Description
•