Closed Bug 949221 Opened 12 years ago Closed 9 years ago

Robocop test fails with Automation Error but tbpl shows green

Categories

(Testing :: General, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: gbrown, Assigned: gbrown)

References

(Blocks 1 open bug)

Details

Android 2.2: https://tbpl.mozilla.org/php/getParsedLog.php?id=31821963&tree=Mozilla-Central&full=1#error0 Android 4.0: https://tbpl.mozilla.org/php/getParsedLog.php?id=31821413&tree=Mozilla-Central&full=1#error0 These fail with: Mochi-Remote ERROR | Automation Error: Missing end of test marker (process crashed?) but tbpl shows a green run. (This seems to happen on all of our robocop runs currently.) I note there are other bugs open for related topics, some with wip patches: Bug 898165 - Panda Android jobs show green on TBPL even if they were aborted due to Python errors Bug 895966 - tbpl shows green for Android 4.0 rc2 job that failed with DMError Bug 677964 - Some mochitest errors are not counted in the totals, resulting in green results in TBPL for failing runs
I don't understand how the "oranging" works. Do we need a new regex in global_errors? tegra_errors?? http://hg.mozilla.org/build/buildbotcustom/annotate/9c71ca2b0e0b/status/errors.py#l8
Neither one - that's used to overrule the buildstep's own evaluateCommand, so that we don't have to teach each separate step of each separate job that if its log included "No space left on device" then it should set RETRY no matter what it otherwise thought it did. Since it looks like buildbot thinks of robocop as a mochitest, and it looks like robocop isn't running under mozharness, I *think* you're looking at http://mxr.mozilla.org/build/source/buildbotcustom/steps/unittest.py#256, which of course doesn't even remotely work for robocop, since it was written for test suites which run their tests, then summarize them. Once. Since robocop runs as though the number of tests equals the number of separate suites each job runs, using an evaluateCommand which doesn't know that probably means that if a single robocop test passes, so there's a single instance of "INFO Failed: 0" in the log, and no instances of "TEST-UNEXPECTED-", then the run passed, end of story. It's not going to care whether every single one of the other hundred tests crashed, since they didn't TEST-UNEXPECTED-FAIL crash.
Blocks: 1048775
I haven't seen anything like this recently.
Assignee: nobody → gbrown
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.