1309436 - Do something about the unstarrability of "java-exception" test failures

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect, P1)

Description

[Phil Ringnalda (:philor)]

They are logged as

PROCESS-CRASH | java-exception | java.lang.NullPointerException at android.util.StateSet.stateSetMatches(StateSet.java:110) 

which means that the "test filename" is "java-exception" and that every bug with that in the summary, kw:intermittent-failure, last modified in the last 5 months, is a candidate for bug suggestions.

Since that's more than 20 bugs, there are no suggestions ("Exceeded max (20) bug suggestions, most of which are likely false positives.") which means there's a 100% chance that I'll have to star it, and about a 60% chance I'll file a duplicate rather than realize it's one I've already filed.

I doubt it's possible to get it under 20 and keep it under 20, which leaves either the option of crashing with the actual test name rather than java-exception, or adding it to the blacklist in https://github.com/mozilla/treeherder/blob/8fc9754f4e0665cd0fd18530ed4196868b3179e8/treeherder/model/error_summary.py#L149 so that instead of everything matching the "filename" we instead get a full-line search. That has it problems too, since the full line includes the "PROCESS-CRASH |" which almost none of the existing bugs have, and getting it on new ones would probably require that bugfiler know about this particular PROCESS-CRASH which needs to include it unlike all the others that don't.

Comment 1

[Geoff Brown [:gbrown]]

(In reply to Phil Ringnalda (:philor) from comment #0)
> I doubt it's possible to get it under 20 and keep it under 20, 

This seemed like a good opportunity to review java-exception bugs, so I had a look and was able to get them down to about 20...but yes, that probably won't stay that way.

Comment 2

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3a398974ca98da72c11f8ecb60c090967dfb25e2 demonstrates java exceptions with the introduction of test names. Does that seem like a good solution?

(Note that the exception in that try push is thrown on startup, which results in a "test name" of remoteautomation.py for reftest and mochitest. That wouldn't normally be the case - if the exception happens during a reftest/mochitest test, the actual test name would be reported.)

Comment 3

[Phil Ringnalda (:philor)]

LGTM

Comment 4

[Geoff Brown [:gbrown]]

Attachment: add test name to java-exception messages

This change adds the test name to java-exception failure messages. Instead of

21:30:05  WARNING -  PROCESS-CRASH | java-exception | java.lang.NullPointerException at org.mozilla.gecko.BrowserApp$14.run(BrowserApp.java:1171)

now mozcrash will report:

21:30:05  WARNING -  PROCESS-CRASH | testGeckoProfile | java-exception java.lang.NullPointerException at org.mozilla.gecko.BrowserApp$14.run(BrowserApp.java:1171)

I've copied the test_name checks from check_for_crashes().

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/cbce412d2211
Add test name to java-exception failure messages; r=ted

Comment 6

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/cbce412d2211