Build java_stack_trace from java_exception for crash pings
Categories
(Socorro :: General, task)
Tracking
(Not tracked)
People
(Reporter: bdanforth, Assigned: bdanforth)
References
(Blocks 1 open bug)
Details
(Whiteboard: [cringe])
Attachments
(1 file, 1 external file)
|
91.97 KB,
application/zip
|
Details |
Java crash reports include the JavaStackTrace annotation. The JavaStackTraceRule turns it into the sanitized public processed field java_stack_trace, and SignatureGenerationRule uses that field with JavaSignatureTool to generate Java signatures. Crash pings don't include JavaStackTrace, only the structured JavaException, so they currently fall back to JavaExceptionSignatureTool. For about 45% of Fenix Java crashes, that produces a different signature than the corresponding crash report (see the options comparison in CRINGE-445 (https://mozilla-hub.atlassian.net/browse/CRINGE-445?focusedCommentId=1888317)).
java_exception already contains everything JavaSignatureTool needs: it only uses the first two lines of java_stack_trace (the outermost exception's class and its top frame). A validation on 1,000 recent Fenix Java crash reports rebuilt those two lines from java_exception and got identical signatures for ~99.6% of crashes.
The goal of this ticket is to add a processor rule that, for crash pings only, builds java_stack_trace from java_exception, so crash pings get the same Java signatures as the corresponding crash reports with virtually no changes to existing signatures.
Acceptance criteria:
- Using the approach as validated, add a new processor rule sets
java_stack_tracefromjava_exceptionfor crash pings only (based on the submission type). It doesn't run for crash reports, and doesn't overwrite an existingjava_stack_trace. - Unit tests to cover all the cases handled in the new rule.
- A validation run on a recent sample of at least 1,000 Fenix Java crash reports (rebuilding from each report's
java_exceptionand comparing with its realjava_stack_traceand signature) shows at least 99% identical signatures. The results are recorded in this ticket. - The
java_stack_tracedescription inprocessed_crash.schema.yamland the data documentation say that, for crash pings, the field is built fromjava_exception.
Known limitation: in about 0.4% of crashes (in the validation sample), java_exception doesn't include the outermost exception at all; it's missing on the client. The rule can't detect that from the crash ping, so those crash pings get a signature built from the outermost exception that is present, which differs from the crash report's. This could be fixed later in android-components (optional follow-up bug).
| Assignee | ||
Updated•17 days ago
|
Updated•17 days ago
|
| Assignee | ||
Comment 1•7 days ago
|
||
| Assignee | ||
Updated•7 days ago
|
| Assignee | ||
Comment 2•3 days ago
•
|
||
I validated the new processor rule against a random sample[1] of 5000 Fenix Java crash reports with a non-null JavaException crash annotation and processed_crash.java_stack_trace (> 98% of crash reports with a processed_crash.java_stack_trace, which is how we identify a Java from a native crash in crash reports, also have a JavaException) and found that 99% of signatures for crash pings with this new rule would match the corresponding crash report's signature.
Method
- Get a list of 5000 random Fenix Java crash ids (from
just shell; see attachedfenix_java_crashids_sample.txt):
$ socorro-cmd fetch_crashids --num=5000 \
--url="https://crash-stats.mozilla.org/search/?product=Fenix&crash_report_keys=JavaException&java_stack_trace=%21__null__&date=%3E%3D2026-05-17&_sort=uuid"
\ fenix_java_crashids_sample.txt
- Since the
socorro-cmd signaturecommand only runs signature generation and not the full processor ruleset, temporarily modify its script to removejava_stack_traceandjava_stack_trace_raw, and addsubmission_type = "ping"from theprocessed_crash, and run the newJavaStackTraceFromJavaExceptionRuleon the processed crash (see attachedrun_new_rule_before_siggen.diff). - Run the signature generation comparison (
socorro-cmd signature; see attachedsiggen_ping_diff.csvandsiggen_ping_errors.txt)
socorro-cmd signature --format=csv --different-only \
< fenix_java_crashids_sample.txt \
> siggen_ping_diff.csv \
2> siggen_ping_errors.txt
[1]: As can be seen in the value of the --url argument into socorro-cmd fetch_crashids, the sample was randomized by sorting SuperSearch results by crash ID, which is a randomly generated UUID (with the exception of the last 7 characters, which include the date and unused throttle result bit).
| Assignee | ||
Comment 3•3 days ago
|
||
| Assignee | ||
Comment 4•3 days ago
•
|
||
This was deployed in v2026.11.17.
I verified in stage and prod:
- Crash report with a
JavaExceptioncrash annotation but noprocessed_crash.java_stack_trace. Signature likely to change.- Stage: https://crash-stats.allizom.org/report/index/c9dbd414-a0cc-4924-9ad0-5493a0261117
- Processor notes confirm the new processing rule ran:
JavaStackTraceFromJavaExceptionRule: built java_stack_trace from java_exception - Signature changed from processing:
- Old:
java.lang.OutOfMemoryError: at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java) - New:
java.lang.OutOfMemoryError: at dalvik.system.VMRuntime.newNonMovableArray(Native Method)
- Old:
- Processor notes confirm the new processing rule ran:
- Prod: https://crash-stats.mozilla.org/report/index/4fe85f99-d5e5-44a7-8ab0-c1ec40261117
- Processor notes confirm the new processing rule ran:
JavaStackTraceFromJavaExceptionRule: built java_stack_trace from java_exception - Signature changed from processing:
- Old:
java.lang.NullPointerException: at mozilla.appservices.fxaclient.FxaStateMachineChecker.checkInternalState(fxa_client.kt) - New:
java.lang.NullPointerException: at mozilla.appservices.fxaclient.FxaStateMachineChecker.checkInternalState(fxa_client.kt:78)
- Old:
- Processor notes confirm the new processing rule ran:
- Stage: https://crash-stats.allizom.org/report/index/c9dbd414-a0cc-4924-9ad0-5493a0261117
- Crash report with both a
JavaExceptioncrash annotation and aprocessed_crash.java_stack_trace. Signature not expected to change.- Stage: https://crash-stats.allizom.org/report/index/4a005b87-4e8e-42e7-8a8c-a50510261117
- Processor notes are missing the new rule line as expected, as the new rule did not run.
- Signature changed from processing:
- Old:
java.lang.IllegalArgumentException: at android.os.Parcel.createExceptionOrNull(Parcel.java) - New (unchanged):
java.lang.IllegalArgumentException: at android.os.Parcel.createExceptionOrNull(Parcel.java)
- Old:
- Prod: https://crash-stats.mozilla.org/report/index/66df87ed-4acf-48fa-b61c-d02a60261117
- Processor notes are missing the new rule line as expected, as the new rule did not run.
- Signature changed from processing:
- Old:
java.lang.IllegalArgumentException: at android.hardware.camera2.CaptureRequest.convertSurfaceToStreamId(CaptureRequest.java) - New (unchanged):
java.lang.IllegalArgumentException: at android.hardware.camera2.CaptureRequest.convertSurfaceToStreamId(CaptureRequest.java)
- Old:
- Stage: https://crash-stats.allizom.org/report/index/4a005b87-4e8e-42e7-8a8c-a50510261117
Marking the ticket as done.
I'll likely reprocess the small number (26k) of crash reports that have a JavaException but are missing a processed_crash.java_stack_trace tomorrow, and when I do, I'll update the ticket to that effect.
Description
•