Closed Bug 2088928 Opened 17 days ago Closed 3 days ago

Build java_stack_trace from java_exception for crash pings

Categories

(Socorro :: General, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bdanforth, Assigned: bdanforth)

References

(Blocks 1 open bug)

Details

(Whiteboard: [cringe])

Attachments

(1 file, 1 external file)

Java crash reports include the JavaStackTrace annotation. The JavaStackTraceRule turns it into the sanitized public processed field java_stack_trace, and SignatureGenerationRule uses that field with JavaSignatureTool to generate Java signatures. Crash pings don't include JavaStackTrace, only the structured JavaException, so they currently fall back to JavaExceptionSignatureTool. For about 45% of Fenix Java crashes, that produces a different signature than the corresponding crash report (see the options comparison in CRINGE-445 (https://mozilla-hub.atlassian.net/browse/CRINGE-445?focusedCommentId=1888317)).

java_exception already contains everything JavaSignatureTool needs: it only uses the first two lines of java_stack_trace (the outermost exception's class and its top frame). A validation on 1,000 recent Fenix Java crash reports rebuilt those two lines from java_exception and got identical signatures for ~99.6% of crashes.

The goal of this ticket is to add a processor rule that, for crash pings only, builds java_stack_trace from java_exception, so crash pings get the same Java signatures as the corresponding crash reports with virtually no changes to existing signatures.

Acceptance criteria:

  • Using the approach as validated, add a new processor rule sets java_stack_trace from java_exception for crash pings only (based on the submission type). It doesn't run for crash reports, and doesn't overwrite an existing java_stack_trace.
  • Unit tests to cover all the cases handled in the new rule.
  • A validation run on a recent sample of at least 1,000 Fenix Java crash reports (rebuilding from each report's java_exception and comparing with its real java_stack_trace and signature) shows at least 99% identical signatures. The results are recorded in this ticket.
  • The java_stack_trace description in processed_crash.schema.yaml and the data documentation say that, for crash pings, the field is built from java_exception.

Known limitation: in about 0.4% of crashes (in the validation sample), java_exception doesn't include the outermost exception at all; it's missing on the client. The rule can't detect that from the crash ping, so those crash pings get a signature built from the outermost exception that is present, which differs from the crash report's. This could be fixed later in android-components (optional follow-up bug).

Whiteboard: [cringe]
Status: NEW → ASSIGNED
See Also: → 1698623
See Also: → 1888997

I validated the new processor rule against a random sample[1] of 5000 Fenix Java crash reports with a non-null JavaException crash annotation and processed_crash.java_stack_trace (> 98% of crash reports with a processed_crash.java_stack_trace, which is how we identify a Java from a native crash in crash reports, also have a JavaException) and found that 99% of signatures for crash pings with this new rule would match the corresponding crash report's signature.

Method

  1. Get a list of 5000 random Fenix Java crash ids (from just shell; see attached fenix_java_crashids_sample.txt):
$ socorro-cmd fetch_crashids --num=5000 \
--url="https://crash-stats.mozilla.org/search/?product=Fenix&crash_report_keys=JavaException&java_stack_trace=%21__null__&date=%3E%3D2026-05-17&_sort=uuid"
\ fenix_java_crashids_sample.txt
  1. Since the socorro-cmd signature command only runs signature generation and not the full processor ruleset, temporarily modify its script to remove java_stack_trace and java_stack_trace_raw, and add submission_type = "ping" from the processed_crash, and run the new JavaStackTraceFromJavaExceptionRule on the processed crash (see attached run_new_rule_before_siggen.diff).
  2. Run the signature generation comparison (socorro-cmd signature; see attached siggen_ping_diff.csv and siggen_ping_errors.txt)
socorro-cmd signature --format=csv --different-only \
  < fenix_java_crashids_sample.txt \
  > siggen_ping_diff.csv \
  2> siggen_ping_errors.txt

[1]: As can be seen in the value of the --url argument into socorro-cmd fetch_crashids, the sample was randomized by sorting SuperSearch results by crash ID, which is a randomly generated UUID (with the exception of the last 7 characters, which include the date and unused throttle result bit).

This was deployed in v2026.11.17.

I verified in stage and prod:

  • Crash report with a JavaException crash annotation but no processed_crash.java_stack_trace. Signature likely to change.
    • Stage: https://crash-stats.allizom.org/report/index/c9dbd414-a0cc-4924-9ad0-5493a0261117
      • Processor notes confirm the new processing rule ran: JavaStackTraceFromJavaExceptionRule: built java_stack_trace from java_exception
      • Signature changed from processing:
        • Old: java.lang.OutOfMemoryError: at dalvik.system.VMRuntime.newNonMovableArray(VMRuntime.java)
        • New: java.lang.OutOfMemoryError: at dalvik.system.VMRuntime.newNonMovableArray(Native Method)
    • Prod: https://crash-stats.mozilla.org/report/index/4fe85f99-d5e5-44a7-8ab0-c1ec40261117
      • Processor notes confirm the new processing rule ran: JavaStackTraceFromJavaExceptionRule: built java_stack_trace from java_exception
      • Signature changed from processing:
        • Old: java.lang.NullPointerException: at mozilla.appservices.fxaclient.FxaStateMachineChecker.checkInternalState(fxa_client.kt)
        • New: java.lang.NullPointerException: at mozilla.appservices.fxaclient.FxaStateMachineChecker.checkInternalState(fxa_client.kt:78)
  • Crash report with both a JavaException crash annotation and a processed_crash.java_stack_trace. Signature not expected to change.

Marking the ticket as done.

I'll likely reprocess the small number (26k) of crash reports that have a JavaException but are missing a processed_crash.java_stack_trace tomorrow, and when I do, I'll update the ticket to that effect.

Status: ASSIGNED → RESOLVED
Closed: 3 days ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: