1837471 - live_backing.log and logcat-emulator log give different crash address

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Sean Feng [:sefeng211]]

For this Android crash https://treeherder.mozilla.org/jobs?repo=try&revision=9fa63cac10f86f9e8d3ee88dcb2d5c2c609d5819&selectedTaskRun=DN_ILQ3nSq-yfONxCK9mkQ.0 , the address from the live_backing.log is libxul.so + 0x0000000001237f63, however if you open the logcat file which is named as logcat-emulator-5554.log for this task, the crash address is displayed as
???[/data/app/org.mozilla.geckoview.test_runner-1/lib/x86_64/libxul.so +0x237cf52]. After manually looking up the address in the symbol file, I believe the one from the logcat is correct.

Comment 1

[Gabriele Svelto [:gsvelto]]

This is an issue with minidump generation which we'll hopefully fix with the new version of the minidump-writer crate which will soon support Android (see bug 1689358). The libxul.so mapping in the process is the following:

78910c9f7000-78910cf71000 r--p 00000000 fd:20 14091                      /data/app/org.mozilla.geckoview.test_runner-1/lib/x86_64/libxul.so
78910cf71000-78910db3c000 ---p 00000000 00:00 0 
78910db3c000-7891146d2000 r-xp 00579000 fd:20 14091                      /data/app/org.mozilla.geckoview.test_runner-1/lib/x86_64/libxul.so
7891146d2000-789115891000 r--p 0710f000 fd:20 14091                      /data/app/org.mozilla.geckoview.test_runner-1/lib/x86_64/libxul.so
789115891000-789115892000 r--p 082ce000 fd:20 14091                      /data/app/org.mozilla.geckoview.test_runner-1/lib/x86_64/libxul.so
789115892000-789118152000 r--p 082cf000 fd:20 14091                      /data/app/org.mozilla.geckoview.test_runner-1/lib/x86_64/libxul.so
789118152000-78911875c000 r--p 0ab8e000 fd:20 14091                      /data/app/org.mozilla.geckoview.test_runner-1/lib/x86_64/libxul.so
78911875c000-7891187ac000 rw-p 0b198000 fd:20 14091                      /data/app/org.mozilla.geckoview.test_runner-1/lib/x86_64/libxul.so

The hole in the second line causes the minidump writer to emit two distinct modules for libxul.so which confuses the stackwalker. It ends up calculating the offsets from the 0x78910db3c000 mapping instead of the 0x78910c9f7000 one.

Comment 2

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:gsvelto, could you have a look please?

For more information, please visit BugBot documentation.

Comment 3

[Gabriele Svelto [:gsvelto]]

Here's a similar issue showing up in this crash with even more holes in the mapping:

9b4a0000-9b931000 r--p 00000000 08:12 393449     /data/app/org.mozilla.firefox-1/lib/x86/libxul.so
9b931000-9bcae000 ---p 00000000 00:00 0
9bcae000-a116b000 r-xp 00490000 08:12 393449     /data/app/org.mozilla.firefox-1/lib/x86/libxul.so
a116b000-a4562000 r--p 0594d000 08:12 393449     /data/app/org.mozilla.firefox-1/lib/x86/libxul.so
a4562000-a4563000 ---p 00000000 00:00 0
a4563000-a4840000 r--p 08d44000 08:12 393449     /data/app/org.mozilla.firefox-1/lib/x86/libxul.so
a4840000-a4873000 rw-p 09021000 08:12 393449     /data/app/org.mozilla.firefox-1/lib/x86/libxul.so

Comment 4

[Gabriele Svelto [:gsvelto]]

I prepared a PR to fix this issue, hopefully we'll be able to pull it in m-c quickly.

Comment 5

[Gabriele Svelto [:gsvelto]]

Attachment: Bug 1837471 - Update the minidump-writer crate to pick up important Android fixes r=KrisWright

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/92ae5e9b1bd6
Update the minidump-writer crate to pick up important Android fixes r=KrisWright,supply-chain-reviewers

Comment 7

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/92ae5e9b1bd6

Comment 8

[Donal Meehan [:dmeehan]]

:gsvelto based on https://bugzilla.mozilla.org/show_bug.cgi?id=1852039#c15, since this has been in nightly for a week should this get a beta uplift request?
I wondered if it's safe for Fx119, or if it should ride the train with Fx120.

Comment 9

[Gabriele Svelto [:gsvelto]]

I think it's safe to uplift it. I've looked at the crashes on nightly for the last week and I don't see any funny signatures anymore so it should be doing its job.

Comment 10

[Gabriele Svelto [:gsvelto]]

Comment on attachment 9354283 [details]
Bug 1837471 - Update the minidump-writer crate to pick up important Android fixes r=KrisWright

Beta/Release Uplift Approval Request

• User impact if declined: None, but some crashes will yield bogus crash reports that make it harder to diagnose problems for our developers
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The code in this patch only runs after a crash, so it's rarely exercised. Additionally it's 100% safe Rust.
• String changes made/needed: none
• Is Android affected?: Yes

Comment 11

[Donal Meehan [:dmeehan]]

Comment on attachment 9354283 [details]
Bug 1837471 - Update the minidump-writer crate to pick up important Android fixes r=KrisWright

Approved for 119.0b6

Comment 12

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/8590342d3bd6