1453326 - Important volume of unhandled crashes by breakpad

Status: NEW

(Toolkit :: Crash Reporting, enhancement)

Description

[Sylvestre Ledru [:Sylvestre]]

We saw recently a huge increase (+145%) of crashes catch by google play instead of breakpad.
The crash rate is also huge (2%)

This is causing a lot of information to be missing in socorro.

I guess we have to update breakpad to handle this.

Bug 1453325 is the most important crash not managed by this.

Comment 1

[:sdaswani]

Liz is this bug distinct from the crashers we've been looking into, i.e., https://bugzilla.mozilla.org/show_bug.cgi?id=1450105 , https://bugzilla.mozilla.org/show_bug.cgi?id=1450793

Comment 2

[Liz Henry (:lizzard)]

It is distinct from bug 1450105.

Comment 3

[(not currently active) Ted Mielczarek]

That crash seems to be in our mozglue linker code. My best guess would be that it's crashing before we even get to load libxul, so we don't have Breakpad initialized at this point.

Comment 4

[Sylvestre Ledru [:Sylvestre]]

Anthony, this is causing a lot of crashes not to be sent to socorro, could you find someone to fix this issue? Thanks

Comment 5

[(not currently active) Ted Mielczarek]

I'm not sure that there's anything fixable here, TBH. If we crash on Android before libxul is even loaded there's no way for us to handle that.

Comment 6

[Liz Henry (:lizzard)]

There is more discussion about mozglue in bug 1450793, and a link there to the Google issue we filed about this problem.

Comment 7

[Calixte Denizet (:calixte)]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #3)
> That crash seems to be in our mozglue linker code. My best guess would be
> that it's crashing before we even get to load libxul, so we don't have
> Breakpad initialized at this point.

ted, would it be possible to have Breakpad stuff in a separate library we could load in first ?

Comment 8

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Gabriele - do you have any thoughts on this?

Comment 9

[Gabriele Svelto [:gsvelto]]

Unfortunately there is no simple way of moving crash reporting earlier in the startup process. It's already tricky enough to have it where it is right now. Besides breakpad is currently built within libxul so there's no simple way of using it earlier short of redesigning the way we do crash reporting (which we might do, it's just a lot of work).

I hope this is not relevant anymore since bug 1450793 has been fixed.

Comment 10

[(not currently active) Ted Mielczarek]

(In reply to Calixte Denizet (:calixte) from comment #7)
> ted, would it be possible to have Breakpad stuff in a separate library we
> could load in first ?

It's certainly possible, but it would require a significant amount of work.

Comment 11

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

I'm going to conclude that this is too difficult for now. Let me know if you think otherwise.

Comment 12

[Sylvestre Ledru [:Sylvestre]]

Attachment: foo.png

Comment 13

[Sylvestre Ledru [:Sylvestre]]

Attachment: foo2.png

Comment 14

[Sylvestre Ledru [:Sylvestre]]

Attachment: release volume

Anthony, i understand it is hard but it is growing (my three attachments are the graphs on nightly, beta & release on google play)

Comment 15

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

I'm guessing it is growing due to bug 1460989.

Comment 16

[Mike Hommey [:glandium]]

(In reply to Gabriele Svelto [:gsvelto] from comment #9)
> Unfortunately there is no simple way of moving crash reporting earlier in
> the startup process. It's already tricky enough to have it where it is right
> now. Besides breakpad is currently built within libxul so there's no simple
> way of using it earlier short of redesigning the way we do crash reporting
> (which we might do, it's just a lot of work).
> 
> I hope this is not relevant anymore since bug 1450793 has been fixed.

ISTR crash reporting is different on android, and is handled by java code. The main blocker remains breakpad being in libxul, but I don't see why we can't move it to libmozglue for android.

Comment 17

[Gabriele Svelto [:gsvelto]]

(In reply to Mike Hommey [:glandium] from comment #16)
> ISTR crash reporting is different on android, and is handled by java code.
> The main blocker remains breakpad being in libxul, but I don't see why we
> can't move it to libmozglue for android.

On Fennec the crash reporter client itself is a Java activity and we have a Java exception handler which is meant to deal with exceptions that aren't caught and bubble all the way up to the top level. Both the Java exception handler and Breakpad's exception handler launch the activity when they catch something. As far as Breakpad is concerned I believe we set the exception handler like on desktop in [1]. There's a whole bunch of stuff going on there including annotating the crash report with product information and whatnot. I don't know how hard it would be to move that stuff into mozglue (or at least the parts that we need) but I'm pretty sure it wouldn't be trivial.

[1] https://searchfox.org/mozilla-central/rev/dc6d85680539cb7e5fe2843d38a30a0581bfefe1/toolkit/xre/nsAppRunner.cpp#3360

Comment 18

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

This is a major project to schedule.