1149608 - FF don't send out all crashes

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Tobias B. Besemer [:BesTo] (QA)]

I recognized that breakpad don't send out all crashes.

First I saw it by the [@ OOM | small]-signature that seems to be now fixed (crashes get send out).
I also saw it by signature belonging to plugins like Flash.
Now I saw it by my crashes with [@ EnterBaseline ].

I think they didn't get send out because they are not actionable?

But I think it falsify the statistics and brings the risk that when this crashes rise higher, that nobody recognize it. Also - because they are not shown in the stats - there is IMHO the risk that they don't get enough attention.

So please send out all crashes even the unactionable.

Comment 1

[(not currently active) Ted Mielczarek]

We don't have any classification like what you're talking about. We attempt to send all crashes. If you're seeing crashes that aren't being submitted it may be that they're plugin crashes. The UI for plugin crashes is not as in-your-face as the crash reporter when the entire browser crashes, so it's possible that you just didn't manage to submit them. We don't submit crashes without user interaction.

Comment 2

[Tobias B. Besemer [:BesTo] (QA)]

I be able to send my crashes. ;-)
But breakpad not. ^^
All my [@ EnterBaseline ]-crashes didn't get out today but the [@ js::jit::EnterBaselineMethod(JSContext*, js::RunState&) ] get all out.
So there have to be a bug.

Comment 3

[Tobias B. Besemer [:BesTo] (QA)]

Also the not send out plugin-crashes are a problem.
Look here: https://bugzilla.mozilla.org/show_bug.cgi?id=1120286#c2
And here: https://bugzilla.mozilla.org/show_bug.cgi?id=1120286#c5
It falsify the statistics.
Plz don't close my bug - instead rename it and send it in the right direction. Thx.

Comment 4

[Tobias B. Besemer [:BesTo] (QA)]

OK, I had this problem again today with this crash:
https://crash-stats.mozilla.com/report/index/75f7c4c7-02cf-4b35-a924-740b82150401

Crash signature is:
[@ shutdownhang | PL_FinishArenaPool | nssArena_Destroy | nssTrustDomain_RemoveCertFromCacheLOCKED | nssTrustDomain_RemoveTokenCertsFromCache | STAN_RemoveModuleFromDefaultTrustDomain ]
So it is definitively no plugin crash.

I also have definitively tried to send out the crash with Breakpad and got a "Send successful".

Is this a bug in Breakpad or on Socorro?

Comment 5

[Benjamin Smedberg]

Tobias, could you provide a more precise description of the steps to reproduce this problem? Perhaps not "steps to reproduce" because you might not be able to reproduce the crash, but help us by identifying what exactly you see on your screen, and what you think isn't correct.

You say that you had a crash today and link to the crash report, so we are receiving the crash properly.

Comment 6

[(not currently active) Ted Mielczarek]

(In reply to Tobias B. Besemer from comment #3)
> Plz don't close my bug - instead rename it and send it in the right
> direction. Thx.

Sorry, we get lots of bugs that are not useful or have a false premise. I suspect this bug could be mostly resummarized as "submitting crashes using the plugin crash UI isn't as straightforward as the crash reporter app".

Comment 7

[Tobias B. Besemer [:BesTo] (QA)]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #6)
> (In reply to Tobias B. Besemer from comment #3)
> > Plz don't close my bug - instead rename it and send it in the right
> > direction. Thx.
> Sorry, we get lots of bugs that are not useful or have a false premise. I
> suspect this bug could be mostly resummarized as "submitting crashes using
> the plugin crash UI isn't as straightforward as the crash reporter app".

Seems I'm running into two problems ... so first that one with the "normal" crashes.


(In reply to Benjamin Smedberg  [:bsmedberg] from comment #5)
> Tobias, could you provide a more precise description of the steps to
> reproduce this problem? Perhaps not "steps to reproduce" because you might
> not be able to reproduce the crash, but help us by identifying what exactly
> you see on your screen, and what you think isn't correct.
> 
> You say that you had a crash today and link to the crash report, so we are
> receiving the crash properly.

OK, I think you can see that I have send out a lot of crashes and I know how Breakpad should work ...
... and select "send" and "Quit after" ...
Because the reporter saves my settings and I never change them it should work ...

I see the following:
I send each crash.
I get an "Send successful".
I open about:crashes and see that on some (or the last one) the "bp-" is missing.
I have to click the crash number and FF change to the "bp-" and send the crash out.

The crash I have reported before have I send this way out.
Next time I will not click on the link in about:crashes and just provide the number so we can compare if he really was send to the server.

I first saw this when I came back to the project and uploaded as described above a lot of crashes in one day. I think I was crashing the statistics in this day, too. :D
(Think I crashed the FF18-Stats ...)
The most of them seemed to be "no thread identified/no data".

Then there was IMHO the "memory small" implemented and I had it often with this ones ...

After that I saw this by the "shutdown in _VEC_ memset" ...

As all this crashes are not actionable I was thinking that they getting rejected because nobody can analyze them.

And now I have it with shutdown-crashes.

As far as I can see this I think that the signatures get "just" better (none -> mem out -> VEC mem out -> shutdown) but the bug that prevent this is always the same.

I don't know if this is a Win-only-bug of if this happens on e.g. Linux, too.
I would suggest that as much as possible persons check there about:crashes to see if they can verify my problem.

Comment 8

[Tobias B. Besemer [:BesTo] (QA)]

*arg* My needinfo is still there ... :-/

Comment 9

[Tobias B. Besemer [:BesTo] (QA)]

https://bugzilla.mozilla.org/show_bug.cgi?id=837835

I think this was me, but I'm not really sure ... ^^

Comment 10

[Tobias B. Besemer [:BesTo] (QA)]

OK, had a look on it ... Not really completely me ... ^^

But it seems (looked at my old send-reports) that I send out on 2013-06-26 547 "hanging/not send out" crashes to the server. :D
Think they all should be empty but I would need a lot of time to recheck this ... :D
They were all from different machines and different Win-Users/FF-Profiles from the company I worked ...

Comment 11

[Tobias B. Besemer [:BesTo] (QA)]

657186a8-a652-42ce-aea8-9bfdada68712	05.04.2015	19:43

Was not send out but I got the message that it wasn't send out.

Is in the logs visible why?

Comment 12

[Tobias B. Besemer [:BesTo] (QA)]

Time was CET (Germany).

Comment 13

[Tobias B. Besemer [:BesTo] (QA)]

I had a look at my submit.log. Here are two parts of it:

[08/29/13 15:30:54] Crash report submission failed: Der Server lieferte eine ungültige oder unbekannte Rückmeldung.

[03/31/15 17:48:59] Crash report submitted successfully
[04/01/15 08:29:03] Crash report submitted successfully
[04/01/15 12:54:52] Crash report submission failed: Der Vorgang wurde erfolgreich beendet.
[04/01/15 17:50:53] Crash report submitted successfully
[04/01/15 19:54:37] Crash report submitted successfully
[04/04/15 14:58:51] Crash report submitted successfully
[04/05/15 19:44:31] Crash report submission failed: Der Vorgang wurde erfolgreich beendet.

But "Crash report submission failed: Der Vorgang wurde erfolgreich beendet." makes IMHO not really sense because "Der Vorgang wurde erfolgreich beendet." means "The task was successfully finished.".
A translation problem or a general one?
Is it possible that I get because of this always the "Submission successful"? Not really, or?

Comment 14

[Tobias B. Besemer [:BesTo] (QA)]

Also: Makes it sense to translate the errors in the log ???

Who have a look at it and what does he with them ???

Comment 15

[Robert Kaiser]

I'm not a breakpad developer, I can't say anything to that. Ted, is it intentional that it says "successfuly" and "failed" in the same log line?

Comment 16

[Tobias B. Besemer [:BesTo] (QA)]

(In reply to Robert Kaiser (:[e-mail]) from comment #15)
> I'm not a breakpad developer, I can't say anything to that.

I tagged you because I was thinking that you done/managed the translation.


Also confusing is for me that ATM the log is using German words because my Aurora is in en-US and my profile, too.

Comment 17

[(not currently active) Ted Mielczarek]

(In reply to Robert Kaiser (:[e-mail]) from comment #15)
> I'm not a breakpad developer, I can't say anything to that. Ted, is it
> intentional that it says "successfuly" and "failed" in the same log line?

That message isn't intentionally localized, we just call the Win32 API FormatMessage with GetLastError:
https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/client/crashreporter_win.cpp#269

Unfortunately sometimes the actual failure doesn't show up in the error code, so you get "the operation completed successfully" (or its equivalent in other languages).

Comment 18

[Tobias B. Besemer [:BesTo] (QA)]

(In reply to Tobias B. Besemer [:BesTo] from comment #0)
> I recognized that breakpad don't send out all crashes.
> 
> First I saw it by the [@ OOM | small]-signature that seems to be now fixed
> (crashes get send out).
> I also saw it by signature belonging to plugins like Flash.
> Now I saw it by my crashes with [@ EnterBaseline ].

I see that somebody have changed it. Can I have a feedback to it.

Comment 19

[Georg Fritzsche [:gfritzsche]]

To just ask the obvious question: Can this be the crash submission throttling?
I don't remember where/how that was implemented exactly.

Comment 20

[Robert Kaiser]

(In reply to Georg Fritzsche [:gfritzsche] [away Sep 5 - 13] from comment #19)
> To just ask the obvious question: Can this be the crash submission
> throttling?

No, there is no throttling on the client side.

Comment 21

[Georg Fritzsche [:gfritzsche]]

Comment 7 here talks about entries in about:crashes being prefixed with bp- or not.
I thought that was simply the throttling in action?

Comment 22

[Benjamin Smedberg]

Every crash is assigned a bp- ID if it is successfully submitted, whether or not the server chooses to process it or not.

If there is no bp- number, that means it either wasn't submitted at all (typically plugin or content crashes) or submission failed for some reason. Comment 13 looks like the clue to follow up on here.

Comment 23

[Tobias B. Besemer [:BesTo] (QA)]

Seems no-one know who changed what and why it works now ???

Comment 24

[(not currently active) Ted Mielczarek]

I don't believe we changed anything--I believe this was simply an issue with your system being unable to submit crashes at some point, perhaps due to networking issues.

Comment 25

[Tobias B. Besemer [:BesTo] (QA)]

And I start to believing that someone is kidding me!

It is really terrible, if I have to search for minutes in a lot of old bug reports I was the last months on, to find the comments from me, to bring evidence for this ...
... and in the past, when I was younger, this would be a reason for me, to leaf the project ...
... and I'm pretty sure, that a lot of (young) volunteers do this, if they feel, that a company/foundation is kidding them ... and it doesn't matter for me/them, in which bug we/they were really kidded and in which one not!

1150028(In reply to Tobias B. Besemer [:BesTo] (QA) from comment #7)
> I see the following:
> I send each crash.
> I get an "Send successful".
> I open about:crashes and see that on some (or the last one) the "bp-" is
> missing.
> I have to click the crash number and FF change to the "bp-" and send the
> crash out.
> 
> The crash I have reported before have I send this way out.
> Next time I will not click on the link in about:crashes and just provide the
> number so we can compare if he really was send to the server.
> 
> I first saw this when I came back to the project and uploaded as described
> above a lot of crashes in one day. I think I was crashing the statistics in
> this day, too. :D
> (Think I crashed the FF18-Stats ...)
> The most of them seemed to be "no thread identified/no data".
> 
> Then there was IMHO the "memory small" implemented and I had it often with
> this ones ...
> 
> After that I saw this by the "shutdown in _VEC_ memset" ...
> 
> As all this crashes are not actionable I was thinking that they getting
> rejected because nobody can analyze them.
> 
> And now I have it with shutdown-crashes.
> 
> As far as I can see this I think that the signatures get "just" better (none
> -> mem out -> VEC mem out -> shutdown) but the bug that prevent this is
> always the same.
> 
> I don't know if this is a Win-only-bug of if this happens on e.g. Linux, too.
> I would suggest that as much as possible persons check there about:crashes
> to see if they can verify my problem.

FYI: The old crashes I send out months before, were from a different firm-network, from a lot of PCs, with a lot of profiles, on a other place, with a other provider. ^^

This bug was filled on 2015-03-31 and the problem "just disappeared" immediately after it ... ^^

Just on day later, on 2015-04-01, I filled bug 1150028, were the problem also "just disappeared" ... ^^

What do you think - I as a volunteer, not really part of the company/foundation, not paid - should think and feel about this ???
And what do you think, how many people wouldn't give you/Mozilla this feedback and just leaf the project ???

Comment 26

[Tobias B. Besemer [:BesTo] (QA)]

Btw.: This "rant" is not against you !!!

... but maybe someone by Mozilla should think a little bit about it and inform after that the colleagues, that maybe some things can upset some volunteers ... ;-)