Closed Bug 1407631 Opened 6 years ago Closed 6 years ago

Crash reporter appears to submit two crash reports per crash

Categories

(Toolkit :: Crash Reporting, defect)

Product:
Toolkit
Component:
Crash Reporting
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla58
Tracking Flags:
Tracking Status
firefox-esr52 --- wontfix
firefox56 --- wontfix
firefox57 - wontfix
firefox58 + fixed

People

(Reporter: spohl, Assigned: mconley)

References

(Blocks 1 open bug)

Details

Attachments

(1 file, 1 obsolete file)

Bug 1407631 - Wait about 10 minutes after browser session start before scanning for unsubmitted crash reports.
59 bytes, text/x-review-board-request
gsvelto
: review+
Details 
Ran into this while looking into bug 1405151. If the parent process is terminated via `kill -s 6` we appear to submit two crash reports, which may skew our statistics.

Steps:
1. Launch Firefox.
2. In Terminal, crash the firefox process with `kill -s 6`.
3. Notice Crash Reporter open. Submit the crash report.
4. Restart Firefox, enter about:crashes in the URL bar.
5. Notice two crash reports for the same crash.

Two example reports for the same crash that I just encountered:
https://crash-stats.mozilla.com/report/index/f5c66f2a-8231-40a5-81da-bc0490171011
https://crash-stats.mozilla.com/report/index/005b86e2-8610-4556-bef9-40ff81171011

I'm not sure if this is specific to macOS or if this affects other platforms as well.
Ted, is this a known issue?
Flags: needinfo?(ted)
This isn't anything I've seen before, certainly. If you look in "~/Library/Application Support/Firefox/Crash Reports/submit.log" does it show two lines with "Crash report submitted successfully" for that time?
Flags: needinfo?(ted)
Yuck. I can reproduce this locally. To answer my own question, no, there's only one line in submit.log.
I think this is from the "auto-submit unsubmitted reports" code (browser.crashReports.unsubmittedCheck.autoSubmit) racing with the native crash reporter. I tried crashing Firefox again, but instead of clicking "Restart Firefox" I clicked "Quit" and let it submit the crash, then started Firefox again manually and it had only submitted one report.

The native crash reporter doesn't remove the minidump file from the pending directory until it submits it successfully (so that it can still be submitted if a transient network error causes submission to fail). The unsubmitted check code must be running soon enough that it notices the dump file in the pending directory before it gets removed, and submits it again.

We should probably just make the unsubmitted check code ignore reports that are too new, like within the past few minutes.
Component: General → Crash Reporting
Product: Core → Toolkit
I've seen as many as 4 reports per crash (it appears), but i easily believe 2 - see tons of those.  This is also warping our stats about crashrate I think...

Perhaps we could have the processing script hash the incoming reports, and block any dups within 24 hours (to limit the search space).

NI lizzard/marcia/ritu so Release Management knows this is probably warping crash rates (we should verify - I'm sure it's warping the stats we see in the crash-stats extension (graphs)).  We may want to try to 'cull' the crashreport database of dups

Because tracking crash rates is important (esp. for 57), we should really hop on getting some sort of solution in place (and probably both client side and server side).  Server-side could be deployed ASAP and block all new ones even from older builds, and as mentioned we could retroactively cull bad entries.
Flags: needinfo?(rkothari)
Flags: needinfo?(mozillamarcia.knous)
Flags: needinfo?(lhenry)
Ted: Who is the right person to take this bug?
Flags: needinfo?(mozillamarcia.knous) → needinfo?(ted)
Anyone on the Firefox team could fix this.

FYI: this won't affect release, because we don't enable the "unsubmitted crash reports" infobar there.
Flags: needinfo?(ted)
(In reply to Randell Jesup [:jesup] from comment #5)
> Perhaps we could have the processing script hash the incoming reports, and

We do already have a hash of the minidump in Socorro. I don't think it's visible in the UI anywhere, though. If you're logged in you can probably see it via the API. For the two crashes spohl linked in comment 0:
https://crash-stats.mozilla.com/api/RawCrash/?crash_id=f5c66f2a-8231-40a5-81da-bc0490171011&format=meta

"dump_checksums": {
"upload_file_minidump": "eb758b7cf5d0360d1b726bf37312debe"
}, 

https://crash-stats.mozilla.com/api/RawCrash/?crash_id=005b86e2-8610-4556-bef9-40ff81171011&format=meta


"dump_checksums": {
"upload_file_minidump": "eb758b7cf5d0360d1b726bf37312debe"
}, 

So those are definitely both the same minidump submitted twice.
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #8)

> So those are definitely both the same minidump submitted twice.

Cool - so that should make it easy for someone to cull the already-submitted reports for dups (and block submission of dups - either just check the db, or check hashes-reported-in-the-last-N.  

Worst effect of culling dups I can imagine would be some bugzilla links to crashes might become stale.

Who can do the DB filtering, and who can modify the submission code?  (or we could just filter periodically until they're not being submitted a lot anymore, but that might take a while).
(In reply to Randell Jesup [:jesup] from comment #9)
> Worst effect of culling dups I can imagine would be some bugzilla links to
> crashes might become stale.

about:crashes lists two submissions as well (this is how I originally noticed this bug). I believe one of the two entries would become stale without some client-side changes.
Thanks Randell. Philipp mentioned this at the channel meeting today. We're taking this as "good news" and hoping beta57 stability is better than what the # Socorro based numbers tell us.
Flags: needinfo?(rkothari)
Mike, can you help find someone to take this on? We do need to be able to rely on our crash rate assessments. I'm not sure who to ask here, if not Ted.
Flags: needinfo?(lhenry) → needinfo?(miket)
blassey wrote the patch for the unsubmitted crash reports feature (Bug 1269998) and mconley reviewed it. I'm assuming Brad doesn't have the time to work on this these days, but maybe he or Mike would have a suggestion for someone to tackle this?

(See Comment #4 for what ted thinks is the culprit)
Flags: needinfo?(miket)
Flags: needinfo?(mconley)
Flags: needinfo?(lassey)
Will - can you or someone look at the issue of culling dups out of our Socorro database?  Or leave them there, but make "normal" searches ignore them, or some such?  

The concern is we make important decisions based on the stats coming from the DB; personally having dups in about:crashes that don't lead anywhere is ok (or if browsing to a dup leads to a dup page, great, but I wouldn't block fixing it on that, personally).

Thanks!
Flags: needinfo?(willkg)
I can probably look at this next week.

The simplest solution is probably to have the unsubmitted crash reporter ignore crash reports from the last X number of minutes.

Alternatively, we could have the crash reporter executable hold some kind of lock on the crash report file until the send goes through (and release the lock if anything goes wrong), and have the unsubmitted crash reporter ignore anything that's locked up.
Flags: needinfo?(lassey)
Socorro has multiple crash storage destinations. The one that most things on Socorro (Super Search, APIs, reports, etc) use is Elasticsearch. To remove a duplicate crash, we'd probably have to at least remove it from Elasticsearch, but it's possible it could get reprocessed and show up again.

I think removing existing duplicates and preventing future duplicates from being saved is technically possible, but I think this is non-trivial and it'll take a couple of weeks to figure out and land.

What was the date that we started getting duplicate crash reports?

I see concerns about duplicates affecting reports people are using, but has anyone looked at the actual rates?
Flags: needinfo?(willkg)
I see tons of dups when searching around signatures.  I'm pretty sure these have been showing up for months.  Ted, when did we start setting "browser.crashReports.unsubmittedCheck.autoSubmit"?  From ContentCrashHandlers.jsm, I see the pref was added around Sept of 2016(!)  But it may not have been enabled then, and in any case this probably only applies to e10s, I'd guess.
Flags: needinfo?(ted)
I looked at 14,000 crashes in Socorro -prod from 10/9 to 19/15 (2,000 per day) and saw:

* 3 crashes with 1 duplicate
* 1 crash with 89 duplicates

That's 92 total duplicates out of 14,000. Generally, I don't think this is a big issue. However, since 1 crash had 89 duplicates, maybe this affects some signatures heavily. I didn't look into that aspect of things.

Data here:

2 ['b30de7a5-471b-4cee-ad54-aa8af0171009', '69e0e8ef-3c4d-4516-8ba2-263d30171009']
2 ['b1961fb0-3b0c-4006-93f4-a95670171014', '200e22b3-9b7e-44da-9f99-e11410171014']
90 ['5cf15602-0819-48f1-977c-6acdd0171014', 'febfa0a4-07df-4cfa-91cc-08c190171014', '0228bdf1-4bbb-4641-a582-ca81d0171009',
    'b53976a2-b2b4-4ad8-9883-661070171015', 'ce4f7c89-4905-43fc-92fb-08f490171014', '19778dd9-f6bb-48ab-935f-9c7030171009', 
    '9274bbcf-2882-4760-9013-d44800171009', '20002f44-c8e0-49c5-9369-87cbb0171015', 'eada86af-34dc-4baf-8df7-df7630171015', 
    '0de43ae1-e5ce-434d-9388-101f70171014', '03f833dc-1c91-46e5-9437-489d50171009', '2d4e82f4-ab70-43fb-a806-0d4380171013', 
    '67d0bf14-ee4a-43b3-8842-a7f870171015', '6210631e-633b-4949-892b-157cd0171014', '66f93aff-1a96-4c77-b399-df5cd0171015', 
    '9e8eebcf-5409-4a6f-9f8c-3a29a0171015', '0a0bf6e1-fcbd-4dc6-9f0d-522540171015', '6910a70f-e376-4cc0-a346-aeb060171014', 
    'f3897e1e-32ea-46af-ac56-c1cd90171014', '8202e0ab-3e3e-4054-bb4d-36ef30171015', 'a346a930-7b95-44c9-900a-419060171015', 
    'bc47a5ce-6b3d-4138-9d6f-96b980171015', 'a7129ac0-5506-4b47-be23-414260171015', '0a6fd16a-b508-4076-aad5-3a6da0171014', 
    'b5e060f3-4ab7-4fa8-99f4-1e6670171009', '144675ef-cb9a-4589-804e-0c7660171015', 'ef281989-612c-4f7a-932f-5c69b0171009', 
    'ce577762-9b25-4481-85c8-abd9d0171009', 'afddb7d2-3841-4def-a029-a6f5b0171014', '1277ef8b-3db6-444b-83ce-9ca390171014', 
    '4d6f3e2d-e4ca-4073-a019-7d4000171014', '1754f0e2-c644-41fd-ad95-45dd30171009', '8ef82889-e7fa-4777-bb26-c6ce40171014', 
    'e0695f53-45a6-466d-93d8-176ea0171014', '90a259a9-5bf9-4152-83cd-e12e40171014', 'e63e3d31-b233-4794-ae77-cb3be0171009', 
    '88bfd4fd-f08d-477a-a131-59a7a0171015', 'd4fe0584-357a-4ece-a867-732400171014', 'eda62ed1-0e25-41d1-a4ab-3362d0171013', 
    'f5b8d402-8316-4bbb-ac90-0ad640171014', '68e2fb73-4afa-4e37-8f95-8fa020171014', 'a0218d7f-e221-483b-a480-46d290171015', 
    'c419d989-bdd8-40da-b783-2e8e70171009', '16310a0f-e3db-404c-a2de-252830171014', 'bb6028a0-068c-4e8d-8d64-0be760171009', 
    '36aca070-32b0-4a08-a033-0e1ef0171009', '49ea3ce3-0f6b-4a41-9d02-2e1130171015', '2ed49318-80c1-4a9e-a437-712390171013', 
    '50fb4fa3-5716-4db5-9719-ff42f0171014', 'd1bcb566-d973-4332-9bff-ab05f0171014', '2ea306de-0138-4d56-9b91-9bcf50171009', 
    '61dd3fa9-2bb9-4f3f-9004-2d76c0171009', '1aa1bc6c-a4e0-40a2-be61-9f3260171009', 'd4c3630d-c59a-4320-b6d0-ff9550171009', 
    '99aa5660-b2bf-49c6-add7-28b830171013', 'c4cfed75-6c42-4c23-b0df-202310171015', '288e9c52-51c4-49cf-b862-e131c0171015', 
    '2f0104d5-2af6-496f-8910-e12160171009', '71d2e3e7-c860-4573-9517-073e80171014', 'ea58cd11-848d-4df9-886c-beaf30171009', 
    'b0159b20-f943-4b4d-ba14-00e690171009', '17e8108f-c897-430e-8acb-ba4390171015', '4224240f-2b25-4ca0-8a12-e75fe0171009', 
    '836a5008-cf2f-4eca-ae07-74f450171015', '42827a49-b3a5-4fb4-83cf-afca10171015', '7af8b66e-934b-4d58-ac1d-b39160171015', 
    'ef97b4a4-9020-45b2-8931-8ac390171009', '14af1414-138b-4cd0-aac7-1765f0171015', '43677b85-1fd0-4865-971f-6ec710171014', 
    '87e0a96a-66d9-4846-8469-853190171014', '3db15701-e794-4062-aef3-3d7a90171009', '2eccb1af-beda-4001-ade2-31a9a0171009', 
    'c495cb94-f964-4fcd-8c1c-74b1e0171009', 'af2a671b-2fc9-421e-a796-6c20e0171009', 'f3d5c77e-2971-4764-bba2-cbc1f0171015', 
    '4ae0eb4b-1dae-407c-a935-56f200171014', '0170a52a-d7e2-460a-9847-5b7e40171009', 'd6055d4b-663c-421e-b1b1-172e20171009', 
    '2474e064-adc6-4935-8cc8-048510171009', '9409dd98-4b7e-4e57-a0c7-793cc0171015', 'af55a0a7-159d-4a1b-a68a-238980171009', 
    'e5a2746d-d2d1-47bb-8cb5-d95ca0171015', '8d540df5-7861-4007-b3d2-a6a040171015', 'ed2b0f09-b69a-44c2-ac15-13a660171015', 
    'ec6ea551-2475-46d2-b80a-732a50171015', '749a5a93-7d7e-4ec9-8af0-08c590171009', 'e54f758f-fa17-4eeb-bc60-307af0171009', 
    '92383fd8-e35a-4aff-a62a-ee4b00171013', '3dff9113-355b-465f-bb94-387b80171013', '802e6d14-2674-4b2e-8285-7ec730171015']
2 ['beb20293-5ae9-4078-aac6-ed6080171015', '35f442b3-a0b2-44c3-84e6-387ba0171015']


I'm pretty sure this is a large chunk of work for Socorro. I wrote up bug #1409138 to look into it further and deal with it. We'll need to implement bug #1408041 first. However, unless someone shows me compelling evidence that this actually is affecting analysis, I don't think this is something we should work on in Socorro until after Firefox 57.
Flags: needinfo?(mconley)
This patch should probably take care of most cases where this occurs - it prevents the unsubmitted crash submitter stuff from running on reports that are less than an hour old.
(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #18)
> I looked at 14,000 crashes in Socorro -prod from 10/9 to 19/15 (2,000 per
> day) and saw:
> 
> * 3 crashes with 1 duplicate
> * 1 crash with 89 duplicates
> 
> That's 92 total duplicates out of 14,000. Generally, I don't think this is a
> big issue. However, since 1 crash had 89 duplicates, maybe this affects some
> signatures heavily. I didn't look into that aspect of things.

That's odd.... I saw a bunch of apparent dups when looking over various 57/58 crash reports -- was the 14000 all reports, or nightly/beta?  IIUC from Ted, this only can happen if you opt into reporting that's only possible in nightly/beta, and also ted and I found some pretty easily.  (Note my searches were limited to 57/58).  Is the time field the time it's uploaded, or  the time the crash happened?  If there are repeated crashes-on-start-and-restore (or on restoring a tab that crashed), that could cause this pattern without being dups - but the uptimes I was seeing didn't seem to support that.

If dups are really that rare, great!, and then I'm not worried about it hurting our stats.
I wasn't filtering on the channel, so it covered all crashes. I'll redo the analysis tomorrow and restrict to just nightly and beta crashes and see if it comes out differently.
(In reply to Mike Conley (:mconley) (:⚙️) - Backlogged on reviews and needinfos from comment #20)
> This patch should probably take care of most cases where this occurs - it
> prevents the unsubmitted crash submitter stuff from running on reports that
> are less than an hour old.

This assumes that a user restarts Firefox within an hour of crashing. I just got back to my desk and noticed that Firefox had crashed, with the crash reporter open. It looks like the crash occurred over an hour ago. If I click submit and restart Firefox, the same race will occur again and two crash reports will be submitted, wouldn't it?

I do like your idea of holding a lock of some sort on the crash report while submitting.
Flags: needinfo?(mconley)
Comment on attachment 8919021 [details]
Bug 1407631 - Have the UnsubmittedCrashReporter ignore reports that are less than an hour old.

(I can take the review back if Gabriele is busy, but for now I'll tentatively send it to him since he has spent a lot of time thinking about crash reports recently)
Attachment #8919021 - Flags: review?(felipc) → review?(gsvelto)
(In reply to Stephen A Pohl [:spohl] from comment #23)
> I do like your idea of holding a lock of some sort on the crash report while
> submitting.

Or, instead of running the UnsubmittedCrashReporter immediately after startup, could we simply delay it by 5-10 minutes? That should take care of the race as well.
Comment on attachment 8919021 [details]
Bug 1407631 - Have the UnsubmittedCrashReporter ignore reports that are less than an hour old.

I'd be more keen on Stephen's suggestion in comment 25. And BTW there's no real way to solve this problem because nothing prevents the user from launching Firefox manually while the crashreporter is still running. In fact it happened to me a couple of times I forgot that the crashreporter was open and it was hidden behind other windows.

As for file locks they're usually frowned upon because they're brittle in the presence of network-mounted filesystems. Additionally I don't think we have the necessary machinery to implement them: NSPR file locking facilities are not used in Gecko so I suspect they wouldn't fit the bill and I couldn't find any other example of actual file locking in our code.
Attachment #8919021 - Flags: review?(gsvelto) → review-
We could alternately make the crashreporter client move or rename the .dmp/.extra files it's submitting when it launches, and then move them back if submission fails. (It deletes them on successful submission.)
Flags: needinfo?(ted)
(In reply to Gabriele Svelto [:gsvelto] from comment #26)
> Comment on attachment 8919021 [details]
> Bug 1407631 - Have the UnsubmittedCrashReporter ignore reports that are less
> than an hour old.
> 
> I'd be more keen on Stephen's suggestion in comment 25. And BTW there's no
> real way to solve this problem because nothing prevents the user from
> launching Firefox manually while the crashreporter is still running. In fact
> it happened to me a couple of times I forgot that the crashreporter was open
> and it was hidden behind other windows.

I assumed that UnsubmittedCrashReporter would remove the crash dump from the file system after submitting. It's not ideal that the crash reporter would still be open once the crash dumps have been removed from the file system, but I didn't expect this to cause duplicate crash reports to be submitted. Is this assumption incorrect?

(Yes, there is a tiny chance of a race if you start Firefox manually and then click submit in crash reporter exactly at the same time that UnsubmittedCrashReporter runs after the suggested 5 minute delay. But this seems negligible.)
Flags: needinfo?(gsvelto)
(In reply to Randell Jesup [:jesup] from comment #17)
> I see tons of dups when searching around signatures.  I'm pretty sure these
> have been showing up for months.  Ted, when did we start setting
> "browser.crashReports.unsubmittedCheck.autoSubmit"?  From
> ContentCrashHandlers.jsm, I see the pref was added around Sept of 2016(!) 
> But it may not have been enabled then, and in any case this probably only
> applies to e10s, I'd guess.

bug 1287178 added that pref and added the UI for "Always Submit", so your timeline is correct. It's not enabled by default in any situation--it's opt-in from the infobar. I don't know that we have any data on what percentage of users actually enable this.

bug 1303067 enabled the infobar for early betas, and it landed in late October of 2016 (Firefox 52) but was also uplifted to Aurora (Firefox 51) shortly afterwards.
(In reply to Stephen A Pohl [:spohl] from comment #28)
> I assumed that UnsubmittedCrashReporter would remove the crash dump from the
> file system after submitting. It's not ideal that the crash reporter would
> still be open once the crash dumps have been removed from the file system,
> but I didn't expect this to cause duplicate crash reports to be submitted.
> Is this assumption incorrect?

Indeed, what will happen then is that the crash reporter will fail to submit the crash, or maybe submit it with an empty minidump since the file isn't there anymore.

> (Yes, there is a tiny chance of a race if you start Firefox manually and
> then click submit in crash reporter exactly at the same time that
> UnsubmittedCrashReporter runs after the suggested 5 minute delay. But this
> seems negligible.)

Yeah, my point was that there's no bulletproof solution to this problem but the good enough solution should make the problem negligible.
Flags: needinfo?(gsvelto)
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #27)
> We could alternately make the crashreporter client move or rename the
> .dmp/.extra files it's submitting when it launches, and then move them back
> if submission fails. (It deletes them on successful submission.)

Yeah, that's also doable, like rename the file to .dmp.locked and .extra.locked or something like that. The question is how far do we want to go to address this problem? IMHO just delaying the unsubmitted crash reporter as Stephen suggested in comment 25 should be sufficient to cover most cases while requiring only minimal changes to the existing code.
That seems fine. We could add an additional check in the crashreporter client for the case where it got lost in the background--if the minidump no longer exists when we try to submit, just exit.
I redid the analysis and looked at 2000 crashes per day from 10/9 through 10/15 for Firefox "nightly" and "beta" channels.

In that period, there were a small handful of duplicates:

2 [('e7b31f58-cdee-44be-b88e-93e530171010', u'beta'), ('e493d58e-5547-47d5-a398-70da40171010', u'beta')]
2 [('ccda6ced-2fc4-468c-acf9-5dc3c0171011', u'beta'), ('dd7629db-287d-4b1f-963c-6bf3f0171011', u'beta')]
2 [('beb20293-5ae9-4078-aac6-ed6080171015', u'nightly'), ('35f442b3-a0b2-44c3-84e6-387ba0171015', u'nightly')]
2 [('cc6b2c00-1414-4af1-b00c-bae7b0171010', u'beta'), ('90099f65-6382-4c4c-a8f8-555020171010', u'beta')]
2 [('b1961fb0-3b0c-4006-93f4-a95670171014', u'beta'), ('200e22b3-9b7e-44da-9f99-e11410171014', u'beta')]
2 [('be40e1d4-3948-4db4-87b7-c96540171011', u'beta'), ('93eb7a24-6af9-437d-bcd0-114620171011', u'beta')]
2 [('3a110e6f-c423-4a7c-8c31-60b880171009', u'beta'), ('1496d13b-a61f-4acc-a9e1-b3cd80171009', u'beta')]
2 [('b30de7a5-471b-4cee-ad54-aa8af0171009', u'beta'), ('69e0e8ef-3c4d-4516-8ba2-263d30171009', u'beta')]


I'm ignoring the one that has 418 duplicates because those all have a 0-byte upload_file_minidump file.

Out of 14,000 crashes, we had 8 duplicates.
Super sorry about this--I was looking at the wrong directory of crash data. Re-re-re-doing the analysis shows a lot more duplicates:

2 [('1fa3c6ae-acb8-4bd9-a104-fce4f0171012', u'nightly'), ('752d2f8c-beb1-4fb9-b7d3-9f9f70171012', u'nightly')]
3 [('fd9a855b-0f43-4bc1-aa0a-ceac40171012', u'beta'), ('a1457f97-d356-4b49-b4ca-ec8bd0171012', u'beta'), ('d836b6ab-3bdd-453f-a46c-4a42a0171012', u'beta')]
2 [('8152cd8f-f536-484e-b01e-a5c5e0171009', u'beta'), ('f90c6f84-7b74-4bf6-97a6-940d10171009', u'beta')]
2 [('707108aa-9c7f-4861-a2a4-8bf770171009', u'beta'), ('8f08ad47-0138-4c9e-8988-699020171009', u'beta')]
2 [('d2096fb5-4cec-4db3-9984-1e9480171011', u'beta'), ('9eaf6105-2360-4a29-b8bb-9b3530171011', u'beta')]
2 [('6845058a-fab3-4f1e-b022-5f4260171011', u'beta'), ('b373d84d-55ce-42ec-8ae2-f63440171011', u'beta')]
2 [('b2a1f342-bb35-4baa-b651-2f7c70171014', u'beta'), ('706e03fa-be9b-4f94-8e08-edb850171014', u'beta')]
2 [('fc474e2d-5f03-4251-a260-5b0ec0171013', u'beta'), ('165155fd-ffc3-4e74-b60b-a75bd0171013', u'beta')]
2 [('67e47cc7-6908-4fc7-a2e2-bdb4a0171010', u'beta'), ('2ab1dba6-5a4a-4b96-a33a-fff280171010', u'beta')]
2 [('9d9cff82-187b-461f-b964-34e480171012', u'beta'), ('8f17e1ce-95ab-4849-8212-d5b050171012', u'beta')]
2 [('d4f8c11c-3f7d-4bd5-a906-ea4030171011', u'nightly'), ('54ae1364-33ca-41e8-8fe1-e1abf0171011', u'nightly')]
3 [('14ec1e66-1b25-4948-be6c-e23bf0171011', u'beta'), ('8763c104-01a7-4855-891d-1b4c30171012', u'beta'), ('2c5f5a28-9b78-48bc-b461-3ca8c0171014', u'beta')]
2 [('a6340d48-9366-4f99-9cd0-0afe40171012', u'beta'), ('3603c07e-030c-4c1b-a643-ee4f30171012', u'beta')]
2 [('4a3c80a9-7590-481f-9ff9-aaffb0171010', u'nightly'), ('ea6b3b6e-771a-4654-977f-9216e0171010', u'nightly')]
2 [('05760874-e9a6-49bb-9537-307a90171013', u'nightly'), ('1bb96877-44e0-4400-8f5f-5c2d30171013', u'nightly')]
2 [('f66647a2-4ddf-4e64-9dde-d63c70171010', u'nightly'), ('686fb07e-5f12-461a-88d4-b2dc20171010', u'nightly')]
2 [('037d55c0-6ead-46d8-8574-258fc0171011', u'beta'), ('c2c23b84-36ac-45d3-b668-2cab10171011', u'beta')]
2 [('8480a2ba-fcff-405f-a783-afa5a0171015', u'nightly'), ('297406ae-13ce-4fa4-90d9-c30670171015', u'nightly')]
2 [('62f5c7b8-758d-4b35-9902-612d00171012', u'beta'), ('0af319f5-0d7e-40df-b49d-2f8c10171012', u'beta')]
2 [('f6075d10-f690-4e54-808c-1e22a0171012', u'beta'), ('fcda55f7-1b95-4e46-893a-a0d910171012', u'beta')]
2 [('8c510915-8d8c-492f-85bc-c22850171015', u'beta'), ('0602bf7b-814d-4873-aa70-16e1d0171015', u'beta')]
2 [('521113b1-efa1-412a-a133-579c10171015', u'beta'), ('06c4505a-ca61-47d2-9ebf-2c3230171015', u'beta')]
2 [('d7bfad73-9800-4ade-8123-cf1d00171009', u'beta'), ('028ec38a-e2c5-446f-bd66-31eb80171009', u'beta')]
2 [('f843db81-aad5-4f4d-a8f0-452490171012', u'beta'), ('19c8ab21-b474-417a-97f5-d85350171012', u'beta')]
2 [('562ca47c-627b-491e-9baa-5732c0171014', u'beta'), ('c087a4d2-702b-436a-883f-fef630171014', u'beta')]
2 [('e635ef07-089b-4a0a-a717-77aa50171009', u'nightly'), ('9b81e99c-dbb5-46f2-8452-575310171009', u'nightly')]
2 [('5848165a-34e7-4346-8ae2-d1b050171009', u'beta'), ('3813b178-628c-405c-84fd-867810171009', u'beta')]
2 [('8e178e36-75cc-4f64-a083-0a7000171012', u'beta'), ('b9edd0ec-81a2-498a-94c7-d1ee90171012', u'beta')]
3 [('72c95bce-4edf-43d7-b39d-ab8a90171010', u'beta'), ('ff496a27-86f6-4b61-ae25-0b2c60171010', u'beta'), ('fef8e6a1-0279-459b-90b9-805f00171010', u'beta')]
2 [('88787fe9-7954-4aef-bb29-09e8f0171012', u'beta'), ('1cf34504-d7e0-422d-8c50-b95180171012', u'beta')]
2 [('d0471af6-b0b0-4e86-aedc-5a0940171009', u'beta'), ('7aee34c1-b5cb-4557-ad4a-108a30171009', u'beta')]
2 [('67f98194-bc65-44c4-a8d0-57d030171015', u'nightly'), ('15ff8065-9f29-4f11-8c26-d5a510171015', u'nightly')]
2 [('70c36724-b26f-4217-8cf7-1e4a40171010', u'nightly'), ('2ad4c93c-dd4c-4efc-87c8-fea1b0171010', u'nightly')]
2 [('e2fc40b6-b789-400d-a0c4-45d560171015', u'beta'), ('7690d5b9-2740-4e8d-808a-eb32b0171015', u'beta')]
2 [('c623c525-821e-4a66-9adc-b4cc50171011', u'beta'), ('e5ed280a-d8e1-4a49-8549-1efc20171011', u'beta')]
2 [('e014a2f3-66db-4f11-b386-872ae0171014', u'beta'), ('66ddbe28-28d5-4126-9b62-546070171014', u'beta')]
2 [('bc3f766d-4cc0-4a40-b881-07c8d0171014', u'beta'), ('e89e6154-3a5b-4c1f-b3fe-3adc90171014', u'beta')]
2 [('f6618d65-92b5-4d6c-8b0f-70d810171014', u'beta'), ('f772dd74-b4b9-4b87-9c77-8da860171014', u'beta')]
2 [('0026f8ac-0e43-4e22-bffa-f588c0171012', u'beta'), ('a69710ca-6097-4211-99ed-047b60171012', u'beta')]
2 [('542ccb03-0e1c-41c1-9a12-e0f570171015', u'beta'), ('c27447bb-d128-4d07-b4c5-93baf0171015', u'beta')]
2 [('7737e575-b274-4cc5-b106-ce2550171011', u'beta'), ('3586a868-df10-4f12-aaec-5654b0171011', u'beta')]
2 [('2dfbc46d-07e0-46b2-8188-33df40171011', u'beta'), ('7207c553-767b-4ca7-86be-5cb850171011', u'beta')]
2 [('c087f4c6-9b8d-4d41-b4ec-56ad40171011', u'beta'), ('2128deb7-f086-49ea-8eda-951b50171011', u'beta')]
2 [('7ed831b6-0b03-442d-81e8-e813f0171013', u'nightly'), ('ae250b43-bceb-44d7-b7c2-0bb170171013', u'nightly')]
2 [('ab111c40-e59d-4ee7-af7f-9407a0171009', u'beta'), ('1ae65570-13f8-426a-bb34-32bd70171009', u'beta')]
2 [('47db551d-a48f-435e-abae-06dd30171014', u'nightly'), ('7bc42e60-34ef-4975-93ff-ba1190171014', u'nightly')]
2 [('63d89e46-d268-4983-948d-6537e0171012', u'nightly'), ('e81d5466-0b81-4d82-a24a-65b5f0171012', u'nightly')]
2 [('061ba9f7-05a2-4123-8fc3-828130171014', u'nightly'), ('49019a88-6ad8-4fb5-bbd5-e47a80171014', u'nightly')]
2 [('4f354a1b-d19a-4677-8faf-f93120171011', u'beta'), ('0c56c2c4-dbc3-4d30-9d3f-59bcd0171011', u'beta')]
2 [('1d82c3f1-48ed-4cb6-8d46-9abbd0171014', u'beta'), ('2df3b1b5-73ba-4821-8f5e-176aa0171014', u'beta')]
2 [('ce65a35e-ef68-422c-a99a-4062c0171009', u'beta'), ('f4235d6f-8570-4180-b556-89de30171009', u'beta')]
2 [('f60e0451-b44f-4181-a417-f173e0171014', u'beta'), ('5020e9e1-1f88-4272-878e-7e30e0171014', u'beta')]
2 [('fa03cb1b-73f0-4d5c-bb2c-8133c0171013', u'nightly'), ('73781260-b427-4faa-b9f2-f57700171013', u'nightly')]
2 [('fa7d1f58-e13f-4dd2-aa9d-70a700171014', u'beta'), ('6c2f69db-089d-42e0-aedb-3046e0171014', u'beta')]
2 [('349ebe0f-0383-4348-9073-de09f0171014', u'nightly'), ('a70f98e8-56f0-4f7e-b2ab-0a77b0171014', u'nightly')]
2 [('ed480247-43a3-4df9-b217-b8f760171014', u'beta'), ('38454b80-4be1-4005-a6e9-0c4cd0171014', u'beta')]
2 [('b1866440-00db-4fca-aa6b-9ead40171012', u'beta'), ('fe34c059-7fdd-4fd4-9607-564f30171013', u'beta')]
2 [('30d8273e-8473-473b-bc48-ecdf40171012', u'beta'), ('a23d94e1-e55a-4a4a-96e8-264a00171012', u'beta')]
2 [('cb7e2cb7-a0bf-4c75-ae4e-c87110171015', u'beta'), ('3dbdf027-08d5-49c6-9b28-0de3a0171015', u'beta')]
2 [('697fb14e-3d15-45c0-b1ee-8c33d0171011', u'beta'), ('a64a2dce-8edd-40fc-a6f3-de40c0171011', u'beta')]
2 [('afcbb97d-9ae1-4d9b-b2eb-5dcac0171011', u'beta'), ('01d0b698-47e3-44ac-8dd9-2b0c60171011', u'beta')]
3 [('5f3b1568-c375-4d2c-85b2-5e2bd0171010', u'beta'), ('54195d5d-4a97-4891-a310-618fc0171010', u'beta'), ('f4fd8627-0287-4576-8e1a-857650171010', u'beta')]
2 [('3ba1fac3-80ec-40e5-90fb-79dc00171015', u'beta'), ('7f729e65-6fb5-4c48-b43b-b03f50171015', u'beta')]
2 [('97a3efa8-8f67-41b3-8a85-6f6580171012', u'nightly'), ('2df240be-8d3e-4d1f-b28a-56afb0171012', u'nightly')]
2 [('afdebbdf-abab-4cfd-9ae6-b038b0171015', u'beta'), ('5ac6a187-676b-4696-90e4-2d0d50171015', u'beta')]
2 [('ddffc0d3-95ef-457b-842b-69f2c0171014', u'beta'), ('2f402fbe-1003-405e-8d6c-717fe0171014', u'beta')]
2 [('12b13f13-d78f-4df5-838f-db1260171015', u'nightly'), ('899cdfd3-f419-4feb-ab5d-9a2c30171015', u'nightly')]
2 [('d632faf2-fa75-4fd7-babb-7f0090171011', u'beta'), ('78684e89-f0d1-4e00-aa96-e0e2c0171011', u'beta')]
2 [('d0732be4-1c05-4357-b489-fb5290171009', u'beta'), ('fd51fd31-ac7a-4098-99c1-35f7d0171009', u'beta')]
2 [('022d556d-a305-4030-9140-534be0171011', u'beta'), ('083cd316-66fb-49bb-a48b-f4afd0171011', u'beta')]
2 [('50f93983-2739-4543-984f-776ec0171010', u'nightly'), ('369e6941-43fa-4272-9437-e70af0171012', u'nightly')]
2 [('a676ccd4-81fb-40b1-8260-b8bb70171009', u'beta'), ('386f2e59-3eae-4115-acbd-54ace0171009', u'beta')]
2 [('13ca4c95-2737-4c3a-b80d-43fb70171009', u'nightly'), ('5703ecd4-31cd-4811-9314-f49d90171009', u'nightly')]
2 [('8370d0da-6d46-46de-a17b-6658f0171014', u'beta'), ('e28707b8-521a-4d5f-b96a-e66710171014', u'beta')]
5 [('d2762282-75af-4116-960a-b0d330171012', u'beta'), ('0aa5d256-01f6-485f-a577-e56d40171012', u'beta'), ('c8298d9d-a1a8-4ba4-b141-b90030171012', u'beta'), ('b46cd307-41f1-4c11-b258-531290171012', u'beta'), ('a9eff7c1-b190-4c3d-b1c6-c2ec00171012', u'beta')]
2 [('49c5faf0-6372-478f-81b0-8c86b0171012', u'nightly'), ('2e73dd60-aa72-4cf1-90e6-42d500171012', u'nightly')]
3 [('11875ec5-e812-4e0b-a548-7a4cd0171010', u'beta'), ('c17f55c7-0f02-415b-89b2-132940171010', u'beta'), ('efc5745b-ed86-42e9-87f0-d10c80171010', u'beta')]
2 [('efa87f8f-a315-47e0-9f2d-0f5ac0171013', u'beta'), ('0c1ed69c-8179-444f-9025-056c40171013', u'beta')]
2 [('9f010563-640b-4d6d-862c-5b2150171011', u'beta'), ('e8c22096-0024-4805-acde-e95850171011', u'beta')]
2 [('9ee518e9-306d-4370-8d18-74bd00171013', u'beta'), ('1878ee40-c6b6-44dd-a581-3aa500171013', u'beta')]
3 [('8db2dda5-18ad-4371-bb46-25f270171010', u'beta'), ('1dbe418e-ad3d-40cf-93c1-79c720171010', u'beta'), ('3c1ea7e4-3114-4466-b7f7-175d70171010', u'beta')]
2 [('7920e573-1677-4e57-ae00-0da8d0171009', u'beta'), ('da4aec1e-eaf4-4dc4-b490-71eaa0171009', u'beta')]
3 [('cc344028-542e-4626-baa5-5d7ff0171010', u'beta'), ('e3bbd31e-f31b-48bb-bb4d-af9f50171010', u'beta'), ('af095061-3d69-4a39-9bd6-518a50171010', u'beta')]
2 [('2305146e-cd8f-46a3-97e4-c89290171014', u'nightly'), ('20583fcb-ccbd-4dee-8b3a-6b3570171014', u'nightly')]
2 [('e9bd0a09-0092-469a-83ab-a9ac00171015', u'beta'), ('f710ef1b-8866-474c-889c-847560171015', u'beta')]
3 [('87a2a035-f131-4174-b074-5be6f0171010', u'beta'), ('19bdb9e6-71aa-405d-9c9a-3cc5f0171010', u'beta'), ('fa1c1368-3fe3-499c-b875-536590171010', u'beta')]
3 [('309c84ed-4a51-4efd-8d6a-0700b0171014', u'nightly'), ('55fd2f60-8974-4ffe-9948-26a2f0171014', u'nightly'), ('6d02fddc-4f12-4b80-82a8-4c6f40171014', u'nightly')]
2 [('ba6e3bd9-7a5e-4d71-8cb5-a3a6f0171014', u'beta'), ('92d5cf5b-cc4e-4bcb-bd87-6674a0171014', u'beta')]
2 [('7977390f-4c01-4f3b-8857-f54250171013', u'beta'), ('1bd7b502-1b44-4dc6-af56-180e90171013', u'beta')]
2 [('d7691221-2ee0-44e8-b0e3-b873c0171014', u'beta'), ('63d8063a-cfc0-4a7b-905d-885bf0171014', u'beta')]
3 [('ee208e6c-2453-41d9-b7b7-b1ec00171009', u'nightly'), ('953b5185-715a-4ab6-a967-6eb150171009', u'nightly'), ('552308d7-7407-4893-b823-7028d0171009', u'nightly')]
2 [('d2fd9aab-0c0a-4e70-9d2c-489350171014', u'beta'), ('882f7820-13aa-4e92-8700-c2d0d0171014', u'beta')]
2 [('61676270-8bdf-431b-9f6f-02e430171010', u'beta'), ('9b92f606-e1ba-430e-b8db-642b10171010', u'beta')]
2 [('85cbe801-3276-469e-92b4-e42f20171011', u'nightly'), ('9ebcc551-8001-4c4f-a685-bacd80171011', u'nightly')]
2 [('f95356ad-09d0-4cf0-ba75-d28740171009', u'beta'), ('f7dd60da-2f1c-4e58-b9b3-b50030171009', u'beta')]
2 [('f664f3f7-160f-4fe2-80c6-5fb1d0171011', u'nightly'), ('0f233095-e032-4684-a4e0-3eb130171011', u'nightly')]
2 [('997d0e34-a9a5-4038-9bd1-a08090171009', u'beta'), ('7f1a4286-bd3b-41ac-9b07-c3a110171009', u'beta')]
2 [('afae16af-e196-4bda-85b1-f86980171011', u'beta'), ('f1c3480c-5f08-4797-82b4-8b8bb0171011', u'beta')]
2 [('450a9c43-91c0-4d14-abf9-98a6a0171009', u'beta'), ('1b52f3b9-3fc2-4a9d-a32d-a3c440171009', u'beta')]
2 [('9c584072-e712-4835-9d79-27e840171009', u'beta'), ('d507e27b-d028-44c6-b73c-1fef50171009', u'beta')]
2 [('dd141fbe-2147-4b5f-96a2-7a9520171011', u'beta'), ('72acd667-66ba-46f7-96bd-fefc20171011', u'beta')]
2 [('9c5d9bcd-708c-40e3-afae-608630171011', u'beta'), ('9d3d557f-84c6-4375-97c7-c2f860171011', u'beta')]
2 [('e9515e75-6cf8-419d-955c-e467d0171014', u'nightly'), ('551c0441-f59c-4a29-a1d0-376490171014', u'nightly')]
2 [('b0682e72-32b7-4182-b537-0d7e80171013', u'nightly'), ('991db844-587e-4fef-b0f4-5f9520171013', u'nightly')]
2 [('4770e9ae-5ca0-42c0-9af9-cc70d0171010', u'beta'), ('696ac3b8-da72-4a15-936f-332240171010', u'beta')]
2 [('9e90f2e6-b63b-4665-82d4-1258f0171012', u'nightly'), ('7f7d17f9-19f5-40af-8c34-93ff00171012', u'nightly')]
2 [('9994f1fb-d9d4-49e3-9494-28f780171013', u'beta'), ('54cf9580-9bf1-4ed0-a255-5e8510171013', u'beta')]
2 [('27817b41-6e72-40c4-a03d-517570171011', u'nightly'), ('abe2e434-4ec0-413d-92fb-4339d0171011', u'nightly')]
2 [('995585d6-0502-426f-9831-586b60171010', u'nightly'), ('3429a45c-277c-41a2-80a0-9f4310171010', u'nightly')]
2 [('93614482-5b84-4084-93bf-705d40171014', u'beta'), ('cb9141cc-4f24-4b21-96b9-128e90171014', u'beta')]
2 [('0145d2b2-7421-4719-bda3-a1dae0171014', u'beta'), ('14d974b5-2fe1-4a03-a928-139230171014', u'beta')]
2 [('f838789a-be8b-4794-8cd7-6d9020171009', u'beta'), ('7b8752f0-0f65-4377-8816-edd8e0171009', u'beta')]
2 [('c72bf735-a6ab-4af2-bdfc-b2ef40171011', u'beta'), ('05f272d0-2760-4a23-b503-bf31f0171011', u'beta')]
2 [('21ab596e-c14f-44a9-bfa7-fc0660171009', u'nightly'), ('83a703c2-deb6-45dc-8f93-737960171009', u'nightly')]
2 [('6f564e6e-fda8-4dd2-8434-fe7a20171015', u'beta'), ('0ba2eda4-841b-4f29-8d75-8b7890171015', u'beta')]
2 [('9d0889e3-9503-4b3e-acdd-29e9e0171013', u'beta'), ('f13e1f15-074e-4fd6-ade3-39f100171013', u'beta')]
2 [('0c0e3c42-dfe9-41ef-8524-070d00171011', u'beta'), ('a6edccc3-193f-415d-afa6-fafac0171011', u'beta')]
2 [('c253ad85-bbc2-4ad9-8e34-61e220171010', u'beta'), ('d3d7a20e-006b-4890-8bb4-11d9f0171010', u'beta')]
2 [('f6ebc0a8-a3b8-4dbe-ab7b-45c420171012', u'nightly'), ('abad3272-a710-46ae-b8e1-7b2220171012', u'nightly')]
2 [('5df6d0cc-5a47-47b6-844d-01a5e0171012', u'nightly'), ('8d0172fc-21af-4c61-8a54-e53eb0171012', u'nightly')]
2 [('7db29624-c145-427e-bc73-5aced0171010', u'nightly'), ('e5522fa4-027e-4907-9ce4-957870171010', u'nightly')]
2 [('82b0f161-8dc9-4d8e-8a9a-78c120171015', u'beta'), ('f39b051a-5d5a-4a77-a5be-436ba0171015', u'beta')]
2 [('f5826d8f-78b5-4c8e-95dd-4a47e0171010', u'nightly'), ('ee90fcba-d765-46a2-a2f6-c548b0171010', u'nightly')]
2 [('736f917d-f649-490d-9862-bd2520171011', u'beta'), ('dff64b5b-77f0-4131-bf0c-ff8980171011', u'beta')]
2 [('82d76678-9f66-4ece-8e55-1b3790171011', u'beta'), ('7e06c5ea-6164-4ff1-9f28-4ac490171011', u'beta')]
2 [('389c7216-7f6e-4385-a721-b28510171009', u'beta'), ('38dfa7cb-0a5a-4433-bdf7-fae210171009', u'beta')]
3 [('aa5c9928-3d6e-45bb-93c8-25bce0171010', u'beta'), ('7a619673-e434-48a1-9a73-120550171010', u'beta'), ('e8e92f72-edae-4028-8e58-156580171010', u'beta')]
2 [('8663a92d-1261-40da-9875-a8ccf0171015', u'beta'), ('b6df075a-1f1e-412f-ab54-a78a60171015', u'beta')]
2 [('f5fe052c-1638-4c14-8668-205a20171009', u'beta'), ('0ade4103-690b-43ad-b113-c22140171009', u'beta')]
2 [('3da05909-a74e-4824-bbdd-311cf0171014', u'nightly'), ('a7a04578-e3ae-436d-b323-658d70171014', u'nightly')]
2 [('26519106-1341-49bf-b780-118e20171011', u'nightly'), ('c037822c-e047-4671-8d96-9a5b80171011', u'nightly')]
3 [('5d219481-6582-4b71-9b37-f21720171015', u'nightly'), ('4ad55619-5fdb-453a-8cb6-9af7e0171015', u'nightly'), ('8dcf254d-6dc4-4da3-8c3f-c05470171015', u'nightly')]
3 [('a3dea81b-0da3-4d41-af88-0cc040171011', u'beta'), ('b48d7882-6e13-44a0-95d5-516870171012', u'beta'), ('2509a5ae-80e1-4457-9359-95b150171014', u'beta')]
3 [('2851a2aa-5678-490d-ad58-b38f80171013', u'beta'), ('0f5b090b-f0b7-447f-b36c-09ef90171013', u'beta'), ('d820c981-9085-4a75-8baf-e30e60171013', u'beta')]
2 [('442d2884-292e-4e1c-88ba-8f8b40171010', u'beta'), ('69d6c7d7-a7d5-4127-89bf-f3da60171010', u'beta')]
2 [('7c3d49f2-97ad-4d28-abf5-90a7d0171015', u'nightly'), ('fa0ea73f-3c58-4145-bc75-99f770171015', u'nightly')]
3 [('b49e8f29-886c-43f3-9a8a-ee9a10171009', u'beta'), ('1fe35eab-416d-49f9-9925-0577f0171009', u'beta'), ('836758ce-3f9e-4ce7-9a9c-1a69d0171009', u'beta')]
2 [('713d08cc-7f66-4923-ba7d-d0fcf0171013', u'beta'), ('e5b55c9c-056c-4e7d-8b50-53c570171013', u'beta')]
2 [('74127832-16ad-40a5-bf43-54ba60171012', u'nightly'), ('0c7490a6-767c-4f80-9a5e-7605c0171012', u'nightly')]
2 [('a7dd5e57-d970-47ed-849f-0dac80171014', u'beta'), ('041dab7e-5148-4929-a60a-c5c310171014', u'beta')]
2 [('fbb182df-a1ba-4f8a-aec5-938e80171011', u'beta'), ('4b512832-2263-48d0-8068-040840171011', u'beta')]
2 [('c8059474-eed7-4f73-ad7a-b7c2b0171009', u'nightly'), ('a5f04b1a-e7dc-4809-a39c-2ecc80171009', u'nightly')]
2 [('66eeb478-9715-4425-a7ae-f5ac90171010', u'beta'), ('0e31f44a-d054-41ea-99ed-6023e0171010', u'beta')]


Out of 14,000 crashes, we had 161 duplicates.
> Out of 14,000 crashes, we had 161 duplicates.

How many were nightly/beta?

if there were a thousand or two, that might match what I believe I've seen, though I'd guess much higher.  How likely is it we're seeing dups that *don't* have the same dump_checksum?
Regarding the breakdown of nightly vs. beta, eyeballing the above data, it looks roughly like half-and-half.

I don't see how we can have one crash report be a duplicate of another if they have two different upload_file_minidumps. Maybe Mike Conley or Ted can talk to that? Are there other situations where Firefox could send two almost-identical crash reports for the same crash or something along those lines?
(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #36)
> Are there other situations where Firefox could send two
> almost-identical crash reports for the same crash or something along those
> lines?

Nothing springs to mind.
Attachment #8919021 - Attachment is obsolete: true
Flags: needinfo?(mconley)
Assignee: nobody → mconley
Comment on attachment 8921629 [details]
Bug 1407631 - Wait about 10 minutes after browser session start before scanning for unsubmitted crash reports.

https://reviewboard.mozilla.org/r/192620/#review197832

Looks good to me, nice and straightforward.
Attachment #8921629 - Flags: review?(gsvelto) → review+
Pushed by mconley@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/188ff8b2f7f6
Wait about 10 minutes after browser session start before scanning for unsubmitted crash reports. r=gsvelto
https://hg.mozilla.org/mozilla-central/rev/188ff8b2f7f6
Status: NEW → RESOLVED
Closed: 6 years ago
status-firefox58: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
Is this something that can ride the 58 train or should we consider nominating it for Beta approval? My main concern is that crash rates are certainly something we'll be watching closely when 57 goes to release, so ensuring that we have accurate data seems important.
status-firefox56: --- → wontfix
status-firefox57: --- → affected
status-firefox-esr52: --- → wontfix
tracking-firefox57: --- → ?
Flags: needinfo?(mconley)
(In reply to Ryan VanderMeulen [:RyanVM] from comment #42)
> Is this something that can ride the 58 train or should we consider
> nominating it for Beta approval? My main concern is that crash rates are
> certainly something we'll be watching closely when 57 goes to release, so
> ensuring that we have accurate data seems important.

We've been shipping things like this for a while. I personally think we can let this ride the 58 train, though I'm open to argument if others (particularly, the people who monitor crash rates) disagree.
Flags: needinfo?(mconley)
Ted mentioned in Comment 7 that this doesn't affect release - so maybe in that case we don't need to worry about this in 57?
(In reply to Marcia Knous [:marcia - use ni] from comment #44)
> Ted mentioned in Comment 7 that this doesn't affect release - so maybe in
> that case we don't need to worry about this in 57?

This is right. I believe the concern here was about our ability to accurately track our crash rate during the 57 beta cycle. Since we're coming to the end of that, letting this ride 58 seems to be the reasonable thing to do here.
Based on the last few comments and the fact we are almost at RC, won't fixing this for 57 and adding tracking -.
status-firefox57: affected → wontfix
tracking-firefox57: ? → -
You need to log in before you can comment on or make changes to this bug.