579136 - Duplicate crashes in Socorro [tracker]

Reporter

Description

•

14 years ago

Seen whilst reviewing crash stats today.

I have seen several stacks which seem to not look right.

http://tinyurl.com/2d5ywqj is one example of the same crash which looks as if it was submitted 38 times. The URL is the same in each instance.

http://tinyurl.com/22rfuj3 is another example.

bc wonders if it is possible that either Soccoro is automatically submitting the same crash over and over or is processing the same report over and over. Either way I have not seen that much identical information being shown in my daily review of crash stats.

Laura Thomson :laura

Assignee

Comment 1

•

14 years ago

These are definitely dupes - the uptime is the same for each.

Did you see others?  

lars, xstevens: could this be related to our monitor problems of yesterday?

Bob Clary [:bc] (inactive)

Comment 2

•

14 years ago

url, buildid, crashdate, time since last crash, install age are also identical. correlations are at 100%. These are essentially two identical sets of crash reports. The libnspr4 signature dates back to 20100703. The mozjs signature dates to 20100714.

Marcia Knous [:marcia]

Reporter

Comment 3

•

14 years ago

http://tinyurl.com/27cb5l8 contains a set of crashes that all seem have to the same uptime as well.

Marcia Knous [:marcia]

Reporter

Comment 4

•

14 years ago

http://tinyurl.com/22rfuj3 is from the Mac set that I caught, I can go back to the Windows machine if you need others.

K Lars Lohn [:lars] [:klohn]

Comment 5

•

14 years ago

while each of these crashes is essentially identical, they have there own uuid
and submission time.  That means that the collector is actually seeing these
arrive individually from the client.  The collector reads the data from the http POST from the client and then adds the uuid and submission time to it.   

The monitor only sees uuids and never the data behind them.  It just assigns uuids to processors.  It is very unlikely that yesterday's monitor trouble is related to this problem.

Could breakpad be submitting the same crash repeatedly?

Bob Clary [:bc] (inactive)

Comment 6

•

14 years ago

assuming the duplicates only differ in uuid and date processed, counting the total lines in each crashdump file and then counting the unique lines after removing the uuid and the date processed can give a count of the actual submitted reports vs the unique reports for a given day. The difference between the two is the number of duplicate submissions for that day.

$ for f in *crashdata.csv.gz; do before=`gunzip -c  $f  | awk -F'\t' '{print $1, $2, $4, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22}'|wc -l`; after=`gunzip -c $f | awk -F'\t' '{print $1, $2, $4, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22}' | sort -u | wc -l`; echo "$f before $before, after $after"; done
20100626-crashdata.csv.gz before 487779, after 486501
20100627-crashdata.csv.gz before 419135, after 418172
20100628-crashdata.csv.gz before 344641, after 343559
20100629-crashdata.csv.gz before 301676, after 300666
20100630-crashdata.csv.gz before 287176, after 286346
20100701-crashdata.csv.gz before 270586, after 269851
20100702-crashdata.csv.gz before 258072, after 257209
20100703-crashdata.csv.gz before 230711, after 230097
20100704-crashdata.csv.gz before 235063, after 234426
20100705-crashdata.csv.gz before 263509, after 262917
20100706-crashdata.csv.gz before 268535, after 267941
20100707-crashdata.csv.gz before 328875, after 328025
20100708-crashdata.csv.gz before 336260, after 335340
20100709-crashdata.csv.gz before 272916, after 271960
20100710-crashdata.csv.gz before 246530, after 245750
20100711-crashdata.csv.gz before 255869, after 255241
20100712-crashdata.csv.gz before 285620, after 284851
20100713-crashdata.csv.gz before 283947, after 283092
20100714-crashdata.csv.gz before 272448, after 269337

Bob Clary [:bc] (inactive)

Comment 7

•

14 years ago

Attached file duplicates 20090801-20100701 (obsolete) — Details

chris hofmann

Comment 8

•

14 years ago

we also have documented dups of the hang reports.  see bug 568849

Daniel Einspanjer [:dre] [:deinspanjer]

Comment 9

•

14 years ago

We are going to run a map reduce tomorrow that will count the number of duplicates broken down by date and product+buildid.  It will also count how many of those duplicates were throttled vs processed.

K Lars Lohn [:lars] [:klohn]

Comment 10

•

14 years ago

with access to the latest database, I've been doing some queries, trolling for duplicates.  I'm using sql that will group unlikely combinations of columns over the data from this week.

select
  install_age,
  last_crash,
  uptime,
  process_type,
  product,
  version,
  url,
  count(*) as total
from reports_20100712
group by
  1,2,3,4,5,6,7
having
  count(*) > 3
order by
  3 desc, 1, 2, 4;

This query will give all the repeats where all those columns in the select list are the same.  I sort by uptime descending because I want to see crashes that aren't immediately on restart first.  This query found 360 duplicates across all products.

Then I reran the query on a week's data from a year ago.   It found only 136. The number of times for each repeat were generally lower than the first dataset. Interestingly, only one had a url associated with it, while most had urls from the current dataset.  

I'm not sure yet what this means, but I thought I should report it before I left for the day.  I'll continue my investigation tomorrow.

chris hofmann

Comment 11

•

14 years ago

we probably need to calibrate the dups against total reports received per day as well.  I have a bit of that data.

A year ago I think we were running 118k-180k processed reports per day, now we are running 240k-280k per day.

$ wc -l 2010071
  246530 20100710-crashdata.csv
  255869 20100711-crashdata.csv
  285620 20100712-crashdata.csv
  283947 20100713-crashdata.csv
  272448 20100714-crashdata.csv

$ wc -l 2009071*  
  186111 20090716-crashdata.csv
  181468 20090717-crashdata.csv
  155158 20090718-crashdata.csv
  118364 20090719-crashdata.csv

Wayne Mery (:wsmwk)

Comment 12

•

14 years ago

we might also be seeing this in some thunderbird crash sigs

Anurag Phadke[:aphadke@mozilla.com]

Comment 13

•

14 years ago

We will be running a map reduce job in production that should get us the duplicate counts and other details for all products that are submitted by Socorro.

Laura Thomson :laura

Assignee

Comment 14

•

14 years ago

I'm nitpicking here and changing the title.  (Socorro doesn't submit anything.)

From Bob's numbers it looks like this issue has existed for at least a year, too.

Summary: Socorro may be automatically submitting the same crash over and over → Duplicate crashes in Socorro

Anurag Phadke[:aphadke@mozilla.com]

Comment 15

•

14 years ago

For 7/14:
Browser BuildId InstallTime StartupTime CrashTime ProcessType legacy_0 legacy_1 total
Firefox 20100202165920 1266538977 1268937980 1268940564 null    16      1073    1089

The OOID 001060ae-c30f-4871-a1ef-c2a072100610 got duplicated 1089 times and only 16 of it got processed. 

Should we (metrics team) go ahead and generate statistics on ranges of dupes etc. ?

Laura Thomson :laura

Assignee

Comment 16

•

14 years ago

One data point doesn't tell me much (why am I telling the metrics team this? ;) )

More data would be good.

Daniel Einspanjer [:dre] [:deinspanjer]

Comment 17

•

14 years ago

Attached file crash report counts of duplicate ranges (obsolete) — Details

Work in progress, haven't fully audited the results, but they are basically tabulating the data aphadke already attached above.

Anurag Phadke[:aphadke@mozilla.com]

Comment 18

•

14 years ago

Attached file attached excel equivalent, easy to read (obsolete) — Details

Laura Thomson :laura

Assignee

Comment 19

•

14 years ago

Questions:
- What percentage of reports are duplicates?  (total, by date, by buildid)
- How are dupes different from non dupes?  Are they over-represented:
  - For hangs vs non hangs?
  - For a particular product?
  - For a particular build?

Assignee: nobody → aphadke

Wayne Mery (:wsmwk)

Comment 20

•

14 years ago

possible example from thunderbird https://crash-stats.mozilla.com/report/list?product=Thunderbird&build_id=&query_search=signature&query_type=exact&query=__delayLoadHelper2&date=07%2F20%2F2010%2012%3A15%3A09&range_value=4&range_unit=weeks&hang_type=any&process_type=all&plugin_field=&plugin_query_type=&plugin_query=&do_query=1&signature=__delayLoadHelper2&missing_sig=&page=1

Daniel Einspanjer [:dre] [:deinspanjer]

Comment 21

•

14 years ago

Attached image Trend lines for startupcrash vs not for a two week period of duplicate crashes — Details

This image shows that most of our duplicates are due to startup crashes.  Also, we had a huge spike in them on 2010-07-14.  We need to run the job on subsequent days to see if that is a new trend or just a single spike.

Attachment #457671 - Attachment is obsolete: true

Attachment #458478 - Attachment is obsolete: true

Attachment #458482 - Attachment is obsolete: true

Daniel Einspanjer [:dre] [:deinspanjer]

Comment 22

•

14 years ago

Attached image Duplicate crashes by top 5 Firefox build ids — Details

It looks like Firefox 3.6.6 Windows is generating significantly more duplicate startup crash submissions.

Laura Thomson :laura

Assignee

Comment 23

•

14 years ago

It looks like the startup crashes went back down from that second graph, too.

Anyone (chofmann? Tomcat? Marcia? LegNeato?) have any ideas why 3.6.6 on Windows might have given us all these duplicate startup crashes?  Seems like there might be an underlying Firefox bug.

Daniel Einspanjer [:dre] [:deinspanjer]

Comment 24

•

14 years ago

Please note that currently, the number shown in the Y axis is a sum of all the duplicate submissions.  That does mean that 3.6.6 will be higher than the other builds since it has a significantly higher user population so there are more total submissions that might possibly be duplicated.

I'm trying to think of how we might be able to change this number to be normalized based on the population so we can tell if it is generating more duplicates per capita than previous builds.

Daniel Einspanjer [:dre] [:deinspanjer]

Comment 25

•

14 years ago

Attached file Socorro Duplicate Crashes By BuildID — Details

PDF export of a cross tab that shows the following:

Avg. Duplicates, Duplicates, Fingerprints and Std. dev. of Duplicates broken down by BuildID vs. IsStartupCrash and SubmitDate. The data is filtered on IsUnique, which keeps 0. The view is filtered on SubmitDate and BuildID. The SubmitDate filter excludes 0. The BuildID filter has multiple members selected.

I still don't have an easy way to provide quartiles or export to a spreadsheet.  I'm going to CC the Portugal team to see if they can do that tomorrow morning.

Anurag Phadke[:aphadke@mozilla.com]

Comment 26

•

14 years ago

should we close this ticket or assign it to someone other than me (aphadke)?

Daniel Einspanjer [:dre] [:deinspanjer]

Comment 27

•

14 years ago

The full analysis view is now available on the metrics site:
https://metrics.mozilla.com/pentaho/Pivot?solution=metrics&path=&action=SocorroDuplicatesStudy.analysisview.xaction

You should be able to use that to explore any of the several dimensions included and either graph or export it to CSV.  If you have any trouble with it, please just let me know what you are trying to do and I'll be happy to help out.

Assignee: aphadke → laura

Severity: critical → normal

OS: Mac OS X → All

Hardware: x86 → All

Laura Thomson :laura

Assignee

Comment 28

•

14 years ago

Daniel, I can't view that page (get the error: This page cannot be accessed directly. It must be linked to from other pages)

chris hofmann

Comment 29

•

14 years ago

duplicate crashes are actually getting in the way of diagnosing and getting traction on the introduction of new crashes on trunk development, pre-beta builds, and experimental branches. Daily volume of top 10-20 crashes on these builds is often around the range of 20-40 crashes today, so its easy for a duplicate to slip into the top crash list, and when it does we waste time trying to sort out the "real frequency" of the crash and the priority it should receive.

We need a way for this to be eliminated. One possibility is surface some info that tells anyone analyzing the reports the number of possible duplicates in any given sample.

Daniel suggest watching time_since_last_crash and install_time as a possible way to quickly fingerprint duplicate reports. That might be a good start.
We might need to expand this definition to improve the fingerprinting.

We should also spin off another bug to get a review of the client code to better understand error handling and transactions conditions between client and server where these dups might be occurring.

For example it looks like ted suggest returning the http status to the client to help it understand the state of the transfer.
http://mxr.mozilla.org/mozilla1.9.2/source/toolkit/crashreporter/client/crashreporter_win.cpp#432
http://code.google.com/p/google-breakpad/issues/detail?id=220

We could also look at the possibility that these are all "manual resubmissions" happening under the conditions in this code http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/client/crashreporter.cpp#312

Maybe the client could also forward info to indicate which crashes are "automatically submitted" and which are "manual."

(not currently active) Ted Mielczarek

Comment 30

•

14 years ago

(In reply to comment #29)
> For example it looks like ted suggest returning the http status to the client
> to help it understand the state of the transfer.
> http://mxr.mozilla.org/mozilla1.9.2/source/toolkit/crashreporter/client/crashreporter_win.cpp#432
> http://code.google.com/p/google-breakpad/issues/detail?id=220

This could be a possible root cause, if the collector returned an error status, but in fact had accepted the report, and then the user manually resubmitted it.

> 
> We could also look at the possibility that these are all "manual resubmissions"
> happening under the conditions in this code
> http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/client/crashreporter.cpp#312

This code is currently unused. We never rolled out that kind of throttling.


> Maybe the client could also forward info to indicate which crashes are
> "automatically submitted" and which are "manual."

We already do this. Crashes submitted by manually clicking in about:crashes send "Throttleable=0".

chris hofmann

Comment 31

•

14 years ago

here is another area to look at for bugs in the client, or for fingerprinting on the server.

it looks like it might be a bit more complicated that just watching for matches on install_time and time_since_last_crash.

here is a sample of crashes for the signature gfxWindowsSurface::GetImageSurface() which is one of these signatures with a suspect early release cycle one-day-spikes. 

in these cases if you take install time minus time since last crash the possible duplicates begin to appear as a result of the client appearing to increment install time before resending.

in the set of reports below it looks like

#  install since itime        uptime
   time    last   - last_time
           crash

1. 6707   115   6592  103 
2. 7988    28   7960   10 
3. 8020    32   7988   20    -- 3 is a retransmission of 2, 32 seconds later
4. 8040    20   8020   12    -- 4 is a retransmission of 3, 20 seconds later
5. 8128    88   8040   28    -- 5 is a retransmission of 4, 88 seconds later
6. 8193    65   8128   59    -- 6 is a retransmission of 5
7. 8236    43   8193   36    -- 7 is a retransmission of 6
8. 8438   168   8270  134 
9. 8555   117   8438  112    -- 9 is a retransmission of 8

here is the set of reports used in this sample if we want to look deeper.

http://crash-stats.mozilla.com/report/index/4b7eaabd-2078-4da0-80b2-a5f8d2100921
http://crash-stats.mozilla.com/report/index/a974adf2-9661-4058-8259-e38cc2100921
http://crash-stats.mozilla.com/report/index/930553de-34fe-4325-891f-aeb562100921
http://crash-stats.mozilla.com/report/index/78b4cdc2-966e-44b1-a6a2-f39d32100921
http://crash-stats.mozilla.com/report/index/8e05cdd7-6634-46ce-b063-501292100921
http://crash-stats.mozilla.com/report/index/77f7d3d8-c06f-4436-924d-b7f1b2100921
http://crash-stats.mozilla.com/report/index/c488053c-1b4a-4532-bc05-7bc952100921
http://crash-stats.mozilla.com/report/index/867c8d99-0e14-4307-a57e-aa1802100921
http://crash-stats.mozilla.com/report/index/55a524a6-7552-4065-98ea-0932c2100921

chris hofmann

Comment 32

•

14 years ago

Attached file same data as above with processed time, and client crash time added — Details

this is also strange. processed time and client crash time doesn't seem to be advancing in synch with the incremental advance of install time and time since last crash.

one theory to explain this is that somehow the dups are queing on the client, with each report advancing time since install and time since last crash, then are submitted in fast succession and the server picks them up as unique reports.

another theory is that these really aren't dups, even though the majority of the reports share a lot in common when trying to fingerprint signature and various correlations and connects of time, os (Windows NT 5.1.2600 Service Pack 2), build id ( 20100921041551 ) , and crash address (0x116) as you can see in
http://crash-stats.mozilla.com/report/list?signature=gfxWindowsSurface::GetImageSurface%28%29

chris hofmann

Comment 33

•

14 years ago

Attached file more fingerprint info on this set of crashes — Details

beyond proximity in summission time, version, branch, os, crash address, e-mail status  addon compat check status, and flash version 8 of these also have the same URL

chris hofmann

Comment 34

•

14 years ago

that last list with urls and other fingerprinting info might support the theory that the original crash event happened around 201009211505 when the user hit that url a http://200.181.114.4, 

then a bunch of reports were dupped on the client, 

then 21 minutes later at 201009211526 the server began receiving and processing a flood of these duplicates over the next 4 minutes until 201009211530

duplicates 20090801-20100701 14 years ago Bob Clary [:bc] (inactive) 1.04 KB, text/plain		Details
crash report counts of duplicate ranges 14 years ago Daniel Einspanjer [:dre] [:deinspanjer] 37.95 KB, text/plain		Details
attached excel equivalent, easy to read 14 years ago Anurag Phadke[:aphadke@mozilla.com] 126.00 KB, application/vnd.ms-excel		Details
Trend lines for startupcrash vs not for a two week period of duplicate crashes 14 years ago Daniel Einspanjer [:dre] [:deinspanjer] 48.43 KB, image/png		Details
Duplicate crashes by top 5 Firefox build ids 14 years ago Daniel Einspanjer [:dre] [:deinspanjer] 71.62 KB, image/png		Details
Socorro Duplicate Crashes By BuildID 14 years ago Daniel Einspanjer [:dre] [:deinspanjer] 311.67 KB, application/octet-stream		Details
same data as above with processed time, and client crash time added 14 years ago chris hofmann 1.31 KB, text/plain		Details
more fingerprint info on this set of crashes 14 years ago chris hofmann 2.13 KB, text/plain		Details