Closed Bug 568849 Opened 16 years ago Closed 9 years ago

duplicated and missing hang reports in hand id pairs in 3.6.4 crash reports

Categories

(Socorro :: General, task)

Product:
Socorro
Component:
General
Platform:
x86
All
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: chofmann, Unassigned)

References

Details

Attachments

(2 files)

hangid and process listing from 2010 05 27
744.20 KB, text/plain
Details
hangs with only a single id
9.51 KB, text/plain
Details
I started generating some reports to sort out firefox and plugin crashes, and firefox/plugin hang reports in 3.6.4 you can see these at http://people.mozilla.com/~chofmann/crash-stats/20100527/crash-counts.txt here is the logic I use to do the counting for any report received during the day on firefox 3.6.4. $23 ~ /\N/ && $25 ~ /\N/ {fxcrash++} ; \ e.g. no hang id in the report and process_type is not plugin (it is firefox) $25 ~ /plugin/ && $23 ~ /\N/ {plugcrash++}; \ process_type is plugin and a hang_id does not exist for the report $25 ~ /\N/ && $23 !~ /\N/ {hang_fx_side++}; \ process_type is firefox and a hang id is in the report $25 ~ /plugin/ && $23 !~ /\N/ {hang_p_side++}; \ proess_type is plugin and a hangid is in the report This seems to generate some pretty reasonable and understable numbers but closer inspection of the hang data shows some unexplained patterns. If I print just the $23 and $25 fields from the csv files I get these counts 3525 \N plugin -- plugin crash 3085 \N \N -- firefox crash 19 516c4e50-14a9-4fdb-ab89-141f7f0df8fb plugin - 19 rpts with the same hangid 19 516c4e50-14a9-4fdb-ab89-141f7f0df8fb \N 7 77f37a51-871f-42b9-b39d-9c0398f01e74 plugin - 17 rpts with the same hangid 7 77f37a51-871f-42b9-b39d-9c0398f01e74 \N 6 b1a6ca48-3be2-4b12-b882-b4b72414ec37 plugin 6 b1a6ca48-3be2-4b12-b882-b4b72414ec37 \N 6 70890070-1cc2-4008-9ff9-8fdb702eac5b plugin 6 70890070-1cc2-4008-9ff9-8fdb702eac5b \N 6 6a6e61ca-903e-46ce-857e-07aabfd1108a plugin 6 6a6e61ca-903e-46ce-857e-07aabfd1108a \N 3 f220ed4b-7e19-42f3-8f60-7bd65a146600 plugin 3 f220ed4b-7e19-42f3-8f60-7bd65a146600 \N 3 d5de4cbc-3261-45ca-ac19-c01f6ce21d04 \N 3 626a03f5-f3bc-4cae-9b78-6535f4d8b9ad plugin 3 626a03f5-f3bc-4cae-9b78-6535f4d8b9ad \N then 106 reports with 2 each of the same hang id like 2 fca2cef5-3a3e-4f26-91ce-08634190b1d8 plugin 2 fca2cef5-3a3e-4f26-91ce-08634190b1d8 \N then 16,0089 reports like we might expect with single hangid's; one for the plugin and one for firefox. 1 ffefe646-8867-4d3e-a090-6c867f1c91f9 plugin 1 ffefe646-8867-4d3e-a090-6c867f1c91f9 \N 1 ffef41b7-bc45-4c35-9489-270352cc8c0a plugin 1 ffef41b7-bc45-4c35-9489-270352cc8c0a \N Anyone have any ideas on why we might be seeing what look like duplicate hang reports?
generated with this command on the crash report .csv file awk -F\t '$8 ~ /3.6.4/ {print $23,$25}' 20100527*| sort | uniq -c | sort -nr
since may 12 when the hangid and process info got added to the .csv files this is the daily breakdown of duplicate reports. Those 19 dups submitted by a single user looks a bit abnormal, although we did see 25 retransmissions on may 16th. looks like we are seeing about 78-128 dup pairs a day and maybe around 200 total dups. date high_dup_cnt 2+_dups 2_dups singles 20100513-crashdata.csv 8 16 85 11565 20100514-crashdata.csv 4 27 88 11244 20100515-crashdata.csv 7 13 78 10752 20100516-crashdata.csv 25 16 82 11317 20100517-crashdata.csv 6 9 69 10996 20100518-crashdata.csv 12 24 53 11440 20100519-crashdata.csv 17 35 87 13182 20100520-crashdata.csv 12 13 102 15399 20100521-crashdata.csv 14 22 110 16050 20100522-crashdata.csv 12 9 103 16541 20100523-crashdata.csv 4 22 105 15395 20100524-crashdata.csv 4 8 128 15522 20100525-crashdata.csv 9 10 120 14554 20100526-crashdata.csv 6 20 115 16269 20100527-crashdata.csv 19 15 105 16089
looks like we also see about 180 reports per day were only half of the two reports get submitted. this is from data midnight to midnight so at least some of these are explained by the other half of the hang report appearing in the previous or next day. both the duplicate and missing reports are probably within the acceptable bounds what we might expect, but it's worth understanding the volume of this activity when analyzing the data, and also keeping an eye out for increases or ways to reduce if we can.
It looks like when there is a missing single half of the hang report its more likely to be the plugin side of the report pair (column 9), than the firefox side (column 8). there is some fluctuation but dups also seem to happen at about the same low rate as missing reports. date 1. high_dup_cnt 2. 2+_dups 3. 2_dups 4. tl_single reports 5. fx_singles 6. plugin_singles 7. matching_singles 8. fx_unmatched_hang 9. plugin_unmatched_hang 1. 2. 3. 4. 5. 6. 7. 8. 9. 20100520 12 13 102 15399 7653 7746 7616 37 130 20100521 14 22 110 16050 7977 8073 7943 34 130 20100522 12 9 103 16541 8226 8315 8193 33 122 20100523 4 22 105 15395 7649 7746 7626 23 120 20100524 4 8 128 15522 7727 7795 7694 33 101 20100525 9 10 120 14554 7194 7360 7054 140 306 20100526 6 20 115 16269 8081 8188 7978 103 210 20100527 19 15 105 16089 7988 8101 7950 38 151
Summary: abnormal counts and non-matching of hang id pairs in 3.6.4 crash reports → duplicated and missing hang reports in hand id pairs in 3.6.4 crash reports
One possible reason for duplicate reports would be if Flash crashes while there are many instances of Flash open, and the user clicks "submit crash report" for each one individually. The UI code is supposed to prevent that, but there might be a bug somewhere. Alternatively they could be resubmissions from about:crashes if the network wasn't responding properly. Unmatched reports don't really worry me at all at this level: any network hiccups like we experienced this week could mean one of a pair wasn't submitted. It's also possible that Firefox crashes or quits between submitting the two reports, but that seems less likely to me.
not sure if this was the a cause or effect of the overloaded processor, backlog, and short outage in yesterday's events, but missing and dup reports spiked dramatically. hang report sanity checking 1. date _____Duplicates_______ 2. highest reported duplicate report count 3. count of 2 or more dups 4. count of 2 dups _____Raw Data______ 5. tl number of single reports 6. firefox side single reports 7. plugin side single reports _____The good stuff______ 8. matching hang pairs _____Missing______ 9. unmatched firfox hang 10.unmatched plugin hang date 2. 3. 4. 5. 6. 7. 8. 9. 10. Duplicates Info------ Raw Counts------- V Missing Rpts 20100620 8 31 201 26542 13202 13340 13133 69 207 20100621 17 32 210 26349 13124 13225 13020 104 205 20100622 2 52 394 78032 38896 39136 38745 151 391 20100623 34 166 1112 335936 167631 168305 152313 15318 15992
formating on that last got botched. easlier to read stuff at http://people.mozilla.com/~chofmann/crash-stats/20100623/crash-counts.txt in the hang report sanity checking section
currently, our hang id index only keeps the last collected uuid for each process type of a hang. This means that if we received more than one browser hang report with the same hang id, only the last uuid will be stored in the "ids:ooid:default" column. Same if we receive more than one plugin hang, only the last one will be stored in the "ids:ooid:plugin" column of the index. It sounds like we might want to open a bug to change this index to be able to record all ooids rather than just the last.
Should this be on the "duplicate crashes" tracker (bug 579136), given that this is also some form of duplicate reports?
bug 579136 deals mostly with firefox crash dups. lets keep this one separate since bsmedberg's scenario in comment 5 probably explains alot of the plugin dups and users just repeatedly hitting the "reload plugin" button. there is not much we can do about that. there are some things we need to do about firefox dups.
Component: Socorro → General
Product: Webtools → Socorro
these jobs are long since deleted
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: