Closed Bug 526992 Opened 15 years ago Closed 15 years ago

Topcrasher for Firefox 3.5.5 [@ WriteFile][@ _PR_MD_WRITE][@ FileWrite]

Categories

(Firefox :: General, defect)

Product:
Firefox
Component:
General
Version:
3.5 Branch
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED INCOMPLETE

People

(Reporter: whimboo, Unassigned)

Details

(Keywords: crash, Whiteboard: [crashkill][crashkill-outreach])

Crash Data

We have a new topcrasher for Firefox 3.5.5 on Windows 7 which is already #11 in the list. So far I am able to see this crash listed for 3.5 and 3.0.15 too. http://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox%3A3.5.5&query_search=signature&query_type=exact&query=&date=&range_value=1&range_unit=weeks&do_query=1&signature=WriteFile Stack: 0 KERNELBASE.dll WriteFile 1 kernel32.dll WriteFileImplementation 2 nspr4.dll _PR_MD_WRITE nsprpub/pr/src/md/windows/w95io.c:528 3 nspr4.dll FileWrite nsprpub/pr/src/io/prfile.c:109 4 xul.dll nsSafeFileOutputStream::Write netwerk/base/src/nsFileStreams.cpp:582 5 xul.dll ConvertAndWrite content/base/src/nsDocumentEncoder.cpp:452 6 xul.dll nsDocumentEncoder::FlushText content/base/src/nsDocumentEncoder.cpp:508 7 xul.dll nsDocumentEncoder::EncodeToStream content/base/src/nsDocumentEncoder.cpp:1014 8 xul.dll nsDOMSerializer::SerializeToStream content/base/src/nsDOMSerializer.cpp:163
cc'ing jst and damon in case the problem is earlier in the stack when network and content code is hit.
110 total crashes for WriteFile on 20091103-crashdata.csv 67 start up crashes inside 3 minutes os breakdown 86 WriteFile Windows NT 6.1.7600 24 WriteFile Windows NT 6.1.7100 distribution of all versions where the WriteFile crash was found on 20091103-crashdata.csv 86 Firefox 3.5.4 8 Firefox 3.5.3 5 Firefox 3.6b1 5 Firefox 3.5 3 Firefox 3.0.14 2 Firefox 3.0.15 1 Firefox 3.0
so this one looks like a crasher that rises (and maybe falls) with each new release. looking back at about 7 days after the release of 3.5.3 we see the about the same for that release distribution of all versions where the WriteFile crash was found on 20091016-crashdata.csv 73 Firefox 3.5.3 3 Firefox 3.0.14 2 Firefox 3.5.2 1 Firefox 3.5.4 1 Firefox 3.5 1 Firefox 3.0.3 this does look like its much worse on 3.5.x than 3.0.x though
there are strange crash volume changes around this that maybe someone can explain by understanding when we are doing update pushes, and when users are likley to be accepting updates, and when they might be reverting to older releases. the 3.5.4 data looks like what might be expected, but the 3.5.3 data looks a bit strange. 1 20091016-crashdata.csv:WriteFile 3.5.4 4 20091017-crashdata.csv:WriteFile 3.5.4 1 20091023-crashdata.csv:WriteFile 3.5.4 1 20091025-crashdata.csv:WriteFile 3.5.4 5 20091027-crashdata.csv:WriteFile 3.5.4 62 20091028-crashdata.csv:WriteFile 3.5.4 -- 3.5.4 Released 76 20091029-crashdata.csv:WriteFile 3.5.4 85 20091030-crashdata.csv:WriteFile 3.5.4 89 20091031-crashdata.csv:WriteFile 3.5.4 105 20091101-crashdata.csv:WriteFile 3.5.4 58 20091102-crashdata.csv:WriteFile 3.5.4 86 20091103-crashdata.csv:WriteFile 3.5.4 82 20091104-crashdata.csv:WriteFile 3.5.4 55 20091105-crashdata.csv:WriteFile 3.5.4 3.5.3 released 9/9 14 20090912-crashdata.csv:WriteFile 3.5.3 26 20090913-crashdata.csv:WriteFile 3.5.3 24 20090914-crashdata.csv:WriteFile 3.5.3 13 20090915-crashdata.csv:WriteFile 3.5.3 9 20090916-crashdata.csv:WriteFile 3.5.3 23 20090917-crashdata.csv:WriteFile 3.5.3 43 20090918-crashdata.csv:WriteFile 3.5.3 why do do we see a ramp starting here? 69 20090919-crashdata.csv:WriteFile 3.5.3 60 20090920-crashdata.csv:WriteFile 3.5.3 58 20090921-crashdata.csv:WriteFile 3.5.3 88 20090922-crashdata.csv:WriteFile 3.5.3 87 20090923-crashdata.csv:WriteFile 3.5.3 101 20090924-crashdata.csv:WriteFile 3.5.3 63 20090925-crashdata.csv:WriteFile 3.5.3 why do we see a drop here? 73 20090926-crashdata.csv:WriteFile 3.5.3 96 20090927-crashdata.csv:WriteFile 3.5.3 85 20090928-crashdata.csv:WriteFile 3.5.3 103 20090929-crashdata.csv:WriteFile 3.5.3 32 20091004-crashdata.csv:WriteFile 3.5.3 and another drop here 18 20091005-crashdata.csv:WriteFile 3.5.3 24 20091006-crashdata.csv:WriteFile 3.5.3 13 20091007-crashdata.csv:WriteFile 3.5.3 12 20091008-crashdata.csv:WriteFile 3.5.3 12 20091009-crashdata.csv:WriteFile 3.5.3 12 20091010-crashdata.csv:WriteFile 3.5.3 14 20091011-crashdata.csv:WriteFile 3.5.3 16 20091012-crashdata.csv:WriteFile 3.5.3 5 20091013-crashdata.csv:WriteFile 3.5.3 12 20091014-crashdata.csv:WriteFile 3.5.3 39 20091015-crashdata.csv:WriteFile 3.5.3 73 20091016-crashdata.csv:WriteFile 3.5.3 another uptick here 57 20091017-crashdata.csv:WriteFile 3.5.3 76 20091018-crashdata.csv:WriteFile 3.5.3 67 20091019-crashdata.csv:WriteFile 3.5.3 77 20091020-crashdata.csv:WriteFile 3.5.3 61 20091021-crashdata.csv:WriteFile 3.5.3 93 20091022-crashdata.csv:WriteFile 3.5.3 95 20091023-crashdata.csv:WriteFile 3.5.3 87 20091024-crashdata.csv:WriteFile 3.5.3 108 20091025-crashdata.csv:WriteFile 3.5.3 146 20091026-crashdata.csv:WriteFile 3.5.3 why do we see this spike on 3.5.3 around release of 3.5.4? 124 20091027-crashdata.csv:WriteFile 3.5.3 72 20091028-crashdata.csv:WriteFile 3.5.3 38 20091029-crashdata.csv:WriteFile 3.5.3 24 20091030-crashdata.csv:WriteFile 3.5.3 18 20091031-crashdata.csv:WriteFile 3.5.3 16 20091101-crashdata.csv:WriteFile 3.5.3 12 20091102-crashdata.csv:WriteFile 3.5.3 8 20091103-crashdata.csv:WriteFile 3.5.3 9 20091104-crashdata.csv:WriteFile 3.5.3 12 20091105-crashdata.csv:WriteFile 3.5.3
Assignee: wtc → nobody
I looked through about a dozen of the stacks in the link in comment 0. They all have a few things in common. NSPR is common to about half of them. They all crash in WriteFile. The exception address is always on a page boundary, but I can't tell if it's a code page or a data page. Some stacks involve NSPR. e.g. http://crash-stats.mozilla.com/report/index/15f3a308-8956-4363-9dc1-297152091105 http://crash-stats.mozilla.com/report/index/92efeeff-f42e-4d81-9b0a-00d0e2091106 A greater number of stacks involve a DLL named avgtbapi.dll http://crash-stats.mozilla.com/report/index/c4dc15a7-8581-4920-838b-9c4972091105 http://crash-stats.mozilla.com/report/index/d19e4c8d-28be-414b-9d80-797f72091106 At least one stack involves sqlite3 http://crash-stats.mozilla.com/report/index/4e8d4564-8141-4e7c-980f-c92292091106 I'd guess these are all writes with absurd lengths that attempt to write past the end of existing pages. That would be the fault of the caller of NSPR, not of NSPR.
http://people.mozilla.com/crash_analysis/20091106/20091106_Firefox_3.5.4-interesting-addons.txt.bz2 says trend mirco is correlated 100% of the time. WriteFile|EXCEPTION_ACCESS_VIOLATION (56 crashes) msvcr80.dll@0xf880|EXCEPTION_ACCESS_VIOLATION (55 crashes) 100% (55/55) vs. 1% (1079/88271) {22181a4d-af90-4ca3-a569-faed9118d6bc} (Trend Micro Toolbar, http://us.trendmicro.com/us/products/personal/internet-security-pro/index.html) 16% (9/55) vs. 4% (3762/88271) {77b819fa-95ad-4f2c-ac7c-486b356188a9} (IE Tab, https://addons.mozilla.org/addon/1419) 35% (19/55) vs. 24% (20948/88271) {CAFEEFAC-0016-0000-0015-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/) 15% (8/55) vs. 5% (4642/88271) {E2883E8F-472F-4fb0-9522-AC9BF37916A7} (Adobe Flash Extension, https://addons.mozilla.org/addon/13845) 9% (5/55) vs. 1% (501/88271) {6e84150a-d526-41f1-a480-a67d3fed910d} (IE View, https://addons.mozilla.org/addon/35) 11% (6/55) vs. 3% (2649/88271) {DDC359D1-844A-42a7-9AA1-88A850A938A8} (DownThemAll!, https://addons.mozilla.org/addon/201) 7% (4/55) vs. 0% (300/88271) smartwebprinting@hp.com 7% (4/55) vs. 0% (405/88271) {53A03D43-5363-4669-8190-99061B2DEBA5} (ScrapBook, https://addons.mozilla.org/addon/427) 16% (9/55) vs. 10% (8465/88271) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865) 11% (6/55) vs. 5% (4513/88271) {CAFEEFAC-0016-0000-0012-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/) 7% (4/55) vs. 2% (1375/88271) {EF522540-89F5-46b9-B6FE-1829E2B572C6} (GooglePreview, https://addons.mozilla.org/addon/189) 9% (5/55) vs. 3% (2996/88271) {D4DD63FA-01E4-46a7-B6B1-EDAB7D6AD389} (Download Statusbar, https://addons.mozilla.org/addon/26) 9% (5/55) vs. 3% (3007/88271) {e4a8a97b-f2ed-450b-b12d-ee082ba24781} (Greasemonkey, https://addons.mozilla.org/addon/748) 7% (4/55) vs. 2% (1545/88271) {73a6fe31-595d-460b-a920-fcc0f8843232} (NoScript, https://addons.mozilla.org/addon/722) 5% (3/55) vs. 0% (29/88271) {7C7F5C11-4ACD-4CDB-9293-2E3F46654E2A} (TableTools, https://addons.mozilla.org/addon/2637) 5% (3/55) vs. 0% (37/88271) chaika@chaika.xrea.jp 5% (3/55) vs. 0% (115/88271) {8b5bea8c-6194-4c7c-a440-d5ca181480c3} 5% (3/55) vs. 0% (182/88271) {5e594888-3e8e-47da-b2c6-b0b545112f84} (Save Image in Folder, https://addons.mozilla.org/addon/614) 20% (11/55) vs. 15% (13061/88271) {CAFEEFAC-0016-0000-0016-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/) 5% (3/55) vs. 0% (350/88271) {1280606b-2510-4fe0-97ef-9b5a22eafe41} (Fission, https://addons.mozilla.org/addon/1951)
Whiteboard: [crashkill][crashkill-outreach]
> says trend mirco is correlated 100% of the time. Does that make this a Tech Evangelism bug?
(In reply to comment #7) > says trend mirco is correlated 100% of the time. > > WriteFile|EXCEPTION_ACCESS_VIOLATION (56 crashes) > > msvcr80.dll@0xf880|EXCEPTION_ACCESS_VIOLATION (55 crashes) > 100% (55/55) vs. 1% (1079/88271) {22181a4d-af90-4ca3-a569-faed9118d6bc} > (Trend Micro Toolbar, > http://us.trendmicro.com/us/products/personal/internet-security-pro/index.html) Chris, aren't this too completely different stacks? I don't see how they match up. As what I can see we don't have any stack info for WriteFile.
Group: core-security
your right. I miss read that file. here is better data I think search for WriteFile| in http://people.mozilla.com/crash_analysis/20091106/20091106_Firefox_3.6b1-interesting-modules.txt In that set of data here is a protector 60% (6/10) vs. 3% (226/6862) avgrsstx.dll and these are attackers. 30% (3/10) vs. 0% (3/6862) E3D7D230.x86.dll 20% (2/10) vs. 0% (2/6862) 44807ED0.x86.dll 20% (2/10) vs. 0% (2/6862) 58B68859.x86.dll 10% (1/10) vs. 0% (1/6862) E5B89D38.x86.dll 10% (1/10) vs. 0% (1/6862) 5B45838C.x86.dll 10% (1/10) vs. 0% (1/6862) 62C14E60.x86.dll
(In reply to comment #10) > http://people.mozilla.com/crash_analysis/20091106/20091106_Firefox_3.6b1-interesting-modules.txt > > In that set of data here is a protector > > 60% (6/10) vs. 3% (226/6862) avgrsstx.dll There should be another bug of that particular issue with AVG. I think we should handle it separately. > 30% (3/10) vs. 0% (3/6862) E3D7D230.x86.dll > 20% (2/10) vs. 0% (2/6862) 44807ED0.x86.dll > 20% (2/10) vs. 0% (2/6862) 58B68859.x86.dll > 10% (1/10) vs. 0% (1/6862) E5B89D38.x86.dll > 10% (1/10) vs. 0% (1/6862) 5B45838C.x86.dll > 10% (1/10) vs. 0% (1/6862) 62C14E60.x86.dll Looks like that those systems are infected by Trojan-Spy.Win32.Agent.azpj. That one is creating DLL files with randomly created names like xxxxxxxx.x86.dll. If that would be the case those modules don't have to appear in the stack of the crashing thread?
crashes can be caused by module hooks returning badly or modules badly hooking core code. if you look at how we intend to fix this problem generically, you'll see that we're stomping on (in memory) system code. nothing prevents a third party library (once loaded) from doing the same. and evil code is more likely to do such things (and more critically: to get it wrong).
I believe we've established that this is not a bug in NSPR. So to what other product/component should this be moved?
Component: NSPR → General
Product: NSPR → Firefox
QA Contact: nspr → general
Whiteboard: [crashkill][crashkill-outreach] → [crashkill][crashkill-outreach] [sg:investigate]
Version: other → 3.5 Branch
ozten is interested in the metrics side of this bug. adding to cc
Summary: Topcrasher for Firefox 3.5.5 [@WriteFile | _PR_MD_WRITE | FileWrite] → Topcrasher for Firefox 3.5.5 [@WriteFile][@ _PR_MD_WRITE][@ FileWrite]
Summary: Topcrasher for Firefox 3.5.5 [@WriteFile][@ _PR_MD_WRITE][@ FileWrite] → Topcrasher for Firefox 3.5.5 [@ WriteFile][@ _PR_MD_WRITE][@ FileWrite]
Not a topcrash anymore.
Keywords: topcrash
Should this be INCO & public? Or is there something actionable here?
Whiteboard: [crashkill][crashkill-outreach] [sg:investigate] → [crashkill][crashkill-outreach]
(In reply to comment #16) > Should this be INCO & public? yeah. we should block the random x86.dl's but I think that might be another bug.
Group: core-security
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → INCOMPLETE
Crash Signature: [@ WriteFile] [@ _PR_MD_WRITE] [@ FileWrite]
You need to log in before you can comment on or make changes to this bug.