Closed Bug 526992 Opened 15 years ago Closed 14 years ago

Topcrasher for Firefox 3.5.5 [@ WriteFile][@ _PR_MD_WRITE][@ FileWrite]

Categories

(Firefox :: General, defect)

3.5 Branch
x86
Windows 7
defect
Not set
critical

Tracking

()

RESOLVED INCOMPLETE

People

(Reporter: whimboo, Unassigned)

Details

(Keywords: crash, Whiteboard: [crashkill][crashkill-outreach])

Crash Data

We have a new topcrasher for Firefox 3.5.5 on Windows 7 which is already #11 in the list. So far I am able to see this crash listed for 3.5 and 3.0.15 too.

http://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox%3A3.5.5&query_search=signature&query_type=exact&query=&date=&range_value=1&range_unit=weeks&do_query=1&signature=WriteFile

Stack:

0  	KERNELBASE.dll  	WriteFile  	
1 	kernel32.dll 	WriteFileImplementation 	
2 	nspr4.dll 	_PR_MD_WRITE 	nsprpub/pr/src/md/windows/w95io.c:528
3 	nspr4.dll 	FileWrite 	nsprpub/pr/src/io/prfile.c:109
4 	xul.dll 	nsSafeFileOutputStream::Write 	netwerk/base/src/nsFileStreams.cpp:582
5 	xul.dll 	ConvertAndWrite 	content/base/src/nsDocumentEncoder.cpp:452
6 	xul.dll 	nsDocumentEncoder::FlushText 	content/base/src/nsDocumentEncoder.cpp:508
7 	xul.dll 	nsDocumentEncoder::EncodeToStream 	content/base/src/nsDocumentEncoder.cpp:1014
8 	xul.dll 	nsDOMSerializer::SerializeToStream 	content/base/src/nsDOMSerializer.cpp:163
cc'ing jst and damon in case the problem is earlier in the stack when network and content code is hit.
110 total crashes for WriteFile on 20091103-crashdata.csv
67 start up crashes inside 3 minutes

os breakdown
  86 WriteFile Windows NT 6.1.7600
  24 WriteFile Windows NT 6.1.7100

distribution of all versions where the WriteFile crash was found on 20091103-crashdata.csv
  86 Firefox 3.5.4
   8 Firefox 3.5.3
   5 Firefox 3.6b1
   5 Firefox 3.5
   3 Firefox 3.0.14
   2 Firefox 3.0.15
   1 Firefox 3.0
so this one looks like a crasher that rises (and maybe falls) with each new release.

looking back at about 7 days after the release of 3.5.3 we see the about the same for that release

distribution of all versions where the WriteFile crash was found on 20091016-crashdata.csv
  73 Firefox 3.5.3
   3 Firefox 3.0.14
   2 Firefox 3.5.2
   1 Firefox 3.5.4
   1 Firefox 3.5
   1 Firefox 3.0.3

this does look like its much worse on 3.5.x than 3.0.x though
there are strange crash volume changes around this that maybe someone can explain  by understanding when we are doing update pushes, and when users are likley to be accepting updates, and when they might be reverting to older releases.   the 3.5.4 data looks like what might be expected, but the 3.5.3 data looks a bit strange.

   1 20091016-crashdata.csv:WriteFile 3.5.4
   4 20091017-crashdata.csv:WriteFile 3.5.4
   1 20091023-crashdata.csv:WriteFile 3.5.4
   1 20091025-crashdata.csv:WriteFile 3.5.4
   5 20091027-crashdata.csv:WriteFile 3.5.4
  62 20091028-crashdata.csv:WriteFile 3.5.4  -- 3.5.4 Released
  76 20091029-crashdata.csv:WriteFile 3.5.4
  85 20091030-crashdata.csv:WriteFile 3.5.4
  89 20091031-crashdata.csv:WriteFile 3.5.4
 105 20091101-crashdata.csv:WriteFile 3.5.4
  58 20091102-crashdata.csv:WriteFile 3.5.4
  86 20091103-crashdata.csv:WriteFile 3.5.4
  82 20091104-crashdata.csv:WriteFile 3.5.4
  55 20091105-crashdata.csv:WriteFile 3.5.4

3.5.3 released  9/9

  14 20090912-crashdata.csv:WriteFile 3.5.3
  26 20090913-crashdata.csv:WriteFile 3.5.3
  24 20090914-crashdata.csv:WriteFile 3.5.3
  13 20090915-crashdata.csv:WriteFile 3.5.3
   9 20090916-crashdata.csv:WriteFile 3.5.3
  23 20090917-crashdata.csv:WriteFile 3.5.3
  43 20090918-crashdata.csv:WriteFile 3.5.3 why do do we see a ramp starting here?
  69 20090919-crashdata.csv:WriteFile 3.5.3
  60 20090920-crashdata.csv:WriteFile 3.5.3
  58 20090921-crashdata.csv:WriteFile 3.5.3
  88 20090922-crashdata.csv:WriteFile 3.5.3
  87 20090923-crashdata.csv:WriteFile 3.5.3
 101 20090924-crashdata.csv:WriteFile 3.5.3
  63 20090925-crashdata.csv:WriteFile 3.5.3 why do we see a drop here?
  73 20090926-crashdata.csv:WriteFile 3.5.3
  96 20090927-crashdata.csv:WriteFile 3.5.3
  85 20090928-crashdata.csv:WriteFile 3.5.3
 103 20090929-crashdata.csv:WriteFile 3.5.3
  32 20091004-crashdata.csv:WriteFile 3.5.3 and another drop here
  18 20091005-crashdata.csv:WriteFile 3.5.3
  24 20091006-crashdata.csv:WriteFile 3.5.3
  13 20091007-crashdata.csv:WriteFile 3.5.3
  12 20091008-crashdata.csv:WriteFile 3.5.3
  12 20091009-crashdata.csv:WriteFile 3.5.3
  12 20091010-crashdata.csv:WriteFile 3.5.3
  14 20091011-crashdata.csv:WriteFile 3.5.3
  16 20091012-crashdata.csv:WriteFile 3.5.3
   5 20091013-crashdata.csv:WriteFile 3.5.3
  12 20091014-crashdata.csv:WriteFile 3.5.3
  39 20091015-crashdata.csv:WriteFile 3.5.3
  73 20091016-crashdata.csv:WriteFile 3.5.3 another uptick here
  57 20091017-crashdata.csv:WriteFile 3.5.3
  76 20091018-crashdata.csv:WriteFile 3.5.3
  67 20091019-crashdata.csv:WriteFile 3.5.3
  77 20091020-crashdata.csv:WriteFile 3.5.3
  61 20091021-crashdata.csv:WriteFile 3.5.3
  93 20091022-crashdata.csv:WriteFile 3.5.3
  95 20091023-crashdata.csv:WriteFile 3.5.3
  87 20091024-crashdata.csv:WriteFile 3.5.3
 108 20091025-crashdata.csv:WriteFile 3.5.3
 146 20091026-crashdata.csv:WriteFile 3.5.3  
                      why do we see this spike on 3.5.3 around release of 3.5.4?
 124 20091027-crashdata.csv:WriteFile 3.5.3
  72 20091028-crashdata.csv:WriteFile 3.5.3
  38 20091029-crashdata.csv:WriteFile 3.5.3
  24 20091030-crashdata.csv:WriteFile 3.5.3
  18 20091031-crashdata.csv:WriteFile 3.5.3
  16 20091101-crashdata.csv:WriteFile 3.5.3
  12 20091102-crashdata.csv:WriteFile 3.5.3
   8 20091103-crashdata.csv:WriteFile 3.5.3
   9 20091104-crashdata.csv:WriteFile 3.5.3
  12 20091105-crashdata.csv:WriteFile 3.5.3
Assignee: wtc → nobody
I looked through about a dozen of the stacks in the link in comment 0.
They all have a few things in common.  NSPR is common to about half of them.
They all crash in WriteFile.  The exception address is always on a page 
boundary, but I can't tell if it's a code page or a data page.  

Some stacks involve NSPR. e.g. 
http://crash-stats.mozilla.com/report/index/15f3a308-8956-4363-9dc1-297152091105
http://crash-stats.mozilla.com/report/index/92efeeff-f42e-4d81-9b0a-00d0e2091106

A greater number of stacks involve a DLL named avgtbapi.dll
http://crash-stats.mozilla.com/report/index/c4dc15a7-8581-4920-838b-9c4972091105
http://crash-stats.mozilla.com/report/index/d19e4c8d-28be-414b-9d80-797f72091106

At least one stack involves sqlite3
http://crash-stats.mozilla.com/report/index/4e8d4564-8141-4e7c-980f-c92292091106

I'd guess these are all writes with absurd lengths that attempt to write 
past the end of existing pages.  That would be the fault of the caller of 
NSPR, not of NSPR.
http://people.mozilla.com/crash_analysis/20091106/20091106_Firefox_3.5.4-interesting-addons.txt.bz2

says trend mirco is correlated 100% of the time.

WriteFile|EXCEPTION_ACCESS_VIOLATION (56 crashes)

  msvcr80.dll@0xf880|EXCEPTION_ACCESS_VIOLATION (55 crashes)
    100% (55/55) vs.   1% (1079/88271) {22181a4d-af90-4ca3-a569-faed9118d6bc} (Trend Micro Toolbar, http://us.trendmicro.com/us/products/personal/internet-security-pro/index.html)
     16% (9/55) vs.   4% (3762/88271) {77b819fa-95ad-4f2c-ac7c-486b356188a9} (IE Tab, https://addons.mozilla.org/addon/1419)
     35% (19/55) vs.  24% (20948/88271) {CAFEEFAC-0016-0000-0015-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/)
     15% (8/55) vs.   5% (4642/88271) {E2883E8F-472F-4fb0-9522-AC9BF37916A7} (Adobe Flash Extension, https://addons.mozilla.org/addon/13845)
      9% (5/55) vs.   1% (501/88271) {6e84150a-d526-41f1-a480-a67d3fed910d} (IE View, https://addons.mozilla.org/addon/35)
     11% (6/55) vs.   3% (2649/88271) {DDC359D1-844A-42a7-9AA1-88A850A938A8} (DownThemAll!, https://addons.mozilla.org/addon/201)
      7% (4/55) vs.   0% (300/88271) smartwebprinting@hp.com
      7% (4/55) vs.   0% (405/88271) {53A03D43-5363-4669-8190-99061B2DEBA5} (ScrapBook, https://addons.mozilla.org/addon/427)
     16% (9/55) vs.  10% (8465/88271) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865)
     11% (6/55) vs.   5% (4513/88271) {CAFEEFAC-0016-0000-0012-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/)
      7% (4/55) vs.   2% (1375/88271) {EF522540-89F5-46b9-B6FE-1829E2B572C6} (GooglePreview, https://addons.mozilla.org/addon/189)
      9% (5/55) vs.   3% (2996/88271) {D4DD63FA-01E4-46a7-B6B1-EDAB7D6AD389} (Download Statusbar, https://addons.mozilla.org/addon/26)
      9% (5/55) vs.   3% (3007/88271) {e4a8a97b-f2ed-450b-b12d-ee082ba24781} (Greasemonkey, https://addons.mozilla.org/addon/748)
      7% (4/55) vs.   2% (1545/88271) {73a6fe31-595d-460b-a920-fcc0f8843232} (NoScript, https://addons.mozilla.org/addon/722)
      5% (3/55) vs.   0% (29/88271) {7C7F5C11-4ACD-4CDB-9293-2E3F46654E2A} (TableTools, https://addons.mozilla.org/addon/2637)
      5% (3/55) vs.   0% (37/88271) chaika@chaika.xrea.jp
      5% (3/55) vs.   0% (115/88271) {8b5bea8c-6194-4c7c-a440-d5ca181480c3}
      5% (3/55) vs.   0% (182/88271) {5e594888-3e8e-47da-b2c6-b0b545112f84} (Save Image in Folder, https://addons.mozilla.org/addon/614)
     20% (11/55) vs.  15% (13061/88271) {CAFEEFAC-0016-0000-0016-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/)
      5% (3/55) vs.   0% (350/88271) {1280606b-2510-4fe0-97ef-9b5a22eafe41} (Fission, https://addons.mozilla.org/addon/1951)
Whiteboard: [crashkill][crashkill-outreach]
> says trend mirco is correlated 100% of the time.
Does that make this a Tech Evangelism bug?
(In reply to comment #7)
> says trend mirco is correlated 100% of the time.
> 
> WriteFile|EXCEPTION_ACCESS_VIOLATION (56 crashes)
> 
> msvcr80.dll@0xf880|EXCEPTION_ACCESS_VIOLATION (55 crashes)
>     100% (55/55) vs.   1% (1079/88271) {22181a4d-af90-4ca3-a569-faed9118d6bc}
> (Trend Micro Toolbar,
> http://us.trendmicro.com/us/products/personal/internet-security-pro/index.html)

Chris, aren't this too completely different stacks? I don't see how they match up. As what I can see we don't have any stack info for WriteFile.
Group: core-security
your right.  I miss read that file.  here is better data I think

search for WriteFile| in

http://people.mozilla.com/crash_analysis/20091106/20091106_Firefox_3.6b1-interesting-modules.txt  

In that set of data here is a protector

    60% (6/10) vs.   3% (226/6862) avgrsstx.dll

and these are attackers.

   30% (3/10) vs.   0% (3/6862) E3D7D230.x86.dll
   20% (2/10) vs.   0% (2/6862) 44807ED0.x86.dll
   20% (2/10) vs.   0% (2/6862) 58B68859.x86.dll
   10% (1/10) vs.   0% (1/6862) E5B89D38.x86.dll
   10% (1/10) vs.   0% (1/6862) 5B45838C.x86.dll
   10% (1/10) vs.   0% (1/6862) 62C14E60.x86.dll
(In reply to comment #10)

> http://people.mozilla.com/crash_analysis/20091106/20091106_Firefox_3.6b1-interesting-modules.txt 
> 
> In that set of data here is a protector
> 
>     60% (6/10) vs.   3% (226/6862) avgrsstx.dll

There should be another bug of that particular issue with AVG. I think we should handle it separately.

>    30% (3/10) vs.   0% (3/6862) E3D7D230.x86.dll
>    20% (2/10) vs.   0% (2/6862) 44807ED0.x86.dll
>    20% (2/10) vs.   0% (2/6862) 58B68859.x86.dll
>    10% (1/10) vs.   0% (1/6862) E5B89D38.x86.dll
>    10% (1/10) vs.   0% (1/6862) 5B45838C.x86.dll
>    10% (1/10) vs.   0% (1/6862) 62C14E60.x86.dll

Looks like that those systems are infected by Trojan-Spy.Win32.Agent.azpj. That one is creating DLL files with randomly created names like xxxxxxxx.x86.dll.

If that would be the case those modules don't have to appear in the stack of the crashing thread?
crashes can be caused by module hooks returning badly or modules badly hooking core code.

if you look at how we intend to fix this problem generically, you'll see that we're stomping on (in memory) system code. nothing prevents a third party library (once loaded) from doing the same. and evil code is more likely to do such things (and more critically: to get it wrong).
I believe we've established that this is not a bug in NSPR.  
So to what other product/component should this be moved?
Component: NSPR → General
Product: NSPR → Firefox
QA Contact: nspr → general
Whiteboard: [crashkill][crashkill-outreach] → [crashkill][crashkill-outreach] [sg:investigate]
Version: other → 3.5 Branch
ozten is interested in the metrics side of this bug.  adding to cc
Summary: Topcrasher for Firefox 3.5.5 [@WriteFile | _PR_MD_WRITE | FileWrite] → Topcrasher for Firefox 3.5.5 [@WriteFile][@ _PR_MD_WRITE][@ FileWrite]
Summary: Topcrasher for Firefox 3.5.5 [@WriteFile][@ _PR_MD_WRITE][@ FileWrite] → Topcrasher for Firefox 3.5.5 [@ WriteFile][@ _PR_MD_WRITE][@ FileWrite]
Not a topcrash anymore.
Keywords: topcrash
Should this be INCO & public?  Or is there something actionable here?
Whiteboard: [crashkill][crashkill-outreach] [sg:investigate] → [crashkill][crashkill-outreach]
(In reply to comment #16)
> Should this be INCO & public? 
yeah.   we should block the random x86.dl's but I think that might be another bug.
Group: core-security
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → INCOMPLETE
Crash Signature: [@ WriteFile] [@ _PR_MD_WRITE] [@ FileWrite]
You need to log in before you can comment on or make changes to this bug.