Topcrasher for Firefox 3.5.5 [@ WriteFile][@ _PR_MD_WRITE][@ FileWrite]

RESOLVED INCOMPLETE

Status

()

--
critical
RESOLVED INCOMPLETE
9 years ago
8 years ago

People

(Reporter: whimboo, Unassigned)

Tracking

({crash})

3.5 Branch
x86
Windows 7
crash
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [crashkill][crashkill-outreach], crash signature)

(Reporter)

Description

9 years ago
We have a new topcrasher for Firefox 3.5.5 on Windows 7 which is already #11 in the list. So far I am able to see this crash listed for 3.5 and 3.0.15 too.

http://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox%3A3.5.5&query_search=signature&query_type=exact&query=&date=&range_value=1&range_unit=weeks&do_query=1&signature=WriteFile

Stack:

0  	KERNELBASE.dll  	WriteFile  	
1 	kernel32.dll 	WriteFileImplementation 	
2 	nspr4.dll 	_PR_MD_WRITE 	nsprpub/pr/src/md/windows/w95io.c:528
3 	nspr4.dll 	FileWrite 	nsprpub/pr/src/io/prfile.c:109
4 	xul.dll 	nsSafeFileOutputStream::Write 	netwerk/base/src/nsFileStreams.cpp:582
5 	xul.dll 	ConvertAndWrite 	content/base/src/nsDocumentEncoder.cpp:452
6 	xul.dll 	nsDocumentEncoder::FlushText 	content/base/src/nsDocumentEncoder.cpp:508
7 	xul.dll 	nsDocumentEncoder::EncodeToStream 	content/base/src/nsDocumentEncoder.cpp:1014
8 	xul.dll 	nsDOMSerializer::SerializeToStream 	content/base/src/nsDOMSerializer.cpp:163

Comment 1

9 years ago
cc'ing jst and damon in case the problem is earlier in the stack when network and content code is hit.

Comment 2

9 years ago
110 total crashes for WriteFile on 20091103-crashdata.csv
67 start up crashes inside 3 minutes

os breakdown
  86 WriteFile Windows NT 6.1.7600
  24 WriteFile Windows NT 6.1.7100

distribution of all versions where the WriteFile crash was found on 20091103-crashdata.csv
  86 Firefox 3.5.4
   8 Firefox 3.5.3
   5 Firefox 3.6b1
   5 Firefox 3.5
   3 Firefox 3.0.14
   2 Firefox 3.0.15
   1 Firefox 3.0

Comment 3

9 years ago
so this one looks like a crasher that rises (and maybe falls) with each new release.

looking back at about 7 days after the release of 3.5.3 we see the about the same for that release

distribution of all versions where the WriteFile crash was found on 20091016-crashdata.csv
  73 Firefox 3.5.3
   3 Firefox 3.0.14
   2 Firefox 3.5.2
   1 Firefox 3.5.4
   1 Firefox 3.5
   1 Firefox 3.0.3

this does look like its much worse on 3.5.x than 3.0.x though

Comment 5

9 years ago
there are strange crash volume changes around this that maybe someone can explain  by understanding when we are doing update pushes, and when users are likley to be accepting updates, and when they might be reverting to older releases.   the 3.5.4 data looks like what might be expected, but the 3.5.3 data looks a bit strange.

   1 20091016-crashdata.csv:WriteFile 3.5.4
   4 20091017-crashdata.csv:WriteFile 3.5.4
   1 20091023-crashdata.csv:WriteFile 3.5.4
   1 20091025-crashdata.csv:WriteFile 3.5.4
   5 20091027-crashdata.csv:WriteFile 3.5.4
  62 20091028-crashdata.csv:WriteFile 3.5.4  -- 3.5.4 Released
  76 20091029-crashdata.csv:WriteFile 3.5.4
  85 20091030-crashdata.csv:WriteFile 3.5.4
  89 20091031-crashdata.csv:WriteFile 3.5.4
 105 20091101-crashdata.csv:WriteFile 3.5.4
  58 20091102-crashdata.csv:WriteFile 3.5.4
  86 20091103-crashdata.csv:WriteFile 3.5.4
  82 20091104-crashdata.csv:WriteFile 3.5.4
  55 20091105-crashdata.csv:WriteFile 3.5.4

3.5.3 released  9/9

  14 20090912-crashdata.csv:WriteFile 3.5.3
  26 20090913-crashdata.csv:WriteFile 3.5.3
  24 20090914-crashdata.csv:WriteFile 3.5.3
  13 20090915-crashdata.csv:WriteFile 3.5.3
   9 20090916-crashdata.csv:WriteFile 3.5.3
  23 20090917-crashdata.csv:WriteFile 3.5.3
  43 20090918-crashdata.csv:WriteFile 3.5.3 why do do we see a ramp starting here?
  69 20090919-crashdata.csv:WriteFile 3.5.3
  60 20090920-crashdata.csv:WriteFile 3.5.3
  58 20090921-crashdata.csv:WriteFile 3.5.3
  88 20090922-crashdata.csv:WriteFile 3.5.3
  87 20090923-crashdata.csv:WriteFile 3.5.3
 101 20090924-crashdata.csv:WriteFile 3.5.3
  63 20090925-crashdata.csv:WriteFile 3.5.3 why do we see a drop here?
  73 20090926-crashdata.csv:WriteFile 3.5.3
  96 20090927-crashdata.csv:WriteFile 3.5.3
  85 20090928-crashdata.csv:WriteFile 3.5.3
 103 20090929-crashdata.csv:WriteFile 3.5.3
  32 20091004-crashdata.csv:WriteFile 3.5.3 and another drop here
  18 20091005-crashdata.csv:WriteFile 3.5.3
  24 20091006-crashdata.csv:WriteFile 3.5.3
  13 20091007-crashdata.csv:WriteFile 3.5.3
  12 20091008-crashdata.csv:WriteFile 3.5.3
  12 20091009-crashdata.csv:WriteFile 3.5.3
  12 20091010-crashdata.csv:WriteFile 3.5.3
  14 20091011-crashdata.csv:WriteFile 3.5.3
  16 20091012-crashdata.csv:WriteFile 3.5.3
   5 20091013-crashdata.csv:WriteFile 3.5.3
  12 20091014-crashdata.csv:WriteFile 3.5.3
  39 20091015-crashdata.csv:WriteFile 3.5.3
  73 20091016-crashdata.csv:WriteFile 3.5.3 another uptick here
  57 20091017-crashdata.csv:WriteFile 3.5.3
  76 20091018-crashdata.csv:WriteFile 3.5.3
  67 20091019-crashdata.csv:WriteFile 3.5.3
  77 20091020-crashdata.csv:WriteFile 3.5.3
  61 20091021-crashdata.csv:WriteFile 3.5.3
  93 20091022-crashdata.csv:WriteFile 3.5.3
  95 20091023-crashdata.csv:WriteFile 3.5.3
  87 20091024-crashdata.csv:WriteFile 3.5.3
 108 20091025-crashdata.csv:WriteFile 3.5.3
 146 20091026-crashdata.csv:WriteFile 3.5.3  
                      why do we see this spike on 3.5.3 around release of 3.5.4?
 124 20091027-crashdata.csv:WriteFile 3.5.3
  72 20091028-crashdata.csv:WriteFile 3.5.3
  38 20091029-crashdata.csv:WriteFile 3.5.3
  24 20091030-crashdata.csv:WriteFile 3.5.3
  18 20091031-crashdata.csv:WriteFile 3.5.3
  16 20091101-crashdata.csv:WriteFile 3.5.3
  12 20091102-crashdata.csv:WriteFile 3.5.3
   8 20091103-crashdata.csv:WriteFile 3.5.3
   9 20091104-crashdata.csv:WriteFile 3.5.3
  12 20091105-crashdata.csv:WriteFile 3.5.3

Updated

9 years ago
Assignee: wtc → nobody
I looked through about a dozen of the stacks in the link in comment 0.
They all have a few things in common.  NSPR is common to about half of them.
They all crash in WriteFile.  The exception address is always on a page 
boundary, but I can't tell if it's a code page or a data page.  

Some stacks involve NSPR. e.g. 
http://crash-stats.mozilla.com/report/index/15f3a308-8956-4363-9dc1-297152091105
http://crash-stats.mozilla.com/report/index/92efeeff-f42e-4d81-9b0a-00d0e2091106

A greater number of stacks involve a DLL named avgtbapi.dll
http://crash-stats.mozilla.com/report/index/c4dc15a7-8581-4920-838b-9c4972091105
http://crash-stats.mozilla.com/report/index/d19e4c8d-28be-414b-9d80-797f72091106

At least one stack involves sqlite3
http://crash-stats.mozilla.com/report/index/4e8d4564-8141-4e7c-980f-c92292091106

I'd guess these are all writes with absurd lengths that attempt to write 
past the end of existing pages.  That would be the fault of the caller of 
NSPR, not of NSPR.

Comment 7

9 years ago
http://people.mozilla.com/crash_analysis/20091106/20091106_Firefox_3.5.4-interesting-addons.txt.bz2

says trend mirco is correlated 100% of the time.

WriteFile|EXCEPTION_ACCESS_VIOLATION (56 crashes)

  msvcr80.dll@0xf880|EXCEPTION_ACCESS_VIOLATION (55 crashes)
    100% (55/55) vs.   1% (1079/88271) {22181a4d-af90-4ca3-a569-faed9118d6bc} (Trend Micro Toolbar, http://us.trendmicro.com/us/products/personal/internet-security-pro/index.html)
     16% (9/55) vs.   4% (3762/88271) {77b819fa-95ad-4f2c-ac7c-486b356188a9} (IE Tab, https://addons.mozilla.org/addon/1419)
     35% (19/55) vs.  24% (20948/88271) {CAFEEFAC-0016-0000-0015-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/)
     15% (8/55) vs.   5% (4642/88271) {E2883E8F-472F-4fb0-9522-AC9BF37916A7} (Adobe Flash Extension, https://addons.mozilla.org/addon/13845)
      9% (5/55) vs.   1% (501/88271) {6e84150a-d526-41f1-a480-a67d3fed910d} (IE View, https://addons.mozilla.org/addon/35)
     11% (6/55) vs.   3% (2649/88271) {DDC359D1-844A-42a7-9AA1-88A850A938A8} (DownThemAll!, https://addons.mozilla.org/addon/201)
      7% (4/55) vs.   0% (300/88271) smartwebprinting@hp.com
      7% (4/55) vs.   0% (405/88271) {53A03D43-5363-4669-8190-99061B2DEBA5} (ScrapBook, https://addons.mozilla.org/addon/427)
     16% (9/55) vs.  10% (8465/88271) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865)
     11% (6/55) vs.   5% (4513/88271) {CAFEEFAC-0016-0000-0012-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/)
      7% (4/55) vs.   2% (1375/88271) {EF522540-89F5-46b9-B6FE-1829E2B572C6} (GooglePreview, https://addons.mozilla.org/addon/189)
      9% (5/55) vs.   3% (2996/88271) {D4DD63FA-01E4-46a7-B6B1-EDAB7D6AD389} (Download Statusbar, https://addons.mozilla.org/addon/26)
      9% (5/55) vs.   3% (3007/88271) {e4a8a97b-f2ed-450b-b12d-ee082ba24781} (Greasemonkey, https://addons.mozilla.org/addon/748)
      7% (4/55) vs.   2% (1545/88271) {73a6fe31-595d-460b-a920-fcc0f8843232} (NoScript, https://addons.mozilla.org/addon/722)
      5% (3/55) vs.   0% (29/88271) {7C7F5C11-4ACD-4CDB-9293-2E3F46654E2A} (TableTools, https://addons.mozilla.org/addon/2637)
      5% (3/55) vs.   0% (37/88271) chaika@chaika.xrea.jp
      5% (3/55) vs.   0% (115/88271) {8b5bea8c-6194-4c7c-a440-d5ca181480c3}
      5% (3/55) vs.   0% (182/88271) {5e594888-3e8e-47da-b2c6-b0b545112f84} (Save Image in Folder, https://addons.mozilla.org/addon/614)
     20% (11/55) vs.  15% (13061/88271) {CAFEEFAC-0016-0000-0016-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/)
      5% (3/55) vs.   0% (350/88271) {1280606b-2510-4fe0-97ef-9b5a22eafe41} (Fission, https://addons.mozilla.org/addon/1951)

Updated

9 years ago
Whiteboard: [crashkill][crashkill-outreach]
> says trend mirco is correlated 100% of the time.
Does that make this a Tech Evangelism bug?
(Reporter)

Comment 9

9 years ago
(In reply to comment #7)
> says trend mirco is correlated 100% of the time.
> 
> WriteFile|EXCEPTION_ACCESS_VIOLATION (56 crashes)
> 
> msvcr80.dll@0xf880|EXCEPTION_ACCESS_VIOLATION (55 crashes)
>     100% (55/55) vs.   1% (1079/88271) {22181a4d-af90-4ca3-a569-faed9118d6bc}
> (Trend Micro Toolbar,
> http://us.trendmicro.com/us/products/personal/internet-security-pro/index.html)

Chris, aren't this too completely different stacks? I don't see how they match up. As what I can see we don't have any stack info for WriteFile.
Group: core-security

Comment 10

9 years ago
your right.  I miss read that file.  here is better data I think

search for WriteFile| in

http://people.mozilla.com/crash_analysis/20091106/20091106_Firefox_3.6b1-interesting-modules.txt  

In that set of data here is a protector

    60% (6/10) vs.   3% (226/6862) avgrsstx.dll

and these are attackers.

   30% (3/10) vs.   0% (3/6862) E3D7D230.x86.dll
   20% (2/10) vs.   0% (2/6862) 44807ED0.x86.dll
   20% (2/10) vs.   0% (2/6862) 58B68859.x86.dll
   10% (1/10) vs.   0% (1/6862) E5B89D38.x86.dll
   10% (1/10) vs.   0% (1/6862) 5B45838C.x86.dll
   10% (1/10) vs.   0% (1/6862) 62C14E60.x86.dll
(Reporter)

Comment 11

9 years ago
(In reply to comment #10)

> http://people.mozilla.com/crash_analysis/20091106/20091106_Firefox_3.6b1-interesting-modules.txt 
> 
> In that set of data here is a protector
> 
>     60% (6/10) vs.   3% (226/6862) avgrsstx.dll

There should be another bug of that particular issue with AVG. I think we should handle it separately.

>    30% (3/10) vs.   0% (3/6862) E3D7D230.x86.dll
>    20% (2/10) vs.   0% (2/6862) 44807ED0.x86.dll
>    20% (2/10) vs.   0% (2/6862) 58B68859.x86.dll
>    10% (1/10) vs.   0% (1/6862) E5B89D38.x86.dll
>    10% (1/10) vs.   0% (1/6862) 5B45838C.x86.dll
>    10% (1/10) vs.   0% (1/6862) 62C14E60.x86.dll

Looks like that those systems are infected by Trojan-Spy.Win32.Agent.azpj. That one is creating DLL files with randomly created names like xxxxxxxx.x86.dll.

If that would be the case those modules don't have to appear in the stack of the crashing thread?

Comment 12

9 years ago
crashes can be caused by module hooks returning badly or modules badly hooking core code.

if you look at how we intend to fix this problem generically, you'll see that we're stomping on (in memory) system code. nothing prevents a third party library (once loaded) from doing the same. and evil code is more likely to do such things (and more critically: to get it wrong).
I believe we've established that this is not a bug in NSPR.  
So to what other product/component should this be moved?
Component: NSPR → General
Product: NSPR → Firefox
QA Contact: nspr → general
Whiteboard: [crashkill][crashkill-outreach] → [crashkill][crashkill-outreach] [sg:investigate]
Version: other → 3.5 Branch

Comment 14

9 years ago
ozten is interested in the metrics side of this bug.  adding to cc
(Reporter)

Updated

9 years ago
Summary: Topcrasher for Firefox 3.5.5 [@WriteFile | _PR_MD_WRITE | FileWrite] → Topcrasher for Firefox 3.5.5 [@WriteFile][@ _PR_MD_WRITE][@ FileWrite]

Updated

9 years ago
Summary: Topcrasher for Firefox 3.5.5 [@WriteFile][@ _PR_MD_WRITE][@ FileWrite] → Topcrasher for Firefox 3.5.5 [@ WriteFile][@ _PR_MD_WRITE][@ FileWrite]
(Reporter)

Comment 15

9 years ago
Not a topcrash anymore.
Keywords: topcrash

Comment 16

9 years ago
Should this be INCO & public?  Or is there something actionable here?
Whiteboard: [crashkill][crashkill-outreach] [sg:investigate] → [crashkill][crashkill-outreach]

Comment 17

9 years ago
(In reply to comment #16)
> Should this be INCO & public? 
yeah.   we should block the random x86.dl's but I think that might be another bug.
Group: core-security
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → INCOMPLETE
(Assignee)

Updated

8 years ago
Crash Signature: [@ WriteFile] [@ _PR_MD_WRITE] [@ FileWrite]
You need to log in before you can comment on or make changes to this bug.