Last Comment Bug 305073 - FIREFOX caused an invalid page fault [@ MSVCRT.DLL + 0xd16a]
: FIREFOX caused an invalid page fault [@ MSVCRT.DLL + 0xd16a]
Status: RESOLVED FIXED
build patch checked-in on 1.8 branch
: crash, fixed1.8.1.15
Product: Core
Classification: Components
Component: JavaScript Engine (show other bugs)
: 1.8 Branch
: x86 Windows 98
: -- critical (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2005-08-18 02:01 PDT by Lord Matt
Modified: 2011-06-13 10:01 PDT (History)
17 users (show)
dveditz: blocking1.8.1.5-
dveditz: wanted1.8.1.x+
bob: in‑testsuite-
bob: in‑litmus-
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Dr Watson trace of such a crash (46.34 KB, application/octet-stream)
2007-03-14 09:18 PDT, Kenneth Heal
no flags Details
i think this is the fix for the build failure in comment 34 (1.58 KB, patch)
2007-04-18 12:52 PDT, timeless
kheal: review+
sdwilsh: review+
dveditz: superreview+
dveditz: approval1.8.1.5-
dveditz: approval1.8.1.8+
Details | Diff | Splinter Review
correct 13 to 1300 (MOZILLA_1_8_BRANCH) (797 bytes, patch)
2008-03-31 00:45 PDT, timeless
dveditz: review+
dveditz: approval1.8.1.15+
Details | Diff | Splinter Review

Description Lord Matt 2005-08-18 02:01:37 PDT
User-Agent:       Mozilla/5.0 (Windows; U; Win98; en-GB; rv:1.7.10) Gecko/20050717 Firefox/1.0.6
Build Identifier: Mozilla/5.0 (Windows; U; Win98; en-GB; rv:1.7.10) Gecko/20050717 Firefox/1.0.6

Sometimes when at http://mad-den.net/overkill/ (when clicking a link from the
topsites) Firefox bails out.  Most times Firefox catches the crash and reports
it.  this time it did not but windows did:

FIREFOX caused an invalid page fault in
module MSVCRT.DLL at 017f:7800d16a.
Registers:
EAX=00f50008 CS=017f EIP=7800d16a EFLGS=00010202
EBX=00f50138 SS=0187 ESP=00e3dd50 EBP=00e3dd6c
ECX=00f539bc DS=0187 ESI=033974ac FS=5177
EDX=00000010 ES=0187 EDI=0000001c GS=0000
Bytes at CS:EIP:
8b 5c d1 04 8d 0c d1 89 5e 04 89 4e 08 89 71 04 
Stack dump:
00000000 00000202 033974b0 00f539bc 00000001 00000031 00000110 00e3ddcc 78011b77
00f50138 033975bc 040b0df0 033974b0 00000110 000001fa 0388ee5c


[I do not know if it's related but I had to disable the Google tool bar to get
FirFox to load correctly]

Reproducible: Sometimes

Steps to Reproduce:
1. go to site
2. start opening pages or use the NUKE ME to do it for you
3. sware and restart firefox

Actual Results:  
This seems to differ each time but I suspect it is simply disagreeing with
something on one of the pages that get's opened.

Expected Results:  
opened the page as normal.

MSVCRT.DLL
Comment 1 Dave Townsend [:mossop] 2005-08-18 02:04:42 PDT
Please submit a talkback report for the crash (http://kb.mozillazine.org/Talkback)
Comment 2 Lord Matt 2005-08-18 06:36:33 PDT
In an update to the crashing issue here is the dump from another crash.

FIREFOX caused an invalid page fault in
module FIREFOX.EXE at 017f:0067cd46.
Registers:
EAX=00000000 CS=017f EIP=0067cd46 EFLGS=00010202
EBX=00e3d820 SS=0187 ESP=00e3d6c0 EBP=00e3d6dc
ECX=00e3d728 DS=0187 ESI=00e3d820 FS=4f0f
EDX=00e3d728 ES=0187 EDI=04af8a20 GS=0000
Bytes at CS:EIP:
8b 78 1c eb 1d 8b 07 8b cf ff 90 cc 00 00 00 3b 
Stack dump:
04af8a20 00e3d820 04adb284 01e575f8 00000000 00587125 01e575f8 00e3d730 0067c82d
00000000 00e3d6f8 0000230a 40000000 00e3d820 00000000 00000000 
Comment 3 timeless 2005-08-18 08:39:31 PDT
matt: unless you're running an os/2 version of mozilla (which you aren't), we
can't use those stack dumps. we need a talkback incident id.
Comment 4 Lord Matt 2005-08-19 00:22:39 PDT
I can't provide any, sorry.  FF bailed and failed to produce any such report. 
Is there some way I can find a log file that might hold something?

From the look of the stack dump (and my patchy knowledge of window OS) Firefox
conflicted with the kernal and windows dropped the entire process tree.  I
understand that from NT this was not possible and also in *nix this was never
possible.  I am reliably informed that windows 98 allows processes to run within
the kernal but I have no idea what that means.
Comment 5 Kenneth Heal 2007-03-14 09:18:52 PDT
Created attachment 258552 [details]
Dr Watson trace of such a crash
Comment 6 Kenneth Heal 2007-03-14 09:19:18 PDT
I had a similar crash and unfortunately no talk-back ID was produced either.  I do, however, have a DrWatson trace (I had it running at the time.)  I shall attach this to the bug -- maybe that will help.
Comment 7 timeless 2007-03-14 18:36:23 PDT
please don't. we can't do anything with it. i'm not sure how many ways i can write this.

we just plain can't take those reports.

drwatson is useful if:
1. you have symbols when drwatson runs
2. the person to whom you send the report can match your stack against symbols

kenneth didn't even indicate what version of firefox he ran. but ignoring that, no one here has access to the symbols for firefox, not the person who runs drwatson and not the people who read bugzilla.

here are two inconsistent attempts at reading the stack. i really don't have time to try to integrate them.
js3250!js_SearchScope
js3250!JS_ResolveStub
...
 [v just finished a call to js3250!JS_DHashTableOperate]
...
user32!EndPaint
...
js3250!JS_GetReservedSlot
...
 [next call would be to js3250!JS_GetContextPrivate]
...
user32!PeekMessageA
...
 [previous calls were nsVoidArray::RemoveElement, nspr4!PR_Unlock]
...
user32!DefWindowProcA
ole32!unknown
...
user32!CallNextHookEx
...
user32!DefWindowProcA

the stack trace is approximately (and decoding this is amazingly expensive and utterly useless):
 [v just finished a call to msvcrt!memcpy]
msvcrt!unknown
 [^ final call before return would be to msvcrt!_unlock]
msvcrt!realloc
js3250!unknown1
js3250!js_Invoke
js3250!unknown2
js3250!unknown3
 [next call would be to js3250!js_ValueToString]
js3250!js_ValueToSource
js3250!unknown4
js3250!js_Invoke
js3250!unknown2
js3250!unknown3
 [next call would be to js3250!js_ValueToString]
js3250!js_ValueToSource
js3250!unknown4
js3250!js_Invoke
js3250!unknown2
js3250!unknown3
 [next call would be to js3250!js_ValueToString]
js3250!js_ValueToSource
js3250!unknown4
js3250!js_Invoke
firefox!unknown
firefox!unknown
xpcom_core!unknown
xpcom_core!unknown
firefox!main
[firefox!WinMain tail call optimized]
KERNEL32!ApplicationStartup

i don't know enough about the structure of realloc, so this could be heap corruption, or it could be something else.
Comment 8 Kenneth Heal 2007-03-15 02:12:23 PDT
I am running FX 2.0.0.2 (When filing a bug it says you must have the latest version and after that I did not think any more of it, sorry.).

In any case I checked Talkback and it seems it did not get updated with fx and so I have reinstalled to make sure Talkback is running -- hopefully it will create a report of such a crash in the near future.

Thanks for your help.
Comment 9 Kenneth Heal 2007-03-18 06:56:16 PDT
Have seen two further crashes, TB30339203Z and TB30348398W.

The crashes are slightly different in as much as the crash is FIREFOX caused an invalid page fault in module MSXML.DLL.  The strange thing was that even after clicking OK firefox did not shut down.

For TB30348398W windows gave this stack trace:
FIREFOX caused an invalid page fault in module MSXML.DLL at 0167:7127d720.
Registers:
EAX=00000000 CS=0167 EIP=7127d720 EFLGS=00010246
EBX=00000000 SS=016f ESP=05a8f5fc EBP=05a8f65c
ECX=ffffffff DS=016f ESI=0a470110 FS=344f
EDX=81786274 ES=016f EDI=00000014 GS=0000
Bytes at CS:EIP:
8b 40 28 f7 d8 1b c0 24 fb 83 c0 04 89 46 04 01 
Stack dump:
71271700 00be6e58 712818ce 00000014 00be6e58 00000001 712816eb 71271700 71280ed1 7127cfda 00000001 00000001 00000000 00000001 712d102c 00000000
Comment 10 Kenneth Heal 2007-03-18 09:21:26 PDT
Bingo! Have a crash with the exact same error, TB30353007Q

Here is the Windows stack trace that goes with it:
FIREFOX caused an invalid page fault in module MSVCRT.DLL at 0167:7800d16a.
Registers:
EAX=00ea0008 CS=0167 EIP=7800d16a EFLGS=00010297
EBX=00000006 SS=016f ESP=00d8ec34 EBP=00d8ec50
ECX=00ea3bc0 DS=016f ESI=0655b71c FS=79ef
EDX=0000000d ES=016f EDI=0000001d GS=0000
Bytes at CS:EIP:
8b 5c d1 04 8d 0c d1 89 5e 04 89 4e 08 89 71 04 
Stack dump:
00000000 00000080 0655b790 00ea3bc0 00000000 00000070 000000e0 00d8ecb0 78011b77 00ea0138 0655b71c 068d0f70 0655b790 00000070 0000000f 00000030 
Comment 11 Kenneth Heal 2007-03-18 18:26:32 PDT
TB30363435W is another such crash.

For the record Windows spits out.
FIREFOX caused an invalid page fault in module MSVCRT.DLL at 0167:7800d16a.
Registers:
EAX=00ea0008 CS=0167 EIP=7800d16a EFLGS=00010202
EBX=00ea0138 SS=016f ESP=00d8dde8 EBP=00d8de04
ECX=00ea3dc4 DS=016f ESI=04a571ec FS=0ebf
EDX=00000035 ES=016f EDI=0000001e GS=0000
Bytes at CS:EIP:
8b 5c d1 04 8d 0c d1 89 5e 04 89 4e 08 89 71 04 
Stack dump:
00000000 000003da 04a571f0 00ea3dc4 00000001 00000011 00000360 00d8de64 78011b77 00ea0138 04a5754c 02ca0c10 04a571f0 00000360 00000002 00000044
Comment 12 Kenneth Heal 2007-03-18 18:37:18 PDT
TB30366578E also.
Comment 13 timeless 2007-03-19 08:32:03 PDT
ok, they're all reallocs, most are fairly deep js stacks, at least of the ones i've poked.
Comment 14 Kenneth Heal 2007-03-20 09:36:35 PDT
Bit confused as to why we are still on UNCONFIRMED.  I think from the data we can safely say that this is not intended behaviour, and therefore NEW seems a more logical choice of bug status.

Let me know if you need any more info.
Comment 15 Brendan Eich [:brendan] 2007-03-20 10:08:09 PDT
Heap corruption, not fun. Need to get a reproducible testcase.

/be
Comment 16 Kenneth Heal 2007-03-20 10:15:53 PDT
It seems to happen quite randomly, though when I did a search for similar crashes I found a lot where the URL was http://mail.yahoo.com so this is probably your best bet to repro the crash.

If the PC is low on disk space it seems to occur more often, so I suspect tight memory conditions (limited swapfile) might help exacerbate the corruption.

If you an fx debug or checked build I could run that to gather more data when the crash happens.
Comment 17 Kenneth Heal 2007-03-20 15:54:54 PDT
I meant to type:
If you have a fx debug or checked build I could run that to gather more data when the crash happens.
Comment 18 Kenneth Heal 2007-04-04 10:06:20 PDT
TB30884438E, TB30877127Y
Don't currently have the chance to create a debug build.
Comment 19 Brian Crowder 2007-04-04 10:28:32 PDT
These last two are very different crashes.
Comment 20 Kenneth Heal 2007-04-04 10:45:36 PDT
Thanks for the quick response -- if I look at TB one looks a bit similar and one does indeed look very different.  Shall I file two new bugs or just one bug to cover both crashes?  (Don't see any other bugs which look relevant.)  Thanks!
Comment 21 Brian Crowder 2007-04-04 11:02:20 PDT
No, I don't think either is necessary to file a new bug for, just observing that the crashes seem very different.  Can you please describe the the extensions you have installed?  Also, are you running version 2.0.0.3 now?

Thanks
Comment 22 Kenneth Heal 2007-04-04 11:35:33 PDT
Hi.
I am running fx 2.0.0.3 now.

I have these extensions installed:

United States English Dictionary 2.0.0.6
Dictionnaire MySpell en Français 1.0.1
Nederlands Woordenboek 1.0.5
Deutsches Wörterbuch, erweitert für Österreich 1.0.1
Talkback 2.0.0.3
Diccionario de Español/España 1.1
British English Dictionary 1.19
Dictionnaire MySpell en Français (réforme 1990) 1.0.1
Geiriadur Cymraeg 0.02
Dizionario italiano 3.0
Deutsches Wörterbuch 1.0.1

I am using the default theme.
Comment 23 Brendan Eich [:brendan] 2007-04-04 11:59:48 PDT
(In reply to comment #19)
> These last two are very different crashes.

They're not obvious bugs in the leaf frames (some of which are in the C runtime libraries), they're due to heap corruption. Heap corruption can kill anyone at any time (as Joe-Bob Briggs used to say about drive-in horror movies ;-). Odds are these are skidmarks from the same death-car bug that's driving through the heap, enabled by a certain addon.

Purify or valgrind time.

/be
Comment 24 Brian Crowder 2007-04-04 12:04:28 PDT
Most of Kenneth's other talkbacks have looked a lot like TB30884438E, however.

Kenneth:  can you try disabling/uninstalling extensions until the crashes go away?
Comment 25 Kenneth Heal 2007-04-04 13:28:27 PDT
All the extensions are language dictionaries (except the Talkback extension which is kind of a must :-/ ) for the spell-checker feature.

Some of these languages are reasonably easy to get rid of.  I would have expected someone else to be hitting this if it were a language dictionary.

I shall get rid of these as this is the route of least pain:
United States English Dictionary 2.0.0.6
Dictionnaire MySpell en Français 1.0.1
Deutsches Wörterbuch, erweitert für Österreich 1.0.1
Diccionario de Español/España 1.1
Dictionnaire MySpell en Français (réforme 1990) 1.0.1
Geiriadur Cymraeg 0.02
Dizionario italiano 3.0

Problem is the crashes are fairly intermittent, so it will be hard to know for sure if they have gone away.
Comment 26 Brian Crowder 2007-04-04 14:37:16 PDT
Don't hesitate to disable talkback, too; knowing whether or not it is affecting your stability would still be useful information.
Comment 27 Brendan Eich [:brendan] 2007-04-04 14:40:22 PDT
Kind of points the finger at spellchecker, at first blush.

/be
Comment 28 Brian Crowder 2007-04-04 14:43:04 PDT
Yeah, I'll browse over the code for the extension (array-heavy stuff particularly) and see if I can spot anything reducible to a shell testcase.
Comment 29 Brendan Eich [:brendan] 2007-04-04 15:54:19 PDT
I do *not* think talkback should be disabled. It doesn't do much to corrupt the in-process heap; it's really only there when a segv or buserr happens. And it may provide crucial clues.

/be
Comment 30 Kenneth Heal 2007-04-04 22:48:32 PDT
Early on in the bug I had these crashes even though talkback was disabled (an uprade to fx 2.0.0.2 failed to upgrade talkback).  At this stage I am inclined to agree with /be and rule out talkback.
Comment 31 Kenneth Heal 2007-04-05 06:53:40 PDT
Just crashed again, TB30911361G

For WIW and that is not much, windows spits out:
FIREFOX caused an invalid page fault in module MSVCRT.DLL at 0167:7800d17d.
Registers:
EAX=037dda80 CS=0167 EIP=7800d17d EFLGS=00010297
EBX=80000001 SS=016f ESP=00d8dff0 EBP=00d8e00c
ECX=80000001 DS=016f ESI=06b4f18c FS=0edf
EDX=00000016 ES=016f EDI=0000001f GS=0000
Bytes at CS:EIP:
89 71 08 8b 4e 04 3b 4e 08 75 39 8a 4c 02 04 83 
Stack dump:
00000000 00000208 06b4f1d0 037e1a40 00000000 00000040 00000170 00d8e06c 78011b77 037dde68 06b4f18c 06b50df0 06b4f1d0 00000130 00000002 00000074
Comment 32 Kenneth Heal 2007-04-05 06:55:31 PDT
Forgot to mention the URL:
http://by126w.bay126.mail.live.com/mail/mail.aspx?gs=true&wa=wsignin1.0
Comment 33 Kenneth Heal 2007-04-05 07:12:34 PDT
Current extensions are:

Talkback 2.0.0.3

British English Dictionary 1.19
Deutsches Wörterbuch 1.0.1
Nederlands Woordenboek 1.0.5

I guess I could proceed to throw out the remaining dictionaries as well, though that is painful as these are the languages I work in on a daily basis.  I would have suspected that if any of these had a problem it would be known (esp. en-GB and de-DE) as they are fairly common languages.
Comment 34 Kenneth Heal 2007-04-17 04:55:30 PDT
TBH a bit disappointed at the lack of progress here.  Have made several attempts to compile myself a debug release, but these have failed and you guys have been unable to help.

As emailed off-line I am using a VMWared XP Pro SP2 with Visual Studio 6 SP6.  I even later tried installing SP5 over this and the processor pack but that hasn't helped.  Neither did using the MOZILLA_1_8_BRANCH instead.  You can see the results at:
http://moodle.senet.nl/mozilla/stderr.wri.gz
http://moodle.senet.nl/mozilla/buildoutput.wri.gz

I notice several other serious JS crashes where also zero progress has been made.  It looks like I am going to have to recommend IE, a step I do not really wish to take.
Comment 35 Brian Crowder 2007-04-17 08:37:13 PDT
Kenneth:  This is a bug which we are not easily able to reproduce, please have some patience with us as we try to work with you to resolve it.  If you're encountering bugs in other pieces of our product which are easy to reproduce, I encourage you to contribute your information on those bugs as well.  The "I'm going to recommend IE" threat isn't a useful contribution to this bug.  I'm sorry this bug hasn't been resolved to your satisfaction, but I doubt you'll find the IE team, or the Microsoft CTO more responsive.
Comment 36 Kenneth Heal 2007-04-17 09:01:14 PDT
As I said IE is not a step I relish.  Enough said.
Can someone provide me with a working debug release from 2.0.0.3 or 1.8_BRANCH?  (I am not a developer and not really that interested to troubleshoot a bunch of build issues if someone can give me a debug release.)
Thanks.
Comment 37 Ted Mielczarek [:ted.mielczarek] 2007-04-17 10:08:19 PDT
Kenneth:

You can get a debuggable Windows trunk build by downloading 1.9a3:
http://releases.mozilla.org/pub/mozilla.org/firefox/releases/granparadiso/alpha3/win32/en-US/

You will need WinDbg, which you can get in the Debugging Tools for Windows pack:
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx

In WinDbg, once you've selected to debug that firefox.exe, you should type the following command into the command box:
SRV*e:\symsrv*http://msdl.microsoft.com/download/symbols*http://mavra.perilith.com/~luser/airbag-symbols/

Replace the e:\symsrv with an empty directory on your computer, that's where the debugging symbols will be downloaded to.  WinDbg should then fetch debug symbols as needed for the process.
Comment 38 Kenneth Heal 2007-04-17 11:31:43 PDT
Ted, thanks for the suggestion; however I am facing the problem on a Win 98 machine so this is not feasible... also my intention in a debug release is to for talkback to gather more data for developers to pinpoint the issue and to do this on a non-alpha release. (i.e. 2.0.0.x or 1.8_BRANCH)

Have had a bit of progress on the build front, I managed to get a 2.0.0.3 debug release to build, though it crashes quite often.  The problem I now see is though that it does not include Talkback... (checked in the custom install and the option was not there) is there a build option I need to specify for this?
Comment 39 Brian Crowder 2007-04-17 11:59:30 PDT
With a working debug build, you don't need talkback anymore.  You will need a debugger of some sort, though.  Once you have that, you can start looking at stack traces.  If you want, e-mail me and we can coordinate meeting on IRC to look at one of your crashes in real-time.

Thanks,
  crowder
Comment 40 timeless 2007-04-18 12:52:56 PDT
Created attachment 262004 [details] [diff] [review]
i think this is the fix for the build failure in comment 34  

I talked w/ neil a bit. I haven't actually tested this. but conceptually this is about right (testing it would be good...)
Comment 41 timeless 2007-04-18 13:08:11 PDT
Comment on attachment 262004 [details] [diff] [review]
i think this is the fix for the build failure in comment 34  

crowder rightly complains that my comment didn't explain what bug I was trying to fix. This isn't fixing the crash...
Comment 42 Kenneth Heal 2007-04-19 11:09:50 PDT
Tried altering review status on attachment but got "You are not authorised to edit attachment 262004 [details] [diff] [review]" so I am updating the bug instead.

Applied patch and was successful in building fx2_0_0_3 using mingw32 and msvc6:

In case it is needed here is about:buildconfig

Build platform
target i686-pc-mingw32

Build tools
Compiler 	Version 	Compiler flags
cl 	12.00.8804 	-TC -nologo -W3 -Gy -Fd$(PDBFILE)
cl 	12.00.8804 	-TP -nologo -W3 -Gy -Fd$(PDBFILE)

Configure arguments
--enable-application=browser --enable-application=browser --enable-svg --enable-canvas --enable-debugger-info-modules=all --enable-debug --disable-optimize --enable-static --disable-shared --enable-official-branding
Comment 43 Kenneth Heal 2007-04-19 11:34:28 PDT
Comment on attachment 262004 [details] [diff] [review]
i think this is the fix for the build failure in comment 34  

Pls see bug update #42.  Note I am not a C developer, so I can only comment as a sysadmin.
Comment 44 Kenneth Heal 2007-05-01 04:17:47 PDT
Wouldn't it be worth checking in the changes from attachment 262004 [details] [diff] [review] into MOZILLA_1_8_BRANCH ?

Might save others tearing their hair out when building. ;-)
Comment 45 Daniel Veditz [:dveditz] 2007-05-01 11:07:22 PDT
I just went through 35 recent talkbacks with the signature "MSVCRT.DLL + 0xd16a". Mostly the latest Firefox2 and Firefox15, but a few Firefox10, Mozilla17, and Thunderbird10/15.

All of them were "Windows 98 4.10 build 67766446" except one "build 67766222".

Starts to sound like a bug in old MSVCRT rather than our bug with that kind of data.
Comment 46 Kenneth Heal 2007-05-01 13:11:55 PDT
Thanks for the update.  Could be.  Though Win 98 is also less stable with any app which misbehaves, so I think it still could be (which is not to say it must be) a fx issue. In any case my msvcrt.dll is version 6.00.9782.0 which I believe is the latest for this platform.  In the meantime I have built latest MOZILLA_1_8_BRANCH and it currently is not crashing.
Comment 47 Daniel Veditz [:dveditz] 2007-05-01 17:56:54 PDT
If you built it as described in comment 42 the fact that you're running a debug build could well sidestep the crash without fixing anything. For one thing you'd be calling into the debug MSVCRTd.dll instead of the system MSVCRT and using the debug heap routines.
Comment 48 Kenneth Heal 2007-05-01 22:57:52 PDT
Nope, am running a build with these options:
about:buildconfig

Build platform
target
i686-pc-mingw32

Build tools
Compiler 	Version 	Compiler flags
cl 	12.00.8804 	-TC -nologo -W3 -Gy -Fd$(PDBFILE)
cl 	12.00.8804 	-TP -nologo -W3 -Gy -Fd$(PDBFILE)

Configure arguments
--enable-application=browser --enable-application=browser --enable-official-branding --disable-optimize --disable-debug --enable-debugger-info-modules --disable-tests --enable-static --disable-shared --enable-svg --enable-canvas --enable-ui-locale=en-GB
Comment 49 Kenneth Heal 2007-05-01 23:28:45 PDT
Have just checked and msvcrt.dll from Visual C++ 6.0 Service Pack 6 is version 6.00.9782.0.  I checked the various versions shipped:

VC 6++ Gold (no-SP)	6.00.8168.0
Windows 98 SE bare	6.00.8397.0
VC 6++ Service Pack 5	6.00.8797.0
VC 6++ Service Pack 6	6.00.9782.0

In the process of troubleshooting this I installed VC 6++ Service Pack 6 on the problem system and so have the newer file.
Comment 50 Kenneth Heal 2007-05-02 05:35:45 PDT
http://support.microsoft.com/kb/895959 says: "The latest version of the Msvcrt.dll file that is compatible with Windows Millennium Edition, with Windows 98, or with Windows 95 is version 6.0.9782.0. This version is included with Microsoft Visual Studio 6.0 Service Pack 6 (SP6)."

As it is a pain to find and extract I shall attach the VC 6++ Service Pack 6 redistributable (vcredist.exe) to this bug.

As the system now seems to be working (will be a while before I can be 100% sure) I shall mark the bug as WORKSFORME based on the following steps:

1) Ensure Windows has latest patches from http://v4.windowsupdate.microsoft.com/en/default.asp.
2) Install VC 6++ Service Pack 6 redistributable package vcredist.exe.
3) Install Firefox 2.0.0.4 or later.
4) Ensure Adobe shockwave and flash are at their latest versions.

Thanks for all your help.
Comment 51 Kenneth Heal 2007-05-02 05:55:06 PDT
Unfortunately bugzilla won't let me upload the attachment as it is too large, so other users will need to get VS 6 SP 6 from the Microsoft website and extract the vcredist.exe by hand.

The file needed is:
vcredist exe     1,821,920  17/02/04  20:34
Comment 52 Kenneth Heal 2007-05-02 06:24:23 PDT
Setting blocking1.8.1.5 so that the patch for the build issues can be considered for inclusion in next firefox release 2.0.0.5.
Comment 53 Kenneth Heal 2007-05-02 15:22:38 PDT
Tip: The easiest way to get the vcredist.exe is to visit http://www.microsoft.com/downloads/details.aspx?familyid=8ecec2b4-9118-4a63-8dbd-b34547ad2c62&displaylang=en and download Vs6sp6S1.exe.  If opened with WinZip you can extract vcredist.exe.
Comment 54 Kenneth Heal 2007-05-28 02:05:30 PDT
After quite some time without any msvcrt.dll crashes I have hit an msvcrt.dll crash again, on 2.0.0.4 rc3 on Win98SE. msvcrt.dll version 6.00.9782.0.

The stack is quite a bit different on this one though.

TB32582494M
FIREFOX caused an invalid page fault in module MSVCRT.DLL at 0167:7800e1f4.
Registers:
EAX=00000004 CS=0167 EIP=7800e1f4 EFLGS=00010297
EBX=00000059 SS=016f ESP=00d896b8 EBP=00d896c0
ECX=00000004 DS=016f ESI=86bffffc FS=0ec7
EDX=00000000 ES=016f EDI=03aed7a0 GS=0000
Bytes at CS:EIP:
8b 44 8e f4 89 44 8f f4 8b 44 8e f8 89 44 8f f8 
Stack dump:
86bfff14 03aed7a0 00d89720 0052aef9 03aed7a0 86bffffc 00000010 00000001 0052ae17 86bffef8 00610042 006b006e 006f0047 00680074 00630069 004d0020

Extensions are still:
British English Dictionary 1.19
Deutsches Wörterbuch 1.0.1
Diccionario de Español/España 1.1
Dictionnaire MySpell en Français 1.0.1
Dictionnaire MySpell en Français (réforme 1990) 1.0.1
Dizionario italiano 3.0
Geiriadur Cymraeg 0.02
Nederlands Woordenboek 1.0.5
Talkback 2.0.0.4
United States English Dictionary 2.0.0.6

I am using the Firefox (default) theme 2.0.
Comment 55 Daniel Veditz [:dveditz] 2007-06-18 10:45:11 PDT
Comment on attachment 262004 [details] [diff] [review]
i think this is the fix for the build failure in comment 34  

I think what kheal wanted was an approval request for the build changes. Would need module owner review before we can approve it.

I'm not sure neil is the right r= for toolkit download manager code.
Comment 56 Daniel Veditz [:dveditz] 2007-06-18 10:46:47 PDT
We wouldn't hold the release for a build/config change so not a blocker.
Comment 57 Daniel Veditz [:dveditz] 2007-06-18 10:50:02 PDT
actually the build change really ought to have gotten a separate bug -- now if we check in that change it gets very confusing about what got "fixed" since it doesn't help the original problem in any way.
Comment 58 Kenneth Heal 2007-06-18 10:52:31 PDT
Probably true, though at the time of this bug I was facing constant crashes and this was an issue on the way to attempting to fix something else.

What I am/was looking for is for the patch in attachment 262004 [details] [diff] [review] to be included
in the MOZILLA_1_8_BRANCH tree, so that it would work its way into future fx
releases.

Probably seems reasonable to create a fresh bug for the build issue -- pls feel free to go ahead and do this.
Comment 59 Kenneth Heal 2007-06-18 11:00:33 PDT
wanted1.8.1.x seems to be greyed out... are you able to set this field?  This seems the most appropriate for this.
Comment 60 timeless 2007-06-19 02:16:07 PDT
it might not be clear from the description "This is for drivers to track bugs that do not block the next 1.5.0.x release, but are important", in which case, please let me know and we can see about changing it. That flag is reserved for drivers (dveditz could set it). Note that its purpose is for drivers to tell engineers that they want someone to fix a bug. That's not really the purpose here. dveditz has already rejected blocking. The way engineers show they want something is by working on it and then asking for approval.
Comment 61 Daniel Veditz [:dveditz] 2007-06-28 11:31:27 PDT
Comment on attachment 262004 [details] [diff] [review]
i think this is the fix for the build failure in comment 34  

approved for 1.8.1.5, a=dveditz for release-drivers.

Unlike our normal process, please do NOT add the "fixed1.8.1.5" keyword when you check in this patch since it is not the real fix for this bug.
Comment 62 Brendan Eich [:brendan] 2007-06-28 17:27:36 PDT
Should this bug be assigned to general@js.bugs, in Core:JavaScript Engine ?

/be
Comment 63 Brian Crowder 2007-06-28 17:31:42 PDT
The stacks are inconsistent, but often in JS code...  I still think it is likely -not- a core JS bug.
Comment 64 Daniel Veditz [:dveditz] 2007-07-08 16:39:40 PDT
Code-freeze for 1.8.1.5 is July 13 -- if the build change is not checked-in by then it'll have to wait for 1.8.1.6. Find check-in help on irc.mozilla.org
Comment 65 Daniel Veditz [:dveditz] 2007-07-11 13:49:53 PDT
Comment on attachment 262004 [details] [diff] [review]
i think this is the fix for the build failure in comment 34  

Tree closing early, will have to wait for next time.
Comment 66 Kenneth Heal 2007-08-05 21:19:08 PDT
Crashed again.  The system had Windows Live mail open and I was not doing anything when it decided to crash.

TB34685061K

FIREFOX caused an invalid page fault in module MSVCRT.DLL at 0167:7800e1f4.
Registers:
EAX=00000004 CS=0167 EIP=7800e1f4 EFLGS=00010297
EBX=00000050 SS=016f ESP=00d8798c EBP=00d87994
ECX=00000004 DS=016f ESI=80f6cffc FS=0eff
EDX=00000000 ES=016f EDI=02e7ab90 GS=0000
Bytes at CS:EIP:
8b 44 8e f4 89 44 8f f4 8b 44 8e f8 89 44 8f f8 
Stack dump:
80f6cf14 02e7ab90 00d879f4 0052b7c4 02e7ab90 80f6cffc 00000010 00000001 0052b6e2 80f6cef8 00610042 006b006e 006f0047 00680074 00630069 004d0020

Apart from the aforementioned dictionaries and the talkback plugin there are no add-ons on this system.
Comment 67 Kenneth Heal 2007-08-05 21:22:11 PDT
Forgot to mention for the record... Now running fx 2.0.0.6
Comment 68 timeless 2007-08-05 23:32:47 PDT
you don't need to mention it, talkback records your build id for us. in fact, it's sufficient for you to *just* give a talkback id + a list of extensions.

please try using just *one* dictionary for two weeks.

If you need to use three, dictionaries, create three profiles, install 3 copies of firefox, configure a shortcut for each to use "firefox.exe -no-remote -P british" / "firefox.exe -no-remote -P german" / "firefox.exe -no-remote -P netherlands".

In each profile, install only the corresponding dictionary. You can run all three shortcuts/browsers at the same time. When you crash, be sure to report the talkback id and which profile/dictionary is installed.
Comment 69 Daniel Veditz [:dveditz] 2007-08-29 15:31:18 PDT
Comment on attachment 262004 [details] [diff] [review]
i think this is the fix for the build failure in comment 34  

Try again this go around, please find someone on #developers to check this in this time while the tree is open -- the earlier the better.

approved for 1.8.1.7, a=dveditz for release-drivers
Comment 70 :Gavin Sharp [email: gavin@gavinsharp.com] 2007-09-04 09:06:52 PDT
Checked in the build patch. I really wish the patch was put in it's own bug... I don't want to mark this fixed1.8.1.7, because the reported bug isn't fixed, but not including it is going to cause this to stay on the 1.8.1.7 team's checkin-needed radar... oh well.

mozilla/toolkit/components/downloads/src/nsDownloadManager.cpp 	1.53.2.11
Comment 71 Mark Smith (:mcs) 2007-09-25 06:10:14 PDT
(In reply to comment #70)
> Checked in the build patch. I really wish the patch was put in it's own bug...

Me too.  My build just failed due to this patch. The first definition of BYTES_TO_KBYTES() is missing a parenthesis (unbalanced):

+#if defined(_MSC_VER) && _MSC_VER < 13
+#define BYTES_TO_KBYTES(bytes) ((PRFloat64)((PRInt64)(bytes >> 8) / 4 + .5)
+#else
+#define BYTES_TO_KBYTES(bytes) ((PRFloat64)bytes / 1024.0 + .5)
+#endif
+

A simple fix is to add a closing parenthesis to the end.
Comment 72 Daniel Veditz [:dveditz] 2007-09-27 22:40:03 PDT
Isn't _MSC_VER for VC 6 1200? "< 13" doesn't seem like it'd match anything. If it does then we've got problems all over, for example http://lxr.mozilla.org/mozilla/source/xpcom/glue/nsCOMPtr.h#1546

But the "13" being wrong could explain why the tinderbox isn't broken by this code change because the branch builds do still use VC6. But what sort of compiler/system is Mark Smith building on if this did fail for him?
Comment 73 Daniel Veditz [:dveditz] 2007-09-27 22:49:05 PDT
I checked in the missing paren to the 1.8 branch, but I'm still worried about that "13"
Comment 74 Mark Smith (:mcs) 2007-09-28 06:58:23 PDT
(In reply to comment #72)
> But the "13" being wrong could explain why the tinderbox isn't broken by this
> code change because the branch builds do still use VC6. But what sort of
> compiler/system is Mark Smith building on if this did fail for him?

Sorry.  I should have mentioned that I had to edit the file to ensure that the first BYTES_TO_KBYTES() definition was used in my build.  It is a long story, but I am building with VC6 Service Pack 6 (which I need for some other non-Mozilla projects).  Of course the official compiler is VC6 SP5.  And I mistakenly assumed that the wrong macro was chosen due to some difference in my build environment.  But I did not look very closely.

Thanks for making the fix; maybe it will also help someone else.  But the < 13 seems wrong to me too.
Comment 75 Mats Palmgren (vacation) 2007-10-06 08:56:56 PDT
timeless, is (_MSC_VER < 1300) what was intended?
http://msdn2.microsoft.com/en-us/library/b0084kay(VS.80).aspx
Comment 76 Kenneth Heal 2007-11-09 05:14:42 PST
One thing that has become apparent is that this crash happens in times of low memory (typically low disk space ∴ low disk swap space ∴ low amount of adressable memory).  Could it be that these crashes are caused by low memory condition?  Would this explain the stack traces containing only one or two frames?
Comment 77 Kenneth Heal 2007-11-09 05:15:01 PST
One thing that has become apparent is that this crash happens in times of low memory (typically low disk space ∴ low disk swap space ∴ low amount of addressable memory).  Could it be that these crashes are caused by low memory condition?  Would this explain the stack traces containing only one or two frames?
Comment 78 Brian Crowder 2007-11-09 09:44:37 PST
Useful information, Kenneth.  I don't think OOM crashes are going to be considered worth-while on the branch at this point, though.  Should we WONTFIX this?
Comment 79 Kenneth Heal 2007-11-09 10:29:51 PST
It would be good to at first confirm this is an OOM bug.  Not being a C developer I cannot fix the bug myself and not being the boss of any of the developers I cannot oblige them to do so either :-) I suspect the problem is still there under XP and 2K, just that these OS's handle it rather better.  In view of the pain and effort invested in this not too keen on WONTFIXing it, though I suspect that will be the logical conclusion.
Comment 80 Ryan VanderMeulen [:RyanVM] 2008-01-06 06:59:58 PST
Is there a new testcase for this bug? The link in comment #0 no longer works. If you can make one, please attach it to the bug so it isn't lost in the future.
Comment 81 Kenneth Heal 2008-01-06 07:39:50 PST
Unfortunately there is not an easily reproducible testcase (in my experience) but this crash seems to happen a lot when accessing either Yahoo! or Hotmail webmail.  I also noticed the crash happening a lot more when disks were full and memory quite stretched.
Comment 82 Brian Crowder 2008-03-12 09:45:35 PDT
Kenneth:  Still seeing this?
Comment 83 Kenneth Heal 2008-03-12 10:49:15 PDT
Hi Brian, how r u?
I am still seeing this on Windows 98 (SE) systems.
Comment 84 timeless 2008-03-16 11:45:39 PDT
yes, it should be 1300 :(
Comment 85 Kenneth Heal 2008-03-16 14:31:21 PDT
timeless, am a bit confused by your last update.
Comment 86 timeless 2008-03-31 00:45:12 PDT
Created attachment 312684 [details] [diff] [review]
correct 13 to 1300 (MOZILLA_1_8_BRANCH)
Comment 87 Daniel Veditz [:dveditz] 2008-04-08 01:26:05 PDT
Comment on attachment 312684 [details] [diff] [review]
correct 13 to 1300 (MOZILLA_1_8_BRANCH)

r=dveditz
Comment 88 Daniel Veditz [:dveditz] 2008-04-21 11:14:15 PDT
Comment on attachment 312684 [details] [diff] [review]
correct 13 to 1300 (MOZILLA_1_8_BRANCH)

approved for 1.8.1.15, a=dveditz for release-drivers
Comment 89 timeless 2008-05-09 01:24:30 PDT
Comment on attachment 312684 [details] [diff] [review]
correct 13 to 1300 (MOZILLA_1_8_BRANCH)

mozilla/toolkit/components/downloads/src/nsDownloadManager.cpp 	1.53.2.13
Comment 90 juan becerra [:juanb] 2008-06-10 14:02:14 PDT
Kenneth, can you help us verify the fix by trying the latest branch nightly, Firefox 2.0.0.15pre? It's been difficult to reproduce the crash using our Win98 machine.

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.8/

Note You need to log in before you can comment on or make changes to this bug.