Closed Bug 1243765 Opened 8 years ago Closed 8 years ago

Users reporting connection issues and windows crash reports on application shutdown after Firefox 44 update

Categories

(Firefox :: Untriaged, defect)

Product:
Firefox
Component:
Untriaged
Version:
44 Branch
Platform:
Unspecified
Windows
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED DUPLICATE of bug 1243098
Tracking Flags:
Tracking Status
firefox44 - affected

People

(Reporter: philipp, Unassigned)

References

Details

(Keywords: regression, Whiteboard: [44dll])

Crash Data

[Tracking Requested - why for this release]:

there are a significant number of user reports describing two new problems appearing in tandem after the firefox 44 update:
- failure to load any websites, the screen stays blank and stuck on "connecting...": https://gyazo.com/d643b08146fd7123519915496fe7f2c9
- a windows app crash report is displayed when firefox is closed:
> Problem signature:
> Problem Event Name:	BEX
> Application Name:	firefox.exe
> Application Version:	44.0.0.5866
> Application Timestamp:	56a4222c
> Fault Module Name:	StackHash_0a9e
> Fault Module Version:	0.0.0.0
> Fault Module Timestamp:	00000000
> Exception Offset:	03fffff9
> Exception Code:	c0000005
> Exception Data:	00000008
> OS Version:	6.1.7601.2.1.0.768.3
> Locale ID:	2057
> Additional Information 1:	0a9e
> Additional Information 2:	0a9e372d3b4ad19135b953a78882e789
> Additional Information 3:	0a9e
> Additional Information 4:	0a9e372d3b4ad19135b953a78882e789

here are the threads about this issue on sumo: https://support.mozilla.org/en-US/questions/firefox?tagged=44-connecting&show=all (issue is reported to happen on win7-win10 so far)

troubleshooting steps that did not help include: firefox safemode, firefox refresh, windows safe mode, 'no proxy'-setting

switching to win64 builds on windows can fix the issue according to feedback from multiple users though.
curiously all affected users who provided some data are on single gpu nvidia hardware, not sure if this is just coincidental though:

NVIDIA GeForce GT 740
10.18.13.5900
https://support.mozilla.org/en-US/questions/1106527

NVIDIA GeForce GTX 680
9.18.13.3221
https://support.mozilla.org/en-US/questions/1106660

NVIDIA GeForce GTX 770
10.18.13.5362
https://support.mozilla.org/en-US/questions/1106792

NVIDIA GeForce G210
9.18.13.593
https://support.mozilla.org/en-US/questions/1106795

NVIDIA GeForce GT 750M
10.18.13.6143
https://support.mozilla.org/en-US/questions/1106524

NVIDIA GeForce GTX 660
10.18.13.6143
https://support.mozilla.org/en-US/questions/1106713

NVIDIA GeForce GTX 680
10.18.13.6143
http://mozhelp.dynvpn.de/paste/?0fa1c1bb281dccf3#1zrGpxPMzFfW3HMLS+Q2fhREmEB/Ojk7sjTiuNlwwWA=
earlier crash report: https://crash-stats.mozilla.com/report/index/40c6e30f-3394-40f2-82e6-073432160106
The fact that switching to Windows 64 helps is interesting.
Hi Milan, it's good to know Blassey (who has similar hardware) was unable to repro this issue with Fx44. Do we need some additional diagnostics information from end-users to investigate this further? Thanks!
Flags: needinfo?(milan)
i forgot to mention before that it is not tied to a particular av/firewall. according to users this happened under avast, eset, ms security essentials or with no provider as well.
Right now, the only thing pointing to graphics related is the correlation on Nvidia? We also have the "works in Windows 64", so I can't but think of bug 1241921 (related to bug 1240848), which have a similar flavour in that it's (mostly) Nvidia, and have different behaviour in 32 and 64-bit builds (although in the bugs above, it's the 64-bit builds that were bad, and we know it isn't the above bugs, as this is showing up in 44.)  I have no data to claim the above, just brainstorming :)
Flags: needinfo?(milan) → needinfo?(aklotz)
i hope this isn't too far fetched, but the following crash signature, which is jumping up in firefox 44 (#4 top crash score at the moment) might be related and also showing some correlations to nvidia stuff. maybe the modules list can also give some clue what is going on:

  RtlpLowFragHeapFree | RtlFreeHeap | FreeWrapper|EXCEPTION_ACCESS_VIOLATION_READ (89 crashes)
     98% (87/89) vs.  58% (17542/30495) WSHTCPIP.DLL
     97% (86/89) vs.  59% (18059/30495) Wldap32.dll
     99% (88/89) vs.  66% (20101/30495) lpk.dll
     30% (27/89) vs.   6% (1836/30495) nvspcap.dll
    100% (89/89) vs.  79% (23981/30495) wkscli.dll
     35% (31/89) vs.  14% (4293/30495) nvwgf2um.dll
     99% (88/89) vs.  82% (25005/30495) DWrite.dll
     99% (88/89) vs.  85% (25861/30495) firefox.exe
     99% (88/89) vs.  85% (25948/30495) dbghelp.dll
    100% (89/89) vs.  87% (26428/30495) srvcli.dll
     99% (88/89) vs.  86% (26287/30495) dnsapi.dll
    100% (89/89) vs.  88% (26753/30495) sspicli.dll
    100% (89/89) vs.  88% (26796/30495) CRYPTBASE.dll
     27% (24/89) vs.  15% (4560/30495) WLIDNSP.DLL
     98% (87/89) vs.  86% (26203/30495) ntmarta.dll
    100% (89/89) vs.  89% (27000/30495) comctl32.dll
     18% (16/89) vs.   7% (2154/30495) nvapi.dll
     98% (87/89) vs.  88% (26765/30495) mswsock.dll
    100% (89/89) vs.  91% (27799/30495) samcli.dll
    100% (89/89) vs.  91% (27803/30495) netutils.dll
    100% (89/89) vs.  92% (28061/30495) devobj.dll
    100% (89/89) vs.  92% (28132/30495) sechost.dll
    100% (89/89) vs.  92% (28132/30495) KERNELBASE.dll
     19% (17/89) vs.  11% (3469/30495) aswJsFlt.dll
    100% (89/89) vs.  92% (28144/30495) cfgmgr32.dll
    100% (89/89) vs.  93% (28476/30495) winnsi.dll
    100% (89/89) vs.  93% (28478/30495) propsys.dll
    100% (89/89) vs.  94% (28551/30495) nsi.dll
    100% (89/89) vs.  94% (28694/30495) powrprof.dll

some user comments in those crashes certainly fit the problem description we have seen in other support channels as well:

After updating to Firefox 44, I am no longer able to browse any websites even though I still can from another web browser. Starting Firefox in safe mode allows it to reach websites but when I run it normally even with all extensions set to disabled, it still will not load websites. This crash occurred after I had re-enabled all extensions again and then clicked "Restart Now".
https://crash-stats.mozilla.com/report/index/3f21f0f9-5b8a-4f7b-9bf7-a88532160127

Im not happy with your new update... Program opens but will NOT produce web pages - just blank with everything trying to que but nothing happens....
https://crash-stats.mozilla.com/report/index/b76ad885-ec3d-4b0e-97e4-bdf0b2160127

when ever I use private browsing, it doesnt even load my homepage google, and so I go to try and exit out of firefox and ups and crash. Never had this happen before 
https://crash-stats.mozilla.com/report/index/46861140-097a-4952-ba81-eb88b2160128

I downloaded the updated driver from NVidia and have been having problems with Firefox ever since.
https://crash-stats.mozilla.com/report/index/777fe6b2-ba15-4b99-8b5b-f5dcd2160128
Crash Signature: [@ RtlpLowFragHeapFree | RtlFreeHeap | FreeWrapper ] [@ RtlpLowFragHeapFree | RtlFreeHeap | _flsbuf_s ]
those two signatures seem just the tip of the iceberg. starting with firefox 44 there is an upsurge of signatures starting with "RtlpLowFragHeapFree | RtlFreeHeap" (some also with dlls particular to nvidia) - there have been 1500 crashes so far in the first days of 44, whereas over the whole 43 cycle there have been just 190 of them:
https://crash-stats.mozilla.com/search/?signature=%24RtlpLowFragHeapFree+|+RtlFreeHeap&date=%3E2015-06-01&version=44.0&_facets=signature&_facets=version&_facets=app_notes&_facets=user_comments&_facets=uptime&_facets=adapter_vendor_id&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature
What about correlation to nvidia network manager? In particular the people with connectivity problems?
Flags: needinfo?(aklotz)
(In reply to Aaron Klotz [:aklotz] (please use needinfo) from comment #8)
> What about correlation to nvidia network manager? In particular the people
> with connectivity problems?

i don't think that nvidia forceware network access manager plays a part in this issue. this has been a technology from the mid-2000s, whereas the cards mentioned here are much more recent. also the modules from bug 1233237 (nvappfilter.dll/nvlsp.dll) don't show up in this new issue...
See Also: → 1243098
I think the number of crash reports may probably still be grossly understated, see https://bugzilla.mozilla.org/show_bug.cgi?id=1243910

In my case, there won't be any crash reports, since the crash reporter crashes, too.

First, I had not noticed the problem, since Firefox runs the whole day. Only after a reboot could I see that there was a crash in the last session. To find that the problem exists only when Firefox was started from the taskbar was even harder (I tried all plugins and extensions first, to no avail).
do you have nvidia geforce experience installed on your system & if yes does it make a difference if you uninstall it for testing purposes?
Flags: needinfo?(f+bugzilla)
I had GF Experience installed, deinstalled it, rebooted, and the crash pattern stays the same: once on exit, once on next startup. I even switched to the Microsoft Standard VGA driver, to no avail. I doubt that it has to to with the Nvidia card. I have a plain vanilla installation on a Win7 VM, though, which gives no problems.

When I restart in safe mode or when I do not start from the taskbar, everything is O.K., although when I deactivate all plugins and extensions manually, the problem persists.

I have no clue what could be the triggering culprit.
Flags: needinfo?(f+bugzilla)
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → DUPLICATE
(In reply to Kohei Yoshino [:kohei] from comment #13)
> 
> *** This bug has been marked as a duplicate of bug 1243237 ***

DNS failures do not cause crashes. Why was this marked as a dupe?
Flags: needinfo?(kohei.yoshino)
Bug 1244013 Comment 2.
Flags: needinfo?(kohei.yoshino)
I can reproduce this locally.
I've been seeing in this and other bugs a correlation with crashes in the Windows heap.

Milan, would you mind trying the steps in https://pastebin.mozilla.org/8858430 to see if we can get a better crash report? Those steps should cause the Windows heap to crash as soon as a heap validation fails.
Flags: needinfo?(milan)
I'm going to reopen this - as comment 17 seems like the most important thing going on towards resolving the issue.. so tracking here
Blocks: 1243237
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Just wanted to make sure you guys saw https://bugzilla.mozilla.org/show_bug.cgi?id=1243237#c39 - which kinda devolved into a discussion around nvidia weirdness and the dll interceptor. The DNS code (which often seems broken around this bug in addition to the crashes) does do a manual LoadLibraryA() call; I don't know a lot about what the interceptor code does - but it sounds related :)

https://mxr.mozilla.org/mozilla-central/source/netwerk/dns/GetAddrInfo.cpp#115
Flags: needinfo?(ehsan)
Flags: needinfo?(aklotz)
I tried gflags, and the session was slow, as suggested, but - no crash, and I actually got to the site (all my testing is just going to yahoo.com.)
I then turned the flags off, tried again, a few times.  I noticed FlashUtil64_20_0_0_286_ActiveX.exe (which I think is just the update), but also two instances of FlashPlayerPlugin_20_0_0_286.exe.  And on next Firefox exit, everything froze again.  This was the "without" gflags.

In general, my computer completely freezes when Firefox crashes (except under debugger), so things are slow to test.

First Firefox run after the reboot, I still had the "can't get to the site" behaviour.
Flags: needinfo?(milan)
And now I can't reproduce the problem with navigation anymore, except that seeing FlashPlayerPlugin_20_0_0_286.exe in the task manager is a sure sign everything is about to crash.
(In reply to Milan Sreckovic [:milan] from comment #22)
> And now I can't reproduce the problem with navigation anymore, except that
> seeing FlashPlayerPlugin_20_0_0_286.exe in the task manager is a sure sign
> everything is about to crash.

Disabling Flash lets me not crash, but otherwise, I can't reproduce the "can't navigate" problem anymore.
Flags: needinfo?(ehsan)
over in
https://bugzilla.mozilla.org/show_bug.cgi?id=1245948#c8

a reporter tried a try build that disabled (via compiling out) the windows DNS TTL code and seemed to have resolved (or improved?) a similar problem to this. That is a feature that has been basically unchanged since gecko 33 so I don't really suspect its logic specifically.

It does however, as comment 20 says, explicitly load a dll via LoadLibraryA().. I need some help in determining whether or not that could be related. The tea leaves seem to point that way..
Is the DNS code called from the DLL interceptor? If so, it is not a good idea to use LoadLibraryA (or any other non-Unicode Windows API in general). The A to W conversion uses a static buffer in the kernel. See bug 1138070 for details.
(In reply to Masatoshi Kimura [:emk] from comment #25)
> Is the DNS code called from the DLL interceptor? If so, it is not a good
> idea to use LoadLibraryA (or any other non-Unicode Windows API in general).
> The A to W conversion uses a static buffer in the kernel. See bug 1138070
> for details.

We don't intercept LoadLibrary directly; we intercept LdrLoadDll which AFAIK does not call any ANSI APIs. OTOH, we know for a fact that third party DLLs (such as nvidia optimus) *do* intercept LoadLibraryExW (See https://bugzilla.mozilla.org/show_bug.cgi?id=1218473#c39). So if they are calling ANSI APIs from within their hook, it is feasible that this could happen.
Flags: needinfo?(aklotz)
Whiteboard: [44dll]
milan, did the issues on your machine happen within a windows admin account or a restricted account? 
(there are some data points that suggest it may be happening just with the latter type of account, but those are not conclusive yet afaik)
Flags: needinfo?(milan)
Admin account in my case.
Flags: needinfo?(milan)
ok, that was a red herring then :-/
Status: REOPENED → RESOLVED
Closed: 8 years ago → 8 years ago
Resolution: --- → DUPLICATE
Please feel free to delete this post if it isn't helpful.

I'm just a computer tech, but I'm seeing this problem across a lot of PCs (5 Win7 and 3 Vista) so far.  There is no similarity hardware across these PCs.  The only fix I found is an inplace upgrade/repair install in Windows 7 (don't know if that fixes it in Vista yet, but I'm running some tests).

The symptoms I'm seeing are: 
1)Very slow to browse to pages - in some cases, the page will load in 10 to 15 seconds, in others, the page never loads.
2)The firefox.exe process continues to run for a long time after closing Firefox (doesn't matter how I close Firefox).
3)After 30 seconds to two minutes, the process closes and about 90% of the time I get a crash report.

Things I've tried:
1)Running FF in safe mode.
2)Booting Windows in safe.
3)Updating video, NIC, chipset and SATA drivers.
4)SFC /scannow
5)Different DNS servers.
6)Completing removing FF (Uninstall, then remove folders, registry entries) and reinstalling.

Once the problem occurs, going back to an older FF doesn't fix the problem in the single case I tried.

I'm quite happy to try any suggestions here and pass on reports.  I have three Vista PCs and one Win7 right now with the problem.

Here's one of my threads on the issue - please note I posted it from an un-affected PC:
https://support.mozilla.org/en-US/questions/1107092

Thanks,

T
You need to log in before you can comment on or make changes to this bug.