browser startup crashes in nprpffbrowserrecordext.dll or similar with Firefox 40+

VERIFIED FIXED in Firefox 39

Status

()

defect
--
critical
VERIFIED FIXED
4 years ago
4 years ago

People

(Reporter: kairo, Assigned: dmajor)

Tracking

({crash})

unspecified
Firefox 42
x86
Windows NT
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox39+ verified, firefox40+ verified, firefox41+ verified, firefox42 verified)

Details

(crash signature)

Attachments

(1 attachment)

This bug was filed from the Socorro interface and is 
report bp-1fa5d9b8-0719-4161-af1b-f3df02150525.
=============================================================

Top frames:
0 	kernelbase.dll 	RaiseException 	
Ø 1 	nprpffbrowserrecordext.dll 	nprpffbrowserrecordext.dll@0x3d0b 	
Ø 2 	nprpffbrowserrecordext.dll 	nprpffbrowserrecordext.dll@0x2b6a 	
3 	xul.dll 	ffi_call 	
4 	xul.dll 	js::ctypes::FunctionType::Call 	js/src/ctypes/CTypes.cpp
5 	xul.dll 	js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) 	js/src/vm/Interpreter.cpp
6 	xul.dll 	js::CrossCompartmentWrapper::call(JSContext*, JS::Handle<JSObject*>, JS::CallArgs const&) 	js/src/proxy/CrossCompartmentWrapper.cpp
7 	xul.dll 	js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) 	js/src/vm/Interpreter.cpp
8 	xul.dll 	Interpret 	js/src/vm/Interpreter.cpp

We have three different signatures with similar DLL names and stacks that are spiking in 39.0b1, all on browser startup. The reason seems to always be 0xc06d007e / 0x00000000 with a 0x7XXXXXXX address, which might just be common for an uncaught exception (which I guess "RaiseException" at the top of the stack points to).
Even though the library name starts with "np" which usually is plugins, this is loaded in an add-on (I think) via JS-CTypes.

I don't have much more clue as to what's happening there but it sounds like people with this installed are unable to use Firefox at all.

David, do you have any more insight there?
Flags: needinfo?(dmajor)
[Tracking Requested - why for this release]:
Requesting Tracking as those 3 signatures are all in the Top 5 Crash Scores for 39.0b1 atm.
This is another component of the RealPlayer Browser Record Plugin, which has caused us problems before, cf. bug 764210 and bug 1132663.

It seems bug 764210 didn't catch enough versions. Can we expand the block there?

     68% (402/589) vs.   1% (410/29965) {0153E448-190B-4987-BDE1-F256CADA672F} (15.0.6)
     17% (98/589) vs.   0% (99/29965) {97E22097-9A2F-45b1-8DAF-36AD648C7EF4} (15.0.4)
     14% (83/589) vs.   0% (83/29965) {C3949AC2-4B17-43ee-B4F1-D26B9D42404D} (15.0.5)
Flags: needinfo?(dmajor) → needinfo?(jorge)
Yeah, though we should try contacting the devs first. I can do that. As you requesting us to block those 3 IDs?
Flags: needinfo?(jorge) → needinfo?(dmajor)
Yes, if this can't be addressed from Real's side. (Given that some of these are years-old versions, I don't know if there's anything they can do about existing installations)
Flags: needinfo?(dmajor)
And this issue prohibits people from using Firefox Beta and they will probably leave to other channels or browsers, eliminating the test coverage we have on their configurations. Any day this takes to fix means lost users and beta coverage.
Jorge - in light of comment 5, maybe we should block the old versions first, and then follow up with Real about fixing this for future versions.
Depends on: 1170633
The reported versions have been blocked now. Let me know if there are any others causing problems.
Tracking; let's verify that the crashes stop, now that these versions are blocked. 
Jorge do you have time to verify this in a day or two?
Flags: qe-verify+
Flags: needinfo?(jorge)
I think Kairo would know how to check on those crashes better than I do.

Quick update from RealNetworks: they responded to me and indicated that those extensions are a legacy product but still have a substantial install base, meaning that they should be okay to block but it's still a good idea to give them a heads up. They are now aware of the deployed blocks.
Flags: needinfo?(jorge) → needinfo?(kairo)
Because they're at startup, those crashes will not stop in an easily detectable way just because we deployed a blocklist on AMO. People who are already seeing the issue will continue to crash until they either disable the add-on, download a new blocklist in safe mode (I hope safe mode does that), or install a build that has that blocklist in the package (39.0b3 and higher should have that, I think, but will only ship on Friday).
Right now, only people who got an updated blocklist *before* updating to 39.0 beta should be safe from the crashes.

I will monitor crash data (as always), but it will be a number of days until we see a significant effect in data, i.e. until those people who already crash will see any help (but we at least helped those that haven't run into this yet).
The signatures are still around and being flagged as issues even in 39.0b3 - I thought a blocklist update would have been shipped in-product as well there? Are we sure that blocking the add-on is stopping the DLLs to be loaded into the Firefox process?
Flags: needinfo?(dmajor)
It seems these are still getting loaded on 39b3: https://crash-analysis.mozilla.com/crash_analysis/20150607/20150607_Firefox_39.0b3-interesting-addons-with-versions.txt.gz

  nprpffbrowserrecordext.dll@0x3d0b|0xc06d007e / 0x00000000 (147 crashes)
     55% (81/147) vs.   0% (82/22645) {0153E448-190B-4987-BDE1-F256CADA672F} (15.0.6)
     31% (45/147) vs.   0% (45/22645) {97E22097-9A2F-45b1-8DAF-36AD648C7EF4} (15.0.4)
     13% (19/147) vs.   0% (19/22645) {C3949AC2-4B17-43ee-B4F1-D26B9D42404D} (15.0.5)

Along with some older versions that we didn't see before:

  nprndlffbrowserrecordext.dll@0xf3a4|0xc06d007e / 0x00000000 (163 crashes)
     43% (70/163) vs.   0% (70/22645) {FCE04E1F-9378-4f39-96F6-5689A9159E45} (1.3.2)
     36% (58/163) vs.   0% (58/22645) {34712C68-7391-4c47-94F3-8F88D49AD632} (1.3.0)
     20% (33/163) vs.   0% (33/22645) {DAC3F861-B30D-40dd-9166-F4E75327FAC7} (1.3.1)

  nprndlffbrowserrecordext.dll@0xf394|0xc06d007e / 0x00000000 (24 crashes)
    100% (24/24) vs.   0% (24/22645) {B1FC07E1-E05B-4567-8891-E63FBE545BA8} (1.2.0)
Could you do another block for the 1.x versions in comment 12? Sorry for the extra round of churn.
Flags: needinfo?(dmajor) → needinfo?(jorge)
If the DLLs are still being loaded with the blocks in place, is there any real benefit in blocking those older versions? Shouldn't someone first verify that the DLLs are part of the extensions?
Flags: needinfo?(jorge)
According to bug 1172560 the blocklist didn't make beta 3.
Okay, I'll get those blocks in then.
Depends on: 1173154
The blocklisting in bug 1170633 was shipped in 39.0b4, but this is still appearing the the top startup crash and top issue in Top Crash Scores. As bug 639524 has been solved long ago, we actually should take the shipped blocklist into account when people have an older blocklist in their profile (because they can't launch Firefox after all). Are the remaining crashes all with the versions that bug 1173154 fixes and that we haven't shipped in 39 yet (will only be updated in builds created after tomorrow due to bug 1172560)?
Flags: needinfo?(kairo)
We don't yet have correlation files for b4 but from a hand-sampling I'm still seeing bug 1170633's versions showing up (in addition to bug 1173154 which has not yet merged):

 	{97E22097-9A2F-45b1-8DAF-36AD648C7EF4} 	15.0.4
 	{C3949AC2-4B17-43ee-B4F1-D26B9D42404D} 	15.0.5 
 	{0153E448-190B-4987-BDE1-F256CADA672F} 	15.0.6

Are we certain that beta 4 blocks the above versions? Are there any known circumstances where the block doesn't work? Do we have a way to tell how many people re-enable the addon anyway?
Flags: needinfo?(jorge)
Flags: needinfo?(benjamin)
I can see those blocklist items in my local profile, which means they are at least being downloaded correctly. I don't see why they would fail, either. As for figuring out if they are being enabled again, it's possible to get that data from FHR, I think. I doubt most users who have it blocked would enable it again, since they generally don't know how to do that.
Flags: needinfo?(jorge)
What kind of block did we deploy? If it's a click-to-play block, users can still use the in-browser UI to enable the plugin on specific sites. Or is this plugin part of an extension? If so the extension could be activating the plugin in code.

Do we have samples of this software to actually test with, or is this just speculative blocking?
Flags: needinfo?(benjamin) → needinfo?(jorge)
It is speculative. I haven't been able to find a copy. The last known version has a timestamp of July 2012.
Speculative, and only extension blocks, no plugin blocks. I have a contact at Real Networks who might be able to help us if we have any questions about this.
Flags: needinfo?(jorge)
We can't ship 39 with this bug, and we're running out of runway. I'm considering preparing a DLL blocking patch as a last resort.

Jorge, if your contact at Real has any idea why this extension might be avoiding the blocklist, I'd be interested in hearing it. Or if they could provide an archived 15.0.x version of Browser Record Plugin, I could try debugging locally.
Flags: needinfo?(jorge)
Okay, I'm asking them for a copy of these older versions of their software.
Flags: needinfo?(jorge)
Thanks David and Jorge. For this to go into Beta 7 we would need to uplift by tomorrow night (or very early Thursday morning, since I'd like to go to build as early as possible on Thursday).
Here's what I got from them. Let me know if that's enough.

> I have a URL here to RealPlayer 15 which was last updated in 2012:
> http://realplayer-download.real.com/free/support/win/15/RealPlayer.exe
> This is version 15.0.6.14
This was my experience:

1. Install Firefox 38.0.5
2. Install RealPlayer 15 from comment 26 (check the box to enable the downloader button)
3. Start Firefox, accept the add-on install, restart
4. Confirmed with a debugger that Real's DLL is loaded
5. Upgrade Firefox installation to 39.0b6
6. Start Firefox successfully. The extension is disabled in Add-ons Manager.
7. Re-enable the extension, restart
8. Perma-startup-crash
I figured out why the extension is crashing. It's a delayload failure due to missing mozalloc.dll. This comes from bug 868814.

Bug 1173683 is going to back out that change from 39. The patch just hit mozilla-beta and I'll try the tinderbox build when it's ready. We'll still need to figure out what to do for 40 though.

Maybe this bug is enough reason not to drop mozalloc? Otherwise I think we'd have to hard-block this extension since there is no hope of a successful browser launch. Opinions?
Blocks: 868814
Depends on: 1173683
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(benjamin)
We're dropping support for binary XPCOM for exactly this kind of reason. Hard-block away.
Flags: needinfo?(benjamin)
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #29)
> We're dropping support for binary XPCOM for exactly this kind of reason.
> Hard-block away.

Except this one is using ctypes.
Flags: needinfo?(mh+mozilla)
The browser launches successfully with this plugin active using https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=37c99d72de9a
And confirmed by crash-stats. This is not seen in 39.0b7. We'll still need to figure out what to do for 40.
And, as expected, it's back in 40 Beta 1.
Glandium, just want to double-check: is the plan still to drop mozalloc.dll in 40?
Flags: needinfo?(mh+mozilla)
Yes, 40 still has no mozalloc.dll.
Flags: needinfo?(mh+mozilla)
Well that makes the path pretty clear. These addons have no chance of launching successfully in version 40. We'll need to hard-block the IDs in comment 12.
Flags: needinfo?(jorge)
Summary: browser startup crashes in nprpffbrowserrecordext.dll or similar with Firefox 39+ → browser startup crashes in nprpffbrowserrecordext.dll or similar with Firefox 40+
The IDs in comment 12 have already been blocked in bug 1170633 and bug 1173154. I can extend the blocks to all versions of those IDs and change it to a hard-block, but that shouldn't make any significant difference in the blocking or crashing rates.
Flags: needinfo?(jorge)
(In reply to Jorge Villalobos [:jorgev] from comment #37)
> The IDs in comment 12 have already been blocked in bug 1170633 and bug
> 1173154. I can extend the blocks to all versions of those IDs and change it
> to a hard-block, but that shouldn't make any significant difference in the
> blocking or crashing rates.

Yes, let's try that please. If even a hard-block doesn't work, it would be very useful to have that information.
All blocks have been updated now. I'll reiterate that it's likely the problem is elsewhere. Maybe the DLLs are being loaded independently from the add-on.
Thanks. I understand the skepticism. In the cases that I've seen so far, the addon was also loaded, not just a standalone DLL. That rather scares me since it suggests a problem with the blocking (I don't think so many people are re-enabling). I'd like to get concrete proof, if that really is the case.

And if I'm simply wrong then we can DLL-block in the worst case.
40 beta 4 is still crashing, both in the wild and on my machine. Here's the change log for beta 4's blocklist: https://hg.mozilla.org/releases/mozilla-beta/log/664b98d33a1b/browser/app/blocklist.xml

There are no changes after the date of comment 39, even though there was supposed to be an update on the 11th if I understand correctly that they run on Saturdays.

Any idea what's going on?
Flags: needinfo?(jorge)
I can confirm the blocks were updated (on my current Nightly profile). I don't know how the tree update process works. That would be a question for RelEng.
Flags: needinfo?(jorge) → needinfo?(catlee)
I see now that the changes arrived to beta on the 14th: https://hg.mozilla.org/releases/mozilla-beta/log/tip/browser/app/blocklist.xml. I'll test in 40b5.
Yeah, it generally gets updated on Saturdays, but it failed on the 11th. It was re-run on the 14th and passed.
Flags: needinfo?(catlee)
I've tested 40b6 locally and it blocks the Real addon both for signing reasons and due to a hard-block (red text). My expectation is that this signature should go away once we get data on 40b6 in the wild. If not, something is very wrong and we'll need to debug.
But these are still happening in the wild: bp-68e4d262-8dc7-40ba-bf7c-cf2152150722

The volume is very low, but we've been misled by that before. It's possible that many of the affected users have left beta. We may want to add a DLL-block as a precaution.
Posted patch DLL blockSplinter Review
Reviewer TBD. To summarize: this is a discontinued product that has no hope of loading correctly, due to a dependency on mozalloc.dll that we no longer ship in 40.
Attachment #8637432 - Flags: review?(ehsan)
Attachment #8637432 - Flags: review?(ehsan) → review+
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/c442588b5ab5
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 42
Comment on attachment 8637432 [details] [diff] [review]
DLL block

Approval Request Comment
[Feature/regressing bug #]: bug 868814
[User impact if declined]: Some users may still crash despite the addon-block. This adds an extra layer of defense.
[Describe test coverage new/current, TreeHerder]: none
[Risks and why]: should be low risk as we've already blocked most of these
[String/UUID change made/needed]: none
Attachment #8637432 - Flags: approval-mozilla-beta?
Attachment #8637432 - Flags: approval-mozilla-aurora?
Comment on attachment 8637432 [details] [diff] [review]
DLL block

DLL block. Approved.
Attachment #8637432 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Comment on attachment 8637432 [details] [diff] [review]
DLL block

Beta+ as well
Attachment #8637432 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Assignee: nobody → dmajor
Unable to reproduce the crash using STR from comment 27 - the RealPlayer Browser Record Plugin is available via Add-ons Manager/Extensions tab, but is properly blocked after enabling it and restarting/upgrading browser; and in RealPlayer App Preferences the 'Download this Video' button is enabled only for IE, unable to activate it for Firefox. Tested with upgrades from 38.0.5 and 39.0b, under Windows 7 64-bit, Windows 8 32-bit and Windows 10 64-bit. 

Any ideas if I am missing something or is it expected based on the changes here?

Also, crash reports number dropped in the last 3 days for all 3 signatures:
* [@ nprpffbrowserrecordext.dll@0x3d0b]: https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=3&signature=nprpffbrowserrecordext.dll%400x3d0b
* [@ nprndlffbrowserrecordext.dll@0xf3a4]: https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=3&signature=nprndlffbrowserrecordext.dll%400xf3a4
* [@ nprndlffbrowserrecordext.dll@0xf394]: https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=3&signature=nprndlffbrowserrecordext.dll%400xf394

Although, we will keep an eye on Socorro to monitor the crash reports.
Flags: needinfo?(dmajor)
That sounds ok. I couldn't reproduce the remaining crashes either.
Flags: needinfo?(dmajor)
(In reply to David Major [:dmajor] from comment #56)
> That sounds ok. I couldn't reproduce the remaining crashes either.

Thanks for confirming. No new reports via Socorro in the last week for any Fx build (39, 40, 41 and 42):
* [@ nprpffbrowserrecordext.dll@0x3d0b]: https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=7&signature=nprpffbrowserrecordext.dll%400x3d0b#tab-reports
* [@ nprndlffbrowserrecordext.dll@0xf3a4]: https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=7&signature=nprndlffbrowserrecordext.dll%400xf3a4#tab-reports
* [@ nprndlffbrowserrecordext.dll@0xf394]: https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=7&signature=nprndlffbrowserrecordext.dll%400xf394#tab-reports

Marking accordingly based on this results.
Status: RESOLVED → VERIFIED
Flags: qe-verify+
You need to log in before you can comment on or make changes to this bug.