1170141 - browser startup crashes in nprpffbrowserrecordext.dll or similar with Firefox 40+

Status: VERIFIED FIXED

(Firefox :: General, defect)

Description

[Robert Kaiser]

This bug was filed from the Socorro interface and is 
report bp-1fa5d9b8-0719-4161-af1b-f3df02150525.
=============================================================

Top frames:
0 	kernelbase.dll 	RaiseException 	
Ø 1 	nprpffbrowserrecordext.dll 	nprpffbrowserrecordext.dll@0x3d0b 	
Ø 2 	nprpffbrowserrecordext.dll 	nprpffbrowserrecordext.dll@0x2b6a 	
3 	xul.dll 	ffi_call 	
4 	xul.dll 	js::ctypes::FunctionType::Call 	js/src/ctypes/CTypes.cpp
5 	xul.dll 	js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) 	js/src/vm/Interpreter.cpp
6 	xul.dll 	js::CrossCompartmentWrapper::call(JSContext*, JS::Handle<JSObject*>, JS::CallArgs const&) 	js/src/proxy/CrossCompartmentWrapper.cpp
7 	xul.dll 	js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) 	js/src/vm/Interpreter.cpp
8 	xul.dll 	Interpret 	js/src/vm/Interpreter.cpp

We have three different signatures with similar DLL names and stacks that are spiking in 39.0b1, all on browser startup. The reason seems to always be 0xc06d007e / 0x00000000 with a 0x7XXXXXXX address, which might just be common for an uncaught exception (which I guess "RaiseException" at the top of the stack points to).
Even though the library name starts with "np" which usually is plugins, this is loaded in an add-on (I think) via JS-CTypes.

I don't have much more clue as to what's happening there but it sounds like people with this installed are unable to use Firefox at all.

David, do you have any more insight there?

Comment 1

[Robert Kaiser]

[Tracking Requested - why for this release]:
Requesting Tracking as those 3 signatures are all in the Top 5 Crash Scores for 39.0b1 atm.

Comment 2

[(Away)]

This is another component of the RealPlayer Browser Record Plugin, which has caused us problems before, cf. bug 764210 and bug 1132663.

It seems bug 764210 didn't catch enough versions. Can we expand the block there?

     68% (402/589) vs.   1% (410/29965) {0153E448-190B-4987-BDE1-F256CADA672F} (15.0.6)
     17% (98/589) vs.   0% (99/29965) {97E22097-9A2F-45b1-8DAF-36AD648C7EF4} (15.0.4)
     14% (83/589) vs.   0% (83/29965) {C3949AC2-4B17-43ee-B4F1-D26B9D42404D} (15.0.5)

Comment 3

[Jorge Villalobos [:jorgev] (he/him)]

Yeah, though we should try contacting the devs first. I can do that. As you requesting us to block those 3 IDs?

Comment 4

[(Away)]

Yes, if this can't be addressed from Real's side. (Given that some of these are years-old versions, I don't know if there's anything they can do about existing installations)

Comment 5

[Robert Kaiser]

And this issue prohibits people from using Firefox Beta and they will probably leave to other channels or browsers, eliminating the test coverage we have on their configurations. Any day this takes to fix means lost users and beta coverage.

Comment 6

[(Away)]

Jorge - in light of comment 5, maybe we should block the old versions first, and then follow up with Real about fixing this for future versions.

Comment 7

[Jorge Villalobos [:jorgev] (he/him)]

The reported versions have been blocked now. Let me know if there are any others causing problems.

Comment 8

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking; let's verify that the crashes stop, now that these versions are blocked. 
Jorge do you have time to verify this in a day or two?

Comment 9

[Jorge Villalobos [:jorgev] (he/him)]

I think Kairo would know how to check on those crashes better than I do.

Quick update from RealNetworks: they responded to me and indicated that those extensions are a legacy product but still have a substantial install base, meaning that they should be okay to block but it's still a good idea to give them a heads up. They are now aware of the deployed blocks.

Comment 10

[Robert Kaiser]

Because they're at startup, those crashes will not stop in an easily detectable way just because we deployed a blocklist on AMO. People who are already seeing the issue will continue to crash until they either disable the add-on, download a new blocklist in safe mode (I hope safe mode does that), or install a build that has that blocklist in the package (39.0b3 and higher should have that, I think, but will only ship on Friday).
Right now, only people who got an updated blocklist *before* updating to 39.0 beta should be safe from the crashes.

I will monitor crash data (as always), but it will be a number of days until we see a significant effect in data, i.e. until those people who already crash will see any help (but we at least helped those that haven't run into this yet).

Comment 11

[Robert Kaiser]

The signatures are still around and being flagged as issues even in 39.0b3 - I thought a blocklist update would have been shipped in-product as well there? Are we sure that blocking the add-on is stopping the DLLs to be loaded into the Firefox process?

Comment 12

[(Away)]

It seems these are still getting loaded on 39b3: https://crash-analysis.mozilla.com/crash_analysis/20150607/20150607_Firefox_39.0b3-interesting-addons-with-versions.txt.gz

  nprpffbrowserrecordext.dll@0x3d0b|0xc06d007e / 0x00000000 (147 crashes)
     55% (81/147) vs.   0% (82/22645) {0153E448-190B-4987-BDE1-F256CADA672F} (15.0.6)
     31% (45/147) vs.   0% (45/22645) {97E22097-9A2F-45b1-8DAF-36AD648C7EF4} (15.0.4)
     13% (19/147) vs.   0% (19/22645) {C3949AC2-4B17-43ee-B4F1-D26B9D42404D} (15.0.5)

Along with some older versions that we didn't see before:

  nprndlffbrowserrecordext.dll@0xf3a4|0xc06d007e / 0x00000000 (163 crashes)
     43% (70/163) vs.   0% (70/22645) {FCE04E1F-9378-4f39-96F6-5689A9159E45} (1.3.2)
     36% (58/163) vs.   0% (58/22645) {34712C68-7391-4c47-94F3-8F88D49AD632} (1.3.0)
     20% (33/163) vs.   0% (33/22645) {DAC3F861-B30D-40dd-9166-F4E75327FAC7} (1.3.1)

  nprndlffbrowserrecordext.dll@0xf394|0xc06d007e / 0x00000000 (24 crashes)
    100% (24/24) vs.   0% (24/22645) {B1FC07E1-E05B-4567-8891-E63FBE545BA8} (1.2.0)

Comment 13

[(Away)]

Could you do another block for the 1.x versions in comment 12? Sorry for the extra round of churn.

Comment 14

[Jorge Villalobos [:jorgev] (he/him)]

If the DLLs are still being loaded with the blocks in place, is there any real benefit in blocking those older versions? Shouldn't someone first verify that the DLLs are part of the extensions?

Comment 15

[(Away)]

According to bug 1172560 the blocklist didn't make beta 3.

Comment 16

[Jorge Villalobos [:jorgev] (he/him)]

Okay, I'll get those blocks in then.

Comment 17

[Robert Kaiser]

The blocklisting in bug 1170633 was shipped in 39.0b4, but this is still appearing the the top startup crash and top issue in Top Crash Scores. As bug 639524 has been solved long ago, we actually should take the shipped blocklist into account when people have an older blocklist in their profile (because they can't launch Firefox after all). Are the remaining crashes all with the versions that bug 1173154 fixes and that we haven't shipped in 39 yet (will only be updated in builds created after tomorrow due to bug 1172560)?

Comment 18

[(Away)]

We don't yet have correlation files for b4 but from a hand-sampling I'm still seeing bug 1170633's versions showing up (in addition to bug 1173154 which has not yet merged):

 	{97E22097-9A2F-45b1-8DAF-36AD648C7EF4} 	15.0.4
 	{C3949AC2-4B17-43ee-B4F1-D26B9D42404D} 	15.0.5 
 	{0153E448-190B-4987-BDE1-F256CADA672F} 	15.0.6

Are we certain that beta 4 blocks the above versions? Are there any known circumstances where the block doesn't work? Do we have a way to tell how many people re-enable the addon anyway?

Comment 19

[Jorge Villalobos [:jorgev] (he/him)]

I can see those blocklist items in my local profile, which means they are at least being downloaded correctly. I don't see why they would fail, either. As for figuring out if they are being enabled again, it's possible to get that data from FHR, I think. I doubt most users who have it blocked would enable it again, since they generally don't know how to do that.

Comment 20

[Benjamin Smedberg]

What kind of block did we deploy? If it's a click-to-play block, users can still use the in-browser UI to enable the plugin on specific sites. Or is this plugin part of an extension? If so the extension could be activating the plugin in code.

Do we have samples of this software to actually test with, or is this just speculative blocking?

Comment 21

[(Away)]

It is speculative. I haven't been able to find a copy. The last known version has a timestamp of July 2012.

Comment 22

[Jorge Villalobos [:jorgev] (he/him)]

Speculative, and only extension blocks, no plugin blocks. I have a contact at Real Networks who might be able to help us if we have any questions about this.

Comment 23

[(Away)]

We can't ship 39 with this bug, and we're running out of runway. I'm considering preparing a DLL blocking patch as a last resort.

Jorge, if your contact at Real has any idea why this extension might be avoiding the blocklist, I'd be interested in hearing it. Or if they could provide an archived 15.0.x version of Browser Record Plugin, I could try debugging locally.

Comment 24

[Jorge Villalobos [:jorgev] (he/him)]

Okay, I'm asking them for a copy of these older versions of their software.

Comment 25

[Liz Henry (:lizzard) (relman/hg->git project)]

Thanks David and Jorge. For this to go into Beta 7 we would need to uplift by tomorrow night (or very early Thursday morning, since I'd like to go to build as early as possible on Thursday).

Comment 26

[Jorge Villalobos [:jorgev] (he/him)]

Here's what I got from them. Let me know if that's enough.

> I have a URL here to RealPlayer 15 which was last updated in 2012:
> http://realplayer-download.real.com/free/support/win/15/RealPlayer.exe
> This is version 15.0.6.14

Comment 27

[(Away)]

This was my experience:

1. Install Firefox 38.0.5
2. Install RealPlayer 15 from comment 26 (check the box to enable the downloader button)
3. Start Firefox, accept the add-on install, restart
4. Confirmed with a debugger that Real's DLL is loaded
5. Upgrade Firefox installation to 39.0b6
6. Start Firefox successfully. The extension is disabled in Add-ons Manager.
7. Re-enable the extension, restart
8. Perma-startup-crash

Comment 28

[(Away)]

I figured out why the extension is crashing. It's a delayload failure due to missing mozalloc.dll. This comes from bug 868814.

Bug 1173683 is going to back out that change from 39. The patch just hit mozilla-beta and I'll try the tinderbox build when it's ready. We'll still need to figure out what to do for 40 though.

Maybe this bug is enough reason not to drop mozalloc? Otherwise I think we'd have to hard-block this extension since there is no hope of a successful browser launch. Opinions?

Comment 29

[Benjamin Smedberg]

We're dropping support for binary XPCOM for exactly this kind of reason. Hard-block away.

Comment 30

[Mike Hommey [:glandium]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #29)
> We're dropping support for binary XPCOM for exactly this kind of reason.
> Hard-block away.

Except this one is using ctypes.

Comment 31

[(Away)]

The browser launches successfully with this plugin active using https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=37c99d72de9a

Comment 32

[(Away)]

And confirmed by crash-stats. This is not seen in 39.0b7. We'll still need to figure out what to do for 40.

Comment 33

[Robert Kaiser]

And, as expected, it's back in 40 Beta 1.

Comment 34

[(Away)]

Glandium, just want to double-check: is the plan still to drop mozalloc.dll in 40?

Comment 35

[Mike Hommey [:glandium]]

Yes, 40 still has no mozalloc.dll.

Comment 36

[(Away)]

Well that makes the path pretty clear. These addons have no chance of launching successfully in version 40. We'll need to hard-block the IDs in comment 12.

Comment 37

[Jorge Villalobos [:jorgev] (he/him)]

The IDs in comment 12 have already been blocked in bug 1170633 and bug 1173154. I can extend the blocks to all versions of those IDs and change it to a hard-block, but that shouldn't make any significant difference in the blocking or crashing rates.

Comment 38

[(Away)]

(In reply to Jorge Villalobos [:jorgev] from comment #37)
> The IDs in comment 12 have already been blocked in bug 1170633 and bug
> 1173154. I can extend the blocks to all versions of those IDs and change it
> to a hard-block, but that shouldn't make any significant difference in the
> blocking or crashing rates.

Yes, let's try that please. If even a hard-block doesn't work, it would be very useful to have that information.

Comment 39

[Jorge Villalobos [:jorgev] (he/him)]

All blocks have been updated now. I'll reiterate that it's likely the problem is elsewhere. Maybe the DLLs are being loaded independently from the add-on.

Comment 40

[(Away)]

Thanks. I understand the skepticism. In the cases that I've seen so far, the addon was also loaded, not just a standalone DLL. That rather scares me since it suggests a problem with the blocking (I don't think so many people are re-enabling). I'd like to get concrete proof, if that really is the case.

And if I'm simply wrong then we can DLL-block in the worst case.

Comment 41

[(Away)]

40 beta 4 is still crashing, both in the wild and on my machine. Here's the change log for beta 4's blocklist: https://hg.mozilla.org/releases/mozilla-beta/log/664b98d33a1b/browser/app/blocklist.xml

There are no changes after the date of comment 39, even though there was supposed to be an update on the 11th if I understand correctly that they run on Saturdays.

Any idea what's going on?

Comment 42

[Jorge Villalobos [:jorgev] (he/him)]

I can confirm the blocks were updated (on my current Nightly profile). I don't know how the tree update process works. That would be a question for RelEng.

Comment 43

[(Away)]

I see now that the changes arrived to beta on the 14th: https://hg.mozilla.org/releases/mozilla-beta/log/tip/browser/app/blocklist.xml. I'll test in 40b5.

Comment 44

[Chris AtLee [:catlee]]

Yeah, it generally gets updated on Saturdays, but it failed on the 11th. It was re-run on the 14th and passed.

Comment 45

[(Away)]

I've tested 40b6 locally and it blocks the Real addon both for signing reasons and due to a hard-block (red text). My expectation is that this signature should go away once we get data on 40b6 in the wild. If not, something is very wrong and we'll need to debug.

Comment 46

[(Away)]

But these are still happening in the wild: bp-68e4d262-8dc7-40ba-bf7c-cf2152150722

The volume is very low, but we've been misled by that before. It's possible that many of the affected users have left beta. We may want to add a DLL-block as a precaution.

Comment 47

[(Away)]

Attachment: DLL block

Reviewer TBD. To summarize: this is a discontinued product that has no hope of loading correctly, due to a dependency on mozalloc.dll that we no longer ship in 40.

Comment 48

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/c442588b5ab5

Comment 49

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/c442588b5ab5

Comment 50

[(Away)]

Comment on attachment 8637432 [details] [diff] [review]
DLL block

Approval Request Comment
[Feature/regressing bug #]: bug 868814
[User impact if declined]: Some users may still crash despite the addon-block. This adds an extra layer of defense.
[Describe test coverage new/current, TreeHerder]: none
[Risks and why]: should be low risk as we've already blocked most of these
[String/UUID change made/needed]: none

Comment 51

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8637432 [details] [diff] [review]
DLL block

DLL block. Approved.

Comment 52

[Lawrence Mandel [:lmandel] (use needinfo)]

Comment on attachment 8637432 [details] [diff] [review]
DLL block

Beta+ as well

Comment 53

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/4c6a13a000f9

Comment 54

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/784d30d6aaa1

Comment 55

[Ada [:adalucinet]]

Unable to reproduce the crash using STR from comment 27 - the RealPlayer Browser Record Plugin is available via Add-ons Manager/Extensions tab, but is properly blocked after enabling it and restarting/upgrading browser; and in RealPlayer App Preferences the 'Download this Video' button is enabled only for IE, unable to activate it for Firefox. Tested with upgrades from 38.0.5 and 39.0b, under Windows 7 64-bit, Windows 8 32-bit and Windows 10 64-bit. 

Any ideas if I am missing something or is it expected based on the changes here?

Also, crash reports number dropped in the last 3 days for all 3 signatures:
* [@ nprpffbrowserrecordext.dll@0x3d0b]: https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=3&signature=nprpffbrowserrecordext.dll%400x3d0b
* [@ nprndlffbrowserrecordext.dll@0xf3a4]: https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=3&signature=nprndlffbrowserrecordext.dll%400xf3a4
* [@ nprndlffbrowserrecordext.dll@0xf394]: https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=3&signature=nprndlffbrowserrecordext.dll%400xf394

Although, we will keep an eye on Socorro to monitor the crash reports.

Comment 56

[(Away)]

That sounds ok. I couldn't reproduce the remaining crashes either.

Comment 57

[Ada [:adalucinet]]

(In reply to David Major [:dmajor] from comment #56)
> That sounds ok. I couldn't reproduce the remaining crashes either.

Thanks for confirming. No new reports via Socorro in the last week for any Fx build (39, 40, 41 and 42):
* [@ nprpffbrowserrecordext.dll@0x3d0b]: https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=7&signature=nprpffbrowserrecordext.dll%400x3d0b#tab-reports
* [@ nprndlffbrowserrecordext.dll@0xf3a4]: https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=7&signature=nprndlffbrowserrecordext.dll%400xf3a4#tab-reports
* [@ nprndlffbrowserrecordext.dll@0xf394]: https://crash-stats.mozilla.com/report/list?range_unit=days&range_value=7&signature=nprndlffbrowserrecordext.dll%400xf394#tab-reports

Marking accordingly based on this results.