1360029 - Crash in TppCallbackCheckThreadAfterCallback

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect)

Description

[Christian Moser]

This bug was filed from the Socorro interface and is 
report bp-f1093f9b-f73a-4816-b428-91bbc0170426.
=============================================================

Hi,

This happens right after starting Firefox. The current Tab crashes, the rest of the GUI is still usable. New Tabs crashes too.

I already tried with no success:
- Safe mode, crash occours there too
- Uninstalled Firefox and removed all Files from %Programfiles%, from Profiles and all Registry entries, then reinstalled Firefox.
- DISM / Online /Scanhealth -> no results
- sfc /scannow -> no results
- tried 64bit version -> same behaviour
- Deleted entire Windows Userprofile, also happens with a new userprofile
- Installed all Windows Updates
- Disabled Antivirus

As Administrator, Firefox works without crashing.

Let me know if I can provide any other helpful information.

some more crash reports:
bp-9251ae6e-8dbc-4b0d-bb94-bad1e0170426
bp-b438ad2f-b490-4b24-a9ea-41fcc0170426
bp-b7a93e85-3b92-4582-b7f3-b8a170170426
bp-9148a552-2380-482d-aac4-00a690170426
bp-5040fbf1-fce4-410d-8e1f-1926d0170426
bp-f1093f9b-f73a-4816-b428-91bbc0170426

Comment 1

[:Gijs (he/him)]

Reporter, am I right in deducing from the stack of thread 0 (not the crashing thread) that your user directory is on a network drive of some sort? (If you could elaborate on exactly what the setup is that might be helpful.)

Does the same crash happen if you test with nightly? ( https://nightly.mozilla.org/ )
In Nightly, in the hamburger menu, there should be a "non-e10s window" option. If you use that, and in that window go to about:config, and change security.sandbox.content.level to 0, then restart the browser, does that help? (If not, do reset it / swap to a new Firefox profile ( https://support.mozilla.org/kb/profile-manager-create-and-remove-firefox-profiles ) as this is security-related).

Bob, is there any chance this crash is due to sandboxing? It looks like we go through http bits of Windows' networking stack to find the user's homedir when opening content tabs that then crash immediately (based on comment #0 and the stack in e.g. https://crash-stats.mozilla.com/report/index/b438ad2f-b490-4b24-a9ea-41fcc0170426#tab-details (thread 0)).

Comment 2

[Bob Owen (:bobowen)]

(In reply to :Gijs from comment #1)

> Bob, is there any chance this crash is due to sandboxing? It looks like we
> go through http bits of Windows' networking stack to find the user's homedir
> when opening content tabs that then crash immediately (based on comment #0
> and the stack in e.g.
> https://crash-stats.mozilla.com/report/index/b438ad2f-b490-4b24-a9ea-
> 41fcc0170426#tab-details (thread 0)).

Possibly, although there are some earlier cases (probably unrelated), it mainly seems to have started in Fx50.
I landed an update to the sandbox at the end of March in Firefox 55, maybe we're seeing that increase because of that.
Hmm the majority of that spiking is in Fx54, so probably not, looks like it is from a relatively small number of installations as well. It isn't surprising that we'd get a lot of duplicate crash reports given the symptoms.

It's not clear to me why running as Administrator would work, as we set the Admin SIDs to deny only in the access token, but there might be other weird things within Windows security that it is affecting.
Generally, of course, running Firefox (or for that matter any web browser) as Administrator is not a good idea.

Christian - hopefully about:config would load even on the release version as it loads in the parent, then you could try setting the pref security.sandbox.content.level to 0 to prove it.

Also - did you have the same problem with Firefox 52?

Comment 3

[Bob Owen (:bobowen)]

The error there (0xC0000710) is STATUS_CALLBACK_RETURNED_WHILE_IMPERSONATING
	
"A thread pool worker thread is impersonating a client, after a callback to 0x%p(0x%p). This is unexpected, indicating that the callback is missing a call to revert the impersonation."

The sandbox uses an impersonation token during start-up to run with more privileges before the sandbox is lowered, so maybe that's interfering with something.

Comment 4

[Christian Moser]

Hi guys,

I can confirm that setting security.sandbox.content.level to 0 prevents the crashing.
After that, I installed Nightly and set security.sandbox.content.level again to 1 and there is also no more crashing.

The Users Homedir is located on an UNC Network path (this is common when using folder redirection), it looks like this:

Profilepath: c:\users\xxx
Home/Documents: \\server\profiles\xxx\Documents
Downloads: \\server\profiles\xxx\Downloads
Appdata Roaming: \\server\profiles\xxx\Appdata\Roaming
Appdata Local: c:\users\xxx\Appdata\Local

The Problem occured after Upgrading to 53. I can't tell what Version it was before.

> looks like it is from a relatively small number of installations as well
...maybe the problem occours only on domain controllers (which it is in my case)

Comment 5

[Bob Owen (:bobowen)]

Thanks for confirming that.

Interesting that is works in Nightly.
Any chance you could use mozregression (http://mozilla.github.io/mozregression/) to find what fixed it?

The only thing I can think of off the top my head is an update to the chromium sandbox, unfortunately I don't think we'd want to uplift that.
If I can reproduce and that does fix it, maybe I could track it down to a smaller change.

Comment 6

[Christian Moser]

I tested a few Versions with mozregression, until i noticed that mozregression starts all builds with security.sandbox.content.level set to 2. 
So I tested my Version 53 again with security.sandbox.content.level = 2 -> no crashes! Set to 1 again, crashes apear again.

Okay, so I ran mozregression again, but now with security.sandbox.content.level set to 1 and using the same profile which I used to reproduce the crash. Unfortunately, I was not able to reproduce the problem. It seems that running Builds with mozregression doesn't cause the crash.

Comment 7

[Bob Owen (:bobowen)]

(In reply to Christian Moser from comment #6)
> I tested a few Versions with mozregression, until i noticed that
> mozregression starts all builds with security.sandbox.content.level set to
> 2. 
> So I tested my Version 53 again with security.sandbox.content.level = 2 ->
> no crashes! Set to 1 again, crashes apear again.

OK thanks for trying.
The fact that it works with level 2, makes me think that it is to do with the fact it uses restricted tokens whereas level 1 doesn't. I've seen something similar in the past, which also involved impersonation.
So, I'll look into reproducing when I get a chance.

Using level 2 on release, will quite probably be OK as a workaround for you for the moment.
The only issue that I can think of that you might get is if you try to open file:// URI pages from a network share.

Comment 8

[Andrew McCreight [:mccr8]]

#3 crash in the 5-13 Windows Nightly. 431 across all of Nightly in the last 7 days, though from only 12 installations.

Comment 9

[Bob Owen (:bobowen)]

Not long got back to this, but I'm having trouble reproducing, probably because I don't have a proper AD set-up.

Christian - would you mind trying the latest Nightly to see if this still reproduces (with security.sandbox.content.level set to 1 or 2).

If it does, would you test the build at:
https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-385e685658ee0e213cd9fa1c405c2c06cb984bab/try-win64/firefox-56.0a1.en-US.win64.installer.exe

This includes some changes to make sure sandbox policy rules work with junction points and symlinks.

Can't remember if I've said before, but you might want to test by running Firefox as follows:

firefox.exe -no-remote -P

This allows you to create a new profile, to make sure you don't mess up your normal one.

Comment 10

[alex_mayorga]

¡Hola Bob!

I'm confirming this one as I got a question in the SuMo forum with this signature over at https://support.mozilla.org/es/questions/1163185 and there are 713 crashes in the past week at https://crash-stats.mozilla.com/signature/?product=Firefox&signature=TppCallbackCheckThreadAfterCallback so thi is definitely a thing.

¡Gracias!

Comment 11

[Christian Moser]

> firefox.exe -no-remote -P
thanks for that hint!:)

I tested your Nightly Build and the current Nightly Build. Both Versions are working with sandbox Policy set to 1 or 2. But as I mentioned before, Nightlys always worked, even older Versions.

I also tested the current beta (55.0b1 64bit), that Version works too. The last Beta I tested a few Weeks Ago wasnt working. So, maybe someone fixed the Bug between 53.0 and 55.0b1.

Comment 12

[alex_mayorga]

¡Hola Bob!

The user from the SuMo question said he tested your build but it still crashes like this:


bp-82d4fb9f-0c26-4147-b3b8-92b1c0170615 6/16/2017	12:24 AM bp-5d7b249b-c641-475f-9d0b-b4a2d0170615 6/16/2017	12:19 AM bp-d881c2b2-96ae-4ef0-ae57-84b170170615 6/16/2017	12:18 AM bp-3655ffce-a285-40bb-a029-e72831170615 6/16/2017	12:17 AM bp-4e996e15-dcc7-45b4-987b-5fa960170615 6/16/2017	12:17 AM bp-74dd69c5-168f-446c-9904-11a511170615 6/16/2017	12:17 AM bp-c3a24829-b590-415d-98d5-a43fa0170615 6/16/2017	12:17 AM bp-9d09ab94-2dc3-4153-85b8-994611170615 6/16/2017	12:17 AM bp-095de241-b019-4e03-9dc3-cd78b0170615 6/16/2017	12:17 AM

Hope this is helpful. 

¡Gracias!
Alex

Comment 13

[Bob Owen (:bobowen)]

There seems to be a high correlation in Nightly with Windows Creators update (version 10.0.15063.0) and Nahimic2OSD.dll

Comment 14

[Marco Castelluccio [:marco]]

Can you take a look? Do you think we can blocklist Nahimic2OSD.dll?

Comment 15

[Bob Owen (:bobowen)]

Just noticed there is also NahimicVRDevProps.dll at 98+%

I see that we already block nahimic2devprops.dll and nahimicmsiosd.dll.

Comment 16

[Carl Corcoran [:ccorcoran]]

Matt, do you have any insight to add on this one? Seems like nahimic may manifest in another DLL name.

Comment 17

[Marco Castelluccio [:marco]]

If we can technically block the DLL, I guess we should just do that.

Comment 18

[Molly Howell (she/her) (no longer active)]

I was focused on the app update crash back when I was looking at this and wasn't aware of any other crashes, so I didn't include Nahimic2OSD.dll, and I didn't see NahimicVRDevProps.dll getting injected on my machine. In light of the information here, I see no reason not to block both of them if possible.

Comment 19

[neikon]

I am the SuMo user, which alex_mayorga said. I updated chipset, lan, realtek audio, gpu and finaly Nahimic drivers. I have nahimic 2.5.12 right now and it does not make anymore crash report but the process to update nightly does not work right yet.

I have updated nightly today and I have had to finish nightly.exe process for it is restarted correctly but this did not any new crash report.

This is the last report that I have when I updated the drivers this morning (I am from Spain) 
bp-c3d23c19-080e-411a-8f98-e65c10170620
	20/06/2017	11:35

Comment 20

[neikon]

I installed Nahimic 2.3.15 and all work perfectly

Comment 21

[Carl Corcoran [:ccorcoran]]

I was not able to reproduce this, but here's a build that adds nahimicvrdevprops and nahimic2osd to the dll blocklist: https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-aede2502fe7ec22f8732409cbd122dc034c22daa/try-win64/firefox-56.0a1.en-US.win64.installer.exe

Are we still in contact with anyone that can see if this fixes the crash?

Comment 22

[Marco Castelluccio [:marco]]

Can you still reproduce the crash if you use the build that Carl provided in comment 21?

Neikon, can you try with the crashy version of Nahimic and Carl's build?

Comment 23

[neikon]

i try with nahimic 2.5.12 The last Carl's build does not make any crash report but I do not know how reproduce the update error because it does not notify any update to today standar nightly build.

I try with nahimic 2.5.12 with the standard build this version does not make any crash report but it has update freeze, with nahimic 2.3.15 all work. finally the more problematic version ,which makes constant crashes report, is the one between 2.3.15 and 2.5.12 but I can not find it.

Comment 24

[neikon]

I tested with the new driver version
Nahimic 2.5.16 "Problem with Firefox fixed. "
https://nahimic.helprace.com/i263-nahimic-2-is-now-ready-for-a-hot-summer

and realtek driver v8192, v8199 and v8210. 

I do not have any more crashes but Nightly does not restart when I click in update button. I have to kill nightly.exe proces for updated's proces can continue

Comment 25

[alex_mayorga]

¡Hola Gijs!

What to do with this one?

Per the 447 or so crashes in the past week at https://crash-stats.mozilla.com/signature/?signature=TppCallbackCheckThreadAfterCallback this is still a thing even on Nightly 57.

But per https://crash-stats.mozilla.com/report/index/13ef624d-261f-4964-b6e4-63ab70170907 this crash is now happening on ntdll.dll not Nahimic2OSD.dll

¡Gracias!
Alex

Comment 26

[:Gijs (he/him)]

I don't really have any insight here. Maybe Matt does. However, I do still see nahimic dlls show up in the 'correlations' and 'modules' tabs, so I assume it's still loaded?

Comment 27

[Molly Howell (she/her) (no longer active)]

I don't really have anything to add to what I said before. My opinion is still that we should add Nahimic2OSD.dll and NahimicVRDevProps.dll to the blocklist. I didn't do that to start with because I was looking at a separate updater crash which did not implicate these, but the correlations with this signature seem strong enough to easily merit it.

Comment 28

[:Gijs (he/him)]

(In reply to Matt Howell [:mhowell] from comment #27)
> I don't really have anything to add to what I said before. My opinion is
> still that we should add Nahimic2OSD.dll and NahimicVRDevProps.dll to the
> blocklist. I didn't do that to start with because I was looking at a
> separate updater crash which did not implicate these, but the correlations
> with this signature seem strong enough to easily merit it.

Who can make this change?

Comment 29

[neikon]

(In reply to :Gijs from comment #28)
> (In reply to Matt Howell [:mhowell] from comment #27)
> > I don't really have anything to add to what I said before. My opinion is
> > still that we should add Nahimic2OSD.dll and NahimicVRDevProps.dll to the
> > blocklist. I didn't do that to start with because I was looking at a
> > separate updater crash which did not implicate these, but the correlations
> > with this signature seem strong enough to easily merit it.
> 
> Who can make this change?

I think that It is not necesary. I had this error and I couldn't update. But now I don't have any problem with the last version drivers. I think.

Here my last crashes report 

Informes de fallos enviados
ID del informe 	Fecha del envío
bp-ae6b1eb6-a198-4404-bc07-087d80170901
	01/09/2017	18:21
bp-996261a9-ccfb-4531-b652-596a60170804
	04/08/2017	15:07
bp-71a2ac82-265c-4b36-b054-cea140170804
	04/08/2017	15:07
bp-5041e311-e48c-43c8-9ebc-5fee00170804
	04/08/2017	15:07
bp-38ff3d71-45c7-4bb3-a51f-b49020170804
	04/08/2017	10:57
bp-8b1145ee-1814-4f8e-9002-577760170804
	04/08/2017	10:57
bp-2179fc66-dd0a-4d66-b75f-19b810170804
	04/08/2017	10:31
bp-df6481ba-e8c6-4151-99b3-461220170804
	04/08/2017	10:11

Comment 30

[:Gijs (he/him)]

(In reply to neikon from comment #29)
> (In reply to :Gijs from comment #28)
> > (In reply to Matt Howell [:mhowell] from comment #27)
> > > I don't really have anything to add to what I said before. My opinion is
> > > still that we should add Nahimic2OSD.dll and NahimicVRDevProps.dll to the
> > > blocklist. I didn't do that to start with because I was looking at a
> > > separate updater crash which did not implicate these, but the correlations
> > > with this signature seem strong enough to easily merit it.
> > 
> > Who can make this change?
> 
> I think that It is not necesary. I had this error and I couldn't update. But
> now I don't have any problem with the last version drivers. I think.

But this only helps people like you who know that they can update, who have the appropriate privileges to do so (ie not people in an enterprise setup without admin rights) and who have figured out that these dlls are the ones causing crashes (which takes significant technical skill). For everyone else, they just get a terrible Firefox experience because someone is dll-injecting Firefox. That's not OK.

I believe it's possible for us to blocklist only specific versions of the dll, so if we're confident that only older versions have this issue I *think* we could block only them. Jorge, do you also take care of the dll blocklist? If not, who does?

Comment 31

[Marco Castelluccio [:marco]]

Attachment: Patch to block Nahimic DLLs

The policy is to block all DLL injections, so in theory we shouldn't gate on version.
Nahimic has fixed the issue in their latest version though and they are asking us to be unblocklisted.
Kev should make the decision here.

Comment 32

[Kev Needham [:kev]]

DLL Injection Blocks are pdol's domain. Adding him and needinfo-ing. Apologies for delay in addressing NI.

Comment 33

[Peter Dolanjski [:pdol]]

(In reply to Marco Castelluccio [:marco] from comment #31)
> Created attachment 8905898 [details] [diff] [review]
> Patch to block Nahimic DLLs
> 
> The policy is to block all DLL injections, so in theory we shouldn't gate on
> version.
> Nahimic has fixed the issue in their latest version though and they are
> asking us to be unblocklisted.
> Kev should make the decision here.

Sorry for the delay.  We should block only the affected DLLs, if possible.  If not, a full block is acceptable, per our policy.

Comment 34

[Marco Castelluccio [:marco]]

Attachment: Patch to block affected versions of the Nahimic DLLs

Here's a report of the Nahimic DLLs in crash reports over the past 21 days: https://gist.github.com/marco-c/9c1f7f336a6dd64256d838ab421af64f.

We've got confirmation from them that they have fixed the crash in version 2.5.20.

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c685d4102fb8
Block Nahimic DLLs affected by a bug which causes Firefox to crash. r=jimm

Comment 36

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/c685d4102fb8