1400637 - Crash in mozilla::layers::ImageBridgeChild::InitForContent

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect, P1)

Description

[Andrew McCreight [:mccr8]]

This bug was filed from the Socorro interface and is 
report bp-0de017b8-9108-46be-99d8-df65c0170917.
=============================================================

Most of these have this as the crash reason:
  MOZ_RELEASE_ASSERT(!XRE_IsContentProcess()) (Content Process should have called InitChild() before first GetPlatform()) 

This is in the top 10 crashes for the 9-16 Nightly, but it is only from about 9 different installations.

Comment 1

[Andrew McCreight [:mccr8]]

This kind of feels like it could be a regression from bug 1397214, like we preempt initialization somehow, but then we end up trying to use that information anyways. Any ideas, Andrew?

Comment 2

[Andrew McCreight [:mccr8]]

Regression range based on the build crashes started showing up in:
  https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=6be5c7d30d2def62a762ac187252eba626b23a92&tochange=27e7b4261e4b5a016e09adb5332a181e43ab520e

Comment 3

[Andrew Osmond [:aosmond] (he/him)]

This is less of a regression and more of a moving the crash further down the chain. This crash effectively replaced the gfxPlatform::InitAcceleration crash. This was considered as a possible outcome (and it shouldn't be more common than the original crash) -- I guess we need to add checking the shutdown canary to all of the ContentChild initialization methods. Hmm. This could get ugly fast depending on how many methods need to check.

Comment 4

[Andrew Osmond [:aosmond] (he/him)]

The following IPDL methods are called (in order) by ContentParent::InitInternal:

AllocPScriptCacheChild
RecvNotifyProcessPriorityChanged
RecvInitRendering
RecvLoadAndRegisterSheet
RecvSetProcessSandbox
RecvSetAudioSessionData
RecvInitServiceWorkers
RecvInitBlobURLs
RecvPluginsToContent

Note the first two get called without any issues and we crash in RecvInitRendering.

Comment 5

[(no longer active)]

This isn't normal processing of our IPC messages though.  Again, our message loop has exited and we're just cleaning up our pending events.  Exactly like bug 1397214.

I wonder what has changed recently to make this happen...

Comment 6

[[:philipp]]

[Tracking Requested - why for this release]:
when discounting infobar crashes, this is by far the top content crash (50%+) for early 57.0b1.

some user comments:
* bp-875d413f-855c-4948-bd8f-b48eb0170920: can no longer open firefox after update
* bp-5e9da1f2-fd8c-4afb-9c7d-54f540170920: Just updated to 57.x Dev edition and no website will successfully load in any tab. This error - your tab just crashed message is seen every single time.
* bp-e36f17d5-31dc-422a-b70a-94f4a0170920: Get a crashed tab on startup, cannot navigate to any page as the tab crashes again. Have tried a fresh install with no success.

numerous comments also mention un-/plugging a headphone when this happened

Comment 7

[Andrew McCreight [:mccr8]]

The headphone thing sounds like bug 1400465.

Comment 8

[Andrew McCreight [:mccr8]]

The fix for that was landed on 57 in bug 1400671, so I'm marking the dependency in case it is related.

Comment 9

[Andrew Osmond [:aosmond] (he/him)]

I went through the pushlog where the original crash was introduced, and these are the most relevant startup/loading URI/shutdown related changes I could find. That's not to say I blame any/all of them, I just wasn't confident enough to eliminate them:

	807e9eadd5c1	Christoph Kerschbaumer — Bug 1331740: Pass correct context for TYPE_DOCUMENT loads within docshell. r=smaug
	47b2e91b1060	Eugen Sawin — Bug 1396065 - [1.3] Add nsILoadURIDelegate to handle load delegation to the window (GeckoView). r=smaug,snorp
	74d087bee9e5	Christoph Kerschbaumer — Bug 1395948: Move blocking of toplevel data: URI navigations before opening a new window. r=smaug
	662fb967d092	Kevin Jones — Bug 1284886 - Implement discardBrowser method. r=dao
	cdecd701b70f	Kris Maglione — Bug 1396652: Ensure ordered destruction of channel entries at shutdown. r=mixedpuppy
[e-mail]
	54c594c4f68c	Andrew Swan — Bug 1396578 Remove startup notification for non-mpc extensions r=rhelmer
	6d9980e17a8c	Alex Gaynor — Bug 1229829 - Part 2 - Use an alternate desktop on the local winstation for content processes; r=bobowen
	d342ab7ef7c3	Alex Gaynor — Bug 1229829 - Part 1 - Apply chromium sandbox patches from upstream which improves alternate desktop support; r=bobowen

Comment 10

[Andrew Osmond [:aosmond] (he/him)]

One line of thought I have considered is perhaps ChildThread::OnChannelError is called when setting up the IO channel. If I read the code correctly, this would cause an early exit of the main thread (and thus initiate a shutdown), and would happen very early in the startup process as ChildThread is a base class of ContentProcess.

Comment 11

[Andrew Osmond [:aosmond] (he/him)]

So based on the feedback, I feel like we have the cause and effect reversed. This is an unexpected shutdown during startup. I have yet to see any evidence which tells us why the shutdown is happening, but I'm guessing it is expectional in nature, rather than the parent process wanting to teardown the content process.

Alex, any chance the sandboxing security level increase could cause something like this?

Comment 12

[Andrew Osmond [:aosmond] (he/him)]

Christoph, your thoughts as well?

If there is any agreement of suspicion, it would be nice to see either bug 1229829 or bug 1395948 backed out to test the theory (although please not both at the same time :)).

Comment 13

[Christoph Kerschbaumer [:ckerschb]]

(In reply to Andrew Osmond [:aosmond] from comment #12)
> If there is any agreement of suspicion, it would be nice to see either bug
> 1229829 or bug 1395948 backed out to test the theory (although please not
> both at the same time :)).

I doubt that Bug 1395948 causes the problem here. It's hard to backout that change since things changed too much in the meantime. However, there is a pref |security.data_uri.block_toplevel_data_uri_navigations| which you can set to false for testing purposes here [1]. If set to 'false', then all of the blocking code is bypassed, see [2]. Hope that helps to find the root cause for this problem.

[1] https://dxr.mozilla.org/mozilla-central/source/modules/libpref/init/all.js#5827
[2] https://dxr.mozilla.org/mozilla-central/source/dom/security/nsContentSecurityManager.cpp#39

Comment 14

[makotech222]

(In reply to Christoph Kerschbaumer [:ckerschb] from comment #13)
> (In reply to Andrew Osmond [:aosmond] from comment #12)
> > If there is any agreement of suspicion, it would be nice to see either bug
> > 1229829 or bug 1395948 backed out to test the theory (although please not
> > both at the same time :)).
> 
> I doubt that Bug 1395948 causes the problem here. It's hard to backout that
> change since things changed too much in the meantime. However, there is a
> pref |security.data_uri.block_toplevel_data_uri_navigations| which you can
> set to false for testing purposes here [1]. If set to 'false', then all of
> the blocking code is bypassed, see [2]. Hope that helps to find the root
> cause for this problem.
> 
> [1]
> https://dxr.mozilla.org/mozilla-central/source/modules/libpref/init/all.
> js#5827
> [2]
> https://dxr.mozilla.org/mozilla-central/source/dom/security/
> nsContentSecurityManager.cpp#39

Just FYI, in my about config, that setting was already set to false, and crashes still occur. Also, in the early days of this bug, i was able to turn of multiprocess and sites would load (FF still crashed on close, though). With latest builds, it won't even save that setting anymore.

Comment 15

[Jim Mathies [:jimm]]

Hello, can you reproduce this crash reliably? If so, would you mind trying the following:

1) open about:config
2) 'security.sandbox.content.level' to 3 (default is 4)
3) restart and see if you still get the crash

If the crash prevents you from editing about:config, you can add this to your firefox prefs in your profile using user prefs [1]. 

[1] http://kb.mozillazine.org/User.js_file

FYI, agaynor is on pto until next week.

Comment 16

[makotech222]

Yeah that actually fixes both the opening of tabs and also closing firefox. No crashes whatsoever. Thanks Jim

Comment 17

[Jim Mathies [:jimm]]

(In reply to makotech222 from comment #16)
> Yeah that actually fixes both the opening of tabs and also closing firefox.
> No crashes whatsoever. Thanks Jim

Sure! I'd appreciate some help in diagnosing why that particular setting caused this failure. Would you mind telling me a bit about your computer -

1) is your device on a corporate network or using roaming?
2) do you run any type of anti-virus software?
3) would you mind posting your about:support text?

Any additional detail about your system's unique qualities would be appreciated.

Thanks!

Comment 18

[Jim Mathies [:jimm]]

Interesting stats:

- 90% of these crashes get submitted through the crashbar
- 50% have active accessibility, which is really high

Comment 19

[u602632]

I was also experiencing this bug (as far as I can tell).
Every page including the new tab page (though none of the about: pages) was instantly crashing.

I changed the setting 'security.sandbox.content.level' to 3 (default is 4) and that definitely fixed it.

If you're interested in my answers to the questions you asked of makotech222, here they are.

1) is your device on a corporate network or using roaming?
No, this is home desktop.
2) do you run any type of anti-virus software?
WebRoot SecureAnywhere 
3) would you mind posting your about:support text?
Here is mine.
 
Application Basics
------------------

Name: Firefox
Version: 57.0b1
Build ID: 20170917031738
Update Channel: aurora
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0
OS: Windows_NT 10.0
Multiprocess Windows: 1/1 (Enabled by default)
Web Content Processes: 2/4
Stylo: true (enabled by default)
Google Key: Found
Mozilla Location Service Key: Found
Safe Mode: false

Crash Reports for the Last 3 Days
---------------------------------

Report ID: bp-46b60c43-958b-4322-b9f9-059fd0170921
Submitted: 17 hours ago

Report ID: bp-2d7a053b-86e0-4d8a-9348-ebc140170921
Submitted: 17 hours ago

Report ID: bp-4c6fb9b9-183a-4002-8ed8-9eb4b0170921
Submitted: 17 hours ago

Report ID: bp-47f191df-1699-41fb-a480-eea080170921
Submitted: 17 hours ago

Report ID: bp-b693f6cc-7b3c-4cd1-8bb0-04c1a0170921
Submitted: 17 hours ago

Report ID: bp-824dba86-8eac-4990-bc12-0e50d0170921
Submitted: 17 hours ago

Report ID: bp-0c223ad8-3dae-440f-84bc-b285d0170921
Submitted: 17 hours ago

Report ID: bp-c505d9b0-a418-45fd-8eab-f74860170921
Submitted: 17 hours ago

Report ID: bp-d20b8d1b-64a4-4793-ba9b-66b680170921
Submitted: 17 hours ago

Report ID: bp-86f370f1-fa25-49d8-b92c-178d60170921
Submitted: 17 hours ago

Report ID: bp-57c552aa-3d91-4a0f-9dbf-ba57a0170921
Submitted: 17 hours ago

Report ID: bp-ebcc65d7-49be-49d7-8156-2f11a0170921
Submitted: 17 hours ago

Report ID: bp-3392b1d9-a486-4e6b-93b6-2479c0170921
Submitted: 17 hours ago

Report ID: bp-656c5945-9dc4-47dd-a4f2-902250170921
Submitted: 17 hours ago

Report ID: bp-9031858c-6e84-4868-bf1e-61bcd0170921
Submitted: 17 hours ago

Report ID: bp-817cd7a4-3b5c-4130-9c32-477dd0170921
Submitted: 17 hours ago

Report ID: bp-10800f32-5463-41d1-9d14-f27900170921
Submitted: 17 hours ago

Report ID: bp-7f8cabb4-e0b5-4891-8105-eacea0170921
Submitted: 17 hours ago

Report ID: bp-95cf430a-1b12-4d4f-976e-b9c8c0170921
Submitted: 17 hours ago

Report ID: bp-3a29ada5-7cfc-42d6-bb61-e667e0170921
Submitted: 17 hours ago

Report ID: bp-b94c6208-96bc-45e3-b6f2-3d3d60170921
Submitted: 17 hours ago

Report ID: bp-fe47d606-a7e0-41f2-add2-0321b0170921
Submitted: 17 hours ago

Report ID: bp-b4115849-d946-49b1-9a10-0731f0170920
Submitted: 24 hours ago

Report ID: bp-b775abaa-f97d-4f1f-97b9-0117f0170920
Submitted: 24 hours ago

Report ID: bp-c9a61f14-859d-47b2-973a-1c5340170920
Submitted: 24 hours ago

Report ID: bp-dddd1987-cdde-4fd7-80b9-937930170920
Submitted: 24 hours ago

Report ID: bp-4befd8b5-83c0-4822-8a3a-27b3c0170920
Submitted: 24 hours ago

Report ID: bp-fb57d482-f069-4685-9bd5-429ad0170920
Submitted: 24 hours ago

Report ID: bp-ee6bbbbf-f252-4865-a4aa-c2ebf0170920
Submitted: 24 hours ago

Report ID: bp-8628521f-ce58-4d7e-b011-ea7820170920
Submitted: 24 hours ago

Report ID: bp-20ca74f6-9939-437d-9310-3018d0170920
Submitted: 24 hours ago

Report ID: bp-77db1d48-c9d3-4196-848e-197930170920
Submitted: 24 hours ago

Report ID: bp-d441973c-30d2-40be-b89c-9c43e0170920
Submitted: 24 hours ago

Report ID: bp-ddb634c1-3213-4915-98f7-2d5420170920
Submitted: 24 hours ago

Report ID: bp-683eebc4-f855-48ee-b38b-ff52a0170920
Submitted: 24 hours ago

Report ID: bp-1ad1d550-1e58-4257-a82a-f8f410170920
Submitted: 24 hours ago

Report ID: bp-239a6df0-5873-49b9-9978-de7520170920
Submitted: 24 hours ago

Report ID: bp-f3839f98-4b3b-464a-b095-628130170920
Submitted: 24 hours ago

Report ID: bp-d14289be-e29d-4cb9-9a46-aafb90170920
Submitted: 24 hours ago

Report ID: bp-8311251f-524b-497a-ba9f-fa26d0170920
Submitted: 24 hours ago

Report ID: bp-4132c469-558a-4e5f-80c8-f51580170920
Submitted: 24 hours ago

Report ID: bp-61023eaf-50b0-43a9-a3a3-b5f800170920
Submitted: 24 hours ago

Report ID: bp-239d0fe7-4337-44aa-a24c-cb2590170920
Submitted: 24 hours ago

Report ID: bp-5650c815-5655-430b-9428-52b7f0170920
Submitted: 24 hours ago

Report ID: bp-85c404e6-df88-491e-99e1-557220170920
Submitted: 24 hours ago

Report ID: bp-2547b827-f3d2-4561-91f1-91d060170920
Submitted: 24 hours ago

Report ID: bp-2ad4025b-d8fb-4ec7-90ae-9bb770170920
Submitted: 24 hours ago

Report ID: bp-f97b14f8-7027-4613-9cfc-00f0d0170920
Submitted: 24 hours ago

Report ID: bp-913b4a0b-2994-43de-a467-541750170920
Submitted: 24 hours ago

Report ID: bp-028974df-0309-40bb-8b69-3c8340170920
Submitted: 24 hours ago

Report ID: bp-7a8b4300-38fb-42eb-baab-0a75d0170920
Submitted: 24 hours ago

Report ID: bp-d27ea70f-94ea-42f3-81aa-a05000170920
Submitted: 24 hours ago

Report ID: bp-f5429f90-911a-4aa2-b9ec-48c290170920
Submitted: 24 hours ago

Report ID: bp-a809dbfd-3c30-42fe-b99a-678b70170920
Submitted: 24 hours ago

Report ID: bp-09996f37-c470-4ca3-97cf-8291b0170920
Submitted: 24 hours ago

Report ID: bp-b4fd20e2-5b78-4e39-8232-f5cac0170920
Submitted: 24 hours ago

Report ID: bp-7e32da46-6222-4a93-8506-4042e0170920
Submitted: 24 hours ago

Report ID: bp-50a1f6b4-ea04-4452-b3b6-7a50d0170920
Submitted: 24 hours ago

Report ID: bp-aa553423-2461-4a75-a76e-83c320170920
Submitted: 24 hours ago

Report ID: bp-8a148d30-4fc4-4ff1-8955-4113a0170920
Submitted: 24 hours ago

Report ID: bp-06bb9834-b5aa-4dfd-8b46-7a7cd0170920
Submitted: 24 hours ago

Report ID: bp-457bca08-94fd-4652-9270-74dd90170920
Submitted: 24 hours ago

Report ID: bp-deeb4e41-a639-4e1d-b2aa-e3d100170920
Submitted: 24 hours ago

Report ID: bp-9f5a54f4-6cfa-43df-b207-aa68f0170920
Submitted: 24 hours ago

Report ID: bp-9f72b7ad-ec43-4fc9-aa95-1f2220170920
Submitted: 24 hours ago

Report ID: bp-0dec801a-0ca8-4f3d-93b8-651190170920
Submitted: 24 hours ago

Report ID: bp-f5913991-b3a2-41ab-afa9-250210170920
Submitted: 24 hours ago

Report ID: bp-2e97fee5-230c-4cf2-a846-1d5e90170920
Submitted: 24 hours ago

Report ID: bp-7049fcc0-04c8-4d4e-a118-03aca0170920
Submitted: 24 hours ago

Report ID: bp-ff8ca01a-d694-4754-a36a-03aa50170920
Submitted: 24 hours ago

Report ID: bp-02cab246-0997-465f-bcff-dbc170170920
Submitted: 24 hours ago

Report ID: bp-78e0757a-c124-4a0d-8ed0-d5fcb0170920
Submitted: 24 hours ago

Report ID: bp-fe0fdc61-2e0d-4501-931e-e879d0170920
Submitted: 24 hours ago

Report ID: bp-f288b70a-31d6-478c-bf57-a305e0170920
Submitted: 24 hours ago

Report ID: bp-67554e9e-7dff-45c2-bfd3-4d6450170920
Submitted: 24 hours ago

Report ID: bp-2a1cea40-79f3-4cfe-8f0f-112fd0170920
Submitted: 24 hours ago

Report ID: bp-3bfb9d29-2a1e-4923-8dae-101140170920
Submitted: 24 hours ago

Report ID: bp-d3a4dd92-96a5-4250-b141-26dac0170920
Submitted: 24 hours ago

Report ID: bp-5f446be2-e340-4539-9ef3-aa57e0170920
Submitted: 24 hours ago

Report ID: bp-17fd5730-7e4c-4057-aa37-65bf30170920
Submitted: 24 hours ago

Report ID: bp-2f5c8c6f-3d13-46e3-aca9-decc90170920
Submitted: 24 hours ago

Report ID: bp-2a0413a6-4f3c-41c0-80a9-058320170920
Submitted: 24 hours ago

Report ID: bp-0e6ef4ab-09ce-4806-bad9-f1a000170920
Submitted: 24 hours ago

Report ID: bp-5ff9a93d-c568-49cb-a89b-b0c5e0170920
Submitted: 24 hours ago

Report ID: bp-5eea2ee0-e949-4326-8bae-7e9800170920
Submitted: 24 hours ago

Report ID: bp-314c9f13-0cd0-45a3-a476-072420170920
Submitted: 24 hours ago

Report ID: bp-88cdebd9-70f1-4d49-a256-5b98b0170920
Submitted: 24 hours ago

Report ID: bp-66d29983-1ce8-4659-97b5-4c87c0170920
Submitted: 1 day ago

Report ID: bp-7332bc4a-e812-4027-a591-c28ac0170920
Submitted: 1 day ago

Report ID: bp-b0e743c8-88f9-4034-8b34-603420170920
Submitted: 1 day ago

Report ID: bp-ca7fedba-5c3a-4b14-bba0-ba62c0170920
Submitted: 1 day ago

Report ID: bp-00986011-1b13-478c-b7f7-1f3c50170920
Submitted: 1 day ago

Report ID: bp-11f4956e-1e56-4236-af1f-a62680170920
Submitted: 1 day ago

Report ID: bp-ebc3bcbe-10c2-4dd7-baa6-276c20170920
Submitted: 1 day ago

Report ID: bp-0632da45-97e9-4964-bd7a-4b1860170920
Submitted: 1 day ago

All Crash Reports (including 10 pending crashes in the given time range)

Firefox Developer Edition Features
----------------------------------

Name: Activity Stream
Version: 2017.09.16.0001-2fc82208
ID: [e-mail]

Name: Application Update Service Helper
Version: 2.0
ID: [e-mail]

Name: Click-to-Play staged rollout
Version: 1.3
ID: [e-mail]

Name: Firefox Screenshots
Version: 16.1.0
ID: [e-mail]

Name: Follow-on Search Telemetry
Version: 0.9.3
ID: [e-mail]

Name: Form Autofill
Version: 1.0
ID: [e-mail]

Name: Multi-process staged rollout
Version: 3.00
ID: [e-mail]

Name: Photon onboarding
Version: 0.1
ID: [e-mail]

Name: Pocket
Version: 1.0.5
ID: [e-mail]

Name: Shield Recipe Client
Version: 73
ID: [e-mail]

Name: Web Compat
Version: 1.1
ID: [e-mail]

Name: WebCompat Reporter
Version: 1.0.0
ID: [e-mail]

Extensions
----------

Name: uBlock Origin
Version: 1.14.8
Enabled: true
ID: [e-mail]

Name: Webroot Filtering Extension
Version: 1.4.0.8
Enabled: false
ID: [e-mail]

Graphics
--------

Features
Compositing: Direct3D 11 (Advanced Layers)
Asynchronous Pan/Zoom: wheel input enabled; scrollbar drag enabled; keyboard enabled; autoscroll enabled
WebGL 1 Driver WSI Info: EGL_VENDOR: Google Inc. (adapter LUID: 0000000000006f15) EGL_VERSION: 1.4 (ANGLE 2.1.0.dec065540d5f) EGL_EXTENSIONS: EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_ANGLE_direct_composition EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_KHR_stream EGL_KHR_stream_consumer_gltexture EGL_NV_stream_consumer_gltexture_yuv EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_stream_producer_d3d_texture_nv12 EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource EGL_EXTENSIONS(nullptr): EGL_EXT_client_extensions EGL_EXT_platform_base EGL_EXT_platform_device EGL_ANGLE_platform_angle EGL_ANGLE_platform_angle_d3d EGL_ANGLE_device_creation EGL_ANGLE_device_creation_d3d11 EGL_ANGLE_experimental_present_path EGL_KHR_client_get_all_proc_addresses
WebGL 1 Driver Renderer: Google Inc. -- ANGLE (NVIDIA GeForce GTX 980 Direct3D11 vs_5_0 ps_5_0)
WebGL 1 Driver Version: OpenGL ES 2.0 (ANGLE 2.1.0.dec065540d5f)
WebGL 1 Driver Extensions: GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_lossy_etc_decode GL_ANGLE_pack_reverse_row_order GL_ANGLE_request_extension GL_ANGLE_robust_client_memory GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_copy_compressed_texture GL_CHROMIUM_copy_texture GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_discard_framebuffer GL_EXT_disjoint_timer_query GL_EXT_draw_buffers GL_EXT_frag_depth GL_EXT_map_buffer_range GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_sRGB GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_rg GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_EGL_stream_consumer_external GL_NV_fence GL_NV_pack_subimage GL_NV_pixel_buffer_object GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object
WebGL 1 Extensions: ANGLE_instanced_arrays EXT_blend_minmax EXT_color_buffer_half_float EXT_frag_depth EXT_shader_texture_lod EXT_texture_filter_anisotropic EXT_disjoint_timer_query OES_element_index_uint OES_standard_derivatives OES_texture_float OES_texture_float_linear OES_texture_half_float OES_texture_half_float_linear OES_vertex_array_object WEBGL_color_buffer_float WEBGL_compressed_texture_s3tc WEBGL_debug_renderer_info WEBGL_debug_shaders WEBGL_depth_texture WEBGL_draw_buffers WEBGL_lose_context MOZ_WEBGL_lose_context MOZ_WEBGL_compressed_texture_s3tc MOZ_WEBGL_depth_texture
WebGL 2 Driver WSI Info: EGL_VENDOR: Google Inc. (adapter LUID: 0000000000006f15) EGL_VERSION: 1.4 (ANGLE 2.1.0.dec065540d5f) EGL_EXTENSIONS: EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_ANGLE_direct_composition EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_KHR_stream EGL_KHR_stream_consumer_gltexture EGL_NV_stream_consumer_gltexture_yuv EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_stream_producer_d3d_texture_nv12 EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource EGL_EXTENSIONS(nullptr): EGL_EXT_client_extensions EGL_EXT_platform_base EGL_EXT_platform_device EGL_ANGLE_platform_angle EGL_ANGLE_platform_angle_d3d EGL_ANGLE_device_creation EGL_ANGLE_device_creation_d3d11 EGL_ANGLE_experimental_present_path EGL_KHR_client_get_all_proc_addresses
WebGL 2 Driver Renderer: Google Inc. -- ANGLE (NVIDIA GeForce GTX 980 Direct3D11 vs_5_0 ps_5_0)
WebGL 2 Driver Version: OpenGL ES 3.0 (ANGLE 2.1.0.dec065540d5f)
WebGL 2 Driver Extensions: GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_lossy_etc_decode GL_ANGLE_pack_reverse_row_order GL_ANGLE_request_extension GL_ANGLE_robust_client_memory GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_copy_compressed_texture GL_CHROMIUM_copy_texture GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_float GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_discard_framebuffer GL_EXT_disjoint_timer_query GL_EXT_draw_buffers GL_EXT_frag_depth GL_EXT_map_buffer_range GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_sRGB GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_norm16 GL_EXT_texture_rg GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_EGL_stream_consumer_external GL_NV_fence GL_NV_pack_subimage GL_NV_pixel_buffer_object GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_image_external_essl3 GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object
WebGL 2 Extensions: EXT_color_buffer_float EXT_texture_filter_anisotropic EXT_disjoint_timer_query OES_texture_float_linear WEBGL_compressed_texture_s3tc WEBGL_debug_renderer_info WEBGL_debug_shaders WEBGL_lose_context MOZ_WEBGL_lose_context MOZ_WEBGL_compressed_texture_s3tc
Direct2D: true
DirectWrite: true (10.0.15063.483)
GPU #1
Active: Yes
Description: NVIDIA GeForce GTX 980
Vendor ID: 0x10de
Device ID: 0x13c0
Driver Version: 22.21.13.8541
Driver Date: 8-21-2017
Drivers: C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumdx.dll,C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumdx.dll,C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumdx.dll,C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumdx.dll C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumd.dll,C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumd.dll,C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumd.dll,C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumd.dll
Subsys ID: 29833842
RAM: 4096

Diagnostics
AzureCanvasAccelerated: 0
AzureCanvasBackend: Direct2D 1.1
AzureCanvasBackend (UI Process): skia
AzureContentBackend: Direct2D 1.1
AzureContentBackend (UI Process): skia
AzureFallbackCanvasBackend (UI Process): cairo
GPUProcessPid: 10336
GPUProcess: Terminate GPU Process
Device Reset: Trigger Device Reset
Decision Log
WEBRENDER:
opt-in by default: WebRender is an opt-in feature
unavailable by runtime: Build doesn't include WebRender




Media
-----

Audio Backend: wasapi
Max Channels: 2
Preferred Channel Layout: stereo
Preferred Sample Rate: 48000
Output Devices
Name: Group
NVIDIA Output (NVIDIA High Definition Audio): HDAUDIO\FUNC_01&VEN_10DE&DEV_0071&SUBSYS_38422983&REV_1001\5&800d03d&0&0001
Headphones (High Definition Audio Device): HDAUDIO\FUNC_01&VEN_10EC&DEV_0899&SUBSYS_18491898&REV_1000\4&1d2c9eed&0&0001
NVIDIA Output (NVIDIA High Definition Audio): HDAUDIO\FUNC_01&VEN_10DE&DEV_0071&SUBSYS_38422983&REV_1001\5&800d03d&0&0001
Digital Audio (S/PDIF) (High Definition Audio Device): HDAUDIO\FUNC_01&VEN_10EC&DEV_0899&SUBSYS_18491898&REV_1000\4&1d2c9eed&0&0001
SPDIF Interface (Audio-gd): USB\VID_040D&PID_3410&MI_00\9&1f79a332&0&0000
Internal AUX Jack (High Definition Audio Device): HDAUDIO\FUNC_01&VEN_10EC&DEV_0899&SUBSYS_18491898&REV_1000\4&1d2c9eed&0&0001
Internal AUX Jack (High Definition Audio Device): HDAUDIO\FUNC_01&VEN_10EC&DEV_0899&SUBSYS_18491898&REV_1000\4&1d2c9eed&0&0001
NVIDIA Output (NVIDIA High Definition Audio): HDAUDIO\FUNC_01&VEN_10DE&DEV_0071&SUBSYS_38422983&REV_1001\5&800d03d&0&0001
NVIDIA Output (NVIDIA High Definition Audio): HDAUDIO\FUNC_01&VEN_10DE&DEV_0071&SUBSYS_38422983&REV_1001\5&800d03d&0&0001
Speakers (High Definition Audio Device): HDAUDIO\FUNC_01&VEN_10EC&DEV_0899&SUBSYS_18491898&REV_1000\4&1d2c9eed&0&0001
Input Devices
Name: Group
Microphone (High Definition Audio Device): HDAUDIO\FUNC_01&VEN_10EC&DEV_0899&SUBSYS_18491898&REV_1000\4&1d2c9eed&0&0001
Microphone (High Definition Audio Device): HDAUDIO\FUNC_01&VEN_10EC&DEV_0899&SUBSYS_18491898&REV_1000\4&1d2c9eed&0&0001
Internal AUX Jack (High Definition Audio Device): HDAUDIO\FUNC_01&VEN_10EC&DEV_0899&SUBSYS_18491898&REV_1000\4&1d2c9eed&0&0001
Microphone (2- AT2020 USB ): USB\VID_0909&PID_0018&MI_00\7&24665fc6&0&0000
Internal AUX Jack (NVIDIA High Definition Audio): HDAUDIO\FUNC_01&VEN_10DE&DEV_0071&SUBSYS_38422983&REV_1001\5&800d03d&0&0001
Internal AUX Jack (NVIDIA High Definition Audio): HDAUDIO\FUNC_01&VEN_10DE&DEV_0071&SUBSYS_38422983&REV_1001\5&800d03d&0&0001
Internal AUX Jack (NVIDIA High Definition Audio): HDAUDIO\FUNC_01&VEN_10DE&DEV_0071&SUBSYS_38422983&REV_1001\5&800d03d&0&0001
Line In (High Definition Audio Device): HDAUDIO\FUNC_01&VEN_10EC&DEV_0899&SUBSYS_18491898&REV_1000\4&1d2c9eed&0&0001
Internal AUX Jack (NVIDIA High Definition Audio): HDAUDIO\FUNC_01&VEN_10DE&DEV_0071&SUBSYS_38422983&REV_1001\5&800d03d&0&0001
Internal AUX Jack (High Definition Audio Device): HDAUDIO\FUNC_01&VEN_10EC&DEV_0899&SUBSYS_18491898&REV_1000\4&1d2c9eed&0&0001

Important Modified Preferences
------------------------------

accessibility.lastLoadDate: 1505874618
accessibility.loadedInLastSession: true
accessibility.typeaheadfind.flashBar: 0
browser.cache.disk.capacity: 358400
browser.cache.disk.filesystem_reported: 1
browser.cache.disk.smart_size.first_run: false
browser.cache.disk.smart_size.use_old_max: false
browser.cache.frecency_experiment: 3
browser.download.manager.alertOnEXEOpen: true
browser.places.smartBookmarksVersion: 8
browser.sessionstore.upgradeBackup.latestBuildID: 20170917031738
browser.startup.homepage_override.buildID: 20170917031738
browser.startup.homepage_override.mstone: 57.0
browser.tabs.warnOnClose: false
browser.urlbar.experiment.unified-urlbar.mirror.browser.urlbar.matchBuckets:
browser.urlbar.experiment.unified-urlbar.mirror.browser.urlbar.matchBuckets.notset: true
browser.urlbar.experiment.unified-urlbar.mirror.browser.urlbar.matchBucketsSearch:
browser.urlbar.experiment.unified-urlbar.mirror.browser.urlbar.matchBucketsSearch.notset: true
browser.urlbar.experiment.unified-urlbar.mirror.browser.urlbar.maxRichResults: 10
browser.urlbar.experiment.unified-urlbar.mirror.browser.urlbar.maxRichResults.notset: true
browser.urlbar.lastSuggestionsPromptDate: 20170619
browser.urlbar.matchBuckets: general:3,suggestion:3
browser.urlbar.matchBucketsSearch: suggestion:3,general:3
browser.urlbar.maxRichResults: 7
browser.urlbar.searchSuggestionsChoice: false
browser.urlbar.suggest.searches: false
browser.urlbar.timesBeforeHidingSuggestionsHint: 0
dom.forms.autocomplete.formautofill: true
extensions.lastAppVersion: 57.0
gfx.crash-guard.d3d11layers.appVersion: 56.0
gfx.crash-guard.d3d11layers.deviceID: 0x13c0
gfx.crash-guard.d3d11layers.driverVersion: 22.21.13.8541
gfx.crash-guard.d3d11layers.feature-d2d: true
gfx.crash-guard.d3d11layers.feature-d3d11: true
gfx.crash-guard.status.d3d11layers: 2
gfx.crash-guard.status.d3d11video: 2
layers.mlgpu.sanity-test-failed: false
media.benchmark.vp9.fps: 276
media.benchmark.vp9.versioncheck: 2
media.gmp-gmpopenh264.abi: x86_64-msvc-x64
media.gmp-gmpopenh264.lastUpdate: 1497915498
media.gmp-gmpopenh264.version: 1.6
media.gmp-manager.buildID: 20170917031738
media.gmp-manager.lastCheck: 1505948652
media.gmp-widevinecdm.abi: x86_64-msvc-x64
media.gmp-widevinecdm.lastUpdate: 1504663402
media.gmp-widevinecdm.version: 1.4.8.1008
media.gmp.storage.version.observed: 1
media.hardware-video-decoding.failed: true
media.webrtc.debug.log_file: C:\Users\m\AppData\Local\Temp\WebRTC.log
network.cookie.prefsMigrated: true
network.dns.disablePrefetch: true
network.http.speculative-parallel-limit: 0
network.predictor.cleaned-up: true
network.predictor.enabled: false
network.prefetch-next: false
places.database.lastMaintenance: 1505758233
places.history.expiration.transient_current_max_pages: 128999
plugin.disable_full_page_plugin_for_types: application/pdf
print.printer_Brother_HL-3170CDW_series_Printer.print_bgcolor: false
print.printer_Brother_HL-3170CDW_series_Printer.print_bgimages: false
print.printer_Brother_HL-3170CDW_series_Printer.print_duplex: -437918235
print.printer_Brother_HL-3170CDW_series_Printer.print_edge_bottom: 0
print.printer_Brother_HL-3170CDW_series_Printer.print_edge_left: 0
print.printer_Brother_HL-3170CDW_series_Printer.print_edge_right: 0
print.printer_Brother_HL-3170CDW_series_Printer.print_edge_top: 0
print.printer_Brother_HL-3170CDW_series_Printer.print_evenpages: true
print.printer_Brother_HL-3170CDW_series_Printer.print_footercenter:
print.printer_Brother_HL-3170CDW_series_Printer.print_footerleft: &PT
print.printer_Brother_HL-3170CDW_series_Printer.print_footerright: &D
print.printer_Brother_HL-3170CDW_series_Printer.print_headercenter:
print.printer_Brother_HL-3170CDW_series_Printer.print_headerleft: &T
print.printer_Brother_HL-3170CDW_series_Printer.print_headerright: &U
print.printer_Brother_HL-3170CDW_series_Printer.print_in_color: true
print.printer_Brother_HL-3170CDW_series_Printer.print_margin_bottom: 0.5
print.printer_Brother_HL-3170CDW_series_Printer.print_margin_left: 0.5
print.printer_Brother_HL-3170CDW_series_Printer.print_margin_right: 0.5
print.printer_Brother_HL-3170CDW_series_Printer.print_margin_top: 0.5
print.printer_Brother_HL-3170CDW_series_Printer.print_oddpages: true
print.printer_Brother_HL-3170CDW_series_Printer.print_orientation: 0
print.printer_Brother_HL-3170CDW_series_Printer.print_page_delay: 50
print.printer_Brother_HL-3170CDW_series_Printer.print_paper_data: 1
print.printer_Brother_HL-3170CDW_series_Printer.print_paper_height: 11.00
print.printer_Brother_HL-3170CDW_series_Printer.print_paper_name:
print.printer_Brother_HL-3170CDW_series_Printer.print_paper_size_unit: 0
print.printer_Brother_HL-3170CDW_series_Printer.print_paper_width: 8.50
print.printer_Brother_HL-3170CDW_series_Printer.print_resolution: 600
print.printer_Brother_HL-3170CDW_series_Printer.print_reversed: false
print.printer_Brother_HL-3170CDW_series_Printer.print_scaling: 1.00
print.printer_Brother_HL-3170CDW_series_Printer.print_shrink_to_fit: true
print.printer_Brother_HL-3170CDW_series_Printer.print_to_file: false
print.printer_Brother_HL-3170CDW_series_Printer.print_unwriteable_margin_bottom: 0
print.printer_Brother_HL-3170CDW_series_Printer.print_unwriteable_margin_left: 0
print.printer_Brother_HL-3170CDW_series_Printer.print_unwriteable_margin_right: 0
print.printer_Brother_HL-3170CDW_series_Printer.print_unwriteable_margin_top: 0
print.printer_Microsoft_Print_to_PDF.print_bgcolor: false
print.printer_Microsoft_Print_to_PDF.print_bgimages: false
print.printer_Microsoft_Print_to_PDF.print_duplex: -437918235
print.printer_Microsoft_Print_to_PDF.print_edge_bottom: 0
print.printer_Microsoft_Print_to_PDF.print_edge_left: 0
print.printer_Microsoft_Print_to_PDF.print_edge_right: 0
print.printer_Microsoft_Print_to_PDF.print_edge_top: 0
print.printer_Microsoft_Print_to_PDF.print_evenpages: true
print.printer_Microsoft_Print_to_PDF.print_footercenter:
print.printer_Microsoft_Print_to_PDF.print_footerleft: &PT
print.printer_Microsoft_Print_to_PDF.print_footerright: &D
print.printer_Microsoft_Print_to_PDF.print_headercenter:
print.printer_Microsoft_Print_to_PDF.print_headerleft: &T
print.printer_Microsoft_Print_to_PDF.print_headerright: &U
print.printer_Microsoft_Print_to_PDF.print_in_color: true
print.printer_Microsoft_Print_to_PDF.print_margin_bottom: 0.5
print.printer_Microsoft_Print_to_PDF.print_margin_left: 0.5
print.printer_Microsoft_Print_to_PDF.print_margin_right: 0.5
print.printer_Microsoft_Print_to_PDF.print_margin_top: 0.5
print.printer_Microsoft_Print_to_PDF.print_oddpages: true
print.printer_Microsoft_Print_to_PDF.print_orientation: 1
print.printer_Microsoft_Print_to_PDF.print_page_delay: 50
print.printer_Microsoft_Print_to_PDF.print_paper_data: 1
print.printer_Microsoft_Print_to_PDF.print_paper_height: -1.00
print.printer_Microsoft_Print_to_PDF.print_paper_name:
print.printer_Microsoft_Print_to_PDF.print_paper_size_unit: 0
print.printer_Microsoft_Print_to_PDF.print_paper_width: -1.00
print.printer_Microsoft_Print_to_PDF.print_resolution: 600
print.printer_Microsoft_Print_to_PDF.print_reversed: false
print.printer_Microsoft_Print_to_PDF.print_scaling: 1.00
print.printer_Microsoft_Print_to_PDF.print_shrink_to_fit: true
print.printer_Microsoft_Print_to_PDF.print_to_file: false
print.printer_Microsoft_Print_to_PDF.print_unwriteable_margin_bottom: 0
print.printer_Microsoft_Print_to_PDF.print_unwriteable_margin_left: 0
print.printer_Microsoft_Print_to_PDF.print_unwriteable_margin_right: 0
print.printer_Microsoft_Print_to_PDF.print_unwriteable_margin_top: 0
security.sandbox.content.level: 3
security.sandbox.content.tempDirSuffix: {5d71749d-c83c-4aef-b8e5-daa66def99f2}
services.sync.declinedEngines:
services.sync.engine.addresses.available: true
storage.vacuum.last.index: 1
storage.vacuum.last.places.sqlite: 1505758233
ui.osk.debug.keyboardDisplayReason: IKPOS: Touch screen not found.

Important Locked Preferences
----------------------------

Places Database
---------------

JavaScript
----------

Incremental GC: true

Accessibility
-------------

Activated: true
Prevent Accessibility: 0
Accessible Handler Used: true
Accessibility Instantiator: UNKNOWN|

Library Versions
----------------

NSPR
Expected minimum version: 4.17 Beta
Version in use: 4.17 Beta

NSS
Expected minimum version: 3.33 Beta
Version in use: 3.33 Beta

NSSSMIME
Expected minimum version: 3.33 Beta
Version in use: 3.33 Beta

NSSSSL
Expected minimum version: 3.33 Beta
Version in use: 3.33 Beta

NSSUTIL
Expected minimum version: 3.33 Beta
Version in use: 3.33 Beta

Experimental Features
---------------------

Sandbox
-------

Content Process Sandbox Level: 3
Effective Content Process Sandbox Level: 3

Comment 20

[u602632]

Also, prior to fixing the crash though that setting change, the bug persisted through safe mode and a refresh.

Comment 21

[Jim Mathies [:jimm]]

Hey [e-mail], do you have any idea why accessibility is active on your laptop?

Comment 22

[Jim Mathies [:jimm]]

Also, if possible, it'd be great if you could turn webroot secureeverywhere browser related protections off and retest with the sandbox pref set to 4 to see if that addresses the issue as well.

Comment 23

[u602632]

No I don't. And I think I remember trying to force it off because I wanted multiprocess when it came to 56. 
I was never able to get it to turn off. 
I'm pretty sure I have nothing enabled within the a11y settings in Windows.

Comment 24

[u602632]

The WebRoot add-on is disabled (there's no option to remove it) and always has been. That's not a feature I use.

Comment 25

[Hugh Nougher [:Hughman]]

(In reply to Jim Mathies [:jimm] from comment #22)
> Also, if possible, it'd be great if you could turn webroot secureeverywhere
> browser related protections off and retest with the sandbox pref set to 4 to
> see if that addresses the issue as well.

I have Webroot, I have crash, I had not touched any settings.
As you suggest, disabling a feature of Webroot (Identity Shield) fixed the crashes.
The other two (Web Shield and Phishing Shield) and the Addon do not have any effect.

Comment 26

[u602632]

(In reply to Hugh Nougher [:Hughman] from comment #25)
> (In reply to Jim Mathies [:jimm] from comment #22)
> > Also, if possible, it'd be great if you could turn webroot secureeverywhere
> > browser related protections off and retest with the sandbox pref set to 4 to
> > see if that addresses the issue as well.
> 
> I have Webroot, I have crash, I had not touched any settings.
> As you suggest, disabling a feature of Webroot (Identity Shield) fixed the
> crashes.
> The other two (Web Shield and Phishing Shield) and the Addon do not have any
> effect.

I didn't think about disabling the shields from within Webroot. 
I disabled Identity Shield and reset my sandbox level to 4 and it fixed the crashes.

Thanks Hugh Nougher

Comment 27

[u602632]

(In reply to mrelliottfawcett from comment #26)
> (In reply to Hugh Nougher [:Hughman] from comment #25)
> > (In reply to Jim Mathies [:jimm] from comment #22)
> > > Also, if possible, it'd be great if you could turn webroot secureeverywhere
> > > browser related protections off and retest with the sandbox pref set to 4 to
> > > see if that addresses the issue as well.
> > 
> > I have Webroot, I have crash, I had not touched any settings.
> > As you suggest, disabling a feature of Webroot (Identity Shield) fixed the
> > crashes.
> > The other two (Web Shield and Phishing Shield) and the Addon do not have any
> > effect.
> 
> I didn't think about disabling the shields from within Webroot. 
> I disabled Identity Shield and reset my sandbox level to 4 and it fixed the
> crashes.
> 
> Thanks Hugh Nougher

Scratch that. It's crashing again. For some reason it worked for a few minutes and then started crashing again. 
I have all web-related shields turned off now.
I noticed something else before it started crashing, uBlock Origin wasn't blocking ads. 

It displayed this on a page that it normally blocks ads on:
requests blocked
 on this page
?
 since install
?

Comment 28

[makotech222]

To add more evidence, my work computer also has webroot security, so maybe that could be the cause. Can't disable though, because its a work computer.

Comment 29

[Jim Mathies [:jimm]]

Next steps here, purchase or install the trial of this anti-virus package and debug the issue.

Comment 30

[[:philipp]]

Correlations for Firefox Nightly
(63.27% in signature vs 05.47% overall) Module "WRusr.dll" = true
(19.91% in signature vs 01.74% overall) Module "k7pswsen.dll" = true [58.99% vs 06.08% if platform_pretty_version = Windows 7]

hi, it looks like your security products are involved in this type of crash - do you have any insights why this would happen with the stricter sandboxing settings that firefox 57 introduces and/or look into the matter?

Comment 31

[Bob Owen (:bobowen)]

(In reply to Andrew Osmond [:aosmond] from comment #9)
> I went through the pushlog where the original crash was introduced, and
> these are the most relevant startup/loading URI/shutdown related changes I
> could find. That's not to say I blame any/all of them, I just wasn't
> confident enough to eliminate them:

Was this the pushlog from comment 2?

Comment 32

[Andrew Osmond [:aosmond] (he/him)]

(In reply to Bob Owen (:bobowen) from comment #31)
> (In reply to Andrew Osmond [:aosmond] from comment #9)
> > I went through the pushlog where the original crash was introduced, and
> > these are the most relevant startup/loading URI/shutdown related changes I
> > could find. That's not to say I blame any/all of them, I just wasn't
> > confident enough to eliminate them:
> 
> Was this the pushlog from comment 2?

No, this would have been from bug 1397214 comment 3. We tried landing a fix by checking the shutdown state during those IPDL calls as an experiment -- the signature in this bug is just a morphed version of that.

Comment 33

[Jim Mathies [:jimm]]

This is also a new signature for OSX starting at 9-7:

https://crash-stats.mozilla.com/search/?signature=%3Dmozilla%3A%3Aipc%3A%3AMessageChannel%3A%3AWillDestroyCurrentMessageLoop&product=Firefox&platform=Mac%20OS%20X&date=%3E%3D2017-08-01T15%3A47%3A00.000Z&date=%3C2017-09-22T15%3A47%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

Comment 34

[Bob Owen (:bobowen)]

(In reply to Andrew Osmond [:aosmond] from comment #32)
...
> > Was this the pushlog from comment 2?
> 
> No, this would have been from bug 1397214 comment 3. We tried landing a fix
> by checking the shutdown state during those IPDL calls as an experiment --
> the signature in this bug is just a morphed version of that.

Ah right thanks, if I include that signature then there is a jump on build 20170905220108 where this landed.
I've managed to reproduce with webroot installed, but it's fairly infrequent, which seems surprising with the sandbox change.

Anyway, we're going to back it out of Beta, so we'll know for sure.

Comment 35

[Bob Owen (:bobowen)]

I'm convinced now that this is an issue with Webroot SecureAnywhere and the Alternate Desktop, or at least it's making an existing crash much worse.

I can reproduce fairly regularly on a VM with Webroot with Alternate Desktop (but not with level 3) and not on a very similar VM without Webroot, but with level 4.

We should be back to level 3 in Beta 3 (being built on Monday) assuming bug 1402340 gets uplifted, we're leaving this turned on for Nightly for the moment.

I'll see if I can work out why Webroot is causing this crash on Monday, my initial guess is we have an existing race that is getting triggered more often because Webroot is delaying something as it's getting blocked trying to use something from the default desktop.

Comment 36

[Calixte Denizet (:calixte)]

For information, there are 13744 for 668 installations in 57.0b0 (dev edition).

Comment 37

[Jim Mathies [:jimm]]

Not sure yet what's going on here, but it appears we have a few users from the 23rd running with level 4. The numbers are pretty small though.

example report:
https://crash-stats.mozilla.com/report/index/f8ad39f6-433d-436c-8edf-52bd00170924#tab-details
checkin:
https://hg.mozilla.org/mozilla-central/rev/80f6fc604e5a
listing:
https://crash-stats.mozilla.com/search/?signature=%3Dmozilla%3A%3Alayers%3A%3AImageBridgeChild%3A%3AInitForContent&build_id=%3E%3D20170923000000&product=Firefox&version=58.0a1&version=57.0a1&date=%3E%3D2017-09-17T12%3A45%3A00.000Z&date=%3C2017-09-24T12%3A45%3A00.000Z&_sort=-build_id&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

Comment 38

[Jim Mathies [:jimm]]

(In reply to Calixte Denizet (:calixte) from comment #36)
> For information, there are 13744 for 668 installations in 57.0b0 (dev
> edition).

Hey Adam, do we have contacts for Webroot [1]? This bug represents yet another stability problem in Firefox introduced by their software.

[1] https://www.webroot.com/

Comment 39

[Nicholas Nethercote [inactive]]

Just to pile on: the crash numbers here are disastrously high :(  On several recently Nightly builds this is the #1 crash, even ahead of ShutdownKill crashes, which are normally by far the highest.

Comment 40

[Gregory Panakkal]

Our QA has simulated the crash (related to k7pswsen.dll) on our end. 
We're investigating the cause of this issue.

Comment 41

[Adam Stevenson [:adamopenweb]]

Jim, we do, I've reached out by email and copied you in.

Comment 42

[Johnny Shaw]

I'm not seeing any steps to reproduce here (maybe I'm just not finding it, sorry). I've tried installing the nightly build with Webroot, going to a few websites, and I'm not getting a crash. If someone could provide some detailed steps to reproduce I'll look into this from our side.

Comment 43

[Bob Owen (:bobowen)]

(In reply to Johnny Shaw from comment #42)
> I'm not seeing any steps to reproduce here (maybe I'm just not finding it,
> sorry). I've tried installing the nightly build with Webroot, going to a few
> websites, and I'm not getting a crash. If someone could provide some
> detailed steps to reproduce I'll look into this from our side.

Hi Johnny thanks for looking at this.

The problem seems to occur when a new process shuts down early, for some not yet understood reason.

So I'm reproducing by:
 * have just one tab open on "about:robots"
 * navigate this tab to "https://example.com/"
 * then use Alt+left arrow/Alt+right arrow to flip between the two

This should cause a new process to start each time you go to https://example.com/ and eventually you'll get the crash.
I've only seen it for 64-bit (which I think is the default Nightly download now).

Comment 44

[[:philipp]]

in current nightlys "security.sandbox.content.level" was set back to 3 to work around the issue for the moment - before testing you'd also need to set it to 4 again...

Comment 45

[Johnny Shaw]

FWIW, my security.sandbox.content.level was already set to 4 in the nightly build (58.0a1).

Just so I know I'm working with the same crash. This is the stack?:

00 0000006a`b4bfe6b0 00007ffe`b9f7119f xul!soundtouch::SoundTouch::operator=+0xffa6
01 0000006a`b4bfe720 00007ffe`b9e315bc xul!soundtouch::SoundTouch::operator=+0x182aa7
02 0000006a`b4bfe760 00007ffe`b90be99e xul!soundtouch::SoundTouch::operator=+0x42ec4
03 0000006a`b4bfe880 00007ffe`b90be352 xul+0x67e99e
04 0000006a`b4bfeae0 00007ffe`b90bcd7b xul+0x67e352
05 0000006a`b4bfebc0 00007ffe`b90bcc5c xul+0x67cd7b
06 0000006a`b4bfec00 00007ffe`b90c1b20 xul+0x67cc5c
07 0000006a`b4bfec30 00007ffe`b90bf1f5 xul+0x681b20
08 0000006a`b4bff280 00007ffe`b9dfef38 xul+0x67f1f5
09 0000006a`b4bff2f0 00007ffe`b92fdde7 xul!soundtouch::SoundTouch::operator=+0x10840
0a 0000006a`b4bff320 00007ffe`b92fdd98 xul+0x8bdde7
0b 0000006a`b4bff350 00007ffe`b9617aa8 xul+0x8bdd98
0c 0000006a`b4bff3a0 00007ffe`b9617794 xul+0xbd7aa8
0d 0000006a`b4bff3d0 00007ffe`bb6361f2 xul+0xbd7794
0e 0000006a`b4bff400 00007ffe`b9dfeef1 xul!workerlz4_decompress+0x9d1ba
0f 0000006a`b4bff430 00007ffe`b92fdde7 xul!soundtouch::SoundTouch::operator=+0x107f9
10 0000006a`b4bff460 00007ffe`b92fdd98 xul+0x8bdde7
11 0000006a`b4bff490 00007ffe`bb63601c xul+0x8bdd98
12 0000006a`b4bff4e0 00007ff7`b0d7c43b xul!workerlz4_decompress+0x9cfe4
13 0000006a`b4bff710 00007ff7`b0d784a9 firefox!TargetEnumDisplayDevicesA64+0x4baf
14 0000006a`b4bff750 00007ff7`b0d711d0 firefox!TargetEnumDisplayDevicesA64+0xc1d
15 0000006a`b4bffb50 00007ff7`b0d76e95 firefox+0x11d0
16 0000006a`b4bffba0 00007fff`201e2774 firefox+0x6e95
17 0000006a`b4bffbe0 00007fff`20960d51 KERNEL32!BaseThreadInitThunk+0x14
18 0000006a`b4bffc10 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Comment 46

[Bob Owen (:bobowen)]

Yes I left it at 4 on Nightly.

Instructions for getting symbols can be found at:
https://developer.mozilla.org/en-US/docs/Mozilla/Using_the_Mozilla_symbol_server

Comment 47

[Bob Owen (:bobowen)]

I've added WRusr.dll to the chromium sandbox blocklist and it does prevent it from being loaded in the content process.
I've not managed to reproduce while this is blocked.

So, I'm tempted to land this for Nightly.
It will mean that further investigations will need to carried out on an older version, but that shouldn't be too much of a pain if updates are turned off in the profile.

Comment 48

[Jim Mathies [:jimm]]

(In reply to Bob Owen (:bobowen) from comment #47)
> I've added WRusr.dll to the chromium sandbox blocklist and it does prevent
> it from being loaded in the content process.
> I've not managed to reproduce while this is blocked.
> 
> So, I'm tempted to land this for Nightly.
> It will mean that further investigations will need to carried out on an
> older version, but that shouldn't be too much of a pain if updates are
> turned off in the profile.

Lets get this landed. If it fixes the issue with the alt-desktop option for content processes we want it. I'm not going to block good security improvements on 3rd parties.

Comment 49

[Bob Owen (:bobowen)]

Getting a bit further ...

For some reason the do_GetService(kAppShellCID)) is failing at [1].
So we never actually start running it and shutdown instead.
The initialisation events get called during the shutdown and crash.

The error further down inside do_GetService is NS_ERROR_FACTORY_NOT_REGISTERED.

[1] https://hg.mozilla.org/mozilla-central/file/bc5672989895/toolkit/xre/nsEmbedFunctions.cpp#l837

Comment 50

[Johnny Shaw]

Just a question from the Webroot side. Blocking WRusr in the sandbox, would that block it from injecting into Firefox completely or just the sandbox? Or, are you sandboxing WRusr to look for exploitative behavior?

I ask since blocking WRusr from injecting completely would disable some major value to our customers and Webroot would likely ask me inject in another manner to get around it. I prefer we work together to solve this and find a compatible solution.

That said, we don't want to prevent any protection your sandbox is providing to Firefox users.

Sorry I didn't find time last night to dig into the problem I'm making time today to handle this.

Comment 51

[Bob Owen (:bobowen)]

(In reply to Johnny Shaw from comment #50)
> Just a question from the Webroot side. Blocking WRusr in the sandbox, would
> that block it from injecting into Firefox completely or just the sandbox?
> Or, are you sandboxing WRusr to look for exploitative behavior?

This would just be into the sandboxed content and Gecko Media Plugin child processes (the ones that use alternate desktop). So it is still loaded in the main process and any other children that don't use alternate desktop.

> I ask since blocking WRusr from injecting completely would disable some
> major value to our customers and Webroot would likely ask me inject in
> another manner to get around it. I prefer we work together to solve this and
> find a compatible solution.

Absolutely, I'm still investigating as well.
This gives us an initial work around though.

Comment 52

[Johnny Shaw]

(In reply to Bob Owen (:bobowen) from comment #51)
> (In reply to Johnny Shaw from comment #50)
> > Just a question from the Webroot side. Blocking WRusr in the sandbox, would
> > that block it from injecting into Firefox completely or just the sandbox?
> > Or, are you sandboxing WRusr to look for exploitative behavior?
> 
> This would just be into the sandboxed content and Gecko Media Plugin child
> processes (the ones that use alternate desktop). So it is still loaded in
> the main process and any other children that don't use alternate desktop.

This is some great info! I think I have an idea of where the problem might be at. Creating WRusr in an alternative desktop might be trigger our own sandboxing. Good to know!

> 
> > I ask since blocking WRusr from injecting completely would disable some
> > major value to our customers and Webroot would likely ask me inject in
> > another manner to get around it. I prefer we work together to solve this and
> > find a compatible solution.
> 
> Absolutely, I'm still investigating as well.
> This gives us an initial work around though.

Sounds good, I'll let you know what I find.

Comment 53

[Jim Mathies [:jimm]]

(In reply to Johnny Shaw from comment #50)
> Just a question from the Webroot side. Blocking WRusr in the sandbox, would
> that block it from injecting into Firefox completely or just the sandbox?
> Or, are you sandboxing WRusr to look for exploitative behavior?
> 
> I ask since blocking WRusr from injecting completely would disable some
> major value to our customers and Webroot would likely ask me inject in
> another manner to get around it. I prefer we work together to solve this and
> find a compatible solution.

Would you detail what your software does and why it does it in each of our processes?

FYI we have standardized extension mechanisms in Firefox we provide and maintain for companies like yours. We have an extension internal mechanism [1] which fyi some anti-virus and corporate snoop vendors are now leveraging. We also have accessibility interfaces which provide complete access to content and events surrounding user actions. More advanced screen reading and content assistance software packages use this to monitor and assist the user in content.

Do these mechanisms not address what you want to provide to customers?

[1] https://developer.mozilla.org/en-US/Add-ons/WebExtensions

Comment 54

[Johnny Shaw]

There are a few different flavors that WRusr runs in. For WRusr, at high level for browser protection, we do key input protection to ensure what the user is typing isn't being pulled from the browser, we also protect against some arbitrary code execution. We also obviously hook various "dangerous" API calls to monitor for malicious intent. I'm not sure how much detail I can't get into here, so I won't risk treading any further.

There are other flavors (determined at runtime) WRusr is instantiated for. But, based on what I've seen in my reproduction so far, don't apply for this, and isn't relevant to browsers.

There are few legacy systems that rely on us scraping the URL from the window for analysis, this isn't common these days and is most always handled by our browser plugins instead of WRusr. Which (I assume) utilize what your describing. However, I don't work on the browser plugins so I couldn't say for sure.

I sent a email to Bob Owen after looking over the reproduction I did. I'm not sure how much technical detail (disassembly and such) you guys want in these bug threads. So, I figured we'd handle the discussion further out-of-band of this thread. Bob, please feel free to add anyone relevant to Mozilla to that email or let me know I'm not reproducing the same issue.

Comment 55

[Bob Owen (:bobowen)]

Tracked it down to a failure in CreateWindowW at [1].

The last error is:
LastErrorValue: (Win32) 0x5 (5) - Access is denied.

A pretty common error when the sandbox is blocking something.

Johhny - is there a chance that you are trying to do something at this point?
Possibly something that tries to access something from another desktop.

Seems slightly odd that the failure is intermittent.

[1] https://hg.mozilla.org/mozilla-central/file/35fbf14b96a6/widget/windows/nsAppShell.cpp#l196

Comment 56

[Johnny Shaw]

Any filtering we do in that area is purely passive. In other words we wouldn't deny access to that resource. Let me give you a build with a patch in it which disables that area of passive monitoring and see if it helps. I'll email it to you directly.

Comment 57

[[:philipp]]

so just that it doesn't get lost: webroot isn't the only software able to trigger this crash. according to correlation data is apparently also happening with https://www.k7computing.com and they are able to reproduce that issue as well (comment #40).

Comment 58

[Bob Owen (:bobowen)]

(In reply to [:philipp] from comment #57)
> so just that it doesn't get lost: webroot isn't the only software able to
> trigger this crash. according to correlation data is apparently also
> happening with https://www.k7computing.com and they are able to reproduce
> that issue as well (comment #40).

Yes and there are others as well.
It looks like we might need to stop these from loading in the content process as well, it's not entirely clear why they would need to be in this process anyway.

(In reply to Gregory Panakkal from comment #40)
> Our QA has simulated the crash (related to k7pswsen.dll) on our end. 
> We're investigating the cause of this issue.

Gregory - have you managed to get to the root cause of the problem in your case?


In the Webroot case, there doesn't appear to be any standard hooking of any of the calls involved in the CreateWindowW call 
One thing I have noticed when it crashes, WRusr.dll seems to have loaded earlier and before any of our DLLs:

ModLoad: 000007fe`feea0000 000007fe`fef7b000   C:\Windows\system32\ADVAPI32.dll
ModLoad: 000007fe`fe860000 000007fe`fe87f000   C:\Windows\SYSTEM32\sechost.dll
ModLoad: 000007fe`ff2a0000 000007fe`ff3cd000   C:\Windows\system32\RPCRT4.dll
*ModLoad: 000007fe`f2360000 000007fe`f2384000   C:\Windows\system32\WRusr.dll*
ModLoad: 00000000`76f60000 00000000`7705a000   C:\Windows\system32\USER32.dll
ModLoad: 000007fe`fe7f0000 000007fe`fe857000   C:\Windows\system32\GDI32.dll
ModLoad: 000007fe`febf0000 000007fe`febfe000   C:\Windows\system32\LPK.dll
ModLoad: 000007fe`fe720000 000007fe`fe7eb000   C:\Windows\system32\USP10.dll
ModLoad: 000007fe`fd230000 000007fe`fdfba000   C:\Windows\system32\SHELL32.dll
ModLoad: 000007fe`fe480000 000007fe`fe4f1000   C:\Windows\system32\SHLWAPI.dll
ModLoad: 000007fe`fec00000 000007fe`fedfc000   C:\Windows\system32\ole32.dll
ModLoad: 00000000`77340000 00000000`77347000   C:\Windows\system32\PSAPI.DLL
ModLoad: 000007fe`f6540000 000007fe`f6594000   C:\Windows\system32\OLEACC.dll
ModLoad: 000007fe`fef80000 000007fe`ff05a000   C:\Windows\system32\OLEAUT32.dll
ModLoad: 000007fe`f6850000 000007fe`f6857000   C:\Windows\system32\MSIMG32.dll
ModLoad: 000007fe`fe6f0000 000007fe`fe71e000   C:\Windows\system32\IMM32.DLL
ModLoad: 000007fe`fe560000 000007fe`fe669000   C:\Windows\system32\MSCTF.dll
ModLoad: 000007fe`fce70000 000007fe`fce7f000   C:\Windows\system32\profapi.dll
*ModLoad: 000007fe`f2e60000 000007fe`f2fd9000   C:\Program Files\Nightly\nss3.dll*

As opposed to when it works:

ModLoad: 000007fe`feea0000 000007fe`fef7b000   C:\Windows\system32\ADVAPI32.dll
ModLoad: 000007fe`fe860000 000007fe`fe87f000   C:\Windows\SYSTEM32\sechost.dll
ModLoad: 000007fe`ff2a0000 000007fe`ff3cd000   C:\Windows\system32\RPCRT4.dll
*ModLoad: 000007fe`f2e60000 000007fe`f2fd9000   C:\Program Files\Nightly\nss3.dll*
ModLoad: 000007fe`face0000 000007fe`fad1b000   C:\Windows\system32\WINMM.dll
ModLoad: 00000000`76f60000 00000000`7705a000   C:\Windows\system32\USER32.dll
ModLoad: 000007fe`fe7f0000 000007fe`fe857000   C:\Windows\system32\GDI32.dll
ModLoad: 000007fe`febf0000 000007fe`febfe000   C:\Windows\system32\LPK.dll
ModLoad: 000007fe`fe720000 000007fe`fe7eb000   C:\Windows\system32\USP10.dll
ModLoad: 000007fe`f3410000 000007fe`f3419000   C:\Windows\system32\WSOCK32.dll
ModLoad: 000007fe`ff250000 000007fe`ff29d000   C:\Windows\system32\WS2_32.dll
ModLoad: 000007fe`ff240000 000007fe`ff248000   C:\Windows\system32\NSI.dll
ModLoad: 000007fe`fe6f0000 000007fe`fe71e000   C:\Windows\system32\IMM32.DLL
ModLoad: 000007fe`fe560000 000007fe`fe669000   C:\Windows\system32\MSCTF.dll
*ModLoad: 000007fe`f2360000 000007fe`f2384000   C:\Windows\system32\WRusr.dll*

Comment 59

[Gregory Panakkal]

(In reply to Bob Owen (:bobowen) from comment #58)
> (In reply to [:philipp] from comment #57)
> > so just that it doesn't get lost: webroot isn't the only software able to
> .....
> ModLoad: 000007fe`fe6f0000 000007fe`fe71e000   C:\Windows\system32\IMM32.DLL
> ModLoad: 000007fe`fe560000 000007fe`fe669000   C:\Windows\system32\MSCTF.dll
> *ModLoad: 000007fe`f2360000 000007fe`f2384000  
> C:\Windows\system32\WRusr.dll*

Our team is still investigating this issue. (We had an extended weekend due to a string of holidays in India)

To answer your question on 'why they would need to be in this process anyway':
We have functionality to intercept http[s] urls within firefox process ( for web-filtering ), without having to use a extension/plugin. We achieve it by subscribing to nsIWebProgressListener::OnStateChange(). In the current firefox stable channel, we needed to do this in content process as well to intercept all the urls.

Comment 60

[Bob Owen (:bobowen)]

(In reply to Gregory Panakkal from comment #59)
...
> To answer your question on 'why they would need to be in this process
> anyway':
> We have functionality to intercept http[s] urls within firefox process ( for
> web-filtering ), without having to use a extension/plugin. We achieve it by
> subscribing to nsIWebProgressListener::OnStateChange(). In the current
> firefox stable channel, we needed to do this in content process as well to
> intercept all the urls.

I believe we have other mechanisms to achieve the same thing that will be much more reliable.

Jason - I'm told you're the best person to point Gregory in the right direction here.

Comment 61

[Nicholas Nethercote [inactive]]

FWIW: This is the #2 Windows topcrash in Nightly 20171007100142, with 51 occurrences so far.

Comment 62

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Looking at the crash volume, this one is not a blocker for 57 anymore. Philipp, FYI, lemme know if you disagree.

Comment 63

[[:philipp]]

yes the workaround in bug 1402340 helped a great deal to reduce the volume on beta.

Comment 64

[Jason Duell]

> We have functionality to intercept http[s] urls within firefox process ( for
> web-filtering ), without having to use a extension/plugin. We achieve it by
> subscribing to nsIWebProgressListener::OnStateChange()

Gregory: can you tell me what you're doing when you intercept the URIs?  Are you 1) blocking/canceling some channels? 2) Sniffing their content?  3) Modifying their content?  4) Modifying things like headers/cookies? 5) Changing/redirecting the URI? 

needinfo me--I get too much bugmail to reliably see your reply otherwise.

Comment 65

[Jason Duell]

Gregory:  also, which StateChange events are you listening for?  We have various necko hooks but the timing of the hooks (vis a vis channel lifetime) varies.

Comment 66

[Gregory Panakkal]

(In reply to Jason Duell [:jduell] (needinfo me) from comment #64)
> > We have functionality to intercept http[s] urls within firefox process ( for
> > web-filtering ), without having to use a extension/plugin. We achieve it by
> > subscribing to nsIWebProgressListener::OnStateChange()
> 
> Gregory: can you tell me what you're doing when you intercept the URIs?  Are
> you 1) blocking/canceling some channels? 2) Sniffing their content?  3)
> Modifying their content?  4) Modifying things like headers/cookies? 5)
> Changing/redirecting the URI? 
> 
> needinfo me--I get too much bugmail to reliably see your reply otherwise.

We have slightly different implementation for older version of Firefox and the live release-channel version.
We basically pull out the URI using aRequest->GetName(). 
In case, we identify that the URI needs to be blocked:
 = For latest version of Firefox we call
    aRequest->Cancel(NS_ERROR_PROXY_CONNECTION_REFUSED) ;
 = For earlier version of Firefox
    We redirect it to a block page by invoking pWebNav->LoadURI(<redirected url>)

An update to the original crash issue, our team has identified that loading of non-kernel32.dll modules [eg:- user32.dll] into the sandboxed process prior to the loading firefox modules was causing the crash. We are testing internally (with sandbox level 4) with a version of K7PSWSEn.dll that delay-loads non-kernel32.dll dlls only after firefox modules start loading - and seems to avoid the crash. We're investigating in parallel what is the root-cause of the original crash.

Comment 67

[Gregory Panakkal]

(In reply to Jason Duell [:jduell] (needinfo me) from comment #65)
> Gregory:  also, which StateChange events are you listening for?  We have
> various necko hooks but the timing of the hooks (vis a vis channel lifetime)
> varies.

We do the following:
1. Get hold of Service Manager [through documented, and undocumented methods]
2. Call svcmgr->GetServiceByContractID(NS_DOCUMENTLOADER_SERVICE_CONTRACTID, .. WebProgress ..)
3. Call pWebProgress->AddProgressListener(pObj, nsIWebProgress::NOTIFY_STATE_REQUEST|nsIWebProgress::NOTIFY_STATE_DOCUMENT)

In CWebProgressListener::OnStateChange(..):
We process only calls where aStateFlags has one of STATE_START STATE_IS_DOCUMENT flags on; and once we pull out the URI, we further process only http[s] URIs (see above post for further info)

Comment 68

[Valentin Gosu [:valentin] (he/him)]

(In reply to Gregory Panakkal from comment #66)
> An update to the original crash issue, our team has identified that loading
> of non-kernel32.dll modules [eg:- user32.dll] into the sandboxed process
> prior to the loading firefox modules was causing the crash. We are testing
> internally (with sandbox level 4) with a version of K7PSWSEn.dll that
> delay-loads non-kernel32.dll dlls only after firefox modules start loading -
> and seems to avoid the crash. We're investigating in parallel what is the
> root-cause of the original crash.

Have you looked at/Are you considering using the webextension APIs?
https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Intercept_HTTP_requests

From what I can tell they should offer most of the features you're looking for, and have the added benefit that they are stable.
Injecting DLLs into Firefox doesn't seem very sustainable, as any change we make to our implementation may eventually cause a crash, and these breaking changes will come at a much faster pace now.

Comment 69

[Gregory Panakkal]

(In reply to Valentin Gosu [:valentin] from comment #68)
> (In reply to Gregory Panakkal from comment #66)
> > An update to the original crash issue, our team has identified that loading
> > of non-kernel32.dll modules [eg:- user32.dll] into the sandboxed process
> > prior to the loading firefox modules was causing the crash. We are testing
> > internally (with sandbox level 4) with a version of K7PSWSEn.dll that
> > delay-loads non-kernel32.dll dlls only after firefox modules start loading -
> > and seems to avoid the crash. We're investigating in parallel what is the
> > root-cause of the original crash.
> 
> Have you looked at/Are you considering using the webextension APIs?
> https://developer.mozilla.org/en-US/Add-ons/WebExtensions/
> Intercept_HTTP_requests
> 
> From what I can tell they should offer most of the features you're looking
> for, and have the added benefit that they are stable.
> Injecting DLLs into Firefox doesn't seem very sustainable, as any change we
> make to our implementation may eventually cause a crash, and these breaking
> changes will come at a much faster pace now.

The main disadvantage we find is that webextension can be easily disabled (for a user with admin-rights, and in a windows workgroup environment). We had taken this route of injecting a DLL to enforce url filtering even in such environments.
We do have an alternate technology (using local transparent web proxy/mitm) that we use in some of our products to intercept the outgoing URLs - we hope to migration all our products to that method in future.

Comment 70

[Bob Owen (:bobowen)]

(In reply to Gregory Panakkal from comment #66)
...
> An update to the original crash issue, our team has identified that loading
> of non-kernel32.dll modules [eg:- user32.dll] into the sandboxed process
> prior to the loading firefox modules was causing the crash. We are testing
> internally (with sandbox level 4) with a version of K7PSWSEn.dll that
> delay-loads non-kernel32.dll dlls only after firefox modules start loading -
> and seems to avoid the crash. We're investigating in parallel what is the
> root-cause of the original crash.

Thanks for this it tallies with what I've found in comment 58.

I suspect the issue might be because user32.dll is not loading on the main thread, although why this only becomes an issue when using the alternate desktop I'm not sure.

Comment 71

[Marco Castelluccio [:marco]]

[Tracking Requested - why for this release]: High crash volume. We were tracking for 57, but then untracked because there was a workaround introduced with bug 1402340. The workaround is not in 58 though.

Comment 72

[Bob Owen (:bobowen)]

(In reply to Gregory Panakkal from comment #66)
...
> An update to the original crash issue, our team has identified that loading
> of non-kernel32.dll modules [eg:- user32.dll] into the sandboxed process
> prior to the loading firefox modules was causing the crash. We are testing
> internally (with sandbox level 4) with a version of K7PSWSEn.dll that
> delay-loads non-kernel32.dll dlls only after firefox modules start loading -
> and seems to avoid the crash. We're investigating in parallel what is the
> root-cause of the original crash.

Gregory - did you deploy a fix based on this finding?

Comment 73

[Gregory Panakkal]

(In reply to Bob Owen (:bobowen) from comment #72)
> (In reply to Gregory Panakkal from comment #66)
> ...
> > An update to the original crash issue, our team has identified that loading
> > of non-kernel32.dll modules [eg:- user32.dll] into the sandboxed process
> > prior to the loading firefox modules was causing the crash. We are testing
> > internally (with sandbox level 4) with a version of K7PSWSEn.dll that
> > delay-loads non-kernel32.dll dlls only after firefox modules start loading -
> > and seems to avoid the crash. We're investigating in parallel what is the
> > root-cause of the original crash.
> 
> Gregory - did you deploy a fix based on this finding?

We have done a limited release (to our Japanese customers). 
Full rollout we expect this week.

Comment 74

[Gerry Chang [:gchang]]

Per comment #71, track 58+.

Comment 76

[[:philipp]]

this signature amounts to over 50% of content crashes again in the early rollout of 58.0b4 to the general beta population.

Comment 77

[Trevor Skywalker]

Signature report for mozilla::layers::ImageBridgeChild::InitForContent

Showing results from 7 days ago

Operating System
Windows 7 	8444 	88.5%
Windows 10 	505 	5.3%
Windows 8.1 	313 	3.3%
Windows 8 	274 	2.9%

Product
Firefox 	59.0a1 	19 	0.2% 	11
Firefox 	58.0b4 	10238 	93.2% 	558
Firefox 	58.0b3 	278 	2.5% 	19
Firefox 	58.0b0 	44 	0.4% 	1
Firefox 	58.0a1 	325 	3.0% 	58
Firefox 	57.0b99	2 	0.0% 	1
Firefox 	57.0b14	7 	0.1% 	2
Firefox 	57.0b13	15 	0.1% 	1
Firefox 	57.0b12	2 	0.0% 	1
Firefox 	57.0b11	7 	0.1% 	1
Firefox 	57.0b0 	27 	0.2% 	5
Firefox 	57.0a1 	15 	0.1% 	2
Firefox 	57.0 	10 	0.1% 	3
Firefox 	56.0.2 	1 	0.0% 	1

Process Type
content 	9355 	98.1%

Uptime Range
< 1 min 	9536 	100.0%

Architecture
x86 	9345 	98.0%
amd64 	191 	2.0%

Flash Version
[blank] 	9536 	100.0%

Comment 78

[Trevor Skywalker]

Top Crashers for Firefox 58.0b4
Top 50 Crashing Signatures. 7 days ago through a few seconds ago. 

Type: Content

Rank  %   Diff   Signature
2 	36.73% 	new 	mozilla::layers::ImageBridgeChild::InitForContent
rocket application
	10058 	10058 	0 	0 	559 	0 	2017-09-09 	1400637

Comment 79

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We have someone who can reliably reproduce this crash by using Selenium and geckodriver to run an automated test against Firefox. In that case it is done via a Windows task. You can find the report here: https://github.com/mozilla/geckodriver/issues/1134

Also this is the crash report: bp-740876c8-0479-4b55-b2d1-e32bb0180117

Comment 80

[Hugh Nougher [:Hughman]]

OH NO! Firefox 58 release is broken EVERYWHERE that Webroot is installed.
I cannot see any crashes but all the symptoms and fix are the same as this.
I'm part of an MSP that is now panicking for a solution... recommend Chrome?

Comment 81

[Julien Cristau [:jcristau]]

(In reply to Hugh Nougher [:Hughman] from comment #80)
> OH NO! Firefox 58 release is broken EVERYWHERE that Webroot is installed.
> I cannot see any crashes but all the symptoms and fix are the same as this.
> I'm part of an MSP that is now panicking for a solution... recommend Chrome?

Johnny, is this something you're aware of?

Comment 82

[Johnny Shaw]

Thank you for requesting info. We've made changes recently to address compatibility issues with newer versions of Firefox. These changes are currently in beta and being rolled out. I've already forwarded this thread to individuals on our team to assist monitoring of this thread. I'll conveen with them tomorrow and follow up. Thanks for your patience!

Comment 83

[Bob Owen (:bobowen)]

Thanks Johnny.

(In reply to Hugh Nougher [:Hughman] from comment #80)

Looks like this might be a different issue with similar symptoms, I've just installed on a Windows 7 VM and Fx58 seems to be running fine.

Comment 84

[Johnny Shaw]

http://anywhere.webrootcloudav.com/zerol/wsainstall.exe

If you download from that link you should get 9.0.19.36 which has fixes in it for Firefox. Let me know if you continue to experience issues with that version, or have already been using that version.

Since you mention you're from an MSP, we haven't yet released this version to business but the roll out plan will have it coming soon.

Comment 85

[Ben H.]

(In reply to Henrik Skupin (:whimboo) from comment #79)
> We have someone who can reliably reproduce this crash by using Selenium and
> geckodriver to run an automated test against Firefox. In that case it is
> done via a Windows task. You can find the report here:
> https://github.com/mozilla/geckodriver/issues/1134
> 
> Also this is the crash report: bp-740876c8-0479-4b55-b2d1-e32bb0180117

Thanks for pushing my report from github to bugzilla, whimboo. 

Popping in here to add that, to my chagrin, I no longer seem to be able to reproduce this "Selenium + geckodriver + Windows scheduled task" crash in either 59.0a1 or 60.0a1. Bizarre.

As an aside, Webroot was never running on the affected system, nor anything security-related beyond Windows Firewall. And the code that's currently functioning is using whatever the default sandbox content level is.

Comment 86

[Hugh Nougher [:Hughman]]

(In reply to Johnny Shaw from comment #84)
> Since you mention you're from an MSP, we haven't yet released this version
> to business but the roll out plan will have it coming soon.

Ah yes that would be it! >90% of our agents are still 9.0.18.* and yes they are the problem ones.
I knew I was not insane for reporting it in a panic when >10 PCs over 3 clients all had the same issue.
We have adjusted our default policy to disable Identity Shield on all PCs until they have updated.

Comment 87

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Bob, we have a couple of people experiencing this crash by using Selenium and geckodriver. Are there any specific information you would like to have, and I can ask people? Most of them reliably hit this crash. Thanks.

Comment 88

[Bob Owen (:bobowen)]

(In reply to Henrik Skupin (:whimboo) from comment #87)
> Bob, we have a couple of people experiencing this crash by using Selenium
> and geckodriver. Are there any specific information you would like to have,
> and I can ask people? Most of them reliably hit this crash. Thanks.

We've actually stopped trying to create the HWND that was failing and causing the initial problem here.
So it would be interesting if they can still reproduce this issue on Nightly.

It's possible that other things could be going wrong that cause the same symptoms.

Comment 89

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Bob Owen (:bobowen) from comment #88)
> We've actually stopped trying to create the HWND that was failing and
> causing the initial problem here.
> So it would be interesting if they can still reproduce this issue on Nightly.

Can you please tell me when this has been landed and which changeset that is?

Comment 90

[Bob Owen (:bobowen)]

(In reply to Henrik Skupin (:whimboo) from comment #89)
> (In reply to Bob Owen (:bobowen) from comment #88)
> > We've actually stopped trying to create the HWND that was failing and
> > causing the initial problem here.
> > So it would be interesting if they can still reproduce this issue on Nightly.
> 
> Can you please tell me when this has been landed and which changeset that is?

It's been in Nightly 60 from very early on, so certainly any recent Nightly won't be trying to create that HWND.
(Unless someone has manually flipped the dom.ipc.useNativeEventProcessing.content pref.)

https://hg.mozilla.org/mozilla-central/rev/3d064be2fade

If this crash can still be reproduced in Nightly then can you provide STR, please.
It's quite possible that other things could cause these same symptoms.

Comment 91

[Jim Mathies [:jimm]]

Hey Bob, can we close this?

Comment 92

[Bob Owen (:bobowen)]

(In reply to Jim Mathies [:jimm] from comment #91)
> Hey Bob, can we close this?

The initial spike, for which I think this was filed, was due to the Alternate Desktop and some AVs.
It shouldn't fail because of that any more.
However we still do get this error, presumably because other initialisation failures can also cause it.

I guess it would be better if whatever those initialisation problems are would crash, instead of letting us carry on to this point, but I'm not sure we know what they are.
I could take a look through some of the dumps later this week, to see if I can work it out, but it might be difficult without STR.

Comment 93

[Marion Daly [:mdaly]]

:bob, update?

Comment 94

[Bob Owen (:bobowen)]

(In reply to Marion Daly [:mdaly] from comment #93)
> :bob, update?

I don't really have anything to add since comment 92.

Given that the original root cause that resulted in this crash (and bug) has gone, I'm going to close this.
That was basically fixed by not generally processing native events in the content process (bug 1423628) and therefore we don't need the CreateWindowW call that was failing with the Alternate Desktop and some Anti-Virus products.

We do still get this crash, but if it becomes a problem again it will be a different root cause, so we're probably better off tracking that in a new bug.