1364563 - Compositor process fails to composite layers from a content process

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[Mike Conley (:mconley) (:⚙️)]

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0

Build id 20170511063838

about:support attached.



I've hit this twice in the past two days. What I'm experiencing is that I'll switch to a tab, and the tab strip will show the switch completing (so the TabParent must have fired MozLayerTreeReady), but then the content area, where I'd expect the layer tree to have been rendered, remains the same - it shows me the previous tab.

I've attached Visual Studio, and I can tell that the content process is indeed painting the layers and sending the layer tree over to the compositor. The tab is actually "invisible", in that the cursor will change to a pointer / cross hair where the page things that text inputs or links are.

In my about:support, I noticed the following entries in the Graphics section:

(#0) 	CP+[GFX1]: Failed to map bitmap (M).
(#5634) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#5635) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#5636) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#5637) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#5638) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#5639) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#5640) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#5641) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#5642) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#5643) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#5644) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#5645) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#5646) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#5647) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#5648) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.

These seem to just repeat and repeat. Every time I reload, the number on the left side has increased quite a bit, so some kind of device reset is obviously happening here.

I'm running Windows 10 64-bit, and I think I just received the Creators Update.

I've chatted about this with dvander in IRC, and he advised me to file this bug and assign it to him.

Comment 1

[Mike Conley (:mconley) (:⚙️)]

I'm still hitting this about once a day. This effectively makes the tab useless, despite the fact that the content process is running just fine. :/

Comment 2

[Ben Kelly [:bkelly, not reviewing]]

I see this periodically as well.

Comment 3

[David Anderson [:dvander] - inactive, e-mail if emergency]

What is strange is that we're getting a device reset in content without a similar one from the GPU process. Usually they happen at the same time and it's safe for content to ignore it. If only content gets the device reset state, it'll never get a new device from the compositor, and it'll effectively stop painting.

Probably the sanest thing to do is to send a message to the compositor telling it that the device reset. If the compositor didn't also get a reset, content would recreate its devices.

Comment 4

[David Anderson [:dvander] - inactive, e-mail if emergency]

Attachment: part 1, fix potential memory leak in TextureSourceProvider

TextureSourceProvider doesn't automatically flush pending texture notifications. The compositor does it manually. It's easier to just make it automatic.

Comment 5

[David Anderson [:dvander] - inactive, e-mail if emergency]

Attachment: part 2, add Matrix4x4::IsRectilinear

Advanced Layers has a critical path that tests whether an item can be treated as opaque, and it's better if we can test it directly on the effective transform instead of converting to a 2D matrix first.

Comment 6

[David Anderson [:dvander] - inactive, e-mail if emergency]

Attachment: part 3, extract compositor helpers

The compositor has a bunch of extremely useful helpers, like computing the blend backdrop rect, and generating textured triangles, that AL would like to access. This just exposes them in a new header + .cpp file. I also added another helper for generating uv coordinates which is a pretty common operation.

Most of this is code motion and some changes to fix unified build bustage.

Comment 7

[David Anderson [:dvander] - inactive, e-mail if emergency]

Attachment: part 4, expose ContentTextureHost properties

This exposes some simple ContentTextureHost properties that are needed to composite painted layers.

The existing compositor has a callback on each Layer and Compositable type, effectively making rendering recursive. AL doesn't do this, so it needs more information exposed from types like this.

Comment 8

[David Anderson [:dvander] - inactive, e-mail if emergency]

Attachment: part 5, safe downcasting for CompositableHost

This lets us use safe accessors instead of static_casts. I'm actually not sure why ContentHostTexture has an extra base class, but it does.

Comment 9

[David Anderson [:dvander] - inactive, e-mail if emergency]

Attachment: part 6, lockless TextureSource acquisition

This is the first meaty patch. In the old compositing model, each draw call was scoped to processing a single layer. It made sense to automatically lock the layer for drawing, acquire its resources, draw, and then unlock. Since AL uses a batching model, this doesn't work. AL instead needs to automatically lock textures as they are sent to the GPU, and unlock them when we finish drawing.

This approach has two advantages: one, we only lock textures once, in the case of plane splitting. Two, we are much more consistent about locking. The existing compositor often forgets to query IDXGIKeyedMutex. (OTOH, locking is quite rare these days, we synchronize through a global texture instead.)

Anyway, the API change here is that TextureHosts now expose a lockless method to acquire a TextureSource. BufferTextureHost, DXGITextureHostD3D11, TextureHostDirectUpload now support this method.

Comment 10

[David Anderson [:dvander] - inactive, e-mail if emergency]

Oh shit. I've been posting all of these to the wrong bug :( sorry.

Comment 11

[Mike Conley (:mconley) (:⚙️)]

Just happened to me again - here are some more logs from about:support:

(#219) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#220) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#221) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#222) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#223) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#224) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#225) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#226) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#227) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#228) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#229) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#230) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#231) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#232) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#233) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().

Comment 12

[Mike Conley (:mconley) (:⚙️)]

And one more:

(#0) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#1164) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#1165) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#1166) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#1167) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#1168) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#1169) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#1170) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#1171) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#1172) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#1173) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#1174) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#1175) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#1176) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#1177) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#1178) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.

For this past one, the "blank out" occurred while the tab was just sitting there. Suddenly, the content just went away.

Comment 13

[Ben Kelly [:bkelly, not reviewing]]

Just ran into this on latest nightly.  My about:support has:

(#0) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#208) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#209) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#210) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#211) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#212) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#213) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#214) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#215) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#216) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#217) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#218) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#219) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#220) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#221) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#222) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.

Comment 14

[Mike Conley (:mconley) (:⚙️)]

[Tracking Requested - why for this release]:

This is a pretty brutal regression - we can get into a state where content is just not displayed at all, and it appears to occur randomly.

Comment 15

[Julien Cristau [:jcristau]]

Sounds like a bad regression, tracking for 55.  Do we know what caused this?

Comment 16

[David Anderson [:dvander] - inactive, e-mail if emergency]

Probably this isn't a regression, we've never handled content-only device resets properly.

Comment 17

[Mike Conley (:mconley) (:⚙️)]

Certainly a regression in behaviour though - but I can understand if you want to go ahead and remove they keyword.

It looks as if a user on Reddit is experiencing this on the release channel as well:

https://www.reddit.com/r/firefox/comments/6e0ygt/problem_with_firefox_blackwhite_screen/di80r0n/?context=3

Screenshot of failure log:

http://i.imgur.com/z6HMfPl.png

Comment 18

[Ben Kelly [:bkelly, not reviewing]]

Do we have a telemetry to know how frequent this failure mode is?

Comment 19

[Milan Sreckovic [:milan] (needinfo for best results)]

Yeah, we should do something about this.  Peter, can somebody on your team that worked on device reset like things help as well?

Comment 20

[David Anderson [:dvander] - inactive, e-mail if emergency]

Yes, we have telemetry on device resets.

I'm not sure why this only started happening recently, maybe a driver change? It sounds like we're running out of texture memory on the GPU and that's triggering a device reset local to the content process.

The problem is that we want a content reset to be synchronized with a compositor reset. If the compositor resets, content should reset, but not vice versa. And content must wait for the compositor to finish resetting before acquiring a new device.

Here's my proposed solution: in D3D11DeviceStatus, we add (1) a 32-bit generation counter, and (2) a process type. That information will then automatically be present in ContentDeviceData. When content sees a device reset, it can send a sync message through PContent asking if the current D3D11DeviceStatus struct is still valid. ContentParent would check if a device reset is present. If a TDR is present, or if the existing device's generation counter + process type don't match, we return that the device is no longer valid.

Then, content can just handle the device reset internally through existing logic [1].

I can try implementing this later this week, unless someone else can get to it first.

[1] http://searchfox.org/mozilla-central/rev/b318c7dca7392bd16c0b11929f55b1be133f0b31/dom/ipc/ContentChild.cpp#1207

Comment 21

[David Anderson [:dvander] - inactive, e-mail if emergency]

(In reply to David Anderson [:dvander] from comment #20)

Sorry, that should read: if a TDR is *not* present, *and* the devices don't match, we return that content owns a stale device.

If the parent has a TDR condition, we know that content will be reset by other means, and we don't have to do anything.

Comment 22

[Peter Chang[:pchang]]

Mike, could you attach about:memory if this issue happens again? I would like to confirm this is related to memory issue. If it does, we also need to find out the root cause why FF consumes so much memory.

After getting the memory report, you can click 'Terminate GPU process' in about:support page to see this problem could be recovered or not.

Comment 23

[Peter Chang[:pchang]]

Kevin can help this after fixing bug 1362321.

Comment 24

[David Anderson [:dvander] - inactive, e-mail if emergency]

Attachment: wip

This is a proof-of-concept patch for a fix. When I trigger a content-only device reset it seems to work, but sometimes it crashes. The PuppetWidget is holding a freed ClientLayerManager, which is pretty mysterious. I haven't had time to investigate yet.

I'm also less convinced we need to synchronize with the parent process. It's probably okay to just have the client grab a new device and clear everything, and do it again if the parent process was delayed in sending a reset message. But this patch does try and avoid it.

Comment 25

[David Anderson [:dvander] - inactive, e-mail if emergency]

Attachment: wip

Oh, SchedulePaintIfDeviceReset can be called from within a ClientLayerManager, and PuppetWidget::GetLayerManager doesn't return a RefPtr, so destroying it can leave dangling refs on the stack.

This patch posts to the event loop. But something's still not right, content doesn't redraw.

Comment 26

[Mike Conley (:mconley) (:⚙️)]

(In reply to Peter Chang[:pchang] from comment #22)
> Mike, could you attach about:memory if this issue happens again? I would
> like to confirm this is related to memory issue. If it does, we also need to
> find out the root cause why FF consumes so much memory.
> 

Sure - it just happened and I took a report. Will attach.

> After getting the memory report, you can click 'Terminate GPU process' in
> about:support page to see this problem could be recovered or not.

I did. Problem persisted, even after terminating the GPU process and reloading the affected tab.

Comment 27

[Mike Conley (:mconley) (:⚙️)]

Attachment: memory-report.json.gz

Comment 28

[Peter Chang[:pchang]]

(In reply to Mike Conley (:mconley) from comment #27)
> Created attachment 8874025 [details]
> memory-report.json.gz

The only thing I saw from the memory report was d3d11 shared texture consumed around 36 MB in total but I'm not sure it was the root cause of device reset on content process.

         25.73 MB ── d3d11-shared-textures
          9.08 MB ── d3d11-shared-textures
          2.78 MB ── d3d11-shared-textures

Comment 29

[David Anderson [:dvander] - inactive, e-mail if emergency]

Peter, feel free to unassign me if you think someone in Taipei can get to this sooner. I won't have time for a few days.

Comment 30

[Kevin Chen[:kechen] (UTC + 8)]

I agree with David's comment in comment 20, this problem might be caused by a local-only device reset in content side; but the root cause of the device reset is still not clear to me.
Maybe the driver version or some bad draw commands we have executed.

Mike could you share your about:support data? Is this problem reproducible without GPU process?

Comment 31

[Mike Conley (:mconley) (:⚙️)]

Attachment: mconley's about:support on affected machine

Certainly, here's my about:support.

I'll disable the GPU process for a few days and see what happens.

Comment 32

[Ben Kelly [:bkelly, not reviewing]]

Note, I got symptoms similar to this (some tabs not painting any more) and then shortly after got this crash:

https://crash-stats.mozilla.com/report/index/5a6d4ba9-8720-4489-b843-68d6e0170605

Comment 33

[David Anderson [:dvander] - inactive, e-mail if emergency]

(In reply to Ben Kelly [reviewing, but slowly][:bkelly] from comment #32)
> Note, I got symptoms similar to this (some tabs not painting any more) and
> then shortly after got this crash:
> 
> https://crash-stats.mozilla.com/report/index/5a6d4ba9-8720-4489-b843-
> 68d6e0170605

That crash is a pre-release assert [1], in code that I'm pretty sure was 100% dead post-e10s. It could be caused by WebExtensions, though I'm not sure why we'd be trying to Paint those. It could also be caused by bug 1351777. If your GPU process died while you opened a tab, maybe we assigned a BasicLayerManager. And maybe the PuppetWidget's mDrawTarget was some broken D2D surface, so that assert filed.

Do you have any WebExtensions installed? If so, they could be involved here.

Either way, I think a fix for this bug needs to include resetting PuppetWidget::mDrawTarget when we create a new layer manager.

[1] http://searchfox.org/mozilla-central/rev/507743376d1ba753d14ab6b9305b7c6358570be8/widget/PuppetWidget.cpp#1050

Comment 34

[Ben Kelly [:bkelly, not reviewing]]

I have an extension installed, but its disabled.  Here is my about:support:

Application Basics
------------------

Name: Firefox
Version: 55.0a1
Build ID: 20170605030204
Update Channel: nightly
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0
OS: Windows_NT 10.0
Multiprocess Windows: 2/2 (Enabled by default)
Google Key: Found
Mozilla Location Service Key: Found
Safe Mode: false

Crash Reports for the Last 3 Days
---------------------------------

Report ID: bp-5a6d4ba9-8720-4489-b843-68d6e0170605
Submitted: 2 hours ago

All Crash Reports

Nightly Features
----------------

Name: Activity Stream
Version: 0.0.0
ID: [e-mail]

Name: Application Update Service Helper
Version: 2.0
ID: [e-mail]

Name: Firefox Screenshots
Version: 8.1.0
ID: [e-mail]

Name: FlyWeb
Version: 1.0.0
ID: [e-mail]

Name: Form Autofill
Version: 1.0
ID: [e-mail]

Name: Multi-process staged rollout
Version: 1.50
ID: [e-mail]

Name: Photon onboarding
Version: 0.1
ID: [e-mail]

Name: Pocket
Version: 1.0.5
ID: [e-mail]

Name: Presentation
Version: 1.0.0
ID: [e-mail]

Name: Shield Recipe Client
Version: 55
ID: [e-mail]

Name: Web Compat
Version: 1.1
ID: [e-mail]

Name: WebCompat Reporter
Version: 1.0.0
ID: [e-mail]

Extensions
----------

Name: Mozilla Tree Status
Version: 1.0.2
Enabled: false
ID: [e-mail]

Graphics
--------

Features
Compositing: Direct3D 11
Asynchronous Pan/Zoom: wheel input enabled; scrollbar drag enabled
WebGL 1 Driver WSI Info: EGL_VENDOR: Google Inc. (adapter LUID: 00000000000104fe) EGL_VERSION: 1.4 (ANGLE 2.1.0.dec065540d5f) EGL_EXTENSIONS: EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_ANGLE_direct_composition EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_KHR_stream EGL_KHR_stream_consumer_gltexture EGL_NV_stream_consumer_gltexture_yuv EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_stream_producer_d3d_texture_nv12 EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource EGL_EXTENSIONS(nullptr): EGL_EXT_client_extensions EGL_EXT_platform_base EGL_EXT_platform_device EGL_ANGLE_platform_angle EGL_ANGLE_platform_angle_d3d EGL_ANGLE_device_creation EGL_ANGLE_device_creation_d3d11 EGL_ANGLE_experimental_present_path EGL_KHR_client_get_all_proc_addresses
WebGL 1 Driver Renderer: Google Inc. -- ANGLE (NVIDIA Quadro M2000 Direct3D11 vs_5_0 ps_5_0)
WebGL 1 Driver Version: OpenGL ES 2.0 (ANGLE 2.1.0.dec065540d5f)
WebGL 1 Driver Extensions: GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_lossy_etc_decode GL_ANGLE_pack_reverse_row_order GL_ANGLE_request_extension GL_ANGLE_robust_client_memory GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_copy_compressed_texture GL_CHROMIUM_copy_texture GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_discard_framebuffer GL_EXT_disjoint_timer_query GL_EXT_draw_buffers GL_EXT_frag_depth GL_EXT_map_buffer_range GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_sRGB GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_rg GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_EGL_stream_consumer_external GL_NV_fence GL_NV_pack_subimage GL_NV_pixel_buffer_object GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object
WebGL 1 Extensions: ANGLE_instanced_arrays EXT_blend_minmax EXT_color_buffer_half_float EXT_frag_depth EXT_shader_texture_lod EXT_texture_filter_anisotropic EXT_disjoint_timer_query MOZ_debug OES_element_index_uint OES_standard_derivatives OES_texture_float OES_texture_float_linear OES_texture_half_float OES_texture_half_float_linear OES_vertex_array_object WEBGL_color_buffer_float WEBGL_compressed_texture_s3tc WEBGL_debug_renderer_info WEBGL_debug_shaders WEBGL_depth_texture WEBGL_draw_buffers WEBGL_lose_context MOZ_WEBGL_lose_context MOZ_WEBGL_compressed_texture_s3tc MOZ_WEBGL_depth_texture
WebGL 2 Driver WSI Info: EGL_VENDOR: Google Inc. (adapter LUID: 00000000000104fe) EGL_VERSION: 1.4 (ANGLE 2.1.0.dec065540d5f) EGL_EXTENSIONS: EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_ANGLE_direct_composition EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_KHR_stream EGL_KHR_stream_consumer_gltexture EGL_NV_stream_consumer_gltexture_yuv EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_stream_producer_d3d_texture_nv12 EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource EGL_EXTENSIONS(nullptr): EGL_EXT_client_extensions EGL_EXT_platform_base EGL_EXT_platform_device EGL_ANGLE_platform_angle EGL_ANGLE_platform_angle_d3d EGL_ANGLE_device_creation EGL_ANGLE_device_creation_d3d11 EGL_ANGLE_experimental_present_path EGL_KHR_client_get_all_proc_addresses
WebGL 2 Driver Renderer: Google Inc. -- ANGLE (NVIDIA Quadro M2000 Direct3D11 vs_5_0 ps_5_0)
WebGL 2 Driver Version: OpenGL ES 3.0 (ANGLE 2.1.0.dec065540d5f)
WebGL 2 Driver Extensions: GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_lossy_etc_decode GL_ANGLE_pack_reverse_row_order GL_ANGLE_request_extension GL_ANGLE_robust_client_memory GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_copy_compressed_texture GL_CHROMIUM_copy_texture GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_float GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_discard_framebuffer GL_EXT_disjoint_timer_query GL_EXT_draw_buffers GL_EXT_frag_depth GL_EXT_map_buffer_range GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_sRGB GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_norm16 GL_EXT_texture_rg GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_EGL_stream_consumer_external GL_NV_fence GL_NV_pack_subimage GL_NV_pixel_buffer_object GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_image_external_essl3 GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object
WebGL 2 Extensions: EXT_color_buffer_float EXT_texture_filter_anisotropic EXT_disjoint_timer_query MOZ_debug OES_texture_float_linear WEBGL_compressed_texture_s3tc WEBGL_debug_renderer_info WEBGL_debug_shaders WEBGL_lose_context MOZ_WEBGL_lose_context MOZ_WEBGL_compressed_texture_s3tc
Audio Backend: wasapi
Direct2D: true
DirectWrite: true (10.0.15063.0)
GPU #1
Active: Yes
Description: NVIDIA Quadro M2000
Vendor ID: 0x10de
Device ID: 0x1430
Driver Version: 21.21.13.6909
Driver Date: 8-1-2016
Drivers: C:\WINDOWS\System32\DriverStore\FileRepository\nvlw.inf_amd64_209d618eb433ea8c\nvd3dumx,C:\WINDOWS\System32\DriverStore\FileRepository\nvlw.inf_amd64_209d618eb433ea8c\nvwgf2umx,C:\WINDOWS\System32\DriverStore\FileRepository\nvlw.inf_amd64_209d618eb433ea8c\nvwgf2umx,C:\WINDOWS\System32\DriverStore\FileRepository\nvlw.inf_amd64_209d618eb433ea8c\nvwgf2umx C:\WINDOWS\System32\DriverStore\FileRepository\nvlw.inf_amd64_209d618eb433ea8c\nvd3dum,C:\WINDOWS\System32\DriverStore\FileRepository\nvlw.inf_amd64_209d618eb433ea8c\nvwgf2um,C:\WINDOWS\System32\DriverStore\FileRepository\nvlw.inf_amd64_209d618eb433ea8c\nvwgf2um,C:\WINDOWS\System32\DriverStore\FileRepository\nvlw.inf_amd64_209d618eb433ea8c\nvwgf2um
Subsys ID: 119010de
RAM: 4096

Diagnostics
AzureCanvasAccelerated: 0
AzureCanvasBackend: Direct2D 1.1
AzureCanvasBackend (UI Process): skia
AzureContentBackend: Direct2D 1.1
AzureContentBackend (UI Process): skia
AzureFallbackCanvasBackend (UI Process): cairo
GPUProcessPid: 15748
GPUProcess: Terminate GPU Process
failures: CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
Decision Log
WEBRENDER:
opt-in by default: WebRender is an opt-in feature


Failure Log
(#0): CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#126): CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#127): CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#128): CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#129): CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#130): CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#131): CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#132): CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#133): CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#134): CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#135): CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#136): CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#137): CP+[GFX1]: [D2D1.1] 3CreateBitmap failure Size(1,1) Code: 0x8899000c format 0
(#138): CP+[GFX1]: Failed to create DrawTarget, Type: 5 Size: Size(1,1)
(#139): CP+[GFX1-]: Invalid target in gfxContext::CreateOrNull 0000000000000000
(#140): CP+[GFX1 28]: PuppetWidget context problem 0000000000000000

Important Modified Preferences
------------------------------

accessibility.typeaheadfind.flashBar: 0
browser.cache.disk.capacity: 358400
browser.cache.disk.filesystem_reported: 1
browser.cache.disk.smart_size.first_run: false
browser.cache.disk.smart_size.use_old_max: false
browser.cache.frecency_experiment: 3
browser.places.smartBookmarksVersion: 8
browser.sessionstore.upgradeBackup.latestBuildID: 20170605030204
browser.startup.homepage: about:home
browser.startup.homepage_override.buildID: 20170605030204
browser.startup.homepage_override.mstone: 55.0a1
browser.tabs.warnOnClose: false
browser.urlbar.daysBeforeHidingSuggestionsPrompt: 0
browser.urlbar.lastSuggestionsPromptDate: 20170524
browser.urlbar.maxRichResults: 12
browser.urlbar.timesBeforeHidingSuggestionsHint: 0
browser.urlbar.usepreloadedtopurls.enabled: false
dom.fetchController.enabled: true
dom.fetchObserver.enabled: true
dom.push.userAgentID: de7734d04db54d7c96652b80efc5d999
dom.timeout.max_consecutive_callbacks: 9999
extensions.lastAppVersion: 55.0a1
font.internaluseonly.changed: true
media.gmp-gmpopenh264.abi: x86_64-msvc-x64
media.gmp-gmpopenh264.lastUpdate: 1489699177
media.gmp-gmpopenh264.version: 1.6
media.gmp-manager.buildID: 20170605030204
media.gmp-manager.lastCheck: 1496668181
media.gmp-widevinecdm.abi: x86_64-msvc-x64
media.gmp-widevinecdm.lastUpdate: 1489699179
media.gmp-widevinecdm.version: 1.4.8.903
media.gmp.storage.version.observed: 1
media.hardware-video-decoding.failed: true
network.cookie.prefsMigrated: true
network.predictor.cleaned-up: true
places.database.lastMaintenance: 1496181590
places.history.expiration.transient_current_max_pages: 108287
plugin.disable_full_page_plugin_for_types: application/pdf
print.printer_Microsoft_Print_to_PDF.print_bgcolor: false
print.printer_Microsoft_Print_to_PDF.print_bgimages: false
print.printer_Microsoft_Print_to_PDF.print_duplex: -437918235
print.printer_Microsoft_Print_to_PDF.print_edge_bottom: 0
print.printer_Microsoft_Print_to_PDF.print_edge_left: 0
print.printer_Microsoft_Print_to_PDF.print_edge_right: 0
print.printer_Microsoft_Print_to_PDF.print_edge_top: 0
print.printer_Microsoft_Print_to_PDF.print_evenpages: true
print.printer_Microsoft_Print_to_PDF.print_footercenter:
print.printer_Microsoft_Print_to_PDF.print_footerleft: &PT
print.printer_Microsoft_Print_to_PDF.print_footerright: &D
print.printer_Microsoft_Print_to_PDF.print_headercenter:
print.printer_Microsoft_Print_to_PDF.print_headerleft: &T
print.printer_Microsoft_Print_to_PDF.print_headerright: &U
print.printer_Microsoft_Print_to_PDF.print_in_color: true
print.printer_Microsoft_Print_to_PDF.print_margin_bottom: 0.5
print.printer_Microsoft_Print_to_PDF.print_margin_left: 0.5
print.printer_Microsoft_Print_to_PDF.print_margin_right: 0.5
print.printer_Microsoft_Print_to_PDF.print_margin_top: 0.5
print.printer_Microsoft_Print_to_PDF.print_oddpages: true
print.printer_Microsoft_Print_to_PDF.print_orientation: 0
print.printer_Microsoft_Print_to_PDF.print_page_delay: 50
print.printer_Microsoft_Print_to_PDF.print_paper_data: 0
print.printer_Microsoft_Print_to_PDF.print_paper_height: 11.00
print.printer_Microsoft_Print_to_PDF.print_paper_name:
print.printer_Microsoft_Print_to_PDF.print_paper_size_unit: 0
print.printer_Microsoft_Print_to_PDF.print_paper_width: 8.50
print.printer_Microsoft_Print_to_PDF.print_resolution: -437918235
print.printer_Microsoft_Print_to_PDF.print_reversed: false
print.printer_Microsoft_Print_to_PDF.print_scaling: 1.00
print.printer_Microsoft_Print_to_PDF.print_shrink_to_fit: true
print.printer_Microsoft_Print_to_PDF.print_to_file: false
print.printer_Microsoft_Print_to_PDF.print_unwriteable_margin_bottom: 0
print.printer_Microsoft_Print_to_PDF.print_unwriteable_margin_left: 0
print.printer_Microsoft_Print_to_PDF.print_unwriteable_margin_right: 0
print.printer_Microsoft_Print_to_PDF.print_unwriteable_margin_top: 0
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_bgcolor: false
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_bgimages: false
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_duplex: -437918235
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_edge_bottom: 0
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_edge_left: 0
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_edge_right: 0
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_edge_top: 0
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_evenpages: true
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_footercenter:
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_footerleft: &PT
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_footerright: &D
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_headercenter:
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_headerleft: &T
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_headerright: &U
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_in_color: true
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_margin_bottom: 0.5
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_margin_left: 0.5
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_margin_right: 0.5
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_margin_top: 0.5
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_oddpages: true
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_orientation: 0
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_page_delay: 50
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_paper_data: 1
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_paper_height: -1.00
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_paper_name:
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_paper_size_unit: 0
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_paper_width: -1.00
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_resolution: 600
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_reversed: false
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_scaling: 1.00
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_shrink_to_fit: true
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_to_file: false
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_unwriteable_margin_bottom: 0
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_unwriteable_margin_left: 0
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_unwriteable_margin_right: 0
print.printer_Xerox_Phaser_6128MFP-N_Class_Driver.print_unwriteable_margin_top: 0
privacy.donottrackheader.enabled: true
security.sandbox.content.tempDirSuffix: {8fcc9e07-dc16-4e8e-b080-4f1ffdc3496d}
security.tls.version.min: 0
services.sync.declinedEngines: addons
services.sync.engine.addons: false
services.sync.engine.bookmarks.validation.lastTime: 1496683524
services.sync.engine.passwords: false
services.sync.engine.prefs.modified: false
services.sync.lastPing: 1496683525
services.sync.lastSync: Mon Jun 05 2017 16:05:43 GMT-0400 (Eastern Standard Time)
storage.vacuum.last.index: 1
storage.vacuum.last.places.sqlite: 1494971361
ui.osk.debug.keyboardDisplayReason: IKPOS: Touch screen not found.

Important Locked Preferences
----------------------------

Places Database
---------------

JavaScript
----------

Incremental GC: true

Accessibility
-------------

Activated: false
Prevent Accessibility: 0

Library Versions
----------------

NSPR
Expected minimum version: 4.15 Beta
Version in use: 4.15 Beta

NSS
Expected minimum version: 3.31 Beta
Version in use: 3.31 Beta

NSSSMIME
Expected minimum version: 3.31 Beta
Version in use: 3.31 Beta

NSSSSL
Expected minimum version: 3.31 Beta
Version in use: 3.31 Beta

NSSUTIL
Expected minimum version: 3.31 Beta
Version in use: 3.31 Beta

Experimental Features
---------------------

Name: Animated Image Discarding Nightly 55
ID: [e-mail]
Description: Measures the impact of enabling animate image discarding in Firefox Nightly 55
Active: false
End Date: 1494720000011
Homepage:
Branch: control

Sandbox
-------

Content Process Sandbox Level: 2
Effective Content Process Sandbox Level: 2

Comment 35

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to David Anderson [:dvander] from comment #20)
> Yes, we have telemetry on device resets.
> 
> I'm not sure why this only started happening recently...

A lot (all?) reports are from Windows Creators update version; can we correlate the driver reset crashes with the version of Windows (matches DWrite.dll version) on Windows 10, and see if the increase started with the update?

Comment 36

[Milan Sreckovic [:milan] (needinfo for best results)]

Keeping this bug separate from bug 1163440, but it could be a duplicate.

Comment 37

[Mike Conley (:mconley) (:⚙️)]

I hit this again with the GPU process disabled. Seems to occur pretty regularly towards the end of the day after a session lifetime of about 8 hours.

Here's what was in about:support:

(#0) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#204) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#205) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#206) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#207) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#208) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#209) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#210) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#211) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#212) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#213) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#214) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#215) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#216) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 1
(#217) 	CP+[GFX1-]: (gfxWindowsPlatform) Finished device reset.
(#218) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().

I'll attach my about:memory next...

Comment 38

[Mike Conley (:mconley) (:⚙️)]

Attachment: about:memory report after bug occurred with GPU process disabled

Comment 39

[David Anderson [:dvander] - inactive, e-mail if emergency]

(In reply to Ben Kelly [reviewing, but slowly][:bkelly] from comment #34)
> I have an extension installed, but its disabled.  Here is my about:support:

Okay, so not WebExtensions related. Probably what happened is you got a device reset in the content process, and PuppetWidget's mDrawTarget is no longer sane. We should make sure to update that as part of handling a device reset.

Comment 40

[Peter Chang[:pchang]]

(In reply to David Anderson [:dvander] from comment #29)
> Peter, feel free to unassign me if you think someone in Taipei can get to
> this sooner. I won't have time for a few days.
Sure, Kevin is working on this issue now and we are able to simulate the device reset on content process by using dxcap tool.

Comment 41

[Kevin Chen[:kechen] (UTC + 8)]

Still working on handling the local device reset, if reseting the device locally, content process will receive the DidComposite message whose transaction id is belonged to the old one.
This will lead the code to a NULL access in[1], still investigating whether there will be any side effect if skip it.

[1] https://dxr.mozilla.org/mozilla-central/rev/2a63a6c35033b5cbc6c98cabc7657c7290284691/gfx/layers/client/ClientLayerManager.cpp#517

Comment 42

[Kevin Chen[:kechen] (UTC + 8)]

After the investigation, I found this bug might be caused by the unsuccessful device reset recovery process when we decided to kill GPU process.

Easier step to reproduce:
1. Add a new integer preference "gfx.device-reset.limit" with value 0 in about:config (It is est to 10 if there is no preference). 
2. Restart firefox and open a web page.
3. Open the terminal as administrator.
4. Type "dxcap -forcetdr" in terminal (This will trigger a device reset signal from system side).

Expected result:
  The browser should be recovered from device reset and works as usual.

Actual result:
  The UI still works but the content in the tabs are broken.

Hello Mike, could you help me to double check if this case is the same as what you originally reported?

Comment 43

[Mike Conley (:mconley) (:⚙️)]

I ran these steps, and it caused the main process to crash with:

https://crash-stats.mozilla.com/report/index/1471bfc9-bb39-4f7a-9d87-c742f0170616

That's apparently bug 1373540, which is a recent regression. I'll try with an older Nightly.

Comment 44

[Mike Conley (:mconley) (:⚙️)]

(In reply to Kevin Chen[:kechen] (UTC + 8) from comment #42)
> Hello Mike, could you help me to double check if this case is the same as
> what you originally reported?

Yes, using the steps provided in comment 42 using an older Nightly (2017-06-13), the behaviour seems very similar.

Comment 45

[Kevin Chen[:kechen] (UTC + 8)]

Attachment: Bug 1364563 - Update render mode and fallback to software solution when GPU process is killed;

Review commit: https://reviewboard.mozilla.org/r/150758/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/150758/

Comment 46

[David Anderson [:dvander] - inactive, e-mail if emergency]

Hrm, it's not clear to me how this patch addresses the issue in this bug. A content-local TDR is not handled at all so it won't go through these code paths, and it seemed pretty likely the report invovled a content-local TDR. That also means dxcap is probably insufficient for testing this - I had to add some manual instrumentation, for example, since the existing methods we have for testing affect the UI process.

Comment 47

[Kevin Chen[:kechen] (UTC + 8)]

Comment on attachment 8879444 [details]
Bug 1364563 - Update render mode and fallback to software solution when GPU process is killed;

Sorry, I submitted the patch by accident. It is not completed.

My current conclution is that the content-local TDR is caused by an unsynchronized device reset. I am still investigating on this part.

When gecko decides to kill GPU process and recreate compositor things on UI process, we will run into a situation that content process holds the invalid devices while UI process chooses to use BasicCompositor without notifying content process.
I think this might be the root cause of this problem, if we could find out a way to prevent this situation at the first place, we might not need to add a new path to trigger TDR process from content side.

This situation can be reproduced via comment 42.

Comment 48

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Since this is fix-optional, I don't see the need to track this for 55. If a fix is ready in time  and deemed low risk, we could uplift to Beta.

Comment 49

[Kevin Chen[:kechen] (UTC + 8)]

Comment on attachment 8879444 [details]
Bug 1364563 - Update render mode and fallback to software solution when GPU process is killed;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/150758/diff/1-2/

Comment 50

[Kevin Chen[:kechen] (UTC + 8)]

Hello David,

I think the reason of the content only TDR is caused by some bugs in GPU process fallback path.

When reaching the TDR limitation in [1], firefox will kill gpu process and fallback to content / UI process model. Since we use skia and don't have D3D11Devices in UI process at the first place, the TDR flow won't be triggered while content process is in device reset status which causes the content-only TDR.

Since we should fallback into software solution in this case, in this patch, I disable the gfxConfigs when deciding to kill GPU process and make the content process to update rendr mode in RecvReinitRendering. 

With this solution maybe we don't need another synced ipc to handle the content only device reset case.

[1] https://dxr.mozilla.org/mozilla-central/rev/9af23c413a1f8d337b19b4f8450e241e91b71136/gfx/ipc/GPUProcessManager.cpp#412
[2] https://dxr.mozilla.org/mozilla-central/rev/9af23c413a1f8d337b19b4f8450e241e91b71136/gfx/thebes/gfxWindowsPlatform.cpp#1431

Comment 51

[David Anderson [:dvander] - inactive, e-mail if emergency]

Comment on attachment 8879444 [details]
Bug 1364563 - Update render mode and fallback to software solution when GPU process is killed;

https://reviewboard.mozilla.org/r/150758/#review158634

Comment 52

[Kevin Chen[:kechen] (UTC + 8)]

I will add another patch for content-only device reset for handling the case that GPU process is still alive.

Comment 53

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/78f86956e191
Update render mode and fallback to software solution when GPU process is killed; r=dvander

Comment 54

[Peter Chang[:pchang]]

Kevin, please monitor this issue for a while in central and then request uplift because users couldn't see content updating if they hit this problem.

Comment 55

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/78f86956e191

Comment 56

[Kevin Chen[:kechen] (UTC + 8)]

Attachment: Add a path for content process only device reset;

Hello David,

The algorithm of this patch is like you proposed in Comment 20, the one only one difference is that I check content-only device reset through PCompositorBridge instead of PContent.

The reason for the difference is that the D3D11DeviceStatus in PContent parent side can only be synced with the D3D11DeviceStatus in GPU process after GPU process send device reset notification[1]; we might get expired D3D11DeviceStatus if we use PContent.

[1] https://dxr.mozilla.org/mozilla-central/rev/5f44d10bacca2d693413b529e0caadc73e634e1e/gfx/ipc/GPUChild.cpp#212

Comment 57

[David Anderson [:dvander] - inactive, e-mail if emergency]

Comment on attachment 8886941 [details] [diff] [review]
Add a path for content process only device reset;

Review of attachment 8886941 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!

Comment 58

[Kevin Chen[:kechen] (UTC + 8)]

There are several fails on try, but they are caused by other intermittent bugs. I think we are good.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=be623df635cb6d1d2bd9ea8deb9d82821886c596&selectedJob=115165571

Comment 59

[Kevin Chen[:kechen] (UTC + 8)]

(In reply to Kevin Chen[:kechen] (UTC + 8) (PTO 7/3 - 7/11) from comment #58)
> There are several fails on try, but they are caused by other intermittent
> bugs. I think we are good.
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=be623df635cb6d1d2bd9ea8deb9d82821886c596&selectedJob=1
> 15165571

Just checkin the "Add a path for content process only device reset;" patch, the other one is already in codebase.
Thank you!

Comment 60

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/bc5d9e5f1c02
Add a path for content process only device reset. r=dvander

Comment 61

[(no longer active)]

I backed this out because of build failures on inbound:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=bc5d9e5f1c02c700d0cd85142ecb26445c070b69
https://hg.mozilla.org/integration/mozilla-inbound/rev/06a1ba17bbaa7794c27927ed7f0e01f576ce263f

Comment 62

[Kevin Chen[:kechen] (UTC + 8)]

Attachment: Add a path for content process only device reset;

Sorry, this is the correct patch.

Comment 63

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/638c3681d5c3
Add a path for content process only device reset. r=dvander

Comment 64

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/638c3681d5c3

Comment 65

[Kevin Chen[:kechen] (UTC + 8)]

I will request uplift later.

Comment 66

[Kevin Chen[:kechen] (UTC + 8)]

Attachment: [beta-uplift] (Part 1) Update render mode and fallback to software solution when GPU process is killed;

Approval Request Comment
[Feature/Bug causing the regression]:
Device reset
[User impact if declined]:
Windows users might encounter the situation that content in tabs can not be rendered correctly when the process model fall back to ui/content model.
[Is this code covered by automated tests?]:
No.
[Has the fix been verified in Nightly?]:
No, but it has been landed into nightly for 7 days and no following errors are found.
[Needs manual test from QE? If yes, steps to reproduce]: 
No.
[List of other uplifts needed for the feature/fix]:
No.
[Is the change risky?]:
Not risky.
[Why is the change risky/not risky?]:
It just change the backend.
[String changes made/needed]:
No.

Comment 67

[Kevin Chen[:kechen] (UTC + 8)]

Attachment: [beta-uplift] (Part 2) Add a path for content process only device reset;

Approval Request Comment
[Feature/Bug causing the regression]:
Device reset.
[User impact if declined]:
Windows users might encounter the situation that content in tabs can not be rendered correctly.
[Is this code covered by automated tests?]:
No.
[Has the fix been verified in Nightly?]:
No, but it has been landed into nightly for 7 days and no following errors are found.
[Needs manual test from QE? If yes, steps to reproduce]: 
No.
[List of other uplifts needed for the feature/fix]:
No.
[Is the change risky?]:
Not risky.
[Why is the change risky/not risky?]:
It's a very rare case, I think only very few users will run into this part of code.
[String changes made/needed]:
No

Comment 68

[Kevin Chen[:kechen] (UTC + 8)]

Following is the try push of uplift patches in comment 66 and comment 67 [1].
There are several failures but not related to these patches and covered by intermittent bugs.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=e2e2ab5554d21d79ee7db088eebcb547da1677e2&selectedJob=117791330

Comment 69

[Julien Cristau [:jcristau]]

Comment on attachment 8890642 [details] [diff] [review]
[beta-uplift] (Part 1) Update render mode and fallback to software solution when GPU process is killed;

i think this should bake on beta and we don't have time for 55