Provide fix to Firefox to prevent TDR on Windows 7 with bugged NVIDIA GPU drivers (385.69-391.35/latest one) on Pascal GPUs

VERIFIED FIXED in Firefox 60

Status

()

P3
critical
VERIFIED FIXED
11 months ago
2 months ago

People

(Reporter: Virtual, Assigned: bas.schouten)

Tracking

({crash, nightly-community})

57 Branch
mozilla61
All
Windows 7
crash, nightly-community
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(relnote-firefox -, firefox-esr52 unaffected, firefox58- wontfix, firefox59+ wontfix, firefox60+ verified, firefox61+ verified)

Details

(Whiteboard: [gfx-noted] [bug fixed completly in NVIDIA GPU drivers version 397.31])

Attachments

(2 attachments)

Follow up of Bug #1419264, as issue wasn't fixed completely per incomplete NVIDIA GPU drivers blacklist.

NVIDIA GPU drivers blacklist have to start from 385.69 without any limit,
as even latest NVIDIA GPU drivers 390.65 are bugged.
Flags: needinfo?(milan)
Has Regression Range: --- → irrelevant
Has STR: --- → irrelevant
Latest NVIDIA GPU drivers without this issue are 385.41.

Comment 3

11 months ago
I'm the author of Bug 1419264.
With Firefox 57, I didn't had any crashes once layers.mlgpu.enable-on-windows7 set to false.
With Firefox 58 (layers.mlgpu.enable-on-windows7 reset to true), no crash either.
OS: Windows 7 x64 up to date.
Graphic card: NVIDIA_DEV.1C03 = "NVIDIA GeForce GTX 1060 6GB" with latest drivers (390.65)
tracking-firefox58: --- → ?
tracking-firefox59: --- → ?
tracking-firefox60: --- → ?
Flags: needinfo?(matt.woodrow)
Flags: needinfo?(jmuizelaar)
Flags: needinfo?(bas)
(Assignee)

Comment 4

10 months ago
The description here is a little confusing to me, which drivers should be blacklisted? On which devices? I'm running locally running on a GTX 1080 Ti with 388.13 and everything seems to be fine for me? (Windows 10 though)
Flags: needinfo?(bas)
Blacklist should starts from version 385.69, which is first bugged NVIDIA GPU drivers version on Windows 7 according to reports. See bug #1419264 comment #116 & bug #1419264 comment #139.
Flags: needinfo?(bas)
This has quite a lot of needinfo requests but it sounds like maybe we want to confirm the affected drivers.

Can you link to some crash reports? Or provide more information on reproducing the issues you're seeing?
Flags: needinfo?(Virtual)
This bug is about adding missing NVIDIA GPU drivers versions to Firefox GPU drivers blacklist for fixing NVIDIA GRPU drivers TDR issues, so there aren't any Firefox crash reports. Missing NVIDIA GPU drivers versions for example, basing on only WHQL versions are: 385.69 & 387.92 & 388.00. NVIDIA GPU drivers blacklist have to start from 385.69 without any limit, as even latest NVIDIA GPU drivers 390.77 are bugged.

All needed information is in bug #1419264. Mostly in bug #1419264 comment #139 & bug #1419264 comment #109 & and latest one. There are also many reports on NVIDIA Forums ( biggest thread about this is here - https://forums.geforce.com/default/topic/1030260/geforce-drivers/windows-7-and-drivers-385-69-388-xx-accelerated-firefox-tdrs-/ ) and Reddit.
Flags: needinfo?(Virtual)
OS: Windows → Windows 7
I don't know why I was need-info'd on this bug report. Please do not flag me without an explanation as to what information you need from me.
I :ni you, as you are from Quality Team, so I was thinking that maybe you could push this bug further to proper person, which will fix it.

Updated

10 months ago
Whiteboard: [gfx-noted]
I don't think I need to be tracking this for 60.  This bug seems way too confusing, much handwaving and little specific information.
tracking-firefox60: ? → ---

Updated

10 months ago
Duplicate of this bug: 1438170
I changed summary to better describe issue,
but if someone still don't understand, please read Comment #7.
Summary: Incomplete NVIDIA GPU drivers blacklist → Firefox have to blacklist all NVIDIA GPU drivers starting from 385.69 version to prevent TDR on Windows 7
Summary: Firefox have to blacklist all NVIDIA GPU drivers starting from 385.69 version to prevent TDR on Windows 7 → Firefox has to blacklist all NVIDIA GPU drivers starting from 385.69 version to prevent TDR on Windows 7
Sounds like the graphics team is aware of the issue, and Virtual, you're trying to say that we need to blocklist more versions of this driver. I'm not sure we have enough information to tell that at this point.
tracking-firefox58: ? → -
tracking-firefox59: ? → -
Flags: needinfo?(matt.woodrow)
Flags: needinfo?(jmuizelaar)
Flags: needinfo?(bas)
Flags: needinfo?(anthony.s.hughes)

Comment 14

10 months ago
Manuel Guzzman has informed the community that nvidia has consistent reproduction of the issue now, They are working on root causing the problem.

All future drivers should be blacklisted (on Pascal hardware) till the fix has been implemented,  Manuel is personally keeping the respective thread updated on the status of the issue.
NVIDIA released patched GPU driver which is 391.01 version.

>Issues and updates:
> [...]
> -Random TDR while using Firefox on Pascal GPUs

Source:
https://forums.geforce.com/default/topic/1038299/geforce-drivers/official-391-01-game-ready-whql-display-driver-feedback-thread-released-2-26-18-/post/5296731/#5296731
Flags: needinfo?(milan)
Summary: Firefox has to blacklist all NVIDIA GPU drivers starting from 385.69 version to prevent TDR on Windows 7 → Firefox has to blacklist all NVIDIA GPU drivers starting from version 385.69 to 390.77 to prevent TDR on Windows 7
Per Comment #15, I'm rerequesting blacklisting all NVIDIA GPU drivers, starting from version 385.69 to 390.77, to prevent TDR on Windows in Firefox with enabled Advanced Layers. I want to add that this issue was kinda ignored or not get proper attention for over 1 month and unfortunately it's extremely annoying on corporate environment, when hardware drivers aren't updated due to various reasons.
tracking-firefox58: - → ?
tracking-firefox59: - → ?
tracking-firefox60: --- → ?
Flags: needinfo?(milan)
Flags: needinfo?(matt.woodrow)
Flags: needinfo?(jmuizelaar)
Flags: needinfo?(bas)
Flags: needinfo?(anthony.s.hughes)
OS: Windows 7 → Windows
Summary: Firefox has to blacklist all NVIDIA GPU drivers starting from version 385.69 to 390.77 to prevent TDR on Windows 7 → Firefox has to blacklist all NVIDIA GPU drivers starting from version 385.69 to 390.77 to prevent TDR on Windows

Comment 17

10 months ago
Please blacklist 391.01 as well, the issues and updates post is for ongoing issues and those reported from multiple sources in the thread.

I have had no messages to test a fix and users have already reproduced the issue under 391.01 on Guru3D's driver comments.
In other words, we don't have a working driver yet?
Flags: needinfo?(milan)
See Also: → bug 1419264

Comment 19

10 months ago
Root causing is ongoing for this and a few other issues in the issues and updates list.

Comment 20

10 months ago
Just hopping on as "me too" so I know when to update and revert settings I've changed. Thanks for all your diligent work, debuggers.

Win7 x64 updated
gtx 1060 v388.13
ff 58.02 x64

Comment 21

10 months ago
(In reply to Milan Sreckovic [:milan] from comment #18)
> In other words, we don't have a working driver yet?

That's correct, 391.01 still needs to be blacklisted.
Too late to fix for 58 but we could still add these versions for 59 if we need to.
status-firefox58: affected → wontfix
tracking-firefox58: ? → -
tracking-firefox59: ? → +
tracking-firefox60: ? → +
Flags: needinfo?(matt.woodrow)
Flags: needinfo?(jmuizelaar)
Flags: needinfo?(bas)
Flags: needinfo?(anthony.s.hughes)
Bas, can you take a look and see if bug 1419264 did take care of the crashes by disabling advanced layers, rather than just move them to a different signature?  If it did help, then lets see what other devices are affected and block advanced layers there.  In code.

I don't have a problem running AL, btw, Nvidia, 0x139b, GeForce GTX 960M
Flags: needinfo?(bas)
(Assignee)

Comment 24

9 months ago
(In reply to Danial Horton from comment #14)
> Manuel Guzzman has informed the community that nvidia has consistent
> reproduction of the issue now, They are working on root causing the problem.
> 
> All future drivers should be blacklisted (on Pascal hardware) till the fix
> has been implemented,  Manuel is personally keeping the respective thread
> updated on the status of the issue.

Can we get more specific details on what is affected? I'm running a Pascal GPU (GTX 1080 Ti) myself on 388.13 and am seeing no such issues. Danial, you seem to have an effective link to NVidia, do you know?
Flags: needinfo?(bas) → needinfo?(danialhorton)
(Assignee)

Comment 25

9 months ago
The most actionable would be a list of DeviceIDs affected by this.

Comment 26

9 months ago
Hey Bas, I only know what is being posted publicly, my task got closed once nvidia started to get repeatable reproductions after a tip off that it was easy to crash the driver on a high activity Twitch stream with heavy emoji spams in the chat.

My unskilled assumption is that the Windows 7 driver has hit an issue with DMA pre-emption since it is occuring mostly with sudden image decodes, and the Advanced Layers changes was only accelerating the occurences. I went back and read through the wddm tech doctuments and there truly is pre-emption differences since wddm 1.2, so that might be where nvidia is looking right now.

I am on waterfox these days and have triggered the issue with that only when actively browsing, but have had 57+ trigger it before even painting the new tab page at a clean start using the old mozilla profile utility.

There is a small group of windows 10 users who have had TDR's since before 385.69 ever existed so i've been doing what i can to put distance between that and this issue specific to drivers newer than 385.41 as there are so many users who have never had the issue under 8 or 10. (Which I have not had any reproductions of with my backup machine)

The affected list of devices seems to cover every Pascal gpu available at the moment, from the GT 1030 to the Titan Xp.

I am not aware of Titan V being affected and the minimal user base doesn't make it easy to find out, 
I've tested a fresh install of 7 vs 10 with a kepler and swapping it with my second pascal card (i have both a 1080 and 1060 driving two displays) with the issue manifesting only on the pascal board under 7.
Flags: needinfo?(danialhorton)

Comment 27

9 months ago
Pascal device ID's are as follows

NVIDIA_DEV.1B00 = "NVIDIA TITAN X (Pascal)"
NVIDIA_DEV.1B02 = "NVIDIA TITAN Xp"
NVIDIA_DEV.1B02.123E.10DE = "NVIDIA TITAN Xp COLLECTORS EDITION"
NVIDIA_DEV.1B02.123F.10DE = "NVIDIA TITAN Xp COLLECTORS EDITION"
NVIDIA_DEV.1B06 = "NVIDIA GeForce GTX 1080 Ti"
NVIDIA_DEV.1B80 = "NVIDIA GeForce GTX 1080"
NVIDIA_DEV.1B81 = "NVIDIA GeForce GTX 1070"
NVIDIA_DEV.1B82 = "NVIDIA GeForce GTX 1070 Ti"
NVIDIA_DEV.1B84 = "NVIDIA GeForce GTX 1060 3GB"
NVIDIA_DEV.1C02 = "NVIDIA GeForce GTX 1060 3GB"
NVIDIA_DEV.1C03 = "NVIDIA GeForce GTX 1060 6GB"
NVIDIA_DEV.1C04 = "NVIDIA GeForce GTX 1060 5GB"
NVIDIA_DEV.1C06 = "NVIDIA GeForce GTX 1060 6GB"
NVIDIA_DEV.1C81 = "NVIDIA GeForce GTX 1050"
NVIDIA_DEV.1C82 = "NVIDIA GeForce GTX 1050 Ti"
NVIDIA_DEV.1D01 = "NVIDIA GeForce GT 1030"

Nvidia did not seperate desktop and mobile chips this generation, so I do not believe the mobile chips have different ID's,
Max-Q laptops probably do, but these are sold with windows 10 only and should not be affected since the cpu's in these systems aren't compatible with windows 7 anymore.

Comment 28

9 months ago
My GTX 1070 is correctly put on blacklist with latest (391.01) drivers here:

 {
  "application": {
    "name": "Firefox",
    "osVersion": "Windows_NT 6.1",
    "version": "58.0.2",
    "buildID": "20180206200532",
    "userAgent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0",
    "safeMode": false,
    "updateChannel": "release",
    "supportURL": "https://support.mozilla.org/1/firefox/58.0.2/WINNT/fr/",
    "numTotalWindows": 1,
    "numRemoteWindows": 1,
    "remoteAutoStart": true,
    "currentContentProcesses": 2,
    "maxContentProcesses": 4,
    "autoStartStatus": 1,
    "styloBuild": true,
    "styloDefault": true,
    "styloResult": true,
    "styloChromeDefault": false,
    "styloChromeResult": false,
    "keyGoogleFound": true,
    "keyMozillaFound": true
  },



  },
  "graphics": {
    "numTotalWindows": 1,
    "numAcceleratedWindows": 1,
    "windowLayerManagerType": "Direct3D 11",
    "windowLayerManagerRemote": true,
    "windowUsingAdvancedLayers": false,
    "adapterDescription": "NVIDIA GeForce GTX 1070",
    "adapterVendorID": "0x10de",
    "adapterDeviceID": "0x1b81",
    "adapterSubsysID": "00000000",
    "adapterRAM": "8192",
    "adapterDrivers": "nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um",
    "driverVersion": "23.21.13.9101",
    "driverDate": "2-23-2018",
    "adapterDescription2": "",
    "adapterVendorID2": "",
    "adapterDeviceID2": "",
    "adapterSubsysID2": "",
    "adapterRAM2": "",
    "adapterDrivers2": "",
    "driverVersion2": "",
    "driverDate2": "",
    "isGPU2Active": false,
    "direct2DEnabled": true,
    "directWriteEnabled": true,
    "directWriteVersion": "6.2.9200.22164",
    "offMainThreadPaintEnabled": true,
    "webgl1Renderer": "Google Inc. -- ANGLE (NVIDIA GeForce GTX 1070 Direct3D11 vs_5_0 ps_5_0)",
    "webgl1Version": "OpenGL ES 2.0 (ANGLE 2.1.0.db3422764a9b)",
    "webgl1DriverExtensions": "GL_ANGLE_client_arrays GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_lossy_etc_decode GL_ANGLE_pack_reverse_row_order GL_ANGLE_program_cache_control GL_ANGLE_request_extension GL_ANGLE_robust_client_memory GL_ANGLE_robust_resource_initialization GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_color_buffer_float_rgb GL_CHROMIUM_color_buffer_float_rgba GL_CHROMIUM_copy_compressed_texture GL_CHROMIUM_copy_texture GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_discard_framebuffer GL_EXT_disjoint_timer_query GL_EXT_draw_buffers GL_EXT_frag_depth GL_EXT_map_buffer_range GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_sRGB GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_s3tc_srgb GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_rg GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_EGL_stream_consumer_external GL_NV_fence GL_NV_pack_subimage GL_NV_pixel_buffer_object GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_surfaceless_context GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object ",
    "webgl1Extensions": "ANGLE_instanced_arrays EXT_blend_minmax EXT_color_buffer_half_float EXT_frag_depth EXT_sRGB EXT_shader_texture_lod EXT_texture_filter_anisotropic EXT_disjoint_timer_query OES_element_index_uint OES_standard_derivatives OES_texture_float OES_texture_float_linear OES_texture_half_float OES_texture_half_float_linear OES_vertex_array_object WEBGL_color_buffer_float WEBGL_compressed_texture_s3tc WEBGL_compressed_texture_s3tc_srgb WEBGL_debug_renderer_info WEBGL_debug_shaders WEBGL_depth_texture WEBGL_draw_buffers WEBGL_lose_context",
    "webgl1WSIInfo": "EGL_VENDOR: Google Inc. (adapter LUID: 000000000000c1f9)\nEGL_VERSION: 1.4 (ANGLE 2.1.0.db3422764a9b)\nEGL_EXTENSIONS: EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_KHR_stream EGL_KHR_stream_consumer_gltexture EGL_NV_stream_consumer_gltexture_yuv EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource EGL_EXT_pixel_format_float EGL_KHR_surfaceless_context EGL_ANGLE_display_texture_share_group EGL_ANGLE_create_context_client_arrays EGL_ANGLE_program_cache_control \nEGL_EXTENSIONS(nullptr): EGL_EXT_client_extensions EGL_EXT_platform_base EGL_EXT_platform_device EGL_ANGLE_platform_angle EGL_ANGLE_platform_angle_d3d EGL_ANGLE_device_creation EGL_ANGLE_device_creation_d3d11 EGL_ANGLE_experimental_present_path EGL_KHR_client_get_all_proc_addresses EGL_ANGLE_display_robust_resource_initialization ",
    "webgl2Renderer": "Google Inc. -- ANGLE (NVIDIA GeForce GTX 1070 Direct3D11 vs_5_0 ps_5_0)",
    "webgl2Version": "OpenGL ES 3.0 (ANGLE 2.1.0.db3422764a9b)",
    "webgl2DriverExtensions": "GL_ANGLE_client_arrays GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_lossy_etc_decode GL_ANGLE_multiview GL_ANGLE_pack_reverse_row_order GL_ANGLE_program_cache_control GL_ANGLE_request_extension GL_ANGLE_robust_client_memory GL_ANGLE_robust_resource_initialization GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_color_buffer_float_rgb GL_CHROMIUM_color_buffer_float_rgba GL_CHROMIUM_copy_compressed_texture GL_CHROMIUM_copy_texture GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_float GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_discard_framebuffer GL_EXT_disjoint_timer_query GL_EXT_draw_buffers GL_EXT_frag_depth GL_EXT_map_buffer_range GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_sRGB GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_s3tc_srgb GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_norm16 GL_EXT_texture_rg GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_EGL_stream_consumer_external GL_NV_fence GL_NV_pack_subimage GL_NV_pixel_buffer_object GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_image_external_essl3 GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_surfaceless_context GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object ",
    "webgl2Extensions": "EXT_color_buffer_float EXT_texture_filter_anisotropic EXT_disjoint_timer_query OES_texture_float_linear WEBGL_compressed_texture_s3tc WEBGL_compressed_texture_s3tc_srgb WEBGL_debug_renderer_info WEBGL_debug_shaders WEBGL_lose_context",
    "webgl2WSIInfo": "EGL_VENDOR: Google Inc. (adapter LUID: 000000000000c1f9)\nEGL_VERSION: 1.4 (ANGLE 2.1.0.db3422764a9b)\nEGL_EXTENSIONS: EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_KHR_stream EGL_KHR_stream_consumer_gltexture EGL_NV_stream_consumer_gltexture_yuv EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource EGL_EXT_pixel_format_float EGL_KHR_surfaceless_context EGL_ANGLE_display_texture_share_group EGL_ANGLE_create_context_client_arrays EGL_ANGLE_program_cache_control \nEGL_EXTENSIONS(nullptr): EGL_EXT_client_extensions EGL_EXT_platform_base EGL_EXT_platform_device EGL_ANGLE_platform_angle EGL_ANGLE_platform_angle_d3d EGL_ANGLE_device_creation EGL_ANGLE_device_creation_d3d11 EGL_ANGLE_experimental_present_path EGL_KHR_client_get_all_proc_addresses EGL_ANGLE_display_robust_resource_initialization ",
    "info": {
      "AzureCanvasBackend (UI Process)": "skia",
      "AzureFallbackCanvasBackend (UI Process)": "cairo",
      "AzureContentBackend (UI Process)": "skia",
      "AzureCanvasBackend": "Direct2D 1.1",
      "AzureContentBackend": "Direct2D 1.1",
      "AzureCanvasAccelerated": 0,
      "ApzWheelInput": 1,
      "ApzDragInput": 1,
      "ApzKeyboardInput": 1,
      "ApzAutoscrollInput": 1
    },
    "featureLog": {
      "features": [
        {
          "name": "HW_COMPOSITING",
          "description": "Compositing",
          "status": "available",
          "log": [
            {
              "type": "default",
              "status": "available"
            }
          ]
        },
        {
          "name": "D3D11_COMPOSITING",
          "description": "Direct3D11 Compositing",
          "status": "available",
          "log": [
            {
              "type": "default",
              "status": "available"
            }
          ]
        },
        {
          "name": "DIRECT2D",
          "description": "Direct2D",
          "status": "available",
          "log": [
            {
              "type": "default",
              "status": "available"
            }
          ]
        },
        {
          "name": "D3D11_HW_ANGLE",
          "description": "Direct3D11 hardware ANGLE",
          "status": "available",
          "log": [
            {
              "type": "default",
              "status": "available"
            }
          ]
        },
        {
          "name": "GPU_PROCESS",
          "description": "GPU Process",
          "status": "available",
          "log": [
            {
              "type": "default",
              "status": "available"
            }
          ]
        },
        {
          "name": "WEBRENDER",
          "description": "WebRender",
          "status": "unavailable",
          "log": [
            {
              "type": "default",
              "status": "opt-in",
              "message": "WebRender is an opt-in feature"
            },
            {
              "type": "runtime",
              "status": "unavailable",
              "message": "Build doesn't include WebRender"
            }
          ]
        },
        {
          "name": "OMTP",
          "description": "Off Main Thread Painting",
          "status": "available",
          "log": [
            {
              "type": "default",
              "status": "available"
            }
          ]
        },
        {
          "name": "ADVANCED_LAYERS",
          "description": "Advanced Layers",
          "status": "blacklisted",
          "log": [
            {
              "type": "default",
              "status": "available"
            },
            {
              "type": "user",
              "status": "available",
              "message": "Enabled for Windows 7 via user-preference"
            },
            {
              "type": "env",
              "status": "blacklisted",
              "message": "#BLOCKLIST_FEATURE_FAILURE_BUG_1419264"
            }
(Assignee)

Comment 29

9 months ago
(In reply to Milan Sreckovic [:milan] from comment #23)
> Bas, can you take a look and see if bug 1419264 did take care of the crashes
> by disabling advanced layers, rather than just move them to a different
> signature?  If it did help, then lets see what other devices are affected
> and block advanced layers there.  In code.
> 
> I don't have a problem running AL, btw, Nvidia, 0x139b, GeForce GTX 960M

So, it's hard to have hard evidence on this, but looking through comments and reports and such. Windows 10 seems completely unaffected. As far as prevalence, AL seems to tickle this driver bug 'more' than regular layers, but it occurs with regular layers as well. All other evidence I found correlates with what Danial said, Pascal based GPU's, driver versions 385.69 and up, Windows 7. We could blacklist that set from acceleration completely to help the less frequently affected people as well.

Comment 30

9 months ago
(In reply to Bas Schouten (:bas.schouten) from comment #29)
> We could blacklist that set from acceleration completely
> to help the less frequently affected people as well.

Please don't, I don't want to lose HW acceleration. Or at least provide a way to override blacklist.
Like Danial Horton wrote already in Comment #17, unfortunately 391.01 didn't fix this issue completely.
There is also new NVIDIA GPU driver 391.05 ( https://nvidia.custhelp.com/app/answers/detail/a_id/4639 ), but it doesn't include fix for this issue.
There is no need to disable completely full Hardware Acceleration, as disabling just "Advanced Layers" in Firefox is enough to stop TDRs.
OS: Windows → Windows 7
Summary: Firefox has to blacklist all NVIDIA GPU drivers starting from version 385.69 to 390.77 to prevent TDR on Windows → Disable "Advanced Layers" in Firefox on all bugged NVIDIA GPU drivers (385.69-391.05/latest one) to prevent TDR on Windows 7

Comment 32

9 months ago
Please stop misconstruing data, the TDR's did not stop with the AL block, they were reduced considerably but still occur at random intervals with a highly active twitch stream reproducing them quite readily.

Comment 33

9 months ago
(In reply to Danial Horton from comment #32)
> Please stop misconstruing data, the TDR's did not stop with the AL block,
> they were reduced considerably but still occur at random intervals with a
> highly active twitch stream reproducing them quite readily.

Confirming crashing during Twitch viewing, but for me it also happens frequently when viewing an embedded stream where there is no chat visible).
I have also had it occur with FireFox Quantum open only to a single tab of a Google search result page in the background while playing a game, and the crash has occurred.
I am certainly going to follow this to keep an eye out for when it's safe. Thought I was going crazy when I started using FF Quantum a week ago and this started happening. Thanks for your hard work, guys.

Windows 7 x64
GTX 1070 with 391.01 
FF Quantum 64-bit 58.0.2
(In reply to Danial Horton from comment #32)
> Please stop misconstruing data, the TDR's did not stop with the AL block,
> they were reduced considerably but still occur at random intervals with a
> highly active twitch stream reproducing them quite readily.

I don't use Twitch, so I didn't know, that this issue is also reproducible with disabled "Advanced Layers" and with enabled "Hardware Acceleration", also nobody said that it's reproducible at start, at least in this bug.
In my case disabling "Advanced Layers" fixed TRDs completely, so please don't exaggerate.
Summary: Disable "Advanced Layers" in Firefox on all bugged NVIDIA GPU drivers (385.69-391.05/latest one) to prevent TDR on Windows 7 → Provide fix to Firefox to prevent TDR on Windows 7 with bugged NVIDIA GPU drivers (385.69-391.05/latest one) on Pascal GPUs

Comment 35

9 months ago
I confirm, no TDR on twitch for me, I tried the most active stream I've found ( https://www.twitch.tv/lck1 )

Comment 36

9 months ago
Running Win7 x64 with a 1080TI powering a 4K monitor via DP, using nvidia 391.01 drivers and Firefox Q 58.0.2 with layers.mlgpu.enable-on-windows7 set to false. Still getting driver crashes at least once per afternoon/ evening.
Comment hidden (spam)

Comment 38

9 months ago
(In reply to Virtual_ManPL [:Virtual] - (please needinfo? me - so I will see your comment/reply/question/etc.) from comment #34)
 
> I don't use Twitch, so I didn't know, that this issue is also reproducible
> with disabled "Advanced Layers" and with enabled "Hardware Acceleration",

Twitch is only the easy reproduction case, It's amusing how you 'didn't know' considering its been posted OVER AND OVER AND OVER that AL isn't the source, and all the users who had the issue remaining after the blocklist, 
You've been taking moom at his completely disproven word that disabling layers is the fix and are ignoring the entire census of a RANDOM issue.


> also nobody said that it's reproducible at start, at least in this bug.
> In my case disabling "Advanced Layers" fixed TRDs completely, so please
> don't exaggerate.

I said it.
To Here
and to Nvidia when filing the bug and getting the investigation going.

> In my case disabling "Advanced Layers" fixed TRDs completely, so please
> don't exaggerate.

No it didn't.
You're simply in a lul of non-reproduction.

"Testing can prove the presence of an issue, but no test can prove the absence of one."

Every driver between 387 and 390 that I thought had finally fixed the issue (in a browser that never has had AL) all ended up hitting the crash eventually. You simply haven't hit it yet.

Since its there in the drivers, you will eventually.
(In reply to Danial Horton from comment #37)
> stop commenting.

You may want to look at the Bugzilla Etiquette ( https://bugzilla.mozilla.org/page.cgi?id=etiquette.html ), before getting personal and etc.

Comment 40

9 months ago
(In reply to Danial Horton from comment #38)
> No it didn't.
> You're simply in a lul of non-reproduction.

That's ridiculous. I didn't encountered a single crash since AL was disabled, 3 months ago.
Some users have issues with TDR even with AL disabled (and even on Windows 10), but it may be a completely different issue for what we know.

Comment 41

9 months ago
Give me your computer and I'll reproduce it within 5 minutes, without AL.

The issue is in the driver, I can reproduce it, nvidia can reproduce it.
 the fact there is a bug screwing up GPC:Tex functions, which corrupts memory in the usermode driver, 

Your experience does not mean your system isn't affected by the bug, It just doesn't reproduce it as rapidly in your case because of an undisclosed factor.

Users crashing on 10 should consider their system unstable elsewhere, I can get no reproductions in any reproduction case that I do under 7 and many of those have had issues since well before 385.41, and included 900 series users.
@ mooms - Don't forget that if we didn't experience any more TDRs with disabled "Advanced Layers" and with enabled "Hardware Acceleration", it not means that TDRs can't happen for someone else.

Also there is no need to start getting personal and getting drama going on, especially when we all want this bug fixed ASAP in proper way for everyone. Let's keep this conversation civil.

Comment 43

9 months ago
(In reply to Danial Horton from comment #41)
> Give me your computer and I'll reproduce it within 5 minutes, without AL.

Then post proper steps to reproduce it. You said it is easily reproducible during Twitch viewing, I tried the most active twitch stream I've found and still no TDR.

(In reply to Virtual_ManPL [:Virtual] - (please needinfo? me - so I will see your comment/reply/question/etc.) from comment #42)
> @ mooms - Don't forget that if we didn't experience any more TDRs with
> disabled "Advanced Layers" and with enabled "Hardware Acceleration", it not
> means that TDRs can't happen for someone else.


I don't, I'm just saying it may be different issue with similar symptoms.
My brother with a similar setup (GTX 1070 & Win7) had frequent driver crashes before blacklisting AL and have none since 58.0
Too late to fix in 59. We can note it with potential workarounds if you can suggest a release note.
status-firefox59: affected → wontfix
Flags: needinfo?(bas)
No reply from Bas Schouten (:bas.schouten),
so maybe something like this will do.
Modifications or other versions are welcome.

> If you are experiencing increased amount of TDRs with Firefox due to buggy last GPU drivers,
> you may want to disable "Advanced Layers" feature in Firefox
> to mitigate most of TDRs without loosing Direct3D 11 compositing
> You can do that by setting to "false" these 2 preferences
> "layers.mlgpu.enable-on-windows7" and "layers.mlgpu.enabled" in "about:config".
+
> If TDRs didn't stop for you completely,
> you will need to disable Hardware Acceleration fully.
> You can do that in "Options" under "General" Tab (about:preferences#general) in "Performance" section
> by disabling "Use recommended performance settings"
> and disabling new shown option "Use Hardware acceleration when available"
> or by setting to "true" preference named "layers.acceleration.disabled" in "about:config".
(Assignee)

Comment 46

9 months ago
(In reply to Virtual_ManPL [:Virtual] - (please needinfo? me - so I will see your comment/reply/question/etc.) from comment #45)
> No reply from Bas Schouten (:bas.schouten),
> so maybe something like this will do.
> Modifications or other versions are welcome.
> 
> > If you are experiencing increased amount of TDRs with Firefox due to buggy last GPU drivers,
> > you may want to disable "Advanced Layers" feature in Firefox
> > to mitigate most of TDRs without loosing Direct3D 11 compositing
> > You can do that by setting to "false" these 2 preferences
> > "layers.mlgpu.enable-on-windows7" and "layers.mlgpu.enabled" in "about:config".
> +
> > If TDRs didn't stop for you completely,
> > you will need to disable Hardware Acceleration fully.
> > You can do that in "Options" under "General" Tab (about:preferences#general) in "Performance" section
> > by disabling "Use recommended performance settings"
> > and disabling new shown option "Use Hardware acceleration when available"
> > or by setting to "true" preference named "layers.acceleration.disabled" in "about:config".

This seems like a very reasonable set of suggestions, fwiw. On the one hand I'd be fully willing to expand the advanced-layers blacklist, however expanding blacklisting layers altogether seems risky considering the amount of people that will lose accelerated video that are -not- seeing this issue, it's a very difficult issue.
Flags: needinfo?(bas)
Joni, is this something you could write up as a SUMO page?  Probably needs some rephrasing. 
Then I can add it into the release notes for 59. 
need-info on myself to do this tomorrow.
relnote-firefox: --- → ?
Flags: needinfo?(lhenry)
Flags: needinfo?(jsavage)
Comment hidden (me-too, metoo)

Comment 49

9 months ago
Nvidia believes the next driver will fix this issue.

> We believe we have fixed this bug. I am confirming the bug fix will be in our next driver although from the bug details it appears it will be.

Comment 50

9 months ago
(In reply to Danial Horton from comment #49)
> Nvidia believes the next driver will fix this issue.
> 
> > We believe we have fixed this bug. I am confirming the bug fix will be in our next driver although from the bug details it appears it will be.

Thank you for the update. Did they provide the exact version that we can expect to involve this fix, or will it simply be the next available update?

Comment 51

9 months ago
Geforce Driver 391.24 lists this issue as resolved.

Comment 52

9 months ago
Indeed, the blacklist should not include drivers 391.24 and up.

In the meantime, is there any possibility to bypass the blacklist to test if the crashes are actually gone ?
(In reply to mooms from comment #52)
> In the meantime, is there any possibility to bypass the blacklist to test if
> the crashes are actually gone ?

No, but there is bug about adding this feature here bug #1442979.

Comment 54

9 months ago
Bat file with

SET MOZ_GFX_SPOOF_DRIVER_VERSION=23.21.13.8541
"Path to firefox" -p -no-remote

Comment 55

9 months ago
I have a feeling nvidia have been testing against the blacklisting builds of firefox, as I got a TDR the first time I tried to start firefox with that driver spoofed.

Comment 57

9 months ago
No crash here with the version spoof (advanced layers enabled).

Tried on Twitch, Reddit, Google images, etc...

Comment 58

9 months ago
sanity-test.device-id;0x1b81
sanity-test.version;20180315233128
sanity-test.driver-version;23.21.13.9124
layers.mlgpu.sanity-test-failed;false
sanity-test.running;false
sanity-test.advanced-layers;true

(#0) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 3
(#7) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 3
(#8) 	CP+[GFX1-]: (gfxWindowsPlatform) scheduled device update.
(#9) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#10) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 3
(#11) 	CP+[GFX1-]: (gfxWindowsPlatform) scheduled device update.
(#12) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 3
(#13) 	CP+[GFX1-]: (gfxWindowsPlatform) scheduled device update.
(#14) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 3
(#15) 	CP+[GFX1-]: (gfxWindowsPlatform) scheduled device update.
(#16) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 3
(#17) 	CP+[GFX1-]: (gfxWindowsPlatform) scheduled device update.
(#18) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().
(#19) 	CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 3
(#20) 	CP+[GFX1-]: (gfxWindowsPlatform) scheduled device update.
(#21) 	CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().

59.0.1 (64-Bit)

7 months for a placebo fix, several TDR today on a perfectly updated Win7 x64 SP1

Guess I roll back to driver 385.41 which is outdated but perfectly stable in every use-case.

GG Nvidia *claps hands*

I see myself out, just dropped in to let everyone know it is too early to celebrate.

FIXED doesn't mean fixed for everyone

Comment 59

9 months ago
(Advanced Layers) is in the Compositing field when you view about:support?

Looks like Firefox is now stable where the older direct3d11 compositor is used.

nvidia QA must have been testing release builds of firefox with the  blacklist active.

Comment 60

9 months ago
>sanity-test.driver-version;23.21.13.9124

seeing as that is in the info, no, you didn't have advanced layer active at all.

Comment 61

9 months ago
Created attachment 8960908 [details]
screenshot.png

adv layers on screenshot
I can confirm that NVIDIA GPU drivers version 391.24 still didn't fix this issue at all.
More over this is no fun anymore. More like madness, ridiculous and incompetency.
Especially when it was stated second time, that this bug was imaginary fixed in NVIDIA GPU drivers:

>NVIDIA GPU drivers version 391.01 version
>Issues and updates:
> [...]
> -Random TDR while using Firefox on Pascal GPUs
source:
https://forums.geforce.com/default/topic/1038299/geforce-drivers/official-391-01-game-ready-whql-display-driver-feedback-thread-released-2-26-18-/post/5296731/#5296731

>NVIDIA GPU drivers version 391.24 version
>Changes and Fixed Issues:
>[Firefox]: Driver TDR error may occur when using Firefox. [2049523]
source:
http://us.download.nvidia.com/Windows/391.24/391.24-win10-win8-win7-desktop-release-notes.pdf

At least NVIDIA fixed media player hang at start in these drivers...
tracking-firefox61: --- → ?
Summary: Provide fix to Firefox to prevent TDR on Windows 7 with bugged NVIDIA GPU drivers (385.69-391.05/latest one) on Pascal GPUs → Provide fix to Firefox to prevent TDR on Windows 7 with bugged NVIDIA GPU drivers (385.69-391.24/latest one) on Pascal GPUs

Comment 63

9 months ago
As I wrote in bug1419264 (https://bugzilla.mozilla.org/show_bug.cgi?id=1419264#c179), driver version 391.24 has NOT fixed my crashes with Firefox.
I had to disable the hardware acceleration again to prevent these crashes.
tracking-firefox61: ? → ---

Comment 64

9 months ago
I'm trying to find out if the issue wasn't mistakenly attached to this driver where it was checked into the upcoming farcry 5 driver.

Comment 65

9 months ago
>ManuelGuzmanNV said:
> If you can capture dump files, upload them to a file sharing service, and then email it to driverfeedback@nvidia.com, I'll have our team look into them. We did fix a bug with todays driver for a TDR issue we were able to reproduce on Windows 7 w/ Pascal GPUs. 

This includes dump files in c:\Windows\LiveKernelReports\WATCHDOG\

Comment 66

9 months ago
It's fixed for some users at least, I don't had any crashes with AL enabled here.

Comment 67

9 months ago
Update:
Emailed twice with 2 TDR dump files, one from Firefox 59 with the batch file trick, and another with Waterfox

Just Received this
>Thanks.  We think we have found the root cause that would only occur under Windows 7.  We are still testing so not fixed yet.
tracking-firefox61: ? → +

Comment 68

9 months ago
NVIDIA GPU drivers 391.24 version has not fixed my problem with Firefox too, I had to disable the hardware acceleration again to prevent these crashes.
win7-64bit up to date
ASUS EX-GTX1050TI-O4G
Firefox 59.0.1 (64-Bit)

Comment 69

9 months ago
Just had my first crash with the 391.24 drivers. Running WIN7-64bit with Firefox 59.0.1 with a 1080ti on a 4k monitor.

Comment 70

9 months ago
Just had a TDR with the 391.24 drivers, without AL enabled. Previous drivers never crashed with AL disabled.
I emailed the crash dump located in c:\Windows\LiveKernelReports\WATCHDOG\ to driverfeedback@nvidia.com

Failures logged in about:support:

    "failures": [
      "CP+[GFX1-]: [D2D1.1] 4CreateBitmap failure Size(154,17528) Code: 0x80070057 format 0",
      "CP+[GFX1-]: (gfxWindowsPlatform) scheduled device update.",
      "CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().",
      "GP+[GFX1-]: GFX: D3D11 skip BeginFrame with device-removed.",
      "CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 3",
      "CP+[GFX1-]: (gfxWindowsPlatform) scheduled device update.",
      "CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 3",
      "CP+[GFX1-]: (gfxWindowsPlatform) scheduled device update.",
      "CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().",
      "CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 3",
      "CP+[GFX1-]: (gfxWindowsPlatform) scheduled device update.",
      "CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().",
      "CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 3",
      "CP+[GFX1-]: (gfxWindowsPlatform) scheduled device update.",
      "CP+[GFX1-]: LayerManager::EndTransaction skip RenderLayer().",
      "CP+[GFX1-]: (gfxWindowsPlatform) Detected device reset: 3"

Comment 71

9 months ago
391.35 were just released.
(In reply to Arthur K. from comment #71)
> 391.35 were just released.

and still it doesn't contain fix for this terrible and awful issue...
Summary: Provide fix to Firefox to prevent TDR on Windows 7 with bugged NVIDIA GPU drivers (385.69-391.24/latest one) on Pascal GPUs → Provide fix to Firefox to prevent TDR on Windows 7 with bugged NVIDIA GPU drivers (385.69-391.35/latest one) on Pascal GPUs

Comment 73

9 months ago
391.35 are mainly for Far Cry 5 release. Next driver may (or may not...) contains a (newer) fix.

Comment 74

9 months ago
(In reply to Virtual_ManPL [:Virtual] - (please needinfo? me - so I will see your comment/reply/question/etc.) from comment #72)
> (In reply to Arthur K. from comment #71)
> > 391.35 were just released.
> 
> and still it doesn't contain fix for this terrible and awful issue...

Bummer. I read the release notes and didn't see anything specific but hope springs eternal.


> 391.35 are mainly for Far Cry 5 release. Next driver may (or may not...) contains a (newer) fix.

Fingers crossed. Thanks.

Comment 75

9 months ago
Tentatively confirming that the "FIX" is in the next driver.

It was approved but didn't make the cut off for 391.35
Sounds like we can start to prep a blocklist for testing but that will more likely be for the 60 release.   

If we end up with another 59 dot release, we can also consider taking an uplift. 

One thing I don't get from these comments is how we know who is affected and not affected. If some users crash in some configurations, but many others don't and get better performance, we might not blocklist, especially if there is a workaround available to users.
Flags: needinfo?(lhenry) → needinfo?(milan)
Bas, let's set up the patches for central and 60 beta, assuming 391.35 is the last release with the problem.
Assignee: nobody → bas
Flags: needinfo?(milan) → needinfo?(bas)
Comment hidden (mozreview-request)
Comment on attachment 8964954 [details]
Bug 1433058: Expand NVidia AL blacklist to include 385.96 and limit blacklist to the last affected version.

https://reviewboard.mozilla.org/r/233680/#review239662
Attachment #8964954 - Flags: review?(milan) → review+

Comment 80

8 months ago
Pushed by bschouten@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d4b2d3cd0ad1
Expand NVidia AL blacklist to include 385.96 and limit blacklist to the last affected version. r=milan
(Assignee)

Updated

8 months ago
Flags: needinfo?(bas)
Comment hidden (advocacy, offtopic)

Comment 82

8 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/d4b2d3cd0ad1
Status: ASSIGNED → RESOLVED
Last Resolved: 8 months ago
status-firefox61: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
Gentle ping and request about uplift request.
Status: RESOLVED → VERIFIED
status-firefox61: fixed → verified
Flags: needinfo?(milan)
Flags: needinfo?(bas)
Comment hidden (offtopic)
Comment hidden (offtopic)
(Assignee)

Comment 86

8 months ago
Comment on attachment 8964954 [details]
Bug 1433058: Expand NVidia AL blacklist to include 385.96 and limit blacklist to the last affected version.

Approval Request Comment
[Feature/Bug causing the regression]: NVidia driver update
[User impact if declined]: TDRs on affected machines and lack of advanced layers on potential future 'newer' machines.
[Is this code covered by automated tests?]: Not really, we have no such hardware in automated tests.
[Has the fix been verified in Nightly?]: Yes
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: No
[Why is the change risky/not risky?]: simply disabling advanced layers on a slightly different subset of machines
[String changes made/needed]: None
Flags: needinfo?(bas)
Attachment #8964954 - Flags: approval-mozilla-beta?
Comment on attachment 8964954 [details]
Bug 1433058: Expand NVidia AL blacklist to include 385.96 and limit blacklist to the last affected version.

Tweak the Advanced Layers blocklisting to work around more buggy nVidia drivers. Approved for 60.0b12.
Attachment #8964954 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Comment 88

8 months ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/8257635b628b
status-firefox60: affected → fixed
Awesome! Thank you very much for patch and its uplift! \o/
status-firefox60: fixed → verified

Comment 90

8 months ago
I've had this bug for months, windows 7, GTX 1070, only stopped when I turned off hardware acceleration in Firefox. That however meant horrible screen tearing when scrolling. The new nvidia patch seems to have stopped the driver crashing, but its brought the tearing back! Is there anything I can do about this?

Comment 91

8 months ago
(In reply to speakbattlesatlas from comment #90)
> I've had this bug for months, windows 7, GTX 1070, only stopped when I
> turned off hardware acceleration in Firefox. That however meant horrible
> screen tearing when scrolling. The new nvidia patch seems to have stopped
> the driver crashing, but its brought the tearing back! Is there anything I
> can do about this?

Just to add, I find the screen tearing more annoying than the occasional driver crash, so this is no fix in my book!
(In reply to speakbattlesatlas from comment #91)
> [...] this is no fix in my book!

You may want to downgrade NVIDIA GPU driver to version 385.41, as it latest NVIDIA GPU drivers without this issue and wait in meantime for future NVIDIA GPU drivers with real fix.

Comment 93

8 months ago
(In reply to Virtual_ManPL [:Virtual] - (please needinfo? me - so I will see your comment/reply/question/etc.) from comment #92)
> (In reply to speakbattlesatlas from comment #91)
> > [...] this is no fix in my book!
> 
> You may want to downgrade NVIDIA GPU driver to version 385.41, as it latest
> NVIDIA GPU drivers without this issue and wait in meantime for future NVIDIA
> GPU drivers with real fix.

Thanks for info, is there a thread about this on their site? I'd like to go and throw some **** at them - this has been going on for months now!

Comment 94

8 months ago
Oh may have spoken too soon - I restarted FF and it actually seems to be OK now, perhaps it is fixed after all.

Comment 95

8 months ago
Nope still crashing :(

Comment 96

8 months ago
can tell you now, 385.41 is a known good driver.

Comment 97

8 months ago
but you have to restart after downgrading, as the bad drivers leave the system in a unstable state after the first TDR has occured.

Comment 98

8 months ago
I have to say, adding drivers to blacklists is not a "fix" unless you can say with absolute certainty that the fault is entirely within the driver. It is odd that Firefox is the only application that is capable of consistently crashing Windows 7 machines with high end Nvidia drivers. I play a lot of demanding games that can run fullscreen windowed without a problem, but firefox can crash my computer when I change tabs. Again, blacklisting hardware acceleration for a large set of drivers that have no issues other than with you, that's not a fix. Firefox should not be a browser that has screen tears and stutters when streaming video.

Comment 99

8 months ago
You clearly haven't paid any attention.

Comment 100

8 months ago
(In reply to Ernie from comment #98)
> I have to say, adding drivers to blacklists is not a "fix" unless you can
> say with absolute certainty that the fault is entirely within the driver. It
> is odd that Firefox is the only application that is capable of consistently
> crashing Windows 7 machines with high end Nvidia drivers. I play a lot of
> demanding games that can run fullscreen windowed without a problem, but
> firefox can crash my computer when I change tabs. Again, blacklisting
> hardware acceleration for a large set of drivers that have no issues other
> than with you, that's not a fix. Firefox should not be a browser that has
> screen tears and stutters when streaming video.

Well. It is the fault of the driver, and it doesn't matter if Firefox is the only program (that we know of) that has this problem. It just means that Firefox is doing something that 99% of other programs don't do. - And as long as what Firefox is doing is within specifications, its not the one at fault.

Think of a glitch/exploit in a videogame and how some of them require extremely specific actions to activate. So specific that normally 99% of players would never experience it, but then there is that one guy who discovers if you do x,y,z something completely unexpected happens, and now has a broken save file or unlimited items.
Firefox is unfortunately 'That guy'.

Comment 101

8 months ago
I have had hardware acceleration turned off for several months now, and it prevents the crashes.

HOWEVER, if I have either Unity 3D or SideFX Houdini (these are 3D content creation applications) open while running Firefox and then maximize a Youtube video, the crash always happens even though hardware acceleration is deactivated.

Comment 102

8 months ago
397.31 is now available (but not all nvidia caches are updated to show as such)

Comment 103

8 months ago
(In reply to Danial Horton from comment #102)
> 397.31 is now available (but not all nvidia caches are updated to show as
> such)

What's New:
Vulcan 1.1, Nvidia RTX, Cuda 9.2
End of support for 32-bit and Fermi GPUs < !!!!!!!!!!!!!!!!!!!!!!!!!!!

Changelog:

Changes and Fixed Issues in Version 397.31
[GeForce GTX 1080 Ti][Doom]: The game crashes due to the driver reverting to OpenGL 1.1 when HDR is enabled. [2049623]
[GeForce GTX 1060][Far Cry 5]: The game crashes after a few minutes of game play. [2096077]
NvfbcPluginWindow temporarily prevents Windows from shutting down after launching a Steam game. [2068833]
[Firefox]: Driver TDR error may occur when using Firefox. [2049523]
[Geforce GTX 1060][Rise of Tomb Raider]: Flickering/corruption occurs when opening the in-game options UI. [200351146]
[NVIDIA Control Panel][SLI][Diablo III]: With V-Sync on and SLI enabled, the game freezes after switching windows (ALT+TAB) a few times. [1951584]

Shou;ld be published shortly but here are direct links to the drivers:
https://us.download.nvidia.com/Windows/397.31/397.31-desktop-win10-64bit-international-whql.exe
https://us.download.nvidia.com/Windows/397.31/397.31-desktop-win7-64bit-international-whql.exe

Comment 104

8 months ago
(In reply to Danial Horton from comment #102)
> 397.31 is now available (but not all nvidia caches are updated to show as
> such)

Is that supposed to be a fixed driver?

Comment 105

8 months ago
As far as I can tell, it is fixed.

Comment 106

8 months ago
Finally, I think the bug is fixed, I can not reproduce it anymore. Thanks to all!
Whiteboard: [gfx-noted] → [gfx-noted] [bug fixed completly in NVIDIA GPU drivers version 397.31]

Comment 107

8 months ago
Funny thing is, 397.40 which is based on a older code has the TDR issue xD

Comment 108

7 months ago
Now that the issue is fixed on Nvidia latest driver, can we lift the blacklist for 397.31, can we lift the blacklist ?
397.31 will not be blocked
(In reply to mooms from comment #108)
> Now that the issue is fixed on Nvidia latest driver, can we lift the blacklist for 397.31
Unnecessary , as NVIDIA GPU driver version 397.31 isn't blocked

(In reply to mooms from comment #108)
> can we lift the blacklist ?
No, because users who are using NVIDIA GPU drivers version 385.69-391.35 are still affected by TDRs due to NVIDIA GPU drivers bug, so they still have to be on blacklist.

Comment 111

7 months ago
Thanks for the great work, kids.
I don't think we need to relnote this since the fix came from an updated driver on nVidia's end, but I do think a SUMO article would still make sense if one hasn't been made yet (unclear to me if anything happened on that front since comment 47).
relnote-firefox: ? → -

Comment 113

7 months ago
(In reply to Virtual_ManPL [:Virtual] - (please needinfo? me - so I will see your comment/reply/question/etc.) from comment #110)

> No, because users who are using NVIDIA GPU drivers version 385.69-391.35 are
> still affected by TDRs due to NVIDIA GPU drivers bug, so they still have to
> be on blacklist.


397.40 needs to be individually added to the black list,  sure its a  vulkan dev driver but some people are using it under the assumption its newer.

Comment 114

7 months ago
Also 397.31 is blacklisted, in FX59, which is where mooms confusion lays, 59+ has an open ended black list
(In reply to Danial Horton from comment #113)
> (In reply to Virtual_ManPL [:Virtual] - (please needinfo? me - so I will see
> your comment/reply/question/etc.) from comment #110)
> 
> > No, because users who are using NVIDIA GPU drivers version 385.69-391.35 are
> > still affected by TDRs due to NVIDIA GPU drivers bug, so they still have to
> > be on blacklist.
> 
> 
> 397.40 needs to be individually added to the black list,  sure its a  vulkan
> dev driver but some people are using it under the assumption its newer.

Yes. I created follow up bug #1458303 about this.

Comment 116

7 months ago
Can someone remind me the Pref to set to adjust the failure limit?

Comment 117

7 months ago
I don't think this should be affecting anyone anymore but NVidia released 397.64 WHQL the other day.
Flags: needinfo?(jsavage)
You need to log in before you can comment on or make changes to this bug.