Closed Bug 1342690 Opened 7 years ago Closed 3 years ago

Assertion failure - Opcode sequence includes commands after JMP

Categories

(Core Graveyard :: Plug-ins, defect, P3)

defect

Tracking

(firefox51 unaffected, firefox52 unaffected, firefox-esr52 unaffected, firefox53 unaffected, firefox54 affected, firefox62 affected)

RESOLVED WONTFIX
Tracking Status
firefox51 --- unaffected
firefox52 --- unaffected
firefox-esr52 --- unaffected
firefox53 --- unaffected
firefox54 --- affected
firefox62 --- affected

People

(Reporter: Oriol, Unassigned)

References

Details

(Keywords: assertion, Whiteboard: tpi:+)

When I attempt to run my compiled Firefox, I get this error:

> DLL blocklist was unable to intercept AppInit DLLs.
> Assertion failure: false (MOZ_ASSERT_UNREACHABLE: Opcode sequence includes commands after JMP), at
> c:\builds\moz2_slave\m-in-w64-d-0000000000000000000\build\src\xpcom\build\nsWindowsDllInterceptor.h:793

I bisected in mozilla-inbound and I got

> The first bad revision is:
> changeset:   344281:8804a3f8fb15
> user:        David Parks <dparks@mozilla.com>
> date:        Wed Feb 08 11:38:40 2017 -0800
> summary:     Bug 1284897 - Hook GetSaveFileNameW/GetOpenFileNameW to record and grant a sandboxed process permission to access user-chosen files. r=jimm
Flags: needinfo?(davidp99)
Component: Untriaged → Plug-ins
Version: unspecified → Trunk
This looks like it was probably a silent failure before the ASSERTs were added.  Oriol, what version of Windows did this happen with?  The specific build would be helpful -- you should be able to get this from the OS by opening System Information and grabbing the Version from the System Summary tab.  This bug may be tied to a specific build of the OS.

Also, would you mind pasting your about:support info into this bug?
Flags: needinfo?(davidp99) → needinfo?(oriol-bugzilla)
System information says it's version 10.0.14393 Build 14393.
Winver says version 1607 (14393.693).

This is my about:support

> Application Basics
> ------------------
> 
> Name: Firefox
> Version: 54.0a1
> Build ID: 20170227030203
> Update Channel: nightly
> User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0
> OS: Windows_NT 10.0
> Multiprocess Windows: 1/1 (Enabled by default)
> Google Key: Found
> Mozilla Location Service Key: Found
> Safe Mode: false
> 
> Crash Reports for the Last 3 Days
> ---------------------------------
> 
> All Crash Reports
> 
> Extensions
> ----------
> 
> Name: Application Update Service Helper
> Version: 2.0
> Enabled: true
> ID: aushelper@mozilla.org
> 
> Name: FlyWeb
> Version: 1.0.0
> Enabled: true
> ID: flyweb@mozilla.org
> 
> Name: Form Autofill
> Version: 1.0
> Enabled: true
> ID: formautofill@mozilla.org
> 
> Name: Multi-process staged rollout
> Version: 1.9
> Enabled: true
> ID: e10srollout@mozilla.org
> 
> Name: Pocket
> Version: 1.0.5
> Enabled: true
> ID: firefox@getpocket.com
> 
> Name: Presentation
> Version: 1.0.0
> Enabled: true
> ID: presentation@mozilla.org
> 
> Name: Shield Recipe Client
> Version: 1.0.0
> Enabled: true
> ID: shield-recipe-client@mozilla.org
> 
> Name: Web Compat
> Version: 1.1
> Enabled: true
> ID: webcompat@mozilla.org
> 
> Name: WebCompat Reporter
> Version: 1.0.0
> Enabled: true
> ID: webcompat-reporter@mozilla.org
> 
> Graphics
> --------
> 
> Features
> Compositing: Direct3D 11
> Asynchronous Pan/Zoom: wheel input enabled; touch input enabled; scrollbar drag enabled
> WebGL 1 Driver WSI Info: EGL_VENDOR: Google Inc. (adapter LUID: 000000000000a11d) EGL_VERSION: 1.4 (ANGLE 2.1.0.2a250c8a0e15) EGL_EXTENSIONS: EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_ANGLE_direct_composition EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_KHR_stream EGL_KHR_stream_consumer_gltexture EGL_NV_stream_consumer_gltexture_yuv EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_stream_producer_d3d_texture_nv12 EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource EGL_EXTENSIONS(nullptr): EGL_EXT_client_extensions EGL_EXT_platform_base EGL_EXT_platform_device EGL_ANGLE_platform_angle EGL_ANGLE_platform_angle_d3d EGL_ANGLE_device_creation EGL_ANGLE_device_creation_d3d11 EGL_ANGLE_experimental_present_path EGL_KHR_client_get_all_proc_addresses
> WebGL 1 Driver Renderer: Google Inc. -- ANGLE (Intel(R) HD Graphics 530 Direct3D11 vs_5_0 ps_5_0)
> WebGL 1 Driver Version: OpenGL ES 2.0 (ANGLE 2.1.0.2a250c8a0e15)
> WebGL 1 Driver Extensions: GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_lossy_etc_decode GL_ANGLE_pack_reverse_row_order GL_ANGLE_robust_client_memory GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_copy_compressed_texture GL_CHROMIUM_copy_texture GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_discard_framebuffer GL_EXT_disjoint_timer_query GL_EXT_draw_buffers GL_EXT_frag_depth GL_EXT_map_buffer_range GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_sRGB GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_rg GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_EGL_stream_consumer_external GL_NV_fence GL_NV_pack_subimage GL_NV_pixel_buffer_object GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object
> WebGL 1 Extensions: ANGLE_instanced_arrays EXT_blend_minmax EXT_color_buffer_half_float EXT_frag_depth EXT_shader_texture_lod EXT_texture_filter_anisotropic EXT_disjoint_timer_query MOZ_debug_get OES_element_index_uint OES_standard_derivatives OES_texture_float OES_texture_float_linear OES_texture_half_float OES_texture_half_float_linear OES_vertex_array_object WEBGL_color_buffer_float WEBGL_compressed_texture_s3tc WEBGL_debug_renderer_info WEBGL_debug_shaders WEBGL_depth_texture WEBGL_draw_buffers WEBGL_lose_context MOZ_WEBGL_lose_context MOZ_WEBGL_compressed_texture_s3tc MOZ_WEBGL_depth_texture
> WebGL 2 Driver WSI Info: EGL_VENDOR: Google Inc. (adapter LUID: 000000000000a11d) EGL_VERSION: 1.4 (ANGLE 2.1.0.2a250c8a0e15) EGL_EXTENSIONS: EGL_EXT_create_context_robustness EGL_ANGLE_d3d_share_handle_client_buffer EGL_ANGLE_d3d_texture_client_buffer EGL_ANGLE_surface_d3d_texture_2d_share_handle EGL_ANGLE_query_surface_pointer EGL_ANGLE_window_fixed_size EGL_ANGLE_keyed_mutex EGL_ANGLE_surface_orientation EGL_ANGLE_direct_composition EGL_NV_post_sub_buffer EGL_KHR_create_context EGL_EXT_device_query EGL_KHR_image EGL_KHR_image_base EGL_KHR_gl_texture_2D_image EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image EGL_KHR_get_all_proc_addresses EGL_KHR_stream EGL_KHR_stream_consumer_gltexture EGL_NV_stream_consumer_gltexture_yuv EGL_ANGLE_flexible_surface_compatibility EGL_ANGLE_stream_producer_d3d_texture_nv12 EGL_ANGLE_create_context_webgl_compatibility EGL_CHROMIUM_create_context_bind_generates_resource EGL_EXTENSIONS(nullptr): EGL_EXT_client_extensions EGL_EXT_platform_base EGL_EXT_platform_device EGL_ANGLE_platform_angle EGL_ANGLE_platform_angle_d3d EGL_ANGLE_device_creation EGL_ANGLE_device_creation_d3d11 EGL_ANGLE_experimental_present_path EGL_KHR_client_get_all_proc_addresses
> WebGL 2 Driver Renderer: Google Inc. -- ANGLE (Intel(R) HD Graphics 530 Direct3D11 vs_5_0 ps_5_0)
> WebGL 2 Driver Version: OpenGL ES 3.0 (ANGLE 2.1.0.2a250c8a0e15)
> WebGL 2 Driver Extensions: GL_ANGLE_depth_texture GL_ANGLE_framebuffer_blit GL_ANGLE_framebuffer_multisample GL_ANGLE_instanced_arrays GL_ANGLE_lossy_etc_decode GL_ANGLE_pack_reverse_row_order GL_ANGLE_robust_client_memory GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ANGLE_texture_usage GL_ANGLE_translated_shader_source GL_CHROMIUM_bind_generates_resource GL_CHROMIUM_bind_uniform_location GL_CHROMIUM_copy_compressed_texture GL_CHROMIUM_copy_texture GL_CHROMIUM_sync_query GL_EXT_blend_minmax GL_EXT_color_buffer_float GL_EXT_color_buffer_half_float GL_EXT_debug_marker GL_EXT_discard_framebuffer GL_EXT_disjoint_timer_query GL_EXT_draw_buffers GL_EXT_frag_depth GL_EXT_map_buffer_range GL_EXT_occlusion_query_boolean GL_EXT_read_format_bgra GL_EXT_robustness GL_EXT_sRGB GL_EXT_shader_texture_lod GL_EXT_texture_compression_dxt1 GL_EXT_texture_filter_anisotropic GL_EXT_texture_format_BGRA8888 GL_EXT_texture_norm16 GL_EXT_texture_rg GL_EXT_texture_storage GL_EXT_unpack_subimage GL_KHR_debug GL_NV_EGL_stream_consumer_external GL_NV_fence GL_NV_pack_subimage GL_NV_pixel_buffer_object GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_image_external_essl3 GL_OES_compressed_ETC1_RGB8_texture GL_OES_depth32 GL_OES_element_index_uint GL_OES_get_program_binary GL_OES_mapbuffer GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_OES_standard_derivatives GL_OES_texture_float GL_OES_texture_float_linear GL_OES_texture_half_float GL_OES_texture_half_float_linear GL_OES_texture_npot GL_OES_vertex_array_object
> WebGL 2 Extensions: EXT_color_buffer_float EXT_texture_filter_anisotropic EXT_disjoint_timer_query MOZ_debug_get OES_texture_float_linear WEBGL_compressed_texture_s3tc WEBGL_debug_renderer_info WEBGL_debug_shaders WEBGL_lose_context MOZ_WEBGL_lose_context MOZ_WEBGL_compressed_texture_s3tc
> Audio Backend: wasapi
> Direct2D: true
> DirectWrite: true (10.0.14393.351)
> GPU #1
> Active: Yes
> Description: Intel(R) HD Graphics 530
> Vendor ID: 0x8086
> Device ID: 0x1912
> Driver Version: 21.20.16.4590
> Driver Date: 1-18-2017
> Drivers: igdumdim64 igd10iumd64 igd10iumd64 igd12umd64 igdumdim32 igd10iumd32 igd10iumd32 igd12umd32
> Subsys ID: 00000000
> RAM: Unknown
> 
> Diagnostics
> AzureCanvasAccelerated: 0
> AzureCanvasBackend: Direct2D 1.1
> AzureCanvasBackend (UI Process): skia
> AzureContentBackend: Direct2D 1.1
> AzureContentBackend (UI Process): skia
> AzureFallbackCanvasBackend (UI Process): cairo
> GPUProcessPid: 4068
> GPUProcess: Terminate GPU Process
> failures: CP+[GFX1]: Potential driver version mismatch ignored due to missing DLLs igd10umd64 v= and igd10iumd64.dll v=0.0.0.0
> Decision Log
> D3D9_COMPOSITING:
> disabled by default: Disabled by default
> WEBRENDER:
> unavailable by runtime: Build doesn't include WebRender
> 
> 
> Failure Log
> (#0): CP+[GFX1]: Potential driver version mismatch ignored due to missing DLLs igd10umd64 v= and igd10iumd64.dll v=0.0.0.0
> (#1): CP+[GFX1]: Potential driver version mismatch ignored due to missing DLLs igd10umd64 v= and igd10iumd64.dll v=0.0.0.0
> (#2): CP+[GFX1]: Potential driver version mismatch ignored due to missing DLLs igd10umd64 v= and igd10iumd64.dll v=0.0.0.0
> 
> Important Modified Preferences
> ------------------------------
> 
> browser.cache.disk.capacity: 1048576
> browser.cache.disk.filesystem_reported: 1
> browser.cache.disk.smart_size.first_run: false
> browser.cache.frecency_experiment: 3
> browser.places.smartBookmarksVersion: 8
> browser.startup.homepage_override.buildID: 20170227030203
> browser.startup.homepage_override.mstone: 54.0a1
> browser.urlbar.daysBeforeHidingSuggestionsPrompt: 3
> browser.urlbar.lastSuggestionsPromptDate: 20170227
> dom.gamepad.extensions.enabled: true
> extensions.lastAppVersion: 54.0a1
> media.gmp.storage.version.observed: 1
> media.hardware-video-decoding.failed: false
> network.cookie.prefsMigrated: true
> places.history.expiration.transient_current_max_pages: 122334
> plugin.disable_full_page_plugin_for_types: application/pdf
> security.sandbox.content.tempDirSuffix: {87e8edde-87c5-4672-97dd-4497a26061ec}
> ui.osk.debug.keyboardDisplayReason: IKPOS: Touch screen not found.
> 
> Important Locked Preferences
> ----------------------------
> 
> Places Database
> ---------------
> 
> JavaScript
> ----------
> 
> Incremental GC: true
> 
> Accessibility
> -------------
> 
> Activated: false
> Prevent Accessibility: 0
> 
> Library Versions
> ----------------
> 
> NSPR
> Expected minimum version: 4.13.1
> Version in use: 4.13.1
> 
> NSS
> Expected minimum version: 3.30 Beta
> Version in use: 3.30 Beta
> 
> NSSSMIME
> Expected minimum version: 3.30 Beta
> Version in use: 3.30 Beta
> 
> NSSSSL
> Expected minimum version: 3.30 Beta
> Version in use: 3.30 Beta
> 
> NSSUTIL
> Expected minimum version: 3.30 Beta
> Version in use: 3.30 Beta
> 
> Experimental Features
> ---------------------
> 
> Sandbox
> -------
> 
> Content Process Sandbox Level: 2
> 

There are some graphics failures (being fixed in bug 1325980), but I think they are unrelated because the failed assert in this bug happens before the failed assert in bug 1325980.
Flags: needinfo?(oriol-bugzilla)
I have bisected again and now I got

> The first bad revision is:
> changeset:   344278:e2609138ebe1
> user:        David Parks <dparks@mozilla.com>
> date:        Tue Feb 07 12:00:45 2017 -0800
> summary:     Bug 1284897 - Add missing hooked methods to TestDllInterceptor. r=aklotz

It's the same bug, but this is the proper change. In comment 0 I got another one, not sure if because I used --enable-artifact-builds or something like that.

It may be worth noting I already got
> DLL blocklist was unable to intercept AppInit DLLs.
before the assert was introduced.

Is this the cause of the problem?
David, does this assertion failure point to a real bug we should fix before we make 64-bit Firefox the default install?
Flags: needinfo?(davidp99)
Chris,
I ran it by jimm and we don't think this should block 64-bit.  Its only an issue in debug and so far its only seen here.

Oriol,
Thanks for the info -- I'm baffled as to why we don't see the same thing.  It probably does have something to do with the DLL blocklist message you mentioned... but I can't guess what.  Since you build debug, would you also run this test:

> obj-dir/dist/bin/TestDllInterceptor.exe

where obj-dir is the folder you build into.  Can you dump the output of that into this bug?  It should tell us which DLL is causing problems, which should be a good a place to start.
Flags: needinfo?(davidp99) → needinfo?(oriol-bugzilla)
I don't have TestDllInterceptor.exe. This is my mozconfig:

> ac_add_options --enable-application=browser
> ac_add_options --target=x86_64-pc-mingw32
> ac_add_options --host=x86_64-pc-mingw32
> ac_add_options --enable-optimize 
> mk_add_options MOZ_MAKE_FLAGS="-j4"
> ac_add_options --disable-crashreporter
> ac_add_options --enable-debug
> ac_add_options --enable-artifact-builds

By the way, another possibly related DLL problem I had was bug 1285356 comment 13.
(In reply to David Parks (dparks) [:handyman] from comment #5)
> I ran it by jimm and we don't think this should block 64-bit.  Its only an
> issue in debug and so far its only seen here.

Thanks. I'll remove this bug from our "win64-rollout" meta bug.
No longer blocks: win64-rollout
Thanks Chris.

Oriol, that's weird.  I can only guess that it has something to do with something in the mozconfig -- can you try one like this:

---
ac_add_options --enable-debug
ac_add_options --disable-optimize
ac_add_options --enable-warnings-as-errors
 
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-debug
---

(building into obj-debug so that you dont stomp on your current work)

This is my mozconfig so I assume that will build it...
OK, with that mozconfig there is the TestDllInterceptor.exe. The output is

> TEST-PASS | WindowsDllInterceptor | Hook added
> TEST-PASS | WindowsDllInterceptor | Hook called
> TEST-PASS | WindowsDllInterceptor | Hook works properly
> TEST-PASS | WindowsDllInterceptor | Hook was not called after unregistration
> TEST-PASS | WindowsDllInterceptor | Original function worked properly
> TEST-PASS | WindowsDllInterceptor | Could hook GetWindowInfo from user32.dll
> Assertion failure: false (MOZ_ASSERT_UNREACHABLE: Opcode sequence includes commands after JMP), at c:\mozilla-build\mozilla-source\inbound\obj-debug\dist\include\nsWindowsDllInterceptor.h:793

I commented out the assert, and then

> TEST-PASS | WindowsDllInterceptor | Hook added
> TEST-PASS | WindowsDllInterceptor | Hook called
> TEST-PASS | WindowsDllInterceptor | Hook works properly
> TEST-PASS | WindowsDllInterceptor | Hook was not called after unregistration
> TEST-PASS | WindowsDllInterceptor | Original function worked properly
> TEST-PASS | WindowsDllInterceptor | Could hook GetWindowInfo from user32.dll
> TEST-UNEXPECTED-FAIL | WindowsDllInterceptor | Failed to hook SetWindowLongPtrA from user32.dll
Flags: needinfo?(oriol-bugzilla)
Bug 1192844 seems related.
See Also: → 1192844
Oriol,

Somehow, I missed that you are running Win10 Anniversary and I'm on Windows 10 Mobile -- so this is much less of a surprise.  Somehow our entire test infrastructure is also missing your case.  I know what to do now.  I may lean on you for testing an such if thats cool.  Thanks for helping me get to the bottom of this.
OK, tell me when you want me to test something.
I have a laptop also with Win10 Anniversary x64. I copied TestDllInterceptor.exe, and there it works:

> TEST-PASS | WindowsDllInterceptor | Hook added
> TEST-PASS | WindowsDllInterceptor | Hook called
> TEST-PASS | WindowsDllInterceptor | Hook works properly
> TEST-PASS | WindowsDllInterceptor | Hook was not called after unregistration
> TEST-PASS | WindowsDllInterceptor | Original function worked properly
> TEST-PASS | WindowsDllInterceptor | Could hook GetWindowInfo from user32.dll
> TEST-PASS | WindowsDllInterceptor | Could hook SetWindowLongPtrA from user32.dll
> TEST-PASS | WindowsDllInterceptor | Could hook SetWindowLongPtrW from user32.dll
> TEST-PASS | WindowsDllInterceptor | Could hook TrackPopupMenu from user32.dll
> TEST-PASS | WindowsDllInterceptor | Could hook NtCreateFile from ntdll.dll
> TEST-PASS | WindowsDllInterceptor | Could hook NtReadFile from ntdll.dll
> TEST-PASS | WindowsDllInterceptor | Could hook NtReadFileScatter from ntdll.dll
> TEST-PASS | WindowsDllInterceptor | Could hook NtWriteFile from ntdll.dll
> TEST-PASS | WindowsDllInterceptor | Could hook NtWriteFileGather from ntdll.dll
> TEST-PASS | WindowsDllInterceptor | Could hook NtQueryFullAttributesFile from ntdll.dll
> TEST-PASS | WindowsDllInterceptor | Could hook SetUnhandledExceptionFilter from kernel32.dll
> TEST-PASS | WindowsDllInterceptor | Could hook CreateFileA from kernel32.dll
> TEST-PASS | WindowsDllInterceptor | Could detour CreateWindowExW from user32.dll
> TEST-PASS | WindowsDllInterceptor | Could hook InSendMessageEx from user32.dll
> TEST-PASS | WindowsDllInterceptor | Could hook ImmGetContext from imm32.dll
> TEST-PASS | WindowsDllInterceptor | Could hook ImmGetCompositionStringW from imm32.dll
> TEST-PASS | WindowsDllInterceptor | Could hook ImmSetCandidateWindow from imm32.dll
> TEST-PASS | WindowsDllInterceptor | Could hook ImmNotifyIME from imm32.dll
> TEST-PASS | WindowsDllInterceptor | Could hook GetSaveFileNameW from comdlg32.dll
> TEST-PASS | WindowsDllInterceptor | Could hook GetOpenFileNameW from comdlg32.dll
> TEST-PASS | WindowsDllInterceptor | Could hook GetKeyState from user32.dll
> TEST-PASS | WindowsDllInterceptor | Could detour LdrLoadDll from ntdll.dll
> TEST-PASS | WindowsDllInterceptor | all checks passed

So the problem is not caused by the Windows version.
(In reply to Oriol from comment #13)
> I have a laptop also with Win10 Anniversary x64. I copied
> TestDllInterceptor.exe, and there it works:
> So the problem is not caused by the Windows version.

Huh.

According to Wikipedia [1], there are both desktop and mobile versions of Win 10 Anniversary (its not clear if they are different as they share build numbers) so its still possible that that's still the issue... but lets check the AppInit DLL stuff.

Can you dump the output of this command into the bug?  (If you could also do it on the laptop that works, even better):

reg query "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows" /v AppInit_DLLs

Its possible your machine has installed something that is hijacking the DLL before we can get to it.  In that case, all bets are off but the culprit could be something widespread enough that we need to know about it.  At the least, it can help eliminate the possibility.

[1] https://en.wikipedia.org/wiki/Windows_10_version_history
Flags: needinfo?(oriol-bugzilla)
In both PCs I get

> HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows
>     AppInit_DLLs    REG_SZ

And by the way, TestDllInterceptor.exe works if I boot in safe mode.
Flags: needinfo?(oriol-bugzilla)
Not sure if this is helpful, in AddHook, GetProcAddress returns this address:
> e9 cb 6e ba fc cc e9 c5 e5 ff ff cc cc cc cc cc cc cc cc cc 48 ...

and ResolveRedirectedAddress converts it to
> ff 25 f2 ff ff ff cc cc 45 33 c9 e9 f0 76 45 03 ff 25 12 00 00 ...

The assert fails at the first 45.
By the way, this only happens when I build 64-bit Firefox, it works correctly with 32-bit Firefox.
I installed Win10 x64 in a PC for testing purposes, and TestDllInterceptor worked. Then I installed my antivirus, COMODO Internet Security, and I got the JMP problem. So this must be the reason. Disabling all the antivirus protections does not seem to fix the problem.
This is strange. Even killing the COMODO processes and services does not seem to fix the problem.

If I boot in safe mode, it works. But then I can start COMODO and enable all protections, and TestDllInterceptor still works.
I found another way to reproduce the problem: inside a Sandboxie sandbox. In fact, it fails earlier:

> TEST-PASS | WindowsDllInterceptor | Hook added
> TEST-PASS | WindowsDllInterceptor | Hook called
> TEST-PASS | WindowsDllInterceptor | Hook works properly
> TEST-PASS | WindowsDllInterceptor | Hook was not called after unregistration
> TEST-PASS | WindowsDllInterceptor | Original function worked properly
> Opcode sequence includes commands after JMP
> TEST-UNEXPECTED-FAIL | WindowsDllInterceptor | Failed to hook GetWindowInfo from user32.dll

Tested with Windows 10 x64 version 1703 (15063.13) and Sandboxie x64 version 5.17.6 beta
https://www.sandboxie.com/SandboxieInstall-517-6.exe
It sounds like Firefox and the AV are both trying to hook the same Windows APIs, and the two are colliding. Unfortunately this isn't the first time this has happened.
Priority: -- → P3
Whiteboard: tpi:+
Per policy at https://wiki.mozilla.org/Bug_Triage/Projects/Bug_Handling/Bug_Husbandry#Inactive_Bugs. If this bug is not an enhancement request or a bug not present in a supported release of Firefox, then it may be reopened.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INACTIVE
This bug is still present. It's so annoying when I need a debug build, I have to import a patch which disables the assert.
Status: RESOLVED → REOPENED
Resolution: INACTIVE → ---
Resolving as wont fix, plugin support deprecated in Firefox 85.
Status: REOPENED → RESOLVED
Closed: 6 years ago3 years ago
Resolution: --- → WONTFIX
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.