These are not stack overflows but rather Control Flow Guard fast-fail failures, which use the same error code. The call stack is: ``` ntdll!RtlFailFast2 ntdll!RtlpHandleInvalidUserCallTarget+0x93 ntdll!RtlDispatchException+0x56c41 ntdll!KiUserExceptionDispatcher+0xf ntdll!LdrpValidateUserCallTargetBitMapCheck user32!DispatchHookW+0xcd user32!CallHookWithSEH+0x59 user32!__fnHkINDWORD+0x26 ntdll!KiUserCallbackDispatcher+0x36 win32u!NtUserPeekMessage+0xc user32!_PeekMessage+0xeb user32!PeekMessageW+0x1b1 xul!mozilla::widget::WinUtils::WaitForMessage+0x99 ``` I am very confident that this crash occurs when a DLL sets up a hook with `SetWindowsHookEx` and that hook propagates to our threads, if that DLL hasn't been compiled with `/cf:guard`. Notably, if a program calls it with `dwThreadId` to 0, that will install the hook on *all existing threads running in the same desktop as the calling thread* per the documentation. The developers might be doing that without realizing the impact -- in any case, we would need to find out which DLLs are doing these calls, and block them. So far I have found that pattern being used in `evpComponentsXE.bpl` and I would thus recommend blocking it (more details about that investigation below). As for the link to bug 1998650, probably? Perhaps if Firefox 32-bit was previously loading an outdated-enough version of the runtime DLLs from the SysWOW64 directory, that might have resulted in Control Flow Guard getting somehow unenforced for our processes, and now that we load up-to-date DLLs the mitigation would be active again? This would deserve further investigation, but this does have the pattern of a new issue affecting old software, so likely linked to a change on our side, and that change is indeed a good candidate. Below are the details for the `evpComponentsXE.bpl` investigation. I've downloaded the crash dumps and looked for the unusual DLLs being the most present: they are the BPL files signed by `Idera, Inc.`, which are loaded from `C:\Program Files (x86)\Sage Evolution` and are present in 26 crashes (out of 31) from 8 unique users (based on `install_time`). But, there are a lot of them. Luckily I've found a zip file that contains those BPL files by following [this download procedure](https://za-kb.sage.com/portal/app/portlets/results/viewsolution.jsp?solutionid=230602122753277&page=1&position=0&q=230602122753277). The BPL files there are not signed with the same vendor name (maybe due to localization), but they are in the same version, so close enough for investigating. These are Delphi packages, and as suspected they don't have the CF Guard tables like our binaries do, this can be confirmed by comparing the output of `dumpbin /loadconfig` for `mozglue.dll` and e.g. `evpComponentsXE.bpl`. Then I created the dependency graph between these DLLs and realized that the ones that start with `evp` are among the most top-level DLLs. I compared that to the files that import `SetWindowsHookExW` and confirmed the use of a zero `dwThreadId` by `evpComponentsXE.bpl` -- the other `.bpl` are just its (long list of) dependencies.
Bug 1400169 Comment 17 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
These are not stack overflows but rather Control Flow Guard fast-fail failures, which use the same error code. The call stack is: ``` ntdll!RtlFailFast2 ntdll!RtlpHandleInvalidUserCallTarget+0x93 ntdll!RtlDispatchException+0x56c41 ntdll!KiUserExceptionDispatcher+0xf ntdll!LdrpValidateUserCallTargetBitMapCheck user32!DispatchHookW+0xcd user32!CallHookWithSEH+0x59 user32!__fnHkINDWORD+0x26 ntdll!KiUserCallbackDispatcher+0x36 win32u!NtUserPeekMessage+0xc user32!_PeekMessage+0xeb user32!PeekMessageW+0x1b1 xul!mozilla::widget::WinUtils::WaitForMessage+0x99 ``` I am very confident that this crash occurs when a DLL sets up a hook with `SetWindowsHookEx` and that hook propagates to our threads, if that DLL hasn't been compiled with `/cf:guard`. Notably, if a program calls it with `dwThreadId` to 0, that will install the hook on *all existing threads running in the same desktop as the calling thread* per the documentation. The developers might be doing that without realizing the impact -- in any case, we would need to figure out which DLLs are doing these calls, and block them. So far I have found that pattern being used in `evpComponentsXE.bpl` and I would thus recommend blocking it (more details about that investigation below). As for the link to bug 1998650, probably? Perhaps if Firefox 32-bit was previously loading an outdated-enough version of the runtime DLLs from the SysWOW64 directory, that might have resulted in Control Flow Guard getting somehow unenforced for our processes, and now that we load up-to-date DLLs the mitigation would be active again? This would deserve further investigation, but the crash volume does have the pattern of a new issue affecting old software, so likely linked to a change on our side, and that change is indeed a good candidate. Below are the details for the `evpComponentsXE.bpl` investigation. I've downloaded the crash dumps and looked for the unusual DLLs being the most present: they are the BPL files signed by `Idera, Inc.`, which are loaded from `C:\Program Files (x86)\Sage Evolution` and are present in 26 crashes (out of 31) from 8 unique users (based on `install_time`). But, there are a lot of them. Luckily I've found a zip file that contains those BPL files by following [this download procedure](https://za-kb.sage.com/portal/app/portlets/results/viewsolution.jsp?solutionid=230602122753277&page=1&position=0&q=230602122753277). The BPL files there are not signed with the same vendor name (maybe due to localization), but they are in the same version, so close enough for investigating. These are Delphi packages, and as suspected they don't have the CF Guard tables like our binaries do, this can be confirmed by comparing the output of `dumpbin /loadconfig` for `mozglue.dll` and e.g. `evpComponentsXE.bpl`. Then I created the dependency graph between these DLLs and realized that the ones that start with `evp` are among the most top-level DLLs. I compared that to the files that import `SetWindowsHookExW` and confirmed the use of a zero `dwThreadId` by `evpComponentsXE.bpl` -- the other `.bpl` are just its (long list of) dependencies.
These are not stack overflows but rather Control Flow Guard fast-fail failures, which use the same error code. The call stack is: ``` ntdll!RtlFailFast2 ntdll!RtlpHandleInvalidUserCallTarget+0x93 ntdll!RtlDispatchException+0x56c41 ntdll!KiUserExceptionDispatcher+0xf ntdll!LdrpValidateUserCallTargetBitMapCheck user32!DispatchHookW+0xcd user32!CallHookWithSEH+0x59 user32!__fnHkINDWORD+0x26 ntdll!KiUserCallbackDispatcher+0x36 win32u!NtUserPeekMessage+0xc user32!_PeekMessage+0xeb user32!PeekMessageW+0x1b1 xul!mozilla::widget::WinUtils::WaitForMessage+0x99 ``` I am very confident that this crash occurs when a DLL sets up a hook with `SetWindowsHookEx` and that hook propagates to our threads, if that DLL hasn't been compiled with `/cf:guard`. Notably, if a program calls it with `dwThreadId` to 0, that will install the hook on *all existing threads running in the same desktop as the calling thread* per the documentation. The developers might be doing that without realizing the impact -- in any case, we would need to figure out which DLLs are doing these calls, and block them. So far I have found that pattern being used in `evpComponentsXE.bpl` and I would thus recommend blocking it (more details about that investigation below). As for the link to bug 1998650, probably? Perhaps if Firefox 32-bit was previously loading an outdated-enough version of the runtime DLLs from the SysWOW64 directory, that might have resulted in Control Flow Guard getting somehow unenforced for our processes, and now that we load up-to-date DLLs the mitigation would be active again? This would deserve further investigation, but the crash volume does have the pattern of a new issue affecting old software, so likely linked to a change on our side, and that change is indeed a good candidate. Below are the details for the `evpComponentsXE.bpl` investigation. I've downloaded the crash dumps and looked for the unusual DLLs being the most present: they are the BPL files signed by `Idera, Inc.`, which are loaded from `C:\Program Files (x86)\Sage Evolution` and are present in 26 crashes (out of 31) from 8 unique users (based on `install_time`). But, there are a lot of different BPL files. Luckily I've found a zip file that contains those BPL files by following [this download procedure](https://za-kb.sage.com/portal/app/portlets/results/viewsolution.jsp?solutionid=230602122753277&page=1&position=0&q=230602122753277). The BPL files there are not signed with the same vendor name (maybe due to localization), but they are in the same version, so close enough for investigating. These are Delphi packages, and as suspected they don't have the CF Guard tables like our binaries do, this can be confirmed by comparing the output of `dumpbin /loadconfig` for `mozglue.dll` and e.g. `evpComponentsXE.bpl`. Then I created the dependency graph between these DLLs and realized that the ones that start with `evp` are among the most top-level DLLs. I compared that to the files that import `SetWindowsHookExW` and confirmed the use of a zero `dwThreadId` by `evpComponentsXE.bpl` -- the other `.bpl` are just its (long list of) dependencies.
These are not stack overflows but rather Control Flow Guard fast-fail failures, which use the same error code. The call stack is: ``` ntdll!RtlFailFast2 ntdll!RtlpHandleInvalidUserCallTarget+0x93 ntdll!RtlDispatchException+0x56c41 ntdll!KiUserExceptionDispatcher+0xf ntdll!LdrpValidateUserCallTargetBitMapCheck user32!DispatchHookW+0xcd user32!CallHookWithSEH+0x59 user32!__fnHkINDWORD+0x26 ntdll!KiUserCallbackDispatcher+0x36 win32u!NtUserPeekMessage+0xc user32!_PeekMessage+0xeb user32!PeekMessageW+0x1b1 xul!mozilla::widget::WinUtils::WaitForMessage+0x99 ``` I am very confident that this crash occurs when a DLL sets up a hook with `SetWindowsHookEx` and that hook propagates to our threads, if that DLL hasn't been compiled with `/cf:guard`. Notably, if a program calls it with `dwThreadId` to 0, that will install the hook on *all existing threads running in the same desktop as the calling thread* per the documentation. The developers might be doing that without realizing the impact -- in any case, we would need to figure out which DLLs are doing these calls, and block them. So far I have found that pattern being used in `evpComponentsXE.bpl` and I would thus recommend blocking it (more details about that investigation below). As for the link to bug 1998650, probably? Perhaps if Firefox 32-bit was previously loading an outdated-enough version of the runtime DLLs from the SysWOW64 directory, that might have resulted in Control Flow Guard getting somehow unenforced for our processes, and now that we load up-to-date DLLs the mitigation would be active again? This would deserve further investigation, but the crash volume does have the pattern of a new issue affecting old software, so likely linked to a change on our side, and that change is indeed a good candidate. Below are the details for the `evpComponentsXE.bpl` investigation. I've downloaded the crash dumps and looked for the unusual DLLs being the most present: they are the BPL files signed by `Idera, Inc.`, which are loaded from `C:\Program Files (x86)\Sage Evolution` and are present in 26 crashes (out of 31) from 8 unique users (based on `install_time`). But, there are a lot of different BPL files. Luckily I've found a zip file that contains those BPL files by following [this download procedure](https://za-kb.sage.com/portal/app/portlets/results/viewsolution.jsp?solutionid=230602122753277&page=1&position=0&q=230602122753277). The BPL files there are not signed with the same vendor name (maybe due to localization), but they are in the same version, so close enough for investigating. These are Delphi packages, and as suspected they don't have the CF Guard tables like our binaries do, this can be confirmed by comparing the output of `dumpbin /loadconfig` for `mozglue.dll` and e.g. `evpComponentsXE.bpl`. Then I created the dependency graph between these DLLs and realized that the ones that start with `evp` are among the most top-level DLLs in that regard. I compared that to the files that import `SetWindowsHookExW` and confirmed the use of a zero `dwThreadId` by `evpComponentsXE.bpl` -- the other `.bpl` are just its (long list of) dependencies.