dgapi on the blocklist is still loaded and causes crash
Categories
(Firefox :: Launcher Process, defect, P2)
Tracking
()
Tracking | Status | |
---|---|---|
firefox77 | --- | fixed |
People
(Reporter: toshi, Assigned: toshi)
References
(Blocks 1 open bug)
Details
Attachments
(8 files)
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review |
We have blocked dgapi/dgapi64 older than 7.5.1.0 as Bug 1318858, but it's still loaded and causes crash. In those crash dumps, there is no hook in ntdll!LdrLoadDll
or ntdll!NtMapViewOfSection
.
Assignee | ||
Comment 1•5 years ago
|
||
Analyzing the launcher failure ping (https://sql.telemetry.mozilla.org/queries/67099#169970), almost all of launcher failures with dgapi/dgapi64 were caused by the same reason as Bug 1587539, which is not in ESR68 or 71.
EarlgreyTea, would you ask the original reporter to try Beta or Nightly?
Comment 2•5 years ago
|
||
(In reply to Toshihito Kikuchi [:toshi] from comment #1)
EarlgreyTea, would you ask the original reporter to try Beta or Nightly?
sorry.
I immediately called the original reporter on the MozillaZine.jp topic, but I haven't received a response yet.
Comment 3•5 years ago
|
||
The original reporter has reported a crash in Beta.
Firefox 72.0b7:
https://crash-stats.mozilla.org/report/index/83f15c32-6cc5-435e-8b02-ec03d0191218
Firefox 72.0b10:
https://crash-stats.mozilla.org/report/index/1f5c0866-ddb0-435d-89db-9229e0191225
Also, the same situation as Bug 1603335 has occurred at the first startup.
Nightly 73.0a1 is in the same situation as Bug 1603335.
Supplementary information:
- dgapi64.dll Version: 7.5.0.0107 (checked from PowerShell)
- dgapi64.dll seems to be loaded in Firefox (checked with tasklist command)
Assignee | ||
Comment 4•5 years ago
|
||
(In reply to EarlgreyTea from comment #3)
The original reporter has reported a crash in Beta.
Firefox 72.0b7:
https://crash-stats.mozilla.org/report/index/83f15c32-6cc5-435e-8b02-ec03d0191218
Firefox 72.0b10:
https://crash-stats.mozilla.org/report/index/1f5c0866-ddb0-435d-89db-9229e0191225Also, the same situation as Bug 1603335 has occurred at the first startup.
Nightly 73.0a1 is in the same situation as Bug 1603335.
Supplementary information:
- dgapi64.dll Version: 7.5.0.0107 (checked from PowerShell)
- dgapi64.dll seems to be loaded in Firefox (checked with tasklist command)
Thank you for sharing these information. I think I got a good understanding on this issue and Bug 1603335.
Here's my theory.
Digital Guardian fixed the crash issue in 7.5.1, and they also changed an injection method between 7.5.0 and 7.5.1. On the other hand, we have two versions of blocklist which interacts with Digital Guardian as below.
New blocklist:
- can block 7.5.1 safely
- can block 7.5.0, but it breaks process launch (Bug 1603335)
Old blocklist:
- cannot block dgapi 7.5.0
Another factor to make the situation more complicated is we enable a different blocklist for a different process type.
Combining these factors together, the reason why dgapi 7.5.0 is still loaded in 72 Beta and crashes the process can be explained like this.
- At a first launch, we enable a new blocklist in the browser process which causes the 0xc0000022 popup.
- As a result of 1), launcher process is disabled.
- Next time Firefox is launched, we use an old blocklist in the browser process and no blocklist in the child process.
- All firefox processes start correctly, but crash due to the bug on dgapi side.
The story of Nightly is different.
- At a first launch, we use a new blocklist in the browser process which causes the 0xc0000022 popup.
- As a result of 1), launcher process is disabled.
- Next time Firefox is launched, we use an old blocklist in the browser process and a new blocklist in the child process.
- The browser process starts correctly, but the child process fails to start, resulting in the blank page problem (Bug 1603335).
This explains why we don't see any crash reports at GetMBNoDefault
from version 73.
Assignee | ||
Comment 5•5 years ago
|
||
The crash at GetMBNoDefault
is not firefox's problem, and as you may know, blocking dgapi is not a solution but a mitigation. Now that Digital Guardian fixed the crash in 7.5.1 (and probably in 7.4.2 as well), upgrading from 7.5.0 to 7.5.1 is always an ideal solution.
Because they have a kernel driver while we don't, what we can do is limited. Moreover, we don't know which injection method DG 7.5.0 uses. I was able to reproduce Bug 1603335 with my test kernel driver, so I'll try to find out what we can, but please note that we cannot commit to fix DG's crash.
Assignee | ||
Comment 6•5 years ago
|
||
I prototyped a new way to block dgapi not to cause 0xc0000022. Could you please install this and see it fixes both 0xc0000022 and dgapi's crash?
Please note that this is an experimental fix. This may not resolve the issue, or the approach may not be approved even if this works good, but we'd like to know a verification result to confirm we understand the issue correctly.
x64 installer (submitted job):
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/SuQkDlzVRhe0jPIHFIxulQ/runs/0/artifacts/public/build/install/sea/target.installer.exe
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 7•5 years ago
|
||
Sorry, I found this prototype was imperfect. I'll update when a next version is ready.
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 8•5 years ago
|
||
Finally a working prototype is ready. Hopefully someone can test this with the crashing Digital Guardian.
Here are the crash reports indicating this issue:
- https://crash-stats.mozilla.org/search/?signature=~GetMBNoDefault&date=%3E%3D2020-02-07T23%3A58%3A00.000Z&date=%3C2020-02-21T23%3A58%3A00.000Z&_facets=signature&_sort=-date&_columns=date&_columns=signature&_columns=version&_columns=build_id&_columns=platform_version#crash-reports
- https://iodide.telemetry.mozilla.org/notebooks/391/?viewMode=report
Assignee | ||
Comment 9•5 years ago
|
||
This patch introduces nt::VirtualQuery
which consumes only ntdll's functions
to reduce dependency in MMPolicy
on kernel32.dll. With this, MMPolicy
still
depends on kernel32.dll, that will be solved by a coming patch.
Updated•5 years ago
|
Assignee | ||
Comment 10•5 years ago
|
||
This patch moves the instantiation of PEHeaders
from CheckBlockInfo
to
IsDllAllowed
so that IsDllAllowed
can use an instance of PEHeaders
.
Depends on D68342
Assignee | ||
Comment 11•5 years ago
|
||
WindowsDllDetourPatcher::CreateTrampoline
does not only create a trampoline
region but also applies a patch on an original function. This patch extracts
the patching part as separate functions.
Depends on D68343
Assignee | ||
Comment 12•5 years ago
|
||
This patch introduces a new policy MMPolicyInProcessEarlyStage
which does
not consume any functions imported from kernel32.dll so that we can use it
in a process's early stage i.e. before IAT is resolved.
Depends on D68344
Assignee | ||
Comment 13•5 years ago
|
||
This patch introduces a new DLL interceptor WindowsDllEntryPointInterceptor
which applies a hook to a target function without backing up the original
function code.
Depends on D68345
Assignee | ||
Comment 14•5 years ago
|
||
This patch introduces Kernel32ExportsSolver
which calculates RVAs of
kernel32's functions and transfers them to a target process, where the
transferred RVAs are resolved into function addresses.
Depends on D68346
Assignee | ||
Comment 15•5 years ago
|
||
This patch introduces a new DLL blocklist type NeuterlistEntry
which neuters
a DLL, hooking the DLL's entrypoint into a no-op function. With this technique,
we give the injected DLL no chance to run its code though we allow it to be
loaded into the process.
This new blocklist type is intended to block a DLL which is injected by IAT
patching which was planted by a kernel callback routine for LoadImage. It's
because blocking such a DLL makes a new process fail to launch.
Depends on D68347
Assignee | ||
Comment 16•5 years ago
|
||
Depends on D68348
Assignee | ||
Comment 18•5 years ago
|
||
(In reply to Aaron Klotz [:aklotz] from comment #17)
Please ask Molly to review this.
She agreed to help with this. I'll update the revisions. Thanks!
Updated•5 years ago
|
Updated•5 years ago
|
Comment 19•5 years ago
|
||
Comment 20•5 years ago
|
||
Backed out 8 changesets (bug 1603974) for causing build bustage
[task 2020-04-07T15:19:41.402Z] 15:19:41 INFO - 1 error generated.
[task 2020-04-07T15:19:41.402Z] 15:19:41 INFO - /builds/worker/checkouts/gecko/config/rules.mk:750: recipe for target 'WindowsDllBlocklist.obj' failed
[task 2020-04-07T15:19:41.402Z] 15:19:41 ERROR - make[4]: *** [WindowsDllBlocklist.obj] Error 1
[task 2020-04-07T15:19:41.402Z] 15:19:41 INFO - make[4]: Leaving directory '/builds/worker/workspace/obj-build/mozglue/dllservices'
[task 2020-04-07T15:19:41.402Z] 15:19:41 INFO - make[4]: *** Waiting for unfinished jobs....
[task 2020-04-07T15:19:41.406Z] 15:19:41 INFO - make[4]: Entering directory '/builds/worker/workspace/obj-build/mozglue/misc'
[task 2020-04-07T15:19:41.406Z] 15:19:41 INFO - mozglue/misc/WindowsUnicode.obj
[task 2020-04-07T15:19:41.406Z] 15:19:41 INFO - make[4]: Leaving directory '/builds/worker/workspace/obj-build/mozglue/misc'
[task 2020-04-07T15:19:41.415Z] 15:19:41 INFO - make[4]: Entering directory '/builds/worker/workspace/obj-build/js/src'
[task 2020-04-07T15:19:41.416Z] 15:19:41 INFO - /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang-cl -Xclang -std=c++17 --target=aarch64-windows-msvc -FoUnified_cpp_js_src23.obj -c -guard:cf -DNDEBUG=1 -DTRIMMED=1 -DWASM_SUPPORTS_HUGE_MEMORY -DJS_CACHEIR_SPEW -DJS_STRUCTURED_SPEW -DJS_HAS_CTYPES -DFFI_BUILDING -DEXPORT_JS_API -DMOZ_HAS_MOZGLUE -I/builds/worker/checkouts/gecko/js/src -I/builds/worker/workspace/obj-build/js/src -I/builds/worker/workspace/obj-build/js/src/ctypes/libffi/include -I/builds/worker/checkouts/gecko/js/src/ctypes/libffi/src/aarch64 -I/builds/worker/workspace/obj-build/dist/include -I/builds/worker/workspace/obj-build/dist/include/nspr -MD -FI /builds/worker/workspace/obj-build/js/src/js-confdefs.h -DMOZILLA_CLIENT -Qunused-arguments -Qunused-arguments -fcrash-diagnostics-dir=/builds/worker/artifacts -TP -D_CRT_SECURE_NO_WARNINGS -Zc:sizedDealloc- -D_HAS_EXCEPTIONS=0 -W3 -Gy -Zc:inline -Gw -Wno-inline-new-delete -Wno-invalid-offsetof -Wno-microsoft-enum-value -Wno-microsoft-include -Wno-unknown-pragmas -Wno-ignored-pragmas -Wno-deprecated-declarations -Wno-invalid-noreturn -Wno-inconsistent-missing-override -Wno-implicit-exception-spec-mismatch -Wno-microsoft-exception-spec -Wno-unused-local-typedef -Wno-ignored-attributes -Wno-used-but-marked-unused -GR- -Z7 -Xclang -load -Xclang /builds/worker/workspace/obj-build/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -Oy- -Werror -Xclang -MP -Xclang -dependency-file -Xclang .deps/Unified_cpp_js_src23.obj.pp -Xclang -MT -Xclang Unified_cpp_js_src23.obj Unified_cpp_js_src23.cpp
[task 2020-04-07T15:19:41.416Z] 15:19:41 INFO - In file included from Unified_cpp_js_src23.cpp:29:
[task 2020-04-07T15:19:41.416Z] 15:19:41 INFO - /builds/worker/checkouts/gecko/js/src/vm/Printer.cpp(403,11): warning: Usage of ASCII file functions (here fopen) is forbidden on Windows.
[task 2020-04-07T15:19:41.416Z] 15:19:41 INFO - file_ = fopen(path, "w");
[task 2020-04-07T15:19:41.416Z] 15:19:41 INFO - ^
[task 2020-04-07T15:19:41.416Z] 15:19:41 INFO - /builds/worker/checkouts/gecko/js/src/vm/Printer.cpp(403,11): note: On Windows executed functions: fopen, fopen_s, open, _open, _sopen, _sopen_s, OpenFile, CreateFileA should never be used due to lossy conversion from UTF8 to ANSI.
[task 2020-04-07T15:19:41.416Z] 15:19:41 INFO - 1 warning generated.
[task 2020-04-07T15:19:41.416Z] 15:19:41 INFO - make[4]: Leaving directory '/builds/worker/workspace/obj-build/js/src'
[task 2020-04-07T15:19:41.416Z] 15:19:41 INFO - make[4]: Entering directory '/builds/worker/workspace/obj-build/js/src'
[task 2020-04-07T15:19:41.417Z] 15:19:41 INFO - js/src/Unified_cpp_js_src28.obj
[task 2020-04-07T15:19:41.417Z] 15:19:41 INFO - make[4]: Leaving directory '/builds/worker/workspace/obj-build/js/src'
[task 2020-04-07T15:19:41.644Z] 15:19:41 INFO - make[4]: Entering directory '/builds/worker/workspace/obj-build/mozglue/dllservices'
[task 2020-04-07T15:19:41.644Z] 15:19:41 INFO - /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang-cl -Xclang -std=c++17 --target=aarch64-windows-msvc -FoUnified_cpp_mozglue_dllservices0.obj -c -I/builds/worker/workspace/obj-build/dist/stl_wrappers -guard:cf -DNDEBUG=1 -DTRIMMED=1 -DIMPL_MFBT -DMOZ_HAS_MOZGLUE -I/builds/worker/checkouts/gecko/mozglue/dllservices -I/builds/worker/workspace/obj-build/mozglue/dllservices -I/builds/worker/workspace/obj-build/dist/include -I/builds/worker/workspace/obj-build/dist/include/nspr -I/builds/worker/workspace/obj-build/dist/include/nss -MD -FI /builds/worker/workspace/obj-build/mozilla-config.h -DMOZILLA_CLIENT -Qunused-arguments -Qunused-arguments -fcrash-diagnostics-dir=/builds/worker/artifacts -TP -Zc:sizedDealloc- -D_HAS_EXCEPTIONS=0 -W3 -Gy -Zc:inline -Gw -Wno-inline-new-delete -Wno-invalid-offsetof -Wno-microsoft-enum-value -Wno-microsoft-include -Wno-unknown-pragmas -Wno-ignored-pragmas -Wno-deprecated-declarations -Wno-invalid-noreturn -Wno-inconsistent-missing-override -Wno-implicit-exception-spec-mismatch -Wno-microsoft-exception-spec -Wno-unused-local-typedef -Wno-ignored-attributes -Wno-used-but-marked-unused -D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING -GR- -Z7 -Xclang -load -Xclang /builds/worker/workspace/obj-build/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -Oy- -Werror -Xclang -MP -Xclang -dependency-file -Xclang .deps/Unified_cpp_mozglue_dllservices0.obj.pp -Xclang -MT -Xclang Unified_cpp_mozglue_dllservices0.obj Unified_cpp_mozglue_dllservices0.cpp
[task 2020-04-07T15:19:41.645Z] 15:19:41 INFO - make[4]: Leaving directory '/builds/worker/workspace/obj-build/mozglue/dllservices'
[task 2020-04-07T15:19:41.645Z] 15:19:41 INFO - /builds/worker/checkouts/gecko/config/recurse.mk:74: recipe for target 'mozglue/dllservices/target-objects' failed
[task 2020-04-07T15:19:41.645Z] 15:19:41 ERROR - make[3]: *** [mozglue/dllservices/target-objects] Error 2
[task 2020-04-07T15:19:41.646Z] 15:19:41 INFO - make[3]: *** Waiting for unfinished jobs....
[task 2020-04-07T15:19:41.670Z] 15:19:41 INFO - make[4]: Entering directory '/builds/worker/workspace/obj-build/js/src'
[task 2020-04-07T15:19:41.671Z] 15:19:41 INFO - js/src/Unified_cpp_js_src3.obj
[task 2020-04-07T15:19:41.672Z] 15:19:41 INFO - make[4]: Leaving directory '/builds/worker/workspace/obj-build/js/src'
[task 2020-04-07T15:19:41.672Z] 15:19:41 INFO - make[4]: Entering directory '/builds/worker/workspace/obj-build/mfbt/tests'
[task 2020-04-07T15:19:41.672Z] 15:19:41 INFO - /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang-cl -Xclang -std=c++17 --target=aarch64-windows-msvc -FoTestWrappingOperations.obj -c -guard:cf -DNDEBUG=1 -DTRIMMED=1 -DIMPL_MFBT -I/builds/worker/checkouts/gecko/mfbt/tests -I/builds/worker/workspace/obj-build/mfbt/tests -I/builds/worker/workspace/obj-build/dist/include -I/builds/worker/workspace/obj-build/dist/include/testing -I/builds/worker/workspace/obj-build/dist/include/nspr -I/builds/worker/workspace/obj-build/dist/include/nss -MD -FI /builds/worker/workspace/obj-build/mozilla-config.h -DMOZILLA_CLIENT -Qunused-arguments -Qunused-arguments -fcrash-diagnostics-dir=/builds/worker/artifacts -TP -Zc:sizedDealloc- -D_HAS_EXCEPTIONS=0 -W3 -Gy -Zc:inline -Gw -Wno-inline-new-delete -Wno-invalid-offsetof -Wno-microsoft-enum-value -Wno-microsoft-include -Wno-unknown-pragmas -Wno-ignored-pragmas -Wno-deprecated-declarations -Wno-invalid-noreturn -Wno-inconsistent-missing-override -Wno-implicit-exception-spec-mismatch -Wno-microsoft-exception-spec -Wno-unused-local-typedef -Wno-ignored-attributes -Wno-used-but-marked-unused -D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING -GR- -Z7 -Xclang -load -Xclang /builds/worker/workspace/obj-build/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -Oy- -Werror -wd4275 -wd4530 -Xclang -MP -Xclang -dependency-file -Xclang .deps/TestWrappingOperations.obj.pp -Xclang -MT -Xclang TestWrappingOperations.obj -FdTestWrappingOperations.pdb /builds/worker/checkouts/gecko/mfbt/tests/TestWrappingOperations.cpp
[task 2020-04-07T15:19:41.672Z] 15:19:41 INFO - make[4]: Leaving directory '/builds/worker/workspace/obj-build/mfbt/tests'
[task 2020-04-07T15:19:41.680Z] 15:19:41 INFO - make[4]: Entering directory '/builds/worker/workspace/obj-build/mfbt/tests'
[task 2020-04-07T15:19:41.680Z] 15:19:41 INFO - /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang-cl -Xclang -std=c++17 --target=aarch64-windows-msvc -FoTestWeakPtr.obj -c -guard:cf -DNDEBUG=1 -DTRIMMED=1 -DIMPL_MFBT -I/builds/worker/checkouts/gecko/mfbt/tests -I/builds/worker/workspace/obj-build/mfbt/tests -I/builds/worker/workspace/obj-build/dist/include -I/builds/worker/workspace/obj-build/dist/include/testing -I/builds/worker/workspace/obj-build/dist/include/nspr -I/builds/worker/workspace/obj-build/dist/include/nss -MD -FI /builds/worker/workspace/obj-build/mozilla-config.h -DMOZILLA_CLIENT -Qunused-arguments -Qunused-arguments -fcrash-diagnostics-dir=/builds/worker/artifacts -TP -Zc:sizedDealloc- -D_HAS_EXCEPTIONS=0 -W3 -Gy -Zc:inline -Gw -Wno-inline-new-delete -Wno-invalid-offsetof -Wno-microsoft-enum-value -Wno-microsoft-include -Wno-unknown-pragmas -Wno-ignored-pragmas -Wno-deprecated-declarations -Wno-invalid-noreturn -Wno-inconsistent-missing-override -Wno-implicit-exception-spec-mismatch -Wno-microsoft-exception-spec -Wno-unused-local-typedef -Wno-ignored-attributes -Wno-used-but-marked-unused -D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING -GR- -Z7 -Xclang -load -Xclang /builds/worker/workspace/obj-build/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -Oy- -Werror -wd4275 -wd4530 -Xclang -MP -Xclang -dependency-file -Xclang .deps/TestWeakPtr.obj.pp -Xclang -MT -Xclang TestWeakPtr.obj -FdTestWeakPtr.pdb /builds/worker/checkouts/gecko/mfbt/tests/TestWeakPtr.cpp
[task 2020-04-07T15:19:41.680Z] 15:19:41 INFO - make[4]: Leaving directory '/builds/worker/workspace/obj-build/mfbt/tests'
[task 2020-04-07T15:19:41.680Z] 15:19:41 INFO - make[4]: Entering directory '/builds/worker/workspace/obj-build/js/src'
Updated•5 years ago
|
Assignee | ||
Comment 21•5 years ago
|
||
Updated the "Part 3" patch to address Win AArch64 build bustage.
Comment 22•5 years ago
|
||
Comment 23•5 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/b609f60a8bd4
https://hg.mozilla.org/mozilla-central/rev/4fef3c3838ea
https://hg.mozilla.org/mozilla-central/rev/bc3b3c2db7f5
https://hg.mozilla.org/mozilla-central/rev/eafbf6ca23d6
https://hg.mozilla.org/mozilla-central/rev/94b98b6632c6
https://hg.mozilla.org/mozilla-central/rev/9d1ff38ff959
https://hg.mozilla.org/mozilla-central/rev/fac73bafaf21
https://hg.mozilla.org/mozilla-central/rev/daecd7ae1e24
Description
•