1615752 - TestDllInterceptor.exe fails to hook CreateFileA and DuplicateHandle on Windows 8 x64

Status: VERIFIED FIXED

(Core :: mozglue, defect, P1)

Description

[David Parks [:handyman]]

This was originally seen as bug 1598650 but that bug ended up resolving related issues in the DLL interceptor. This bug is specifically about Windows 8.0 x64 (with all of the updates installed) failing to hook CreateFileA. The output:

TEST-UNEXPECTED-FAIL | WindowsDllInterceptor | Failed to hook CreateFileA from k
ernel32.dll
First 13 bytes of function:
FF 25 D6 BE 11 00 90 90 90 90 90 90 48

Unfortunately, 13 bytes isn't enough to see all of what's going on but it disassembles to:

0x0000000000000000: FF 25 D6 BE 11 00 jmp qword ptr [rip + 0x11bed6]
0x0000000000000006: 90 nop
0x0000000000000007: 90 nop
0x0000000000000008: 90 nop
0x0000000000000009: 90 nop
0x000000000000000a: 90 nop
0x000000000000000b: 90 nop

with the last byte (0x48) as the REX byte for another instruction. Unless that instruction turns out to be something we can move with the trampoline, this may be another case that we need to resolve with a 10-byte trampoline.

Comment 1

[David Parks [:handyman]]

We need to use 10-byte patches for these. This is the disassembly of the first 20 bytes these functions:

---

CreateFileA:
0x0000000000000000: FF 25 D6 BE 11 00 jmp qword ptr [rip + 0x11bed6]
0x0000000000000006: 90 nop
0x0000000000000007: 90 nop
0x0000000000000008: 90 nop
0x0000000000000009: 90 nop
0x000000000000000a: 90 nop
0x000000000000000b: 90 nop
0x000000000000000c: 48 FF 25 29 B9 11 00 jmp qword ptr [rip + 0x11b929]
0x0000000000000013: 90 nop

DuplicateHandle:
0x0000000000000000: FF 25 B2 BB 11 00 jmp qword ptr [rip + 0x11bbb2]
0x0000000000000006: 90 nop
0x0000000000000007: 90 nop
0x0000000000000008: 90 nop
0x0000000000000009: 90 nop
0x000000000000000a: 90 nop
0x000000000000000b: 90 nop
0x000000000000000c: 48 FF 25 65 B8 11 00 jmp qword ptr [rip + 0x11b865]
0x0000000000000013: 90 nop

---

We have to assume that the second jmp in each function, which starts at byte #13, is a target of some other jmp (or else they would be useless) so we don't want to overwrite them. Hence, 10-byte patches instead of the usual 13.

Comment 2

[David Parks [:handyman]]

Attachment: Bug 1615752: Require 10-byte detour for Win 8.0 x64 CreateFileA and DuplicateHandle

In the current Win 8.0, these functions both start with a RIP-relative JMP (6 bytes) followed by 6 nops (6-bytes), which does not give us the 13-bytes we need for a trampoline so we require the trampoline to fit into 10 bytes.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dc744340e336
Require 10-byte detour for Win 8.0 x64 CreateFileA and DuplicateHandle r=aklotz

Comment 4

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/dc744340e336

Comment 5

[Bogdan Maris, Desktop Test Engineering]

Trying to verify this since I also encountered the fail in older builds but running the tests on a Windows 8 64bit (VM) from 75.0b10 I still get this fail.
TEST-UNEXPECTED-FAIL : WindowsDllInterceptor : Failed to hook CreateFileA from kernel32.dll
First 13 bytes of function:
FF 25 D6 CE 11 00 90 90 90 90 90 90 46

Though using the tests from Latest Nightly 76.0a1 I am not seeing that fail.

I have to note that we tested on Windows 8 64bit installed on a proper machine and we did not get the above fail using tests from 75.0b10.
David is this something we should worry about?

Comment 6

[David Parks [:handyman]]

I don't know what could be causing the VM version to fail with the 75 beta but not the 76 alpha. I think this is small enough that we should let the apparent 76 fix continue as is. If it breaks down at some point then we can spend some time figuring out why but this will be difficult to track down so I'd prefer not to try unless it's really necessary.

Comment 7

[Bogdan Maris, Desktop Test Engineering]

Keeping in mind that this did not reproduce anymore using a Win8 installed on a proper machine I'll mark this as verified on Fx75. But will keep an eye on what happens with Fx 76 further on. Thanks David.