1316415 - Some Windows methods hooked in nsWindowsDllInterceptor fail

Status: NEW

(Core :: XPCOM, defect, P3)

Description

[David Parks [:handyman]]

As part of bug 1284897, I've found a few x86 instructions that we needed to add to the nsWindowsDllInterceptor for some of the methods that we hook.  This was not previously known because the methods were not listed in the TestDllInterceptor, which is what we use to test that the OS's DLLs (which are sometimes updated without warning) can be hooked on all versions we support.  I've added the missing methods to the test and added some missing opcodes but a few failing cases remain:

* ImmReleaseContext is commented out in TestDllInterceptor because the Win7 and WinXP 32-bit fail on try runs due to a RET opcode that we don't support.  Note that the method is effectively a no-op on these OSs -- it sets a return value in EAX and then RETurns.  So the RET opcode may be hard to support generically but the method itself would be trivial to duplicate.

* GetKeyState is #defined to only test on 64-bit but is hooked on 32-bit as well.  I'm not sure if changing this would cause a failure.  I don't know why this was restricted to 64-bit to begin with.

* CreateFileW is the flip of GetKeyState -- it is only tested on 32-bit but is used on both 32 and 64-bit.  Again, I don't know if this would pass if tested on 64-bit, nor is it clear why it is set up this way.

Comment 1

[(No longer employed by Mozilla) Aaron Klotz]

(In reply to David Parks [:handyman] from comment #0)
> 
> * CreateFileW is the flip of GetKeyState -- it is only tested on 32-bit but
> is used on both 32 and 64-bit.  Again, I don't know if this would pass if
> tested on 64-bit, nor is it clear why it is set up this way.

We already hook NtCreateFile elsewhere, so I think that in this case it might just be better to replace that CreateFileW hook with the former.