1018477 - Fix R/M check in x86 detour code

Status: RESOLVED FIXED

(Core :: General, defect)

Description

[(Away)]

I tried to hook a function containing this instruction:

8b7c240c        mov     edi,dword ptr [esp+0Ch]

CreateTrampoline thought that this was a 3-byte instruction, so then it failed to understand the next byte 0x0C. 

The code was trying to avoid REG 5 (with the 0x38 mask) but it should actually avoid R/M 5 (using 0x07 mask). I fixed that and added the case for R/M==5 as well.

I used this reference: http://www.c-jump.com/CIS77/CPU/x86/lecture.html#X77_0090_addressing_modes

Comment 1

[(Away)]

Attachment: Fix R/M check in x86 detour code

Comment 2

[(Away)]

> The code was trying to avoid REG 5 (with the 0x38 mask) but it should
> actually avoid R/M 5 (using 0x07 mask). I fixed that and added the case for
> R/M==5 as well.

Typo: The special case for SIB is R/M==4.

Comment 3

[(Away)]

remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/8c3664b5e1a2

Comment 4

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/8c3664b5e1a2