688997 - Pointer truncation in waveOutProc callback (x64)

Status: RESOLVED FIXED

(Core :: Audio/Video, defect)

Description

[bugzilla.10.animefan]

User Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)

Steps to reproduce:

Trying to play the video at http://ww.mozilla.org/projects/firefox/prerelease.html crashes Firefox 9.0 x64 nightly on a system with more than 4 GB RAM and AllocationPreference set to 0x100000 in the registry.


Actual results:

Firefox 9.0 x64 crashed in

xul!waveOutProc+0x1f [e:\builds\moz2_slave\m-cen-w64-ntly\build\media\libsydneyaudio\src\sydney_audio_waveapi.c @ 639]

Firefox uses this signature for the callback:

void CALLBACK waveOutProc(
     HWAVEOUT hWaveOut, 
     UINT uMsg, 
     DWORD dwInstance,  
     DWORD dwParam1,    
     DWORD dwParam2     
)
This is not correct. According to MSDN this is the correct signature for this callback:
void CALLBACK waveOutProc(
  HWAVEOUT hwo,
  UINT uMsg,
  DWORD_PTR dwInstance,
  DWORD_PTR dwParam1,
  DWORD_PTR dwParam2
);

( reference : http://msdn.microsoft.com/en-us/library/dd743869(v=vs.85).aspx )

By assigning the dwInstance to a DWORD the pointer to the handle can get truncated on 64 bit systems.


Expected results:

Firefox should not crash.

Fix:
Use the correct calling convention for the callback.

Comment 1

[Matthew Gregan [:kinetik]]

Attachment: patch v0

Comment 2

[Matthew Gregan [:kinetik]]

http://hg.mozilla.org/integration/mozilla-inbound/rev/2cd423eb0392

Comment 3

[(no longer active)]

https://hg.mozilla.org/mozilla-central/rev/2cd423eb0392