Last Comment Bug 688997 - Pointer truncation in waveOutProc callback (x64)
: Pointer truncation in waveOutProc callback (x64)
Status: RESOLVED FIXED
: crash
Product: Core
Classification: Components
Component: Audio/Video (show other bugs)
: Trunk
: x86_64 Windows 7
: -- normal (vote)
: mozilla9
Assigned To: Matthew Gregan [:kinetik]
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-09-24 13:20 PDT by bugzilla.10.animefan
Modified: 2011-09-26 07:44 PDT (History)
3 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
patch v0 (2.02 KB, patch)
2011-09-24 17:47 PDT, Matthew Gregan [:kinetik]
cpearce: review+
Details | Diff | Review

Description bugzilla.10.animefan 2011-09-24 13:20:14 PDT
User Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)

Steps to reproduce:

Trying to play the video at http://ww.mozilla.org/projects/firefox/prerelease.html crashes Firefox 9.0 x64 nightly on a system with more than 4 GB RAM and AllocationPreference set to 0x100000 in the registry.


Actual results:

Firefox 9.0 x64 crashed in

xul!waveOutProc+0x1f [e:\builds\moz2_slave\m-cen-w64-ntly\build\media\libsydneyaudio\src\sydney_audio_waveapi.c @ 639]

Firefox uses this signature for the callback:

void CALLBACK waveOutProc(
     HWAVEOUT hWaveOut, 
     UINT uMsg, 
     DWORD dwInstance,  
     DWORD dwParam1,    
     DWORD dwParam2     
)
This is not correct. According to MSDN this is the correct signature for this callback:
void CALLBACK waveOutProc(
  HWAVEOUT hwo,
  UINT uMsg,
  DWORD_PTR dwInstance,
  DWORD_PTR dwParam1,
  DWORD_PTR dwParam2
);

( reference : http://msdn.microsoft.com/en-us/library/dd743869(v=vs.85).aspx )

By assigning the dwInstance to a DWORD the pointer to the handle can get truncated on 64 bit systems.


Expected results:

Firefox should not crash.

Fix:
Use the correct calling convention for the callback.
Comment 1 Matthew Gregan [:kinetik] 2011-09-24 17:47:18 PDT
Created attachment 562269 [details] [diff] [review]
patch v0
Comment 2 Matthew Gregan [:kinetik] 2011-09-25 18:53:28 PDT
http://hg.mozilla.org/integration/mozilla-inbound/rev/2cd423eb0392
Comment 3 :Ehsan Akhgari (out sick) 2011-09-26 07:44:48 PDT
https://hg.mozilla.org/mozilla-central/rev/2cd423eb0392

Note You need to log in before you can comment on or make changes to this bug.