The default bug view has changed. See this FAQ.

Pointer truncation in waveOutProc callback (x64)

RESOLVED FIXED in mozilla9

Status

()

Core
Audio/Video
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: bugzilla.10.animefan, Assigned: kinetik)

Tracking

({crash})

Trunk
mozilla9
x86_64
Windows 7
crash
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

6 years ago
User Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)

Steps to reproduce:

Trying to play the video at http://ww.mozilla.org/projects/firefox/prerelease.html crashes Firefox 9.0 x64 nightly on a system with more than 4 GB RAM and AllocationPreference set to 0x100000 in the registry.


Actual results:

Firefox 9.0 x64 crashed in

xul!waveOutProc+0x1f [e:\builds\moz2_slave\m-cen-w64-ntly\build\media\libsydneyaudio\src\sydney_audio_waveapi.c @ 639]

Firefox uses this signature for the callback:

void CALLBACK waveOutProc(
     HWAVEOUT hWaveOut, 
     UINT uMsg, 
     DWORD dwInstance,  
     DWORD dwParam1,    
     DWORD dwParam2     
)
This is not correct. According to MSDN this is the correct signature for this callback:
void CALLBACK waveOutProc(
  HWAVEOUT hwo,
  UINT uMsg,
  DWORD_PTR dwInstance,
  DWORD_PTR dwParam1,
  DWORD_PTR dwParam2
);

( reference : http://msdn.microsoft.com/en-us/library/dd743869(v=vs.85).aspx )

By assigning the dwInstance to a DWORD the pointer to the handle can get truncated on 64 bit systems.


Expected results:

Firefox should not crash.

Fix:
Use the correct calling convention for the callback.

Updated

6 years ago
Component: General → Video/Audio
Keywords: crash
Product: Firefox → Core
QA Contact: general → video.audio
Version: unspecified → Trunk
(Assignee)

Comment 1

6 years ago
Created attachment 562269 [details] [diff] [review]
patch v0
Assignee: nobody → kinetik
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Attachment #562269 - Flags: review?(chris)
Attachment #562269 - Flags: review?(chris) → review+
(Assignee)

Comment 2

6 years ago
http://hg.mozilla.org/integration/mozilla-inbound/rev/2cd423eb0392
Target Milestone: --- → mozilla9
https://hg.mozilla.org/mozilla-central/rev/2cd423eb0392
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.