569812 - VP8 asm may conflict with win64 ABI when SSE is enabled.

Status: RESOLVED FIXED

(Core :: Audio/Video, defect)

Description

[Timothy B. Terriberry (:derf)]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.11) Gecko/20071216 SeaMonkey/1.1.7
Build Identifier: 

The win64 ABI requires xmm6...xmm15 to be callee-saved: http://msdn.microsoft.com/en-us/library/9z1stfyw(v=VS.90).aspx

MSVC stores temporary floating-point values in these registers, and expects them to survive function calls. VP8's SSE2 asm optimizations currently clobber these registers. This causes at least one problem in the 0.9 release of libvpx, which was fixed by commit c15652bce127b3967ec34a95b0cfaa366daddfb3:
http://review.webmproject.org/gitweb?p=libvpx.git;a=commit;h=c15652bce127b3967ec34a95b0cfaa366daddfb3

However, that commit just moves a local floating-point value used internally by VP8 to after the offending asm routine. It still does not preserve the registers as required by the ABI, and thus may cause breakage in application code that calls libvpx.

ASM is currently not enabled for win64 builds, so this does not cause any known breakage. This is simply a tracking bug to ensure this issue does not get forgotten.

Reproducible: Didn't try

Comment 1

[cajbir (:cajbir)]

Bug 570882 appears to have enabled the ASM on Win64 and landed on trunk. Is this an issue?

Comment 2

[Makoto Kato [:m_kato]]

I cannot meet crash even if turned on SSE2.

But XMM6 and XMM7 is non-valtile.  So I create a patch for upstream code.
https://review.webmproject.org/#change,130

Comment 3

[Makoto Kato [:m_kato]]

fixed by libvpx 0.9.1

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Sounds like we're not planning to block the release on Win64 issues.  However, it sounds like this one is well on its way to being fixed anyway.

Comment 5

[Timothy B. Terriberry (:derf)]

Fixed by bug 593753.