1122745 - Build failure on toolchains without ssse3/sse4.1 (especially libvpx)

Status: NEW

(Core :: WebRTC, defect)

Description

[Landry Breuil (:gaston)]

Sibling bug splitted from bug #1028827, 35 builds fine, 36 fails to build on OpenBSD/i386 with gcc 4.8.3/ld 2.15 where the toolchain doesnt support ssse3/sse4.1.

So far i've tried:

- making -mssse3/-msse4.1 conditional to toolchain support (ie more or less revert https://hg.mozilla.org/integration/mozilla-inbound/rev/2013f538837c) :

-            if 'ssse3' in f:
+            if 'ssse3' in f and CONFIG['HAVE_TOOLCHAIN_SUPPORT_MSSSE3']:
                 SOURCES[f].flags += ['-mssse3']
-            if 'sse4' in f:
+            if 'sse4' in f and CONFIG['HAVE_TOOLCHAIN_SUPPORT_MSSE4_1']:
                 SOURCES[f].flags += ['-msse4.1']

That helps a bit since the flags arent passed, but all the files are built inconditionally, thus the build badly fails on quantize_sse4.c:

In file included from /usr/obj/ports/firefox-36.0beta1/mozilla-beta/media/libvpx/vp8/encoder/x86/quantize_sse4.c:12:0:
/usr/local/lib/gcc/i386-unknown-openbsd5.6/4.8.3/include/smmintrin.h:31:3: error: #error "SSE4.1 instruction set not enabled"
 # error "SSE4.1 instruction set not enabled"
   ^
/usr/obj/ports/firefox-36.0beta1/mozilla-beta/media/libvpx/vp8/encoder/x86/quantize_sse4.c: In function 'vp8_regular_quantize_b_sse4_1':
/usr/obj/ports/firefox-36.0beta1/mozilla-beta/media/libvpx/vp8/encoder/x86/quantize_sse4.c:36:5: error: unknown type name '__m128i'
     __m128i sz0, x0, sz1, x1, y0, y1, x_minus_zbin0, x_minus_zbin1,
     ^
/usr/obj/ports/firefox-36.0beta1/mozilla-beta/media/libvpx/vp8/encoder/x86/quantize_sse4.c:38:5: error: unknown type name '__m128i'
     __m128i quant_shift0 = _mm_load_si128((__m128i *)(b->quant_shift));

I *think* it was building fine in 35 because we had vp8/encoder/x86/quantize_sse4.asm while in beta we now have vp8/encoder/x86/quantize_sse4.c. Maybe that's the cause, or a major libvpx update in the meantime.

Next i tried switching HAVE_SSE4_1 / HAVE_SSSE3 to 0 in vpx_config_x86-linux-gcc.h/asm. Didnt help as-is.

Now i'm trying to sprinkle some #if HAVE_SSE4_1/#if HAVE_SSSE3 guards on all the files including *mmintrin* headers, but that looks wrong to me. I wonder if we shouldnt make those files build *depending* on toolchain support, like we try to do in skia (see bug 1028827 again, attachement 8512829)


Another workaround is to build using --with-system-libvpx but that's awkward.

Comment 1

[Landry Breuil (:gaston)]

That's really ugly, but if i sprinkle HAVE_SSE4_1/HAVE_SSSE3 checks on:
media/libvpx/vp8/encoder/x86/quantize_sse4.c
media/libvpx/vp8/encoder/x86/quantize_ssse3.c
media/libvpx/vp9/common/x86/vp9_idct_intrin_ssse3.c
media/libvpx/vp9/common/x86/vp9_subpixel_8t_intrin_ssse3.c

(with the config.h/asm files defining those macros to 0) the build goes past libvpx.

Comment 2

[Landry Breuil (:gaston)]

At some point is probably a regression from the last libvpx optim import in 1063356

Comment 3

[Landry Breuil (:gaston)]

Build goes further, but linking libxul of course fails:

../../media/libvpx/rtcd.o(.text.setup_rtcd_internal+0x28e): In function `setup_rtcd_internal':
: undefined reference to `vp8_fast_quantize_b_ssse3'
../../media/libvpx/rtcd.o(.text.setup_rtcd_internal+0x54c): In function `setup_rtcd_internal':
: undefined reference to `vp8_regular_quantize_b_sse4_1'
../../media/libvpx/rtcd.o(.text.setup_rtcd_internal+0x83b): In function `setup_rtcd_internal':
: undefined reference to `vp8_sixtap_predict16x16_ssse3'
../../media/libvpx/rtcd.o(.text.setup_rtcd_internal+0x85f): In function `setup_rtcd_internal':
: undefined reference to `vp8_sixtap_predict4x4_ssse3'
../../media/libvpx/rtcd.o(.text.setup_rtcd_internal+0x893): In function `setup_rtcd_internal':
: undefined reference to `vp8_sixtap_predict8x4_ssse3'
../../media/libvpx/rtcd.o(.text.setup_rtcd_internal+0x8c7): In function `setup_rtcd_internal':
: undefined reference to `vp8_sixtap_predict8x8_ssse3'
../../media/libvpx/vp9_rtcd.o(.text.setup_rtcd_internal+0x99): In function `setup_rtcd_internal':
: undefined reference to `vp9_convolve8_avg_ssse3'
../../media/libvpx/vp9_rtcd.o(.text.setup_rtcd_internal+0xbd): In function `setup_rtcd_internal':
: undefined reference to `vp9_convolve8_avg_horiz_ssse3'
../../media/libvpx/vp9_rtcd.o(.text.setup_rtcd_internal+0xe1): In function `setup_rtcd_internal':
: undefined reference to `vp9_convolve8_avg_vert_ssse3'
../../media/libvpx/vp9_rtcd.o(.text.setup_rtcd_internal+0x105): In function `setup_rtcd_internal':
: undefined reference to `vp9_convolve8_horiz_ssse3'
../../media/libvpx/vp9_rtcd.o(.text.setup_rtcd_internal+0x129): In function `setup_rtcd_internal':
: undefined reference to `vp9_convolve8_vert_ssse3'
../../media/libvpx/vp9_rtcd.o(.text.setup_rtcd_internal+0x218): In function `setup_rtcd_internal':
: undefined reference to `vp9_idct16x16_10_add_ssse3'
../../media/libvpx/vp9_rtcd.o(.text.setup_rtcd_internal+0x254): In function `setup_rtcd_internal':
: undefined reference to `vp9_idct16x16_256_add_ssse3'
../../media/libvpx/vp9_rtcd.o(.text.setup_rtcd_internal+0x5f4): In function `setup_rtcd_internal':
: undefined reference to `vp9_convolve8_ssse3'
/usr/bin/ld: libxul.so.53.0: hidden symbol `vp9_convolve8_horiz_ssse3' isn't defined

This *so* looks like previous failures.... sigh :(

Comment 4

[Landry Breuil (:gaston)]

Using clang "workarounds" the issue but it is insanely ugly.

Comment 5

[johannkoenig]

Chromium has an ugly bit of separating the files into bins so it can apply the correct flags to the C files with intrinsics:
https://chromium.googlesource.com/chromium/src/+/master/third_party/libvpx_new/generate_gypi.sh

Let me know if this helps or if you'd like a more detailed explanation.

Comment 6

[johannkoenig]

If you really want to support toolchains without ssse3/sse4.1, there is now a 'generic' target in moz.build:
https://dxr.mozilla.org/mozilla-central/source/media/libvpx/moz.build#67

It does not contain any assembly/intrinsics optimizations.

Alternatively you could create a *new* config in generate_sources_mozbuild.sh which sets --disable-ssse3 (which will also disable sse4.1/avx/avx2)
https://dxr.mozilla.org/mozilla-central/source/media/libvpx/generate_sources_mozbuild.sh#193

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

fyi, see comment 5

Comment 8

[Landry Breuil (:gaston)]

There's a 404 on the link in comment 5 - but i gave up on this aeons ago, and just build with clang which workarounds the issue. Life is too short for this.

Comment 9

[johannkoenig]

I happen to agree that using a compiler which doesn't support ssse3 is folly (but I'm very surprised gcc 4.8 does not support them). But if someone were interested, the links in comment 6 are a good starting point.

The link in comment 5 was not relevant. I misunderstood the original issue. For posterity, that file has moved to https://chromium.googlesource.com/chromium/src/+/master/third_party/libvpx/generate_gni.sh