1321378 - freebl's intel-gcm-wrap.c fails to compile with clang-cl

Status: RESOLVED FIXED

(NSS :: Build, defect)

Description

[Nathan Froyd [:froydnj]]

clang-cl complains thusly:

82:13.98 c:/m-c/security/nss/lib/freebl/intel-gcm-wrap.c(125,43):  error: always_inline function '_mm_shuffle_epi8' requires target feature 'ssse3', but would be inlined into function 'intel_AES_GCM_CreateContext' that is compiled without support for 'ssse3'
82:13.98     _mm_storeu_si128((__m128i *)gcm->CTR, _mm_shuffle_epi8(_mm_add_epi32(ONE, _mm_shuffle_epi8(_mm_loadu_si128((__m128i *)gcm->CTR), BSWAP_MASK)), BSWAP_MASK));
82:13.98                                           ^
82:13.98 c:/m-c/security/nss/lib/freebl/intel-gcm-wrap.c(125,79):  error: always_inline function '_mm_shuffle_epi8' requires target feature 'ssse3', but would be inlined into function 'intel_AES_GCM_CreateContext' that is compiled without support for 'ssse3'
82:13.98     _mm_storeu_si128((__m128i *)gcm->CTR, _mm_shuffle_epi8(_mm_add_epi32(ONE, _mm_shuffle_epi8(_mm_loadu_si128((__m128i *)gcm->CTR), BSWAP_MASK)), BSWAP_MASK));
82:13.98                                                                               ^

This error happens because MSVC enables support for all relevant instruction sets, whereas clang-cl really wants you to pass the individual -m options appropriate to the instruction set that you're using.

There's a TODO about this in freebl.gyp.  I think I understand what to fix, but I'll need help landing the NSS side of the fix.

Comment 1

[Nathan Froyd [:froydnj]]

Attachment: clang-cl.patch

This patch works for me with a clang-cl build when transplanting it into mozilla-central; we already build a small library for intel-gcm-wrap.c to pass along -mssse3 and this patch simply reuses that library if we're on Windows and using clang-cl.  I have not tried an actual NSS standalone build, though I suppose it wouldn't work very well since check_cc_clang.py always prints 0 on Windows:

https://hg.mozilla.org/projects/nss/file/tip/coreconf/check_cc_clang.py#l8

I haven't tried with MSVC, either, though I suspect things should Just Work.

The NSS makefiles have a separate variable for the clang-cl-ness of the compiler, which I did not attempt to replicate here.

Flagging Franziskus for review as the reviewer of the great NSS gyp build rewrite.

Comment 2

[Franziskus Kiefer [:franziskus]]

This looks good to me but I have to fix the NSS build on Windows before I can test this properly.

Comment 3

[Nathan Froyd [:froydnj]]

Attachment: pass -msss3 when building intel-gcm-wrap.c with clang-cl

Removed tabs and whatnot.

Comment 4

[Franziskus Kiefer [:franziskus]]

Comment on attachment 8817483 [details] [diff] [review]
pass -msss3 when building intel-gcm-wrap.c with clang-cl

Review of attachment 8817483 [details] [diff] [review]:
-----------------------------------------------------------------

Windows gyp builds need to be fixed (I'll put a patch in bug 1315231). But this patch works fine for msvc. I'll check clang-cl later.

Comment 5

[Franziskus Kiefer [:franziskus]]

https://hg.mozilla.org/projects/nss/rev/7dac1f61ea4b9aa57ad675800f70b2a7069c920e