1854795 - NSS bug: Update AVX2 detection method

Status: RESOLVED FIXED

(NSS :: Libraries, defect, P2)

Description

[longjmp]

Steps to reproduce:

Tried to open website https://9p.sdf.org/about.html with firefox

Firefox version: Mozilla Firefox 117.0.1
nss package version: 3.93
Distro: Arch Linux
Kernel: 6.5.4-hardened1-1-hardened

Actual results:

The process crashed with a SIGILL
Tried to executed the instruction "vmovd %r9d,%xmm0"

Code in NSS (freebl) that tried to use an AVX2 instruction, even tho my configuration didn't allow that.
https://hg.mozilla.org/projects/nss/file/tip/lib/freebl/chacha20poly1305.c#l433

I'm using a hardened version of the linux kernel, recently a new mitigation was added to address some the Gather Data Sampling CPU bug. That mitigation turned off vector instructions (AVX, AVX2).
https://kernel.org/doc/html/next/admin-guide/hw-vuln/gather_data_sampling.html
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=553a5c03e90a6087e88f8ff878335ef0621536fb

The current way of checking if AVX2 is supported is to use the cpuid instruction
https://hg.mozilla.org/projects/nss/file/tip/lib/freebl/blinit.c#l121
https://hg.mozilla.org/projects/nss/file/tip/lib/freebl/mpi/mpcpucache.c#l42

According to the kernel documentation:
"If used, these options will disable AVX use by turning off XSAVE YMM support. However, the processor will still enumerate AVX support. Userspace that does not follow proper AVX enumeration to check both AVX and XSAVE YMM support will break."

My current workaround is to disable these features manually using the NSS_DISABLE_AVX2 and NSS_DISABLE_AVX environment variables.

In the Arch Linux forums I was directed to this bug report, which discusses a similar bug in another application.
https://bugs.archlinux.org/task/79444#comment221291

(Warning, I'm a little bit familiar with AVX instructions, but I'm not an expert)

Expected results:

Not use the avx2 instructions.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:beurdouche, could you have a look please?

For more information, please visit BugBot documentation.

Comment 3

[John Schanck [:jschanck]]

Attachment: Bug 1854795 - add OSXSAVE and XCR0 tests to AVX2 detection. r=nkulatova

Comment 4

[John Schanck [:jschanck]]

Could you test whether the attached patch fixes the issue?

Comment 5

[John Schanck [:jschanck]]

https://hg.mozilla.org/projects/nss/rev/d6cd135df187c6b1ec669e2c53b0cc43869f507a

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

Copying crash signatures from duplicate bugs.