1596565 - Investigate JIT workarounds for Intel JCC Erratum performance degradation

Status: RESOLVED WONTFIX

(Core :: JavaScript Engine: JIT, task, P3)

Description

[Ted Campbell [:tcampbell]]

Intel has patched security concerns around conditional jump instructions by issuing a microcode update. There are reports that this may have measurable performance impacts to general software. In particular we should investigate if JIT mitigations such as inserting padding can reduce the impact to Firefox.

Related Links:
https://www.phoronix.com/scan.php?page=article&item=jcc-firefox-chrome&num=1
https://www.phoronix.com/scan.php?page=news_item&px=GNU-Assembler-Patches-JCC
https://reviews.llvm.org/D70157
https://sourceware.org/ml/binutils/2019-11/msg00173.html

Comment 1

[Ted Campbell [:tcampbell]]

We would need to modify the ExecutableAllocatorAlignment in [1] to be at least 32-bytes. We could then pad appropriate instructions in the MacroAssembler.

[1] https://searchfox.org/mozilla-central/rev/e7c61f4a68b974d5fecd216dc7407b631a24eb8f/js/src/jit/Linker.cpp#25

Comment 2

[Ted Campbell [:tcampbell]]

This document is also useful: https://www.intel.com/content/dam/support/us/en/documents/processors/mitigations-jump-conditional-code-erratum.pdf

Section 2.0 explains that this applies to both crossing the cache line or ending at the end of a cacheline.

Comment 3

[Jan de Mooij [:jandem]]

I benchmarked the 2019-09-18 and 2019-11-15 microcode versions on my (personal) laptop with an i7-8750H CPU.

How to update microcode on Ubuntu:

• Download tar.gz files from https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/releases and extract
• Do the following as root:

# cp -rf the-microcode/intel-ucode /lib/firmware
# echo 1 > /sys/devices/system/cpu/microcode/reload
# update-initramfs -u
# reboot now

Then after reboot do grep microcode /proc/cpuinfo to check the version. On my machine 0xb4 for old microcode, 0xca for the new one.

I'm mostly interested in workloads that spend a lot of time in JIT/Wasm code, to get an upper bound for potential speedups. I ran embenchen 4 times and calculated the min and average values, and then the fraction of those:

test	avg fraction	min fraction
box2d	100.85%	100.94%
bullet	103.36%	103.52%
conditionals	101.11%	100.75%
copy	99.67%	99.43%
corrections	99.90%	100.00%
fannkuch	107.77%	102.84%
fasta	101.17%	100.57%
fib	102.66%	102.93%
ifs	99.49%	98.96%
binarytrees	101.73%	101.36%
memops	158.09%	115.57%
primes	101.96%	99.35%
raybench	100.77%	100.47%
rust-fannkuch	111.74%	111.28%
skinning	103.42%	105.87%
zlib	101.02%	102.63%

Note that rust-fannkuch in particular regresses a lot. It goes from ~2625 to ~2930 on that machine.

On Kraken I see a 7-9% regression.

Comment 4

[Jan de Mooij [:jandem]]

The next step is to prototype the MacroAssembler padding for {jmp, call, ret} and collect data on code size and perf with/without that patch.

The most annoying part is that macro-fused op-jcc is also affected. That will likely complicate the patch and result in more bloat (certain cmp + jcc ops can be pretty big).

Comment 5

[Jan de Mooij [:jandem]]

I looked into this and we decided to hold off on this for now.

If I align all JIT code to 32 bytes and ensure all unconditional jump/call/ret instructions don't cross (or end on) a 32-byte boundary I don't see any improvements (a small regression, if anything). Then I did the same for JCC (conditional jumps) and for cmp/test (here I pretend they're 12-byte instructions to hopefully enable macro-op fusion). With this I see an improvement on Kraken, 875 => 839 ms but it doesn't close the gap (we got <= 810 ms before). On rust-fannkuch I get 2922 => 2847 and before we got ~2650.

Furthermore, branches are so common that it will likely bloat generated code quite a bit. For JIT code we'll also waste space from 32-byte alignment. This could backfire in practice on non-microbenchmarks (using more memory => wasting instruction cache memory).

Finally, implementing this well is hard with our backend: because of REX prefixes and variable size encodings it's often hard to know the instruction size beforehand so either we need to get that right or we'll add more padding than necessary. Handling cmp/test/add/sub + jcc efficiently is even worse: these are fused into a single op so the whole thing (could be 10 bytes at least?) can't cross or end on a 32-byte boundary...

Comment 6

[Mike Hommey [:glandium]]

Intel has patched security concerns around conditional jump instructions

I'm curious, where did you see it explicitly said this was about security concerns? All I saw described as being fixed so far was "erratic behavior".

Comment 7

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] (high latency) from comment #6)

I'm curious, where did you see it explicitly said this was about security concerns? All I saw described as being fixed so far was "erratic behavior".

The word was actually "unpredictable", not "erratic".

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

From the compiler toolchain bug 1596844, froyd wrote:
The discussions on the relevant patches (which aren't even committed yet!) do not sound super enthusiastic about turning those workarounds on by default, as they can cause slowdowns on non-affected machines.

So the answer may end up being "those processors just got slower", and we can't compensate without hurting others. (Plus the concerns above)

Comment 9

[Matthew Gaudet (he/him) [:mgaudet]]

I don't see us doing anything here.