1833294 - Differential testing generates different hash values when target is Error()

Status: UNCONFIRMED

(Core :: JavaScript Engine, defect, P3)

Description

[2628388509]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36

Steps to reproduce:

=============test.js==========
for (let v0 = 0; v0 < 100; v0++) {
async function* f1() {
const v10 = new Error();
fuzzilli_hash(v10);
}
const v14 = f1();
v14.next(v14, f1, v14, f1);
}

$ ./js --no-threads --differential-testing --blinterp-warmup-threshold=0 --baseline-warmup-threshold=0 --trial-inlining-warmup-threshold=0 --ion-warmup-threshold=3 test.js
output: executionHash is 0x60b2eb6f with 100 inputs

$ ./js --no-threads --differential-testing --blinterp-warmup-threshold=50 --baseline-warmup-threshold=60 --trial-inlining-warmup-threshold=70 --ion-warmup-threshold=80 test.js
output: executionHash is 0xfe24e42e with 100 inputs

SpiderMonkey should be compiled with --enable-js-fuzzilli to enable fuzzilli_hash() in js.cpp.

Actual results:

Differential testing generates different hash values when target is Error()

Expected results:

Differential testing should generate the same hash value

Comment 1

[Nicolas B. Pierron [:nbp]]

I am not sure if this is a valid point, as Error are not defined in the spec and are simply a best effort in term of JS engine implementation.
Having a differential behavior in the error reporting based on the JIT tier which is used is unfortunate.

Arai / Decoder, what do you think? Would it make sense to filter out such non-determinism from js::FuzzilliHashObject, or should we fix the source of non-determinism?

Comment 2

[Christian Holler (:decoder)]

If we can easily fix the source of non-determinism we should do that when running with --differential-testing.

Comment 3

[Tooru Fujisawa [:arai]]

it's due to different implementation for generator, between interpreter, baseline, and ion.
baseline uses InterpretGeneratorResume in self-hosted JS, but interpreter and ion don't use it.

fuzzilli_hash uses structured-clone, and it serializes the SavedFrame objects.
during baseline execution, InterpretGeneratorResume is included in the frames.
during interpreter or ion execution, it doesn't appear there.

This is not observable from web content, because it's self-hosted JS and it's hidden from web-exposed properties like Error.prototype.stack.

maybe we can filter self-hosted frames out of structured-clone ?
or do we need self-hosted frames across serialization?
sfink, can I have your input here?