1943365 - Add JetStream 3 benchmark

Status: ASSIGNED

(Testing :: Performance, task, P2)

Description

[Jan de Mooij [:jandem]]

We have JetStream 2 in CI, but browser vendors are currently working on the next version of this benchmark.

https://github.com/WebKit/JetStream/

For our JS/Wasm performance work it'd be great to have this running in CI. The benchmark is still changing a lot so we probably want to update this every few weeks or so.

We should keep running JetStream 2 for now because that's the latest stable version and we want to track our performance on that until the next version is released.

Comment 1

[Kash Shampur [:kshampur] ⌚EST]

We had Bug 1894894 filed for 2.2. Should we assume getting js3 running supersedes 2.2?

I still don't see any js3 in https://github.com/WebKit/WebKit/tree/8c16d17d9bf3bcdf3414da111498366bb0e700c1/Websites/browserbench.org (but do see 2.2)
so is there a chance it is not quite ready yet to even be running (reliably) ?

Comment 2

[Ryan Hunt [:rhunt]]

The repository has moved to here [1]. It's very new and changes are landing pretty frequently right now. The wasm team is hoping to start using it for some performance work though, so it'd be useful for us to start getting this tracked.

[1] https://github.com/webkit/jetstream

Comment 3

[Ryan Hunt [:rhunt]]

I personally don't care about 2.2, but I don't know what changes happened from 2 -> 2.2. I think the hope is that version 3 will be the only thing we care about when it is finalized.

Comment 4

[Kash Shampur [:kshampur] ⌚EST]

ah thanks- i just realized I was using outdated links. (and I see now the commit there Initial copy from WebKit/PerformanceTests/JetStream3)

Having it in it's own repo like this makes it slightly easier to set up an external cloning approach like we recently did with motionmark 1.3 and update the revision as needed e.g. so that's great.

I'd be in favour of closing 2.2 (which I'll do shortly)

Comment 5

[Jan de Mooij [:jandem]]

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #4)

I'd be in favour of closing 2.2 (which I'll do shortly)

That works for me. For now we should keep running the current JetStream 2 version also but we can remove that one after the first stable release of JetStream 3.

Comment 6

[Matthew Gaudet (he/him) [:mgaudet]]

JS2.2 is a relatively minor update apparently:

JetStream 2.2 runs the same benchmarks as JetStream 2.1, but includes fixes for several minor issues. One such issue happened when running on fast hardware where the timer resolution, as returned by the VM, was occasionally 0. This causes a problem when calculating a scores. There were two issues with the sub-test segmentation, where the code didn’t create the initial Float32Array for the test, and the second a race condition in the task queue code. The command line scripts cli.js and wasm-cli.js had issues with the D8 and SpiderMonkey shell programs.

(I'll leave it to others to decide if it makes sense to update our JS2 stuff to 2.2 or not -- I suspect no)

Comment 7

[Jan de Mooij [:jandem]]

Changing this to P2 after Matrix discussion with Bas.

Comment 8

[Kash Shampur [:kshampur] ⌚EST]

Attachment: WIP: Bug 1943365 - Add JetStream 3 benchmark. r?#perftest