1597297 - The JS debugging option Async Stacks is enabled in the Talos performance profile, sometimes lowering test scores, sometimes introducing noise

Status: NEW

(Testing :: Talos, task, P3)

Description

[Mike Conley (:mconley) (:⚙️)]

This bug is modeled after bug 1595537, where a similar issue was investigated for the Raptor test suite.

Here's a comparison of all of our Talos tests with async stacks on (base) vs off (new): https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=af7016983c70a191c39889d9ffdce9345ebe2e55&newProject=try&newRevision=4be37d95375d3fd905f90d8296e92d93cdb438db&framework=1

There are gains in sessionrestore_many_windows opt e10s stylo, startup_about_home_paint_realworld_webextensions and tp5o_webext.

Interestingly, this also seems to drop some of the noise from our measurements on Linux and Mac OS. Unfortunately, in some cases it seems to increase the noise for Windows.

Comment 1

[:Gijs (he/him)]

Mike, can you help me understand where this fits in the perf keyword hierarchy? I'm not quite sure...

Comment 2

[Mike Conley (:mconley) (:⚙️)]

I think this is a good idea to reduce Talos test noise, but ultimately isn't something our users will notice.