1802953 - Enable wpt tests to run on try - isolateNothing mode

Status: RESOLVED FIXED

(GeckoView Graveyard :: Sandboxing, task, P1)

Description

[[:owlish] 🦉 PST]

+++ This bug was initially created as a clone of Bug #1788712 +++

We need to enable wpt fission tests to run on try so that the wpt-sync bot could update their expectations.

Because not all the tests are currently green, it's best to make them tier 3.

Comment 1

[[:owlish] 🦉 PST]

Attachment: Bug 1802953 - add module

Comment 2

[[:owlish] 🦉 PST]

Attachment: Bug 1802953 - Temporarily enable Fission tests on Android in isolateNothing mode

We want these tests to run by default on Android Fission to get some statistics on the failures, and to let the wpt-sync bot adjust the expectations. Once the work on fixing wpt tests on Android Fission is complete, we plan to revisit this and make the tier of these tests to be default

Depends on D163261

Comment 3

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/2d613025a659

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Can this change be reverted after the next time wptsync bot runs? (Last sync with upstream was on Nov 21 - next merge is probably blocked on something).

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

There has been a synchronization of the web platfom tests with the upstream repository yesterday but many of the tests enabled here still fail.
What is the plan to get them passing (or turn them or the whole test configuration off)? Reason for the question: We want to avoid permanently failing tasks in CI.

Comment 6

[[:owlish] 🦉 PST]

Thank you Sebastian! We will work on fixing those tests. Meanwhile, it looks like we can turn them back off.

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Attachment: Bug 1802953 - Disable Android fission tests running in isolateNothing mode

They had only been enabled temporarily to get their expectations updated by
the web-platform-tests sync automation.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/33cc95c4cec2
Disable Android fission tests running in isolateNothing mode

Comment 9

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/33cc95c4cec2

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

A patch has been attached on this bug, which was already closed. Filing a separate bug will ensure better tracking. If this was not by mistake and further action is needed, please alert the appropriate party. (Or: if the patch doesn't change behavior -- e.g. landing a test case, or fixing a typo -- then feel free to disregard this message)

Comment 11

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #36726 (as of Thu, 12 Jan 2023 20:57:14 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
111%	google-search-restaurants fcp	android-hw-a51-11-0-aarch64-shippable-qr	cold fission webrender	404.67 -> 853.62
106%	google fcp	android-hw-a51-11-0-aarch64-shippable-qr	cold fission webrender	397.46 -> 816.92
83%	wikipedia fcp	android-hw-a51-11-0-aarch64-shippable-qr	cold fission webrender	531.92 -> 975.75
82%	google-search-restaurants loadtime	android-hw-a51-11-0-aarch64-shippable-qr	cold fission webrender	547.42 -> 997.21
81%	google ContentfulSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	cold fission webrender	513.67 -> 930.08
81%	google FirstVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	cold fission webrender	511.58 -> 924.08
80%	google SpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	cold fission webrender	513.42 -> 926.67
80%	google PerceptualSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	cold fission webrender	518.92 -> 931.75
70%	google-search-restaurants FirstVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	cold fission webrender	613.67 -> 1,042.33
68%	google-search-restaurants PerceptualSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	cold fission webrender	629.58 -> 1,055.50
...	...	...	...	...
9%	microsoft-support loadtime	android-hw-a51-11-0-aarch64-shippable-qr	cold fission webrender	2,280.10 -> 2,477.58
6%	microsoft-support LastVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	cold fission webrender	2,904.17 -> 3,087.00
6%	espn loadtime	android-hw-a51-11-0-aarch64-shippable-qr	fission warm webrender	1,111.54 -> 1,179.75
5%	allrecipes LastVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	cold fission webrender	8,170.33 -> 8,579.08
4%	amazon loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	355.35 -> 368.33

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
32%	facebook fcp	android-hw-a51-11-0-aarch64-shippable-qr	fission warm webrender	463.58 -> 316.12
27%	facebook FirstVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	fission warm webrender	509.83 -> 371.42
25%	facebook SpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	fission warm webrender	542.54 -> 407.25
25%	facebook ContentfulSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	fission warm webrender	551.04 -> 414.75
22%	facebook PerceptualSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	fission warm webrender	570.17 -> 442.42
...	...	...	...	...
6%	facebook LastVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	fission warm webrender	1,446.08 -> 1,360.83

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=36726

Comment 12

[Chris Peterson [:cpeterson]]

== Change summary for alert #36726 (as of Thu, 12 Jan 2023 20:57:14 GMT) ==

The change in comment 8 and 9 restores the Android Fission mode from IsolateNothing to IsolateEverything mode, so the perf regression is expected.