1714585 - Enable Fission for the AWSY base overhead test

Status: RESOLVED FIXED

(Testing :: AWSY, task, P2)

Description

[Andrew McCreight [:mccr8]]

The AWSY base overhead test was added before Fission was ready for use in testing, but I think it is safe to say that it is ready now. Kashav mentioned that he recently was able to avoid loading a frame script in content processes, but that only applies when Fission is enabled, because only Fission is currently using SHIP, so it won't show up in the base overhead test. We should just flip the pref in the test so we are making the test a bit more realistic. Process overhead is of greater interest for Fission, because there are many more processes.

Comment 1

[Chris Peterson [:cpeterson]]

Tracking for Fission M7a because we should fix this test soon.

Comment 2

[Andrew McCreight [:mccr8]]

I tried adding setting fission.autostart to true in testing/awsy/conf/base-prefs.json, but the test still resulted in a bunch of web processes loading about:blank. I'm not sure if that's sufficient to test the differences we might see with Fission enabled. Total content process memory was about 1.5MB lower, if that means anything.

Comment 3

[Andrew McCreight [:mccr8]]

Given that my simple attempt at this doesn't seem to have worked, I don't think I'm going to have time to investigate this further in the M7a timeframe.

Comment 4

[Andrew McCreight [:mccr8]]

Here's the try run, FWIW: https://treeherder.mozilla.org/jobs?repo=try&revision=259025234a321a4a241823f404d25b463f6a77ff&selectedTaskRun=G7dqoADDQ7GYWCMsWzwtMQ.0

Comment 5

[Chris Peterson [:cpeterson]]

(In reply to Andrew McCreight [:mccr8] from comment #3)

Given that my simple attempt at this doesn't seem to have worked, I don't think I'm going to have time to investigate this further in the M7a timeframe.

In that case, we can revisit this bug in Fission M8 or later.

Comment 6

[Neha Kochar [:neha]]

Not a release experiment blocker, moving to MVP.

Comment 7

[Nika Layzell [:nika] (ni? for response)]

(In reply to Andrew McCreight [:mccr8] from comment #2)

I tried adding setting fission.autostart to true in testing/awsy/conf/base-prefs.json, but the test still resulted in a bunch of web processes loading about:blank. I'm not sure if that's sufficient to test the differences we might see with Fission enabled. Total content process memory was about 1.5MB lower, if that means anything.

The about:blank pages loading in "web" content processes is expected, as there's no specific webIsolated process type for them to load in. If we get any advantages from enabling fission, they should also appear in "web" content processes, so this basic approach should work fine. Can we perhaps try landing that?

Comment 8

[Chris Peterson [:cpeterson]]

We should run AWSY with Fission enabled, but we do expect about:blank to load in webIsolated processes.

We will want to move base overhead test from job group SY to SY-fis.

SY-fis currently runs only as tier-2 on mozilla-central. We would like to schedule this base overhead test as tier-1 on autoland, though we don't need the rest of SY-fis to run as tier-1 on autoland. It is expensive to run and doesn't find useful regressions.

Comment 9

[Chris Peterson [:cpeterson]]

Assigning to mccr8 because he said he could take a look at this bug.

Comment 10

[Chris Peterson [:cpeterson]]

This bug is a soft blocker for Fission MVP. We'd like to fix it before our Release channel rollout, but we won't delay the rollout waiting for it.

Comment 11

[Andrew McCreight [:mccr8]]

Attachment: Bug 1714585 - Make Fission the primary way to run AWSY base.

Comment 12

[Andrew McCreight [:mccr8]]

Here's my work in progress patch. I set it so that we continue to run e10s AWSY base on mozilla-central as tier 2, but it turned out we weren't ever running Fission (due to the lack of the "variants: [fission]" annotation) so maybe we can just skip it. I have no idea what the
bugbug-push-schedules.json file is for, so maybe those changes are not needed.

A bigger issue is that unlike the e10s version of the test, the number of processes we end up with varies quite a bit. For some reason, more than one about:blank can end up in a process. That seems like something we'd want to fix before landing this.

Comment 13

[Andrew McCreight [:mccr8]]

Nika suggested I try turning off bfcache in parent, and that seems to have fixed the issue.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/11db8586ad42
Make Fission the primary way to run AWSY base. r=perftest-reviewers,AlexandruIonescu

Comment 15

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/11db8586ad42

Comment 16

[Acasandrei Beatrice (needinfo me)]

This bug has caused the following improvement:

== Change summary for alert #30855 (as of Tue, 10 Aug 2021 08:01:53 GMT) ==

Improvements:

Ratio	Suite	Test	Platform	Options	Absolute values (old vs new)
10%	Base Content JS		macosx1015-64-shippable		1,944,489.33 -> 1,752,442.67
10%	Base Content JS		macosx1015-64-shippable-qr		1,944,489.33 -> 1,752,442.67
10%	Base Content JS		linux1804-64-shippable-qr		1,944,076.00 -> 1,752,234.00
10%	Base Content JS		windows10-64-2004-shippable-qr		1,949,844.67 -> 1,758,632.00
10%	Base Content JS		windows10-64-2004-shippable-qr		1,949,737.33 -> 1,758,802.67
...	...	...	...	...	...
3%	Base Content Explicit		linux1804-64-shippable-qr		7,301,104.00 -> 7,051,760.00

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=30855

Comment 17

[Andrew McCreight [:mccr8]]

The improvement is expected, due to SHIP. Unfortunately, this is an indication of something going wrong on the Perfherder side because the Fission version should be separate from the e10s version. I'm going to find somebody to talk to about this.

Comment 18

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Andrew McCreight [:mccr8] from comment #17)

The improvement is expected, due to SHIP. Unfortunately, this is an indication of something going wrong on the Perfherder side because the Fission version should be separate from the e10s version. I'm going to find somebody to talk to about this.

For other harnesses we add 'fission' to the extraOptions key in the Perfherder JSON. This shows up in Perfherder as a label, and creates a distinct performance signature.

Comment 19

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This is also done in AWSY, but perhaps not in the case of this test? https://searchfox.org/mozilla-central/rev/d3683dbb252506400c71256ef3994cdbdfb71ada/testing/awsy/awsy/test_memory_usage.py#94

Comment 20

[Andrew McCreight [:mccr8]]

Thanks for the explanation. That makes sense now that you've explained it. I'll attach a patch to fix this in bug 1726411.