Closed Bug 1704129 Opened 1 month ago Closed 17 days ago

[fission] Add a test to measure fission process switch time

Categories

(Testing :: Performance, task, P1)

Default
task

Tracking

(Fission Milestone:M7a, firefox90 fixed)

RESOLVED FIXED
90 Branch
Fission Milestone M7a
Tracking Status
firefox90 --- fixed

People

(Reporter: sefeng, Assigned: sparky)

References

Details

Attachments

(2 files, 1 obsolete file)

We'd like to add a test to measure the fission process switch time. The idea is the browser first navigates to a simple website like https://mozilla.seanfeng.dev/files/red.html in site origin 1 (e.g 127.0.0.1), and start the recording, and then navigate to another simple website like https://mozilla.seanfeng.dev/files/blue.html in site origin 2 (e.g 192.168.0.1).

The second navigation is the one that we want to measure.

A sample browsertime preload.js loos like

module.exports = async function(context, commands) {
  let url = context.options.browsertime.url;
  await commands.navigate('https://www.mozilla.org/en-US/');
  await commands.wait.byTime(5000);
  await commands.navigate('http://127.0.0.1:8080/red.html');
  await commands.wait.byTime(1000);
  return commands.measure.start(url);
};

Instead of using live sites, recorded sites are better because we want to eliminate all other side effects.

sparky, what do you think of the best ways to accomplish this? Thanks!

Flags: needinfo?(gmierz2)

M7a aims to add more testing coverage.

Fission Milestone: --- → M7a

:sefeng, so I think the best place for this would be in mozperftest. We could put it in raptor, but we'll have to create a new test type for it (i.e. a test type like benchmark, scenario, pageload) which isn't ideal.

Regarding the need to run it through a recording, since the test pages are very simple and don't seem to need to be hosted to work, I wonder if it would be possible to open them like file:///tmp/red.html? I don't know if browsertime supports that though and I'm not sure if this would prevent the test from testing process switch time.

Flags: needinfo?(gmierz2) → needinfo?(sefeng)

The purpose of using recording is that we'd like to make sure the red.html and blue.html use different eTDL + 1, to ensure fission uses a different process for them. I don't think file:// would work..

Do you have time to tackle this sparky? I can help if you don't have time, although I need you to provide some pointers for me, for things like how to create a new test type etc.

Flags: needinfo?(sefeng) → needinfo?(gmierz2)

:sefeng, have you checked to see if playing back those two recordings simultaneously still gives you the different eTDL+1? I'm guessing yes, but I'd like to make sure before we start making changes in raptor to accommodate this new test.

Flags: needinfo?(gmierz2) → needinfo?(sefeng)
Severity: -- → S2
Priority: -- → P1

Thanks to sparky for providing recordings for me.

The recordings were collected by loading https://mozilla.seanfeng.dev/files/red.html and https://mozilla.pettay.fi/moztests/blue.html.

Yeah, they are different eTLD+1 for sure, and fission with browsertime uses different processes to load them.

needinfo spark for the next steps.

Flags: needinfo?(sefeng) → needinfo?(gmierz2)

This patch adds a test_script field to the raptor manifests that allows us to customize the browsertime test script to use. However, we only allow it to be used on pageload tests for the time being.

Assignee: nobody → gmierz2
Status: NEW → ASSIGNED

This patch adds a test to measure fission process switch time. It uses two recordings replayed at the same time, along with a custom browsertime pageload script to do this. Two urls are provided through the manifest and the second url is the one that gets measured.

Depends on D113219

These 3 patches make it possible to run the test, and also add the test for local and CI testing. We need to get the recordings uploaded now

Flags: needinfo?(gmierz2)
Attachment #9217987 - Attachment is obsolete: true
Pushed by gmierz2@outlook.com:
https://hg.mozilla.org/integration/autoland/rev/2801837a06b2
Add a test to measure process switch time. r=sefeng,perftest-reviewers,Bebe
https://hg.mozilla.org/integration/autoland/rev/3d3d8f849862
Run raptor process-switch test on mozilla-central. r=sefeng,perftest-reviewers,Bebe
Status: ASSIGNED → RESOLVED
Closed: 17 days ago
Resolution: --- → FIXED
Target Milestone: --- → 90 Branch
Duplicate of this bug: 1701988

Sean, I'm trying to find the new process-switch test's results in Perfherder, but I see more tests labeled "process-switch" than I expected.

Which tests should I monitor for Fission?
Also, is this process-switch test applicable for e10s or just Fission?

Steps to reproduce:

  1. Load https://treeherder.mozilla.org/perfherder/graphs
  2. Click "Add test data" button.
  3. Change Framework menu from "talos" to "browsertime".
  4. Type "process-switch" in the filter tests field.

Result:

In the Tests list, there are 18 test names mentioning "process-switch" just for linux-1806-64-shippable:

process-switch fnbpaint opt cold firefox
process-switch dcf opt cold firefox
process-switch loadtime opt cold firefox
process-switch fnbpaint opt warm firefox
process-switch dcf opt warm firefox
process-switch loadtime opt warm firefox
...
process-switch PerceptualSpeedIndex opt warm webrender firefox

Flags: needinfo?(sefeng)

It works for both e10s and fission. You need to select linux1804-64-shippable-qr build, because we only run fission with webrender.

I'd say fnbpaint is suffient to monitor this.

Here's a link you could use: https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&series=mozilla-central,3593005,1,13&series=mozilla-central,3592931,1,13&timerange=1209600

Does this answer the question?

Flags: needinfo?(sefeng)

(In reply to Sean Feng [:sefeng] from comment #15)

I'd say fnbpaint is suffient to monitor this.
...
Does this answer the question?

Yes. Thank you!

You need to log in before you can comment on or make changes to this bug.