Open Bug 1591124 Opened 5 years ago Updated 3 months ago

sessionrestore Talos tests should actually load pages and include app tabs

Categories

(Testing :: Raptor, task, P2)

task

Tracking

(Fission Milestone:Future)

Fission Milestone Future

People

(Reporter: mconley, Unassigned)

References

(Depends on 1 open bug, Blocks 2 open bugs)

Details

(Whiteboard: fission-perf, [fxp])

Running the sessionrestore and sessionrestore_many_windows Talos tests, it looks like none of the pages that the restored session includes are actually being loaded in tabs. Instead, we get a bunch of "Can't connect to page" error tabs.

I suspect we should be including the overhead of loading the initially selected tab(s) and any app tabs as part of the measurement, and for Fission, should be ensuring that these loaded pages include third-party iframes that can be loaded out-of-process.

Fission Milestone: --- → M7
Flags: needinfo?(rjesup)

This is broken even without Fission. We'd like to fix this, of course, to get accurate performance measurements but it doesn't have to block M7 for Beta. Pushing this out to M8.
We're currently rewriting session store in M7 so M8 will be a good time to fix these session restore tests.

Fission Milestone: M7 → M8
Flags: needinfo?(rjesup)
See Also: → 1588165
Whiteboard: fission-perf

It's unfortunate that these are still broken but since these are measuring incorrect even without fission, we have decided to push this out to Future. These tests are not part of the Fission release criteria and therefore will not block shipping Fission.
mconley is probably the only one most familiar with these tests and is currently occupied with other work. Leaving a NI for mconley in case he disagrees with this assessment or if he can find time soon-ish to fix these.

Fission Milestone: M8 → Future
Flags: needinfo?(mconley)

I think you've made the right call, and this shouldn't block Fission.

Flags: needinfo?(mconley)
Severity: normal → S3

@mconley this is blocking bug 1588230 but looking at the comments, perhaps that's not necessary, would you mind adding your latest thoughts?

Flags: needinfo?(mconley)

My latest thoughts are that trying to rig up the Talos test to load a meaningful set of pages will likely be hampered by the challenge of setting up the local server to point at some site snapshots that are meaningfully representative of the modern web (it's straight-forward to use the tp5 or tp6 set, but last I checked, those snapshots was pretty old... tp6 is from ~2017 I think?).

It's been a while since I've thought about performance testing. Reading through https://firefox-source-docs.mozilla.org/testing/perfdocs/index.html, we have lots of options to choose from, but it's not immediately obvious which one is the right fit.

In your opinion, which framework is the best one to:

  1. Run using a prepared user profile / preferences
  2. Allows us to realistically load modern real-world pages (or more accurately, recordings of modern real-world pages)
  3. Allows us to measure how long it takes to do that loading of those pages in both the foreground and background tabs

?

Is it still Talos that best fits this? Or is there something better?

Flags: needinfo?(mconley) → needinfo?(dave.hunt)

Given these requirements, I believe Raptor could be a more suitable framework for these tests due to its use of mitmproxy to replay page loads, and will benefit from future work on refreshing our page set and recording abilities.

@sparky what do you think about moving this into Testing :: Raptor? It sounds like we're missing valuable coverage currently, and this could be an opportunity to migrate these tests away from Talos.

Flags: needinfo?(dave.hunt) → needinfo?(gmierz2)

I agree with moving this Testing :: Raptor. I'll also file a blocking bug for this.

Component: Session Restore → Raptor
Flags: needinfo?(gmierz2)
Product: Firefox → Testing
Depends on: 1882766
Whiteboard: fission-perf → fission-perf, [fxp]
Component: Raptor → Talos
Component: Talos → Raptor
Priority: -- → P2
You need to log in before you can comment on or make changes to this bug.