1712317 - Review page load site list and build TP7

Status: NEW

(Testing :: Performance, task, P2)

Description

[Greg Mierzwinski [:sparky]]

This bug is for reviewing our page load site list and building a TP7 pageload test set.

Here is the doc for our meetings: https://docs.google.com/document/d/14vStjz8_VvCpuiZo2-t49L5HJTPsyWjdehQNeldRtr4/edit?usp=sharing

Comment 1

[Greg Mierzwinski [:sparky]]

Summary from first meeting:

• For TP7, we’ll expand the number of test sites we have to 200. We’ll use the following format for how we schedule them:
  • Perf-tier (PT) 1: Essential tests that run on autoland and on each mozilla-central commit. These should be tests which often catch changes. We will need to regularly refresh this list.
    • Max of 25 sites, with a handful of them running as very important tests.
  • Perf-tier (PT) 2: Weekly tests on the weekends for sites that we want to measure and monitor but without alerting on them and running them constantly.
    • Rest of the 175 sites will run here.
  • Limit login sites to 25 so we don’t overwhelm bebe and alexandrui with recordings.
  • Business as usual with PT1/tp6, rest of them go into PT2.
  • Start with this and see if we need to modify cadence.
  • Distinguish between mobile and desktop.
• Blockers for this work:
  • [P1] Automate recordings for non-login sites.
  • [P2] Implement a smart page/browser cycles count selection method.
    • We could start by limiting PT2 tests to 10-15 cycles and expand on that going forward.
  • [P1] Use pre-built profiles rather than conditioned profiles to save the 30 seconds post startup delay.
• Discuss with aerickson about setting up a system to use machines during their downtime for our PT2 tests.

Comment 2

[Greg Mierzwinski [:sparky]]

Priority remains the same after triage.

Comment 3

[Greg Mierzwinski [:sparky]]

New docs for this: https://docs.google.com/document/d/1ayCZYJQzOR0_QVzkGleBhDtQEEfrS_DWRlSFYmikOdM/edit#