Run live site tests three times a week
Categories
(Testing :: Raptor, enhancement, P1)
Tracking
(firefox90 fixed)
| Tracking | Status | |
|---|---|---|
| firefox90 | --- | fixed |
People
(Reporter: davehunt, Assigned: davehunt)
Details
Attachments
(3 files)
We currently have some live site tests running three times a week, and others running once a week. We're also inconsistent in the applications and variants we run live site tests against. Let's consolidate and run live site tests on the same frequency as our other infrequent tests and include more variants such as Fission, Fenix and GeckoView.
|
||
Comment 1•3 years ago
|
||
:davehunt, is anyone actively using the metrics from those tests? Maybe we should drop all of them down to once a week?
|
Assignee | |
Comment 2•3 years ago
|
||
|
|
Assignee | |
Comment 3•3 years ago
|
||
Depends on D113534
|
|
Assignee | |
Comment 4•3 years ago
|
||
Depends on D113535
|
|
Assignee | |
Comment 5•3 years ago
|
||
(In reply to Greg Mierzwinski [:sparky] from comment #1)
:davehunt, is anyone actively using the metrics from those tests? Maybe we should drop all of them down to once a week?
I believe :acreskey is, perhaps :jesup too? I'm open to us changing the schedule but would still like to consolidate to reduce the complexity and maintenance burden. The problem with once/week is that if there's a build failure or intermittent test failure we could go two weeks without results.
|
||
Comment 6•3 years ago
|
||
I do look at the Fenix live site performance (via arewefastyet.com), arguably our most representative metrics :), on a weekly basis.
But because they are live sites and somewhat noisy, only having one data point for a site per week would make it harder to pick up potential problems.
If resources weren't a concern, I would vote for daily runs.
|
||
Comment 7•3 years ago
|
||
I agree with andew; I check them on a roughly weekly basis, but running once/week is too infrequent (given noise) to be of any real use. I'd love 1/day
Pushed by dhunt@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/14ba92195463 Consolidate live performance tests into a single schedule; r=perftest-reviewers,sparky https://hg.mozilla.org/integration/autoland/rev/d8f85235df7c Run live performance tests three times a week; r=perftest-reviewers,sparky https://hg.mozilla.org/integration/autoland/rev/93a998c7d0c9 Allow live site tests to run against GeckoView; r=perftest-reviewers,sparky
|
||
Comment 9•3 years ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/14ba92195463
https://hg.mozilla.org/mozilla-central/rev/d8f85235df7c
https://hg.mozilla.org/mozilla-central/rev/93a998c7d0c9
Description
•