Add limited set of --browsertime Raptor tests to make it easier to smoke-test changes
Categories
(Testing :: Performance, enhancement, P1)
Tracking
(firefox71 fixed)
| Tracking | Status | |
|---|---|---|
| firefox71 | --- | fixed |
People
(Reporter: nalexander, Assigned: stephend)
References
Details
Attachments
(1 file, 1 obsolete file)
Bug 1573208 added a --browsertime flag to turn Raptor tests into Browsertime tests. That's great but it doesn't make it easy to run Raptor and Browsertime test in the same try push, which is useful for smoketesting harness changes.
This ticket tracks adding a limited test set of Browsertime jobs. Just adding --browsertime to the mozharness.extra-options should be enough to trigger the existing mechanisms.
|
Assignee | |
Updated•5 years ago
|
|
Reporter | |
Comment 1•5 years ago
•
|
||
This requires a new, limited, test-set for Browsertime, and a few test definitions to be cloned and extra-options: ["--browsertime"] added. I'd clone just:
raptor-tp6-1raptor-tp6-cold-1raptor-tp6m-cold-1raptor-tp6m-cold-1
to start. We're not trying to compare results, we're trying to smoketest the harness. These tests can run on the same schedule as the Raptor test set, or only on try if that's easy to do.
|
|
Assignee | |
Comment 2•5 years ago
|
||
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 3•5 years ago
|
||
Hi @ahal! Pardon the need-info, but, skimming over my initial patch up at https://phabricator.services.mozilla.com/D46232 (which adds duplication with slight variations), and in light of your assessment and recommendations in https://bugzilla.mozilla.org/show_bug.cgi?id=1541420#c0, is there a cleaner way to do what I'm trying to do there, with a transform, something else? Thanks!
|
||
Comment 4•5 years ago
|
||
You probably could, but there's no precedent. So I'd just continue following how the other tasks are added for now until we get a chance to refactor this stuff properly.
But yes, we should really do something soon :). If anyone has time to tackle things like this, let me know. I have ideas on how to improve things but not much time to do it.
|
|
Assignee | |
Comment 5•5 years ago
•
|
||
Latest try push here: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=edfb3f6169bb6477d4aab023edc746434a9c932e
The newest patch removed mobile support for now, to get this landable and slightly cleaner.
|
|
Assignee | |
Comment 6•5 years ago
|
||
|
|
Assignee | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 8•5 years ago
|
||
Thx :sparky, all; I've filed follow-up bug 1585120 to take care of the mobile side (free for whomever wants to work on it; conditioned profiles are my top priority).
|
||
Comment 9•5 years ago
|
||
Backed out changeset 3bd92e48c74a (bug 1581580) for causing Perma tier2 Raptor Btime failures
push that caused the failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&searchStr=btime&revision=3bd92e48c74af4ce3c8b5b76cb2b2e09e0fd193f
backout: https://hg.mozilla.org/integration/autoland/rev/1c4e8032c1a4349a7d0d1f4db870a961bc1a4dc5
|
|
Assignee | |
Comment 10•5 years ago
•
|
||
Sorry about the noise and subsequent (need for) backout; I've posted a new diff to Phab which includes the accidentally omitted -- and critical -- test['tier'] = max(test['tier'], 2) change, re-added.
New try push here: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=a68dc3d9a6b7b4a979e5f113bdf79d81dcded457
|
||
Comment 11•5 years ago
|
||
|
||
Comment 12•5 years ago
|
||
| bugherder | ||
Description
•