Closed Bug 1581580 Opened 4 months ago Closed 4 months ago

Add limited set of --browsertime Raptor tests to make it easier to smoke-test changes

Categories

(Testing :: Performance, enhancement, P1)

Version 3
enhancement

Tracking

(firefox71 fixed)

RESOLVED FIXED
mozilla71
Tracking Status
firefox71 --- fixed

People

(Reporter: nalexander, Assigned: stephend)

References

Details

Attachments

(1 file, 1 obsolete file)

Bug 1573208 added a --browsertime flag to turn Raptor tests into Browsertime tests. That's great but it doesn't make it easy to run Raptor and Browsertime test in the same try push, which is useful for smoketesting harness changes.

This ticket tracks adding a limited test set of Browsertime jobs. Just adding --browsertime to the mozharness.extra-options should be enough to trigger the existing mechanisms.

Assignee: nobody → stephen.donner

This requires a new, limited, test-set for Browsertime, and a few test definitions to be cloned and extra-options: ["--browsertime"] added. I'd clone just:

  • raptor-tp6-1
  • raptor-tp6-cold-1
  • raptor-tp6m-cold-1
  • raptor-tp6m-cold-1

to start. We're not trying to compare results, we're trying to smoketest the harness. These tests can run on the same schedule as the Raptor test set, or only on try if that's easy to do.

Status: NEW → ASSIGNED
Priority: -- → P1
Summary: Add limited set of --browsertime Raptor tests to make it easier to smoketest changes → Add limited set of --browsertime Raptor tests to make it easier to smoke-test changes

Hi @ahal! Pardon the need-info, but, skimming over my initial patch up at https://phabricator.services.mozilla.com/D46232 (which adds duplication with slight variations), and in light of your assessment and recommendations in https://bugzilla.mozilla.org/show_bug.cgi?id=1541420#c0, is there a cleaner way to do what I'm trying to do there, with a transform, something else? Thanks!

Flags: needinfo?(ahal)

You probably could, but there's no precedent. So I'd just continue following how the other tasks are added for now until we get a chance to refactor this stuff properly.

But yes, we should really do something soon :). If anyone has time to tackle things like this, let me know. I have ideas on how to improve things but not much time to do it.

Flags: needinfo?(ahal)

Latest try push here: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=edfb3f6169bb6477d4aab023edc746434a9c932e

The newest patch removed mobile support for now, to get this landable and slightly cleaner.

Attachment #9093455 - Attachment is obsolete: true
Pushed by sdonner@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3bd92e48c74a
Add limited set of --browsertime Raptor tests to make it easier to smoke-test changes. r=perftest-reviewers,sparky

Thx :sparky, all; I've filed follow-up bug 1585120 to take care of the mobile side (free for whomever wants to work on it; conditioned profiles are my top priority).

Regressions: 1585155
Regressions: 1585199
Regressions: 1585198

Sorry about the noise and subsequent (need for) backout; I've posted a new diff to Phab which includes the accidentally omitted -- and critical -- test['tier'] = max(test['tier'], 2) change, re-added.

New try push here: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=a68dc3d9a6b7b4a979e5f113bdf79d81dcded457

Flags: needinfo?(stephen.donner)
Pushed by sdonner@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b53f9101ae60
Add limited set of --browsertime Raptor tests to make it easier to smoke-test changes. r=perftest-reviewers,sparky
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla71
Blocks: 1587154
You need to log in before you can comment on or make changes to this bug.