Closed
Bug 1462776
Opened 6 years ago
Closed 6 years ago
Add subtest support to raptor
Categories
(Testing :: Raptor, enhancement)
Tracking
(firefox62 fixed)
RESOLVED
FIXED
mozilla62
Tracking | Status | |
---|---|---|
firefox62 | --- | fixed |
People
(Reporter: rwood, Assigned: rwood)
References
Details
Attachments
(2 files)
Right now raptor tests can only have one single test page (i.e raptor-firefox-tp6 only is currently testing the amazon test page). Add subtest support so that a raptor test can run against several test pages (i.e. so I can add facebook, google, and youtube test pages to the same raptor-firefox-tp6 test). Each test page will be listed inside the raptor-test.ini.
Assignee | ||
Comment 1•6 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=71f9ff90ee9b9317f5c0c8cbc07a1686ddb72e44
Assignee | ||
Comment 2•6 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b7447f02f4a91bef7a60a15c766bd8c708ec5f14
Comment hidden (mozreview-request) |
Comment 4•6 years ago
|
||
mozreview-review |
Comment on attachment 8977096 [details] Bug 1462776 - Run single raptor test or multiple subtests based on test name cmd line arg; https://reviewboard.mozilla.org/r/245168/#review251766 ::: testing/raptor/raptor/manifest.py:128 (Diff revision 1) > - tests_to_run = [next_test] > + tests_to_run.append(next_test) > - break > + break What happens if there are subtests with the same name in different manifests? This looks like it'll run a random test, would it be a good idea to remove the `break` and run all subtests with that name instead?
Attachment #8977096 -
Flags: review?(ahal) → review+
Assignee | ||
Comment 5•6 years ago
|
||
(In reply to Andrew Halberstadt [:ahal] from comment #4) > What happens if there are subtests with the same name in different > manifests? This looks like it'll run a random test, would it be a good idea > to remove the `break` and run all subtests with that name instead? Thanks. I think we'll just have to make sure each subtest name is unique. Don't want to run all subtests with that name because in production you only want to run specific tests in specific jobs. Trying to avoid requiring a separate raptor configuration file (like talos has - talos.json - that tells which subtests belong in which suites).
Assignee | ||
Comment 6•6 years ago
|
||
(In reply to Robert Wood [:rwood] from comment #5) > (In reply to Andrew Halberstadt [:ahal] from comment #4) > > > What happens if there are subtests with the same name in different > > manifests? This looks like it'll run a random test, would it be a good idea > > to remove the `break` and run all subtests with that name instead? > > Thanks. I think we'll just have to make sure each subtest name is unique. > Don't want to run all subtests with that name because in production you only > want to run specific tests in specific jobs. Trying to avoid requiring a > separate raptor configuration file (like talos has - talos.json - that tells > which subtests belong in which suites). Actually maybe a solution is when a matching subtest name is found, check it's manifest name and it should match the subtest name itself, if it doesn't then it's a subtest with the same name but from a different manifest and shouldn't be included.
Assignee | ||
Comment 7•6 years ago
|
||
> Actually maybe a solution is when a matching subtest name is found, check
> it's manifest name and it should match the subtest name itself, if it
> doesn't then it's a subtest with the same name but from a different manifest
> and shouldn't be included.
No wait that doesn't work as the manifest name doesn't necessarily match the manifest name. There's really no good solution for this short of requiring either a specific file that says what subtests belong to which suites.
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 11•6 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d72d00f8aa11f8dd265e2ad1fe0796dae70c5269
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 14•6 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=957a23b717fe9faab39330adeee3a5490798a109
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Comment 17•6 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1cafce7bb5805e4118ad4822fb9231edfbe43383
Comment 18•6 years ago
|
||
mozreview-review |
Comment on attachment 8980061 [details] Bug 1462776 - Add subtest support to raptor; https://reviewboard.mozilla.org/r/246222/#review252514
Attachment #8980061 -
Flags: review?(ahal) → review+
Comment 19•6 years ago
|
||
Pushed by rwood@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/2a0331afdbef Run single raptor test or multiple subtests based on test name cmd line arg; r=ahal https://hg.mozilla.org/integration/autoland/rev/b1b3f1e7320d Add subtest support to raptor; r=ahal
Comment 20•6 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/2a0331afdbef https://hg.mozilla.org/mozilla-central/rev/b1b3f1e7320d
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox62:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
You need to log in
before you can comment on or make changes to this bug.
Description
•