Closed Bug 1462776 Opened 6 years ago Closed 6 years ago

Add subtest support to raptor

Categories

(Testing :: Raptor, enhancement)

Version 3
enhancement
Not set
normal

Tracking

(firefox62 fixed)

RESOLVED FIXED
mozilla62
Tracking Status
firefox62 --- fixed

People

(Reporter: rwood, Assigned: rwood)

References

Details

Attachments

(2 files)

Right now raptor tests can only have one single test page (i.e raptor-firefox-tp6 only is currently testing the amazon test page).

Add subtest support so that a raptor test can run against several test pages (i.e. so I can add facebook, google, and youtube test pages to the same raptor-firefox-tp6 test). Each test page will be listed inside the raptor-test.ini.
Comment on attachment 8977096 [details]
Bug 1462776 - Run single raptor test or multiple subtests based on test name cmd line arg;

https://reviewboard.mozilla.org/r/245168/#review251766

::: testing/raptor/raptor/manifest.py:128
(Diff revision 1)
> -                tests_to_run = [next_test]
> +            tests_to_run.append(next_test)
> -                break
> +            break

What happens if there are subtests with the same name in different manifests? This looks like it'll run a random test, would it be a good idea to remove the `break` and run all subtests with that name instead?
Attachment #8977096 - Flags: review?(ahal) → review+
(In reply to Andrew Halberstadt [:ahal] from comment #4)

> What happens if there are subtests with the same name in different
> manifests? This looks like it'll run a random test, would it be a good idea
> to remove the `break` and run all subtests with that name instead?

Thanks. I think we'll just have to make sure each subtest name is unique. Don't want to run all subtests with that name because in production you only want to run specific tests in specific jobs. Trying to avoid requiring a separate raptor configuration file (like talos has - talos.json - that tells which subtests belong in which suites).
(In reply to Robert Wood [:rwood] from comment #5)
> (In reply to Andrew Halberstadt [:ahal] from comment #4)
> 
> > What happens if there are subtests with the same name in different
> > manifests? This looks like it'll run a random test, would it be a good idea
> > to remove the `break` and run all subtests with that name instead?
> 
> Thanks. I think we'll just have to make sure each subtest name is unique.
> Don't want to run all subtests with that name because in production you only
> want to run specific tests in specific jobs. Trying to avoid requiring a
> separate raptor configuration file (like talos has - talos.json - that tells
> which subtests belong in which suites).

Actually maybe a solution is when a matching subtest name is found, check it's manifest name and it should match the subtest name itself, if it doesn't then it's a subtest with the same name but from a different manifest and shouldn't be included.
> Actually maybe a solution is when a matching subtest name is found, check
> it's manifest name and it should match the subtest name itself, if it
> doesn't then it's a subtest with the same name but from a different manifest
> and shouldn't be included.

No wait that doesn't work as the manifest name doesn't necessarily match the manifest name. There's really no good solution for this short of requiring either a specific file that says what subtests belong to which suites.
Blocks: 1460741
Comment on attachment 8980061 [details]
Bug 1462776 - Add subtest support to raptor;

https://reviewboard.mozilla.org/r/246222/#review252514
Attachment #8980061 - Flags: review?(ahal) → review+
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2a0331afdbef
Run single raptor test or multiple subtests based on test name cmd line arg; r=ahal
https://hg.mozilla.org/integration/autoland/rev/b1b3f1e7320d
Add subtest support to raptor; r=ahal
https://hg.mozilla.org/mozilla-central/rev/2a0331afdbef
https://hg.mozilla.org/mozilla-central/rev/b1b3f1e7320d
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: