Closed Bug 1478773 Opened 3 years ago Closed 3 years ago

Add the option to aggregate test page results inside a single raptor test


(Testing :: Raptor, enhancement)

Version 3
Not set


(Not tracked)



(Reporter: rwood, Unassigned)



Add an option so that test pages within a raptor test can be aggregated into one overall suite result. For example, to add a new test page to a test like raptor-tp6, currently you would add a new section in the test INI. It's done that way so that each page can be run individually (i.e. raptor-tp6-google-firefox) instead of running the entire test with all the pages. Each test INI section results in a new test output line on treeherder/perfherder.

What if we want to add a new raptor test though that has say 25 test pages? That's way too many data points, so we need the ability to have the results aggregated. Possibly add a new 'aggregate_tests = true' option in the INI. If that is set, when the overall test is run (top level INI) then only one median overall result value will appear on treeherder/perfherder. Again all the replicates for all the pages will be available in the detailed result JSON.
Blocks: 1473365
Depends on: 1478057
Blocks: 1473078
No longer blocks: 1473365
Closing this issue - it's better to keep the existing functionality i.e. keep each tp6 test page separate, because:

- allows each test page to be run individually when investigating a regression or debugging
- isolates each page, if one test page is crashing you can still run the others
- easier and faster for mitmproxy, just playing back the recording for the one test page and not a group of them
- won't miss a regression if one page gets slower but another improves, as that will be separate
- can add/remove pages and not cause a new overall baseline to be generated
Closed: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.