1584940 - Make measurement units in perf tests mandatory

Status: NEW

(Testing :: Performance, enhancement, P3)

Description

[Ionuț Goldan [:igoldan]]

Make the unit field from bug 1590346 mandatory.

Comment 1

[Ionuț Goldan [:igoldan]]

When we'll 1st push this to Try, we'll get a lot of perf test failures, from those which don't provide the unit field or one of its predefined values. We'll need to file bugs for each of the problematic tests, so they provide that measurement unit. These bugs must block bug 1584940.

Once all of them are resolved, can this bug be deployed to production.

Comment 2

[Octavian Negru]

Attachment: Bug 1584940 - Make measurement units in perf tests mandatory

Comment 3

[Octavian Negru]

I have some questions related to this Bug.
The following subtests have different measurement units (found in testing/talos/talos.json):

suites:
        g1: 
            subtests: 
                tp5o_scroll:  unit = '1/FPS'
                glterrain:      unit = 'frame interval'
        g4: 
            subtests: 
                glvideo:                         unit = 'ms'
                displaylist_mutate:       unit = 'ms'
                rasterflood_svg:           unit = 'ms'
                rasterflood_gradient:   unit = 'score'
        svgr: 
            subtests: 
                tsvgx:                     unit = 'ms'
                tsvgr_opacity:       unit = 'ms'
                tart:                       unit = 'ms'
                tscrollx:                 unit = '1/FPS'
                tsvg_static:            unit = 'ms'

For most of the test suites, the measurement unit is the same.
Talos and Raptor share a convenient mechanism for calculating the suite's geomean.
Therefore, the test suite can have a score (geomean), because all suite's subtests share the same measurement unit.
However, for the above suites/subtests this does not apply, because measurement units are different.

1.Is there any logic to calculate the suite score (geomean) for such subtests?
2.If yes, what will be the displayed measurement unit for the test suite, given the existence of different measurement units?

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Octavian Negru from comment #3)

1.Is there any logic to calculate the suite score (geomean) for such subtests?
2.If yes, what will be the displayed measurement unit for the test suite, given the existence of different measurement units?

The geometric mean allows us to track a single figure that represents potentially different items, so I believe there is still value in calculating this. We're primarily using this to detect regressions, where the unit is less relevant. I would suggest that whenever we calculate a geometric mean we set the unit to 'geomean'.

Comment 5

[Ionuț Goldan [:igoldan]]

2019/Q4 work for bug 1520739 is considered done. I'm detaching this dependency from it.