1642362 - Use unique hash ID to define raptor/browsertime tests

Status: NEW

(Testing :: Raptor, task, P2)

Description

[Greg Mierzwinski [:sparky]]

We are constantly dealing with dashboard name changes these days and we will continue to have to devote resources to this into the future (and away from more important work) if we don't do something.

I think we should consider adding a unique hash to our tests to overcome this issue. Here's the flow that I am thinking:

1. Make an X character unique hash (of the test url or just a random hash).
2. Associate this hash with a raptor/btime taskcluster task within a new field, i.e. amazon-search raptor/btime fenix would have one.
   1. We would allow tasks to have duplicates (in case of migrations).
3. Make a new field in perfherder to hold this hash.
4. Use the hash to find all data for a given test.

With this process, we can change names as much as we want without having to worry about breaking the dashboards.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This sounds like the test signatures that we already have in Perfherder. We should provide some recent use cases that any solution would need to support. Sometimes the name changes are temporary regressions (such as dropping the '-live' suffix), which we should not spend additional effort to support, we should just aim to identify and fix as soon as possible. Sometimes the changes are significant, such as the expansion of the YouTube playback tests, which may not easily map to existing tests.

The valid use cases that come to mind are when we decide to rename a test (such as dropping '-cold' suffix in favour of extraOptions), or if we dropped the 'raptor-' prefix (we decided not to do this). Another would be when the platform changes, which is mostly outside of our influence (such as adding '-shippable', or opt/pgo changes). I'm not sure having a unique test hash would help here.

Comment 2

[Ionuț Goldan [:igoldan]]

I'm quoting Kyle's reply from the related email thread, as it's a valuable alternate solution:

I believe Gregs problem has been solved by the Firefox Health Dashboard by using (JSON) expressions instead of hash identifiers. The strategy is to define the equivalence class using an expression instead of assigning hashes. This leaves the data simple, and provides a platform for defining other equivalence classes for other problems without changing the data.

To use expressions we need the perfherder API to support simple filtering, or we can use the BigQuery extract (SQL already has expressions), or we can do what the Heath dashboards does: Download all signatures, filter by expression, and request by signature.

I included an example expression below. This format is easy to serialize and parse for communicating with other systems about what data is required.

Here is an expression for Google Tp6 on Chromium

{
or: [{
and: [{ lt: { push_timestamp: { date: '2019-09-01' } } }, {
eq: {
suite: 'raptor-tp6-google-chrome',
framework: 10,
repo: 'autoland',
},
}],
}, {
eq: {
suite: 'raptor-tp6-google-chromium',
framework: 10,
repo: 'autoland',
},
}],
}

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

@andrej this is related to providing friendly test names so graphs are not affected by platform name changes