Open Bug 1501239 Opened Last year Updated 28 days ago

Test validity dashboard

Categories

(Tree Management :: Perfherder, enhancement, P3)

enhancement

Tracking

(Not tracked)

People

(Reporter: igoldan, Unassigned)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

Attachments

(1 file)

Attached file Talos Tests Ownership
This will be a new page on Perfherder.

It should help us find tests which aren’t "valid". It should automate the way we collected data similar to that from the attached Talos Tests Ownership document.

Things we’re interested in seeing:
* Name of test (fairly easy to get)
* Platforms it’s running on (fairly easy to get)
* Branch (fairly easy to get)
* Perfherder tracking bugs linked to it
* Summary on their Bugzilla status
* Various relevant frequencies
  * alerts
  * invalid
  * wontfix
  * fixed
  * average fix time
  * Does it have owners?
  * Is it up to date? Meaning that docs are up to date and tests are relevant
  * How noisy it is - automatically detected
  * Distribution (i.e. bi-modal, noise level) - this might be different for each config/branch

Similar to valid/invalid - maybe we define buckets for tests and categorize them: invalid, noisy, consistent.
How should we figure out the owners? Could we use Moz Wiki docs such as [1] so there's a single source of truth OR rather write the name/email address manually for each test Perfherder collected results for?

[1] https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests
Flags: needinfo?(jmaher)
I also think we need a Bugzilla Components field, with at least one component mentioned there. We agreed that each test should be owned by a team, which in turn has expertise and is concerned with multiple components.
(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #1)
> How should we figure out the owners? Could we use Moz Wiki docs such as [1]
> so there's a single source of truth OR rather write the name/email address
> manually for each test Perfherder collected results for?
> 
> [1] https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests

Using Moz Wiki will require stricter doc formatting, so we don't have issues scrapping those pages.
> * Is it up to date? Meaning that docs are up to date and tests are relevant

This will consist of 2 separate fields: Docs and Relevant? with updated/outdated and yes/no respectively as choices.
I think the dashboard should also present the justification for them: why/when/whom considered docs and relevance are in that specific state. If need be, we can easily track these justifications historically.
mozwiki would work, but that could get out of date, we could have bugs with whiteboard tags- one bug for each test every 6 months or so- then we could query the bugs with the whiteboard tags and determine the status - new/assigned/resolved, etc.
Flags: needinfo?(jmaher)
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #5)
> mozwiki would work, but that could get out of date, we could have bugs with
> whiteboard tags- one bug for each test every 6 months or so- then we could
> query the bugs with the whiteboard tags and determine the status -
> new/assigned/resolved, etc.

If we employ this for determining ownership, state of docs and relevance for the tests, then I think this is a good option.
Making maintenance more transparent is in either case a big plus.
Priority: P1 → P3
Assignee: igoldan → nobody

I think we should scale this back and add to it over time. For the initial view I would propose the following:

  • Test name
  • Platform
  • Branch
  • Tier
  • of alerts

Blocks: 1563748
Priority: P3 → P2
Depends on: 1571411
Priority: P2 → P3
You need to log in before you can comment on or make changes to this bug.