1501241 - [meta] Make states more manageable

Status: NEW

(Tree Management :: Perfherder, enhancement, P3)

Description

[Ionuț Goldan [:igoldan]]

As we're speaking, Perfherder exposes a big amount of alert & alert summary states which tend to be confusing and make the learning curve for beginners more steep.

This is the current list we have:
* untriaged
* downstream
* reassign
* invalid
* improvement
* investigating
* wontfix
* fixed
* backedout
* confirming

The transition between all these states is tricky to follow, so the solution we see is to reduce the number of states to just 3 (say *untriaged/investigating/resolved*) and turn the rest of them into simple and intuitive tags.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

we also want to include the ability to add better tags, specifically I am thinking about reasons:
infra
tooling
test
product

Comment 2

[Ionuț Goldan [:igoldan]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #1)
> we also want to include the ability to add better tags, specifically I am
> thinking about reasons:
> infra
> tooling
> test
> product

Yes, we can make a custom tag designer feature similar to that from Github. That way, if we discover we need some new tags, we create them from Perfherder and use them directly, without doing any source code commit.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

can we outline what is frontend and backend?  will we need to change apis?

Comment 4

[Ionuț Goldan [:igoldan]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3)
> can we outline what is frontend and backend?
Will do.

> will we need to change apis?
I'm not entirely sure, but I think it's possible to simply use the existing backend APIs unaltered.
For the Github-like tag designer, we do need at least one new backend API.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

maybe the question should be, will removing existing states break any APIs?

Comment 6

[Ionuț Goldan [:igoldan]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #5)
> maybe the question should be, will removing existing states break any APIs?

I think we are breaking APIs if we don't migrate the rest of states to tags. That's because alerts validate their state when operated upon.

Comment 7

[Ionuț Goldan [:igoldan]]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #6)
> (In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #5)
> > maybe the question should be, will removing existing states break any APIs?
> 
> I think we are breaking APIs if we don't migrate the rest of states to tags.
> That's because alerts validate their state when operated upon.

There may be other backend places were this may break, though I have to look for them more closely.

Comment 8

[William Lachance (:wlach)]

I'm not sure if the downstream/reassigned states map very cleanly to tags, as they also depend on the "related" field.

One alternative would be to move a system that's closer to bugzilla, which is the system we are interacting with and thus people would already have a mental model for.

In that case, we would have the following fields:

* status: untriaged/investigating/resolved
* resolution (only if resolved): improvement/wontfix/fixed/...
* whiteboard/keywords (could be mapped to tags): infra/tooling/test/confirming

I think separating out resolution from other fields might be a good idea, as it will make analysis of the database easier.

Comment 9

[Ionuț Goldan [:igoldan]]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #8)
> I'm not sure if the downstream/reassigned states map very cleanly to tags,
> as they also depend on the "related" field.
> 
> One alternative would be to move a system that's closer to bugzilla, which
> is the system we are interacting with and thus people would already have a
> mental model for.
> 
> In that case, we would have the following fields:
> 
> * status: untriaged/investigating/resolved
> * resolution (only if resolved): improvement/wontfix/fixed/...
> * whiteboard/keywords (could be mapped to tags):
> infra/tooling/test/confirming
> 
> I think separating out resolution from other fields might be a good idea, as
> it will make analysis of the database easier.

I think this may be a good alternative.
Currently, the resolutions are:
* improvement
* fixed
* wontfix
* invalid
* downstream
* backout
* reassign

As long as this list doesn't get bigger than that from Bugzilla, I think it is manageable.
:jmaher what do you think?

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

:igoldan, how often do we resolve alerts?  Are most left in investigating status?

As for resolutions, while those are interesting for bugs and would make sense here- I am more concerned with adding generic support for dynamic fields, specifically infra, tooling, test changes so we can filter out issues not related to product changes.

I am not sure if this means we would have status/resolution as different hard coded fields, then in addition a set of tags- why not just have tags in general?

Comment 11

[William Lachance (:wlach)]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #10)
> I am not sure if this means we would have status/resolution as different
> hard coded fields, then in addition a set of tags- why not just have tags in
> general?

As I said in comment 8, tags don't encapsulate the duplicate/reassigned statuses, which also depend on some "related summary/alert" metadata which we store in the db. So unless you want to just get rid of that metadata (I don't think we do?), we need something that ties the duplicate/reassigned id to a "status". The only other alternative I can think of would be "special case" tags which the UI would key off of, and that sounds like a giant mess.

In general I think mirroring what bugzilla does is a good strategy here, it's the system that perfherder needs to interact with in most cases.

It also occurs to me that the "notes" field that perfherder already has is pretty similar to bugzilla's whiteboard, which we already use for annotating this kind of information (e.g. `[good first bug]`). We might be able to skip using tags just by just getting the sheriffs to apply some conventions on how they tag things (e.g. "write [infra] in the notes field if the regression was infra-related).

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

as long as we can easily query the notes field and show them in the popup.  Maybe that is something we can parse from there.

Maybe there are 3 states- new, investigating, resolved and when resolved is hit, it requires a different state (i.e. downstream, reassigned) - but this isn't available by default, nor is it in the normal filters, etc.  any way to reduce our states/actions would be a win over today.

Comment 13

[Ionuț Goldan [:igoldan]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #10)
> :igoldan, how often do we resolve alerts?
Well, most of the alerts we have are the resolved ones: majority are wontfixes, then we have fixes & improvements, backouts, invalids.

> Are most left in investigating status?
Not really. They stay so only for a while, until there's a resolution on Bugzilla. Basically, if we consider the average fix time for filed perf bugs, then that's the lifespan of an summary in investigating state.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

ok, I see a lot of the alerts adjusted in state- but there are many in investigation state which do not have bugs- in fact there are more in investigation state than fixed or wontfix- this is why I asked the question.

If we can make a solution easier to ensure all alerts get a final state, I would be happier

Comment 15

[Ionuț Goldan [:igoldan]]

Summaries with investigating state but with missing bugs are accumulated leftovers: either their alerts were acknowledged because they weren't investigated for a long time OR there was way too much noise to look in OR they were simply forgotten when bulks of rest of alerts were reassigned/downstreamed/marked otherwise.

Comment 16

[Ionuț Goldan [:igoldan]]

"investigating" state is a misnomer in our case. Most of the accumulated leftovers aren't investigated by neither Perf sheriffs nor regression authors. We could actually use a new tag like "forgotten" OR "inactive" OR "archived" or something appropriate in this regard, to make a clear distinction.

Comment 17

[Ionuț Goldan [:igoldan]]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #11)
> It also occurs to me that the "notes" field that perfherder already has is
> pretty similar to bugzilla's whiteboard, which we already use for annotating
> this kind of information (e.g. `[good first bug]`). We might be able to skip
> using tags just by just getting the sheriffs to apply some conventions on
> how they tag things (e.g. "write [infra] in the notes field if the
> regression was infra-related).

I would like to also tag at alert scope, not just at summary scope.
I especially require the confirming/confirmed set of tags here.