1397698 - Update Treeherder bugzilla components

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Ed Morley [:emorley]]

As part of our effort to improve the triage of Treeherder bugs, we should:
1) Check if we need any additional components (eg for failure classification/auto-classify)
2) Update the descriptions for the existing components (particularly since they still talk about buildbot and not pulse etc)

The current components/descriptions are are follows:

* Perfherder
Bugs regarding Perfherder, which allows for analysis of performance job data.

* Treeherder
A web interface for the management of continuous integration build & test results (more info)

* Treeherder: API
Bugs regarding Treeherder's API, including submitting results from non-buildbot jobs.

* Treeherder: Client Libraries
Bugs regarding the client libraries used to retrieve and submit information to Treeherder.

* Treeherder: Data Ingestion
Bugs regarding Treeherder's ingestion of data sources such as Buildbot, Hg pushlog & Bugzilla. Note: Bugs in the API for non-buildbot job submission belong in 'Treeherder: API'.

* Treeherder: Docs & Development
Bugs regarding Treeherder's Read the Docs pages, local VM environment & development tools/scripts.

* Treeherder: Infrastructure
Bugs regarding the administration and maintenance of the servers and infrastructure running treeherder.mozilla.org.

* Treeherder: Log Viewer
Bugs regarding Treeherder's viewer for the logs of build & test jobs.

* Treeherder: SETA
Issues related to the automatic prioritization and selection of which test jobs are run (<a href="https://treeherder.readthedocs.io/seta.html">more info</a>).

Comment 1

[Cameron Dawson [:camd]]

I think it would be worth adding "Treeherder: Test View" for the TestGroup UI.

Comment 2

[Ed Morley [:emorley]]

Yeah good idea. Which of the following wordings should we use?
* Treeherder: Test View
* Treeherder: Test Centric View
* Treeherder: Test-centric View
* Treeherder: Test Based View
* Treeherder: Test-based View
* Treeherder: Test Summary View
...or something else?

Also I was thinking we should add a component for anything to do with log failure analysis - including log parsing, error summary generation, bug suggestions and human/automatic classification.

For that, which of the following names do you think is best:
* Treeherder: Log Parsing & Classification
* Treeherder: Log Parsing & Analysis
* Treeherder: Log Parsing & Failure Classification
* Treeherder: Log Failure Classification
* Treeherder: Log Failure Analysis
* Treeherder: Failure Classification
* Treeherder: Failure Analysis
...or something else?

Comment 3

[James Graham [:jgraham]]

I like
* Treeherder: Test-based View
and
* Treeherder: Log Parsing & Classification

Comment 4

[Ed Morley [:emorley]]

Attachment: Proposed component changes

How about this?

Comment 5

[Cameron Dawson [:camd]]

That sounds great to me, Ed.  Thanks for putting that patch together.

Comment 6

[Ed Morley [:emorley]]

I've added those two new components (**NB: people will need to manually add them to their watchlist**):

https://bugzilla.mozilla.org/buglist.cgi?product=Tree%20Management&component=Treeherder%3A%20Log%20Parsing%20%26%20Classification&resolution=---&list_id=13768456
https://bugzilla.mozilla.org/buglist.cgi?product=Tree%20Management&component=Treeherder%3A%20Test-based%20View&resolution=---&list_id=13768457

In addition to the description updates in the attachment, I changed the "Treeherder: Log Viewer" component description to:

"""
Bugs regarding the page for viewing build & test job logs. The scrollable log content section in the centre is an iframe holding the Taskcluster Unified Log Viewer - bugs for that should be filed <a href="https://github.com/taskcluster/unified-logviewer">here</a>.
"""

I also cleaned up the suggested reviewers slightly. See the list at:
https://bugzilla.mozilla.org/page.cgi?id=review_suggestions.html#Tree%20Management

Comment 7

[Ed Morley [:emorley]]

I've been going through the ~250 open bugs in the general "Treeherder" component, trying to find other big chunks of bugs that could also be split out into their own category. (Since having 50% of our open bugs in one component makes triage and spotting duplicates harder.)

One possibility is a component to cover {retriggers, backfill, "add new jobs", "custom action", cancellations}. A rough look shows about 30 open bugs over this area, which is only a small fraction of the 250, but still fairly comparable to the number of open bugs in some of the other components we have:
https://bugzilla.mozilla.org/report.cgi?y_axis_field=component&product=Tree+Management&resolution=---&f1=component&o1=anywordssubstr&v1=Treeherder+Perfherder&format=table&action=wrap

Possible names:
- Treeherder: Job Control
- Treeherder: Job Triggering
- Treeherder: Job Triggering & Cancellation    (Probably the most accurate, but a bit verbose)
- Treeherder: Job Scheduling   (Not entirely accurate and possible confusing given the SETA component)
- ...?

Questions:
1) Should we make such a component, and if so, with what name?
2) Are there any other areas that could have their own component? I was hoping for fewer open bugs in the general Treeherder section after adding these new components, however we'll still have about 40% of our bugs there. 
  - The only thing I can think of is a way to split out UI/jobs view bugs from those about the backend; though there's often overlap and it might be confusing for people filing?

Comment 8

[Ed Morley [:emorley]]

I've just updated the "Treeherder: Log Parsing & Classification" description to also include the bug filer tool.

Comment 9

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #7)
> Questions:
> 1) Should we make such a component, and if so, with what name?
> 2) Are there any other areas that could have their own component? I was
> hoping for fewer open bugs in the general Treeherder section after adding
> these new components, however we'll still have about 40% of our bugs there. 
>   - The only thing I can think of is a way to split out UI/jobs view bugs
> from those about the backend; though there's often overlap and it might be
> confusing for people filing?

Comment 10

[Brian Stack [:bstack]]

I would definitely like to have a component for either Job Control or UI. Most of my work would fall into one or the other (but generally in both, so maybe let's only make one new compontent?). As it stands I just dump them in the general component (as you can see) since nothing else seems to fit.

Comment 11

[Cameron Dawson [:camd]]

A couple other ideas that might be worthwhile:

Auth
Filtering    (I saw a few filtering-ish items in "Treeherder")
Classifying  (Including auto-classifying)
Unit tests?  (or maybe this is "docs & development")

A couple ideas for Job Control:
Job State Control  (will anybody but us devs know what "state" means in contrast to "result" ?) :)
Job State
Job Execution      (kill them jobs!)
Job Administration
Job Regulation
Job Start/Stop

That being said, I think that "Job Triggering & Cancellation" is my favorite.  I kept feeling like there was a word that would adequately mean both (Control does) but also make it easy for Joe-Bug-Filter to get it in the right place.  Control can mean lots of different things.  Yeah, it's verbose, but we're not going to need to type it most of the time.

Having a "UI & UX" component would be nice to differentiate items that are mostly back-end or mostly front-end.  "Treeherder" could be used if it's back-end or unclear.  UI & UX if it's at least MOSTLY in the front-end.  This distinction would be helpful when searching for Good First Bugs for someone who wants to work more in Python or JS.  So, in contrast to :bstack, I'd vote for both.  :)

All this being said, it'd be great to keep it at <=15 items, if possible.  Too many and I've found it has diminishing returns.

Comment 12

[Ed Morley [:emorley]]

I've added:

* Treeherder: Job Triggering & Cancellation
"Bugs regarding ingestion of data from sources such as Buildbot, Hg pushlog, Pulse & Bugzilla."
...with the same suggested reviewers as the other components (Cameron, James, myself) plus Brian.


(In reply to Cameron Dawson [:camd] from comment #11)
> A couple other ideas that might be worthwhile:
> 
> Auth

Yeah this one occurred to me too; there are a few of these. I'll have a think.

> Filtering    (I saw a few filtering-ish items in "Treeherder")

Now that it's a menu (and the feature scope reduced), I think there aren't enough of these to warrant a new component perhaps?

> Classifying  (Including auto-classifying)

This goes under the existing "Treeherder: Log Parsing & Classification" :-)

> Unit tests?  (or maybe this is "docs & development")

I've put these under "Docs & development" in the past, though I've also wondered if they belong in the same component as the feature being tested?

> Having a "UI & UX" component would be nice to differentiate items that are
> mostly back-end or mostly front-end.  "Treeherder" could be used if it's
> back-end or unclear.  UI & UX if it's at least MOSTLY in the front-end. 
> This distinction would be helpful when searching for Good First Bugs for
> someone who wants to work more in Python or JS.  So, in contrast to :bstack,
> I'd vote for both.  :)

Yeah good points - I think a UI/Frontend type component is the next most likely one we should add at this point.

Comment 13

[Ed Morley [:emorley]]

I've also created a component for the UI, called "Treeherder: Frontend" [1] since I think this include bugs that aren't necessarily "UI" (eg the build system). It has description:

  "Bugs regarding Treeherder's UI/frontend that aren't covered by a more specific component."

There ended up being 40 bugs that could be moved into "Job Triggering & Cancellation" and 112 into "Frontend". 

Between that and the triage done in recent weeks, this leaves a much more manageable 43 bugs in the general "Treeherder" component. Subtotals by component, here:
https://bugzilla.mozilla.org/report.cgi?y_axis_field=component&product=Tree+Management&resolution=---&f1=component&o1=anywordssubstr&v1=Treeherder+Perfherder&format=table&action=wrap

As such, I think this is a good point to stop :-)


[1] I chose that name out of {frontend, front-end, front end} since (a) hyphenated is only correct for usage as a compound adjective rather than a noun, (b) "Front End" looked strange title case in the component list, (c) there's more prior art for "Frontend" in the name of other Bugzilla components, (d) https://www.designernews.co/comments/127903 / https://dzone.com/articles/blogging-programmers-style

Comment 14

[Ed Morley [:emorley]]

For those using component watching, the new components to add were:
* Treeherder: Frontend
* Treeherder: Job Triggering & Cancellation
* Treeherder: Test-based View

Comment 15

[Ed Morley [:emorley]]

Oh sorry and:
* Treeherder: Log Parsing & Classification