1110240 - Scope MVP requirements for an intermittent bug filer

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Ed Morley [:emorley]]

Discussed at the Portland work week:
https://etherpad.mozilla.org/ateam-pdx-sheriff-automation

Comment 1

[Ed Morley [:emorley]]

Current manual steps:

1) Decide which failure line corresponds to the root cause of the failure.
2) Interpret that failure line, to ascertain what {test file, assertion, crash top frame} is the item of interest. (eg for an assertion/crash the test name may not be the most relevant thing - the bug will likely want to be filed against the component for the crash/assertion).
3) Search for related/potential duplicate bugs, and if appropriate modify the summary of an existing bug to make it match against the failure lines here, then skip the remaining steps here.
4) If the failure was in a specific test, check to see if the test was recently created/modified, by checking |hg log|. Typical way to get this is to MXR filename search the correct repo (mozilla-central vs gaia vs ...) and follow the |hg/git log| link from the results page.
5) If so, and the change seems relevant: back out the offending bug, comment in that bug & skip the remaining steps.
6) Pick correct product/component for that item of interest, using either prior bugs mentioned in |hg log| for the file or best guess based on location in the repo.
7) Using prior relevant bugs (from |hg log|) or else best guess based on location in the repo, CC developers on the bug.
8) If |hg log| shows another bug recently made changes that may be relevant (but not recent/confident enough to back out), then mark the new bug as blocking the prior bug.
9) Compose summary that will match future bug suggestion searches, using knowledge of the algorithm that determines the search term(s). See below for example summary formats, depending on the type of failure.
10) Add keyword "intermittent-failure".
11) Set OS/Platform based on the job details (needs mapping, since the names aren't the same).
12) Set version according to the repo where the failure occurred.
13) Compose the bug description/comment:
  a) Basic job details (machine name, start/end times, repo, revision, job name).
  b) Link to the logviewer for that job.
  c) Excerpt of the log:
     * pick which failure lines are relevant.
     * include context leading up to those failure lines.
     * strip out spammy content (eg data URI screenshot dumps).
  d) (New) Link to the treeherder page for that push/job.
  e) (New) Link to DXR for the file
  f) (New) Who used the bug filer (since filed using bot account).
14) If crash:
  a) set severity to critical
  b) add keyword "crash"
  c) add top frame of stack to "crash signature" field
15) If assertion, add keyword assertion
16) If memory leak, add keyword mlk
17) Classify the failed job in question on Treeherder, using the newly created bug number.


Complications:

* When trying to CC users to the bug, using the author from hg/git doesn't always work, since they may use a different email address in Bugzilla. Workaround is to open the bug referenced in the commit message, and use the assignee email (if set) instead.
* Not all failures occur in files in mozilla-central. eg gaia-ui-test failures filenames are instead present in gaia-central. Same for release engineering failures etc.
* Each Bugzilla product has different versioning. eg Core uses mozilla34, Firefox uses firefox34, B2G/releng/... something else etc.
* The test filepath referenced in the failure line almost always isn't a path relative to the root of the source directory, and in some cases is entirely different, even with the common prefix removed.
* The summary used varies depending on the type of failure/by the bug filer using their experience to guess as to what parts are relevant. Example summary formats:
  For 'typical' test failures:
  Intermittent <test filename, or partial path if not clear> | <failure message>
  ...for crashes:
  Intermittent crash in <test filename, or partial path if not clear> [@ <crash_signature>] ("<any additional assertions or things of note")
  ...if shutdown/awkward crash something like :
  Intermittent <OS> talos tp5 crash at <URL> [@ <crash_signature>]
  Intermittent <OS> shutdown crash [@ <crash_signature>]
  ...for leaks
  Intermittent <test suite name> leak of <size> bytes (<summary of leaking objections shown in annotated summary box, albeit just the top N alphabetically listed>)
* There may be several unrelated failures that occurred in the job, so:
  ** More than one bug may need to be filed
  ** Not all failure lines may want to be included in the filed bug descriptions
  ** The job may need classifying with several bug numbers, not all of which will have been handled by the bug filer.

Comment 2

[Ed Morley [:emorley]]

Minimum viable product:

* Link/button shown on the job details panel's failure summary tab: "File new bug".
* This links to a Treeherder hosted bug filer form (target=_blank), that can be used by any user logged into Treeherder with Persona.
* The form's fields are pre-populated using the job/log info as well as data from other sources, that can be corrected as required by the user before submission.
* The form:
  ** Displays the error lines excerpt for the job, capped at N lines. Next to each error line is:
    *** a checkbox, which indicates whether to include that line in the pre-filled bug summary + the bug description.
    *** a link to MXR/DXR file search for the extracted filename (if found; for now using hardcoded regexes)
  ** Has radio buttons for the common Bugzilla products used ({Firefox, Firefox for Android, Firefox OS, Core, Toolkit, Release Engineering}) and a dropdown containing all products, under "Other". The product/component lists are fetched using Bugzilla's REST API, but the common products given radio buttons are hardcoded.
  ** Once a product is selected manually by the user, the component field is populated with the components within that product, one of which must then be selected manually.
  ** Displays platform/OS dropdowns that have been pre-populated using the job os/platform and a hardcoded mapping between Treeherder and Bugzilla's names.
  ** Keywords
  ** Summary
  ** Bug description
  ** Version
  ** Whilst future versions will do more, for now if the user needs anything else (eg needinfo), they can just add these after the bug has been created.
* Submitting the form sends the revised fields to a new Treeherder API endpoint, which:
  ** checks the user is authenticated
  ** possibly validates the fields (though both the Bugzilla API and the form will have performed validation, so this may be redundant)
  ** uses a new shared "[e-mail]" account (whose credentials are kept server side) to create a bug using Bugzilla's native REST API.
  ** If bug creation was successful, adds the failure classification to the job using the bug number response from the Bugzilla API.
* After submission, the form provides feedback as to bug number or failure message, and the page is then closed by the user.


Stretch goal:

* In addition to the fields above, also supporting editing the following (since otherwise 95% of the time the MVP above will result in a bug that the user has to manually edit after):
  ** CC list (this either needs to autocomplete using Bugzilla's API, or else we'll have to handle incorrect entries)
  ** Blocks/depends
  ** Crash signature
* Support not just test filename type failure lines, but those for crashes/memory leaks/whole line too.

Comment 3

[Ed Morley [:emorley]]

Missed the descriptions for these:

>   ** Keywords

The keyword "intermittent-failure" is listed and mandatory. Stretch goal: The user can optionally add additional keywords, with possibly the common ones ({crash, mlk, ...}) being listed as quick-access checkboxes.

>   ** Summary

This is generated using the checkboxes next to the failure lines mentioned in comment 2.

>   ** Bug description

Ditto. Plus additional information extracted from the job metadata.

>   ** Version

Likely hardcode this for now (eg use Trunk or choose the last value in the list, since it's normally the most recent).

Comment 4

[Ed Morley [:emorley]]

Wes, is there anything that needs to be added or removed from the list above? I'd like to keep the feature set for this v1.0 MVP as short as possible, and expect it to only cover 80% of the cases.

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

MVP might omit the radio buttons for common products in favor of the "All Products" dropdown, but this looks like a good list.

I'm unsure if you can needinfo from the original filing submission (the actual new bug page on bugzilla itself does not have a needinfo field), so that part might need to be dropped from the "Whilst future versions" section if it's not technically feasible.

Comment 6

[Ed Morley [:emorley]]

(In reply to Wes Kocher (:KWierso) from comment #5)
> MVP might omit the radio buttons for common products in favor of the "All
> Products" dropdown, but this looks like a good list.

Yeah agree :-)

> I'm unsure if you can needinfo from the original filing submission (the
> actual new bug page on bugzilla itself does not have a needinfo field), so
> that part might need to be dropped from the "Whilst future versions" section
> if it's not technically feasible.

It's there, it's just hidden under "Set bug flags".

Cool, I think we're good to close this out now :-)