1757663 - when data is annotated against a simplified intermittent bug, create a tool to get error messages

Status: RESOLVED FIXED

(Testing :: General, task)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

one problem of having a single bug per test case is that multiple errors can occur and as a developer trying to reproduce an issue to fix the test case it is hard to know what error to fix.

we need a tool that would work like:
./mach intermittent-failures netwerk/test/unit/test_odoh.js

and it would output a summary of:

[test_connection_closed - [test_connection_closed : 198] Checking status - 2152398878 == 0]
 * total: 7
 * android-em-7-0-x86_64-lite-qr: 4
  ** opt: 2
  ** debug: 2
* android-em-7-0-x86_64-qr: 2
  ** debug: 1
  ** opt: 1
* android-em-7-0-x86_64-shippable-lite-qr: 1
  ** opt: 1
[testODoHConfig - [testODoHConfig : 64] "undefined" == true]
* total: 1
* macosx1100-64-shippable-qr: 1
 ** opt: 1

preferably we could link to the failures so logs could be referenced.

I imagine this could be done by querying treeherder for failure instances, then having mozci inspect artifacts for the failing tasks and harvest error messages.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I spent some time in redash doing a query that should get this:

select
  jl.job_id,
  fl.test,
  fl.message
from
  bug_job_map bjm,
  job_log jl,
  failure_line fl
where
  bjm.bug_id in (1710170,1682215,1687343,1689428,1687896) and
  bjm.created>'2022-02-01' and
  bjm.job_id=jl.job_id and
  jl.name='errorsummary_json' and
  jl.id=fl.job_log_id

a few hardcoded things in the query, specifically the bug list- I wanted an example of a test bc6 that had multiple suggestions so we could see what data is returned. I would also like to return the platform, build, variant information as well as a date.

My conclusion is that we could return a bit more information from the query, then parse it into something useful for the end user. I am not sure if there is a ./mach command which does a redash query, then this would be fairly quick to implement given the prior art.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

doing a query to redash isn't so straightforward, prior art examples don't work. There is a treeherder api for the intermittent failure view that has basically everything we want:
https://treeherder.mozilla.org/api/failuresbybug/?startday=2022-02-24&endday=2022-03-03&tree=trunk&bug=1708856

I will focus more on the existing treeherder api - maybe we could have a toggle view in treeherder itself that gives us what we want.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

this has been fixed, ./mach supports this as well as intermittent failure view has a drop down to view the specific errors and filter them out of the list and graph.