1399923 - Add ability to search for a test name and identify which chunk it belongs to for that push

Status: RESOLVED INVALID

(Tree Management Graveyard :: Treeherder: Test-based View, enhancement)

Description

[Cameron Dawson [:camd]]

I wonder if it's possible for this UI to initiate a search on Active Data to find which chunk a test name exists for in a given push.  This came up in a discussion on dev-platform.  This would help users be able to retrigger a particular test for intermittent testing.

Comment 1

[Ed Morley [:emorley]]

I don't think this is something Treeherder should do. The mach fuzzy try tool seems like a better fit for this.

Comment 2

[Ed Morley [:emorley]]

In fact the test based UI won't have the exact test names, only the manfiest directories, so won't be able to do this anyway.

Comment 3

[Kyle Lahnakoski [:ekyle]]

Here is an example query that shows the chunk number for each run/build combo in the past day.  

https://activedata.allizom.org/tools/query.html#query_id=4HHuBgDu

> {
>     "from":"unittest",
>     "select":[
>         {"aggregate":"count"},
>         {"value":"action.start_time","aggregate":"max"}
>     ],
>     "groupby":[
>         "run.suite",
>         "run.chunk",
>         "result.test",
>         "build.platform",
>         "build.type",
>         "run.type"
>     ],
>     "where":{"and":[
>         {"eq":{"build.branch":"mozilla-inbound"}},
>         {"prefix":{"run.suite":"moch"}},
>         {"gt":{"action.start_time":{"date":"today-day"}}},
>         {"regex":{"result.test":".*browser_623779.js.*"}}
>     ]},
>     "limit":1000
> }

The ActiveData query tool is HTML/Javascript that POSTs tothe ActiveData query endpoint. Open the debugger to watch its network requests.

ActiveData is an API to Elasticsearch that returns JSON documents, so we can use curl

> curl  http://activedata.allizom.org/query -XPOST -v -d $'{"from":"unittest",\n"select":[\n{"aggregate":"count"},\n{"value":"action.start_time","aggregate":"max"}\n],\n"groupby":[\n"run.suite",\n"run.chunk",\n"result.test",\n"build.platform",\n"build.type",\n"run.type"\n],\n"where":{"and":[\n{"eq":{"build.branch":"mozilla-inbound"}},\n{"prefix":{"run.suite":"moch"}},\n{"gt":{"action.start_time":{"date":"today-day"}}},\n{"regex":{"result.test":".*browser_623779.js.*"}}\n]},\n"limit":1000\n}'

The above examples return a default format, which may not suit your application.  There are three formats to choose from: https://github.com/klahnakoski/ActiveData/blob/dev/docs/jx_tutorial.md#format-clause

Comment 4

[Cameron Dawson [:camd]]

It has the filename that the test ran in, which would certainly be enough to locate the chunk that filename ran in.  So it would be really simple to add a way to query active data to get this info.  I guess I don't see this as a bad fit for the Test-based UI since you're already in that context of a push and the tests.

Comment 5

[Ed Morley [:emorley]]

The use case here is:
"I want to push to try and only need to run certain tests, so need to figure out what job/chunk number to use in my try syntax, so I can do the right thing and save resources"

In this situation:
* the user doesn't have an existing try push (so won't be looking at treeherder)
* needs to generate a try string (so will already be interacting with the mach tool)

Which is why I really don't think the right place for this is treeherder.

Comment 6

[Cameron Dawson [:camd]]

Oh I see, I was misunderstanding his use-case.  When I read Marco's email:

> When I need to retrigger a mochitest-browser test multiple times (to investigate an intermittent)

I took that as he had a push already and wanted to retrigger a suspect test multiple times.  But I can see he's likely doing (as you say) several pushes to try to test his fix(es).

So while it's totally possible to do here, it wouldn't be the right answer to this use-case.