1124689 - Test informant should use structured logs instead of parsing manifests

Status: RESOLVED FIXED

(Testing Graveyard :: Test Informant, defect)

Description

[Andrew Halberstadt [:ahal]]

Currently test-informant does the following steps:

1) Listens for build messages on pulse
2) Downloads and unpacks the associated tests.zip
3) Downloads the associated mozinfo.json file
4) Tries to parse as many manifests as it knows about using the mozinfo data
5) Stores results in the database

This system has three really big limitations:

1) It is not possible to use on suites outside of the tree (like all the gaia tests)
2) It is not possible to use on suites that don't have manifestparser manifests (like reftest)
3) It is not possible to know about configuration that harnesses set at runtime. If this happens, the results could actually be inaccurate.

For these reasons, test-informant should be re-written to use structured logs from the test jobs instead of parsing manifest files itself.

This will be a little bit tricky, since now we need to listen to all test jobs on pulse. We'll need to correlate which test jobs belong to which builds, and sometimes (in the case of chunking) correlate which test jobs belong to which suites.

Ideally we wouldn't need to change the schema of the database, but that might be hard to do in practice. We'll see.

The first step is to run a Test-Informant instance locally. To do that, see:
https://wiki.mozilla.org/Auto-tools/Projects/Test-Informant#Contribute

Comment 1

[Alice Scarpa [:adusca]]

[0] is a first attempt to use structured logs. I just adapted code from [1]. It doesn't deal with chunking yet.

[0] https://github.com/adusca/test-informant/blob/master/informant/worker.py#L98-L114
[1] http://ahal.ca/blog/2014/consume-structured-test-results/

Comment 2

[Ankit Goyal]

I am interested in working on this.

Comment 3

[Andrew Halberstadt [:ahal]]

Sorry Ankit, Alice is already working on this, I forgot to change the assignee. If you are interested in the Test Informant project, you could look into bug 1125863. Please note, I don't consider that a "good first bug", so if you prefer, we can also help you find a better bug somewhere else.

Comment 4

[Andrew Halberstadt [:ahal]]

(In reply to Alice Scarpa[:adusca] from comment #1)
> [0] is a first attempt to use structured logs. I just adapted code from [1].
> It doesn't deal with chunking yet.
> 
> [0]
> https://github.com/adusca/test-informant/blob/master/informant/worker.py#L98-
> L114
> [1] http://ahal.ca/blog/2014/consume-structured-test-results/

This looks great! I just noticed that you knew it doesn't deal with chunking, so ignore my comment about it in the PR (as usual, one step ahead of me :p).

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Test report

Alice's changes were merged into: https://github.com/mozilla/test-informant/tree/structured_log

I ran this for awhile on Friday, and then again today so I could generate a report (attached). It looks like it is working, with a few issues:

1) tests under the marionette suite are not the full path, rather the file (or file + subtest). For some reason other suites display the full path, I imagine this has something to do with a slightly different definition of a "test" under marionette. We'll need to special case it so file paths are stored in the database.

2) there are only half as many browser-chrome tests now. It's possible that they are being filtered out at runtime (and the old reports have been wrong all this time), but this should be investigated.

3) no more mochitest-a11y/chrome. This is a known issue and they should show up under "m-oth" when that suite is added.

Happily mochitest-plain and xpcshell report very similar numbers using this branch. Overall I trust the results from the structured logs more than I do from the old method.

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Test report

Here's a new report with all of the new suites Alice added.

Note that the test id's for these suites are not always a file path relative to the gecko source. Notably the following suites differ:

* gaia-ui-tests: just the file name (these live outside m-c so we'll need to update the report generator as well)

* marionette: file name + subtest, it's not currently possible to get the path out of the logs (see bug 1129493)

* web-platform-tests: relative file paths, but not to the root of gecko source. They still live in m-c, so this is likely just a matter of normalizing them for that suite.

I think we'll need a per-suite method that can look at a log and extract the test path based on the test id and/or the 'path' attribute proposed in bug 1129495.

Comment 7

[James Graham [:jgraham]]

(In reply to Andrew Halberstadt [:ahal] from comment #6)
 
> * web-platform-tests: relative file paths, but not to the root of gecko
> source. They still live in m-c, so this is likely just a matter of
> normalizing them for that suite.

This isn't quite true; they are urls which are normally, but not always, equivalent to relative paths to the test file. In some cases the path can be different (e.g. tets urls ending .worker are derived from filenames ending .worker.js).

I will update wptrunner to log the path directly as planned for marionette.

Comment 8

[Andrew Halberstadt [:ahal]]

Comment on attachment 8559354 [details]
Test report

Erm, seems I accidentally uploaded the same report, but the new report is too big for a bugzilla attachment anyway. So here it is:
http://people.mozilla.org/~ahalberstadt/temp-report.html

Comment 9

[Andrew Halberstadt [:ahal]]

A bunch of things happened since this bug was worked on, most importantly ActiveData becamse available, and the VM that was running the test-informant service got shut down. I am re-writing test-informant as a static web page that queries ActiveData, wip here:
http://people.mozilla.org/~ahalberstadt/test-informant/

Closing this bug WONTFIX.

Comment 10

[Andrew Halberstadt [:ahal]]

Jgraham pointed out that this was fixed, just never got deployed. Sorry Alice!