Test informant should use structured logs instead of parsing manifests

RESOLVED FIXED

Status

Testing
Test Informant
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: ahal, Assigned: adusca, Mentored)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

User Story

Hello contributor!

This is a mentored project for Test Informant. For more information on Test Informant see:
https://wiki.mozilla.org/Auto-tools/Projects/Test-Informant

This project spans multiple bugs. It requires a basic understanding of python, version control (+github) and a fair amount of time.

Please comment below or contact ahal in #ateam on irc.mozilla.org if you are interested in taking on this project or if you have any questions.

Attachments

(2 obsolete attachments)

(Reporter)

Description

3 years ago
Currently test-informant does the following steps:

1) Listens for build messages on pulse
2) Downloads and unpacks the associated tests.zip
3) Downloads the associated mozinfo.json file
4) Tries to parse as many manifests as it knows about using the mozinfo data
5) Stores results in the database

This system has three really big limitations:

1) It is not possible to use on suites outside of the tree (like all the gaia tests)
2) It is not possible to use on suites that don't have manifestparser manifests (like reftest)
3) It is not possible to know about configuration that harnesses set at runtime. If this happens, the results could actually be inaccurate.

For these reasons, test-informant should be re-written to use structured logs from the test jobs instead of parsing manifest files itself.

This will be a little bit tricky, since now we need to listen to all test jobs on pulse. We'll need to correlate which test jobs belong to which builds, and sometimes (in the case of chunking) correlate which test jobs belong to which suites.

Ideally we wouldn't need to change the schema of the database, but that might be hard to do in practice. We'll see.

The first step is to run a Test-Informant instance locally. To do that, see:
https://wiki.mozilla.org/Auto-tools/Projects/Test-Informant#Contribute
(Reporter)

Updated

3 years ago
Depends on: 1124720
(Assignee)

Comment 1

3 years ago
[0] is a first attempt to use structured logs. I just adapted code from [1]. It doesn't deal with chunking yet.

[0] https://github.com/adusca/test-informant/blob/master/informant/worker.py#L98-L114
[1] http://ahal.ca/blog/2014/consume-structured-test-results/

Comment 2

3 years ago
I am interested in working on this.
(Reporter)

Comment 3

3 years ago
Sorry Ankit, Alice is already working on this, I forgot to change the assignee. If you are interested in the Test Informant project, you could look into bug 1125863. Please note, I don't consider that a "good first bug", so if you prefer, we can also help you find a better bug somewhere else.
Assignee: nobody → alicescarpa
(Reporter)

Updated

3 years ago
Status: NEW → ASSIGNED
Summary: [Tracking] test informant should use structured logs instead of parsing manifests → Test informant should use structured logs instead of parsing manifests
(Reporter)

Comment 4

3 years ago
(In reply to Alice Scarpa[:adusca] from comment #1)
> [0] is a first attempt to use structured logs. I just adapted code from [1].
> It doesn't deal with chunking yet.
> 
> [0]
> https://github.com/adusca/test-informant/blob/master/informant/worker.py#L98-
> L114
> [1] http://ahal.ca/blog/2014/consume-structured-test-results/

This looks great! I just noticed that you knew it doesn't deal with chunking, so ignore my comment about it in the PR (as usual, one step ahead of me :p).
(Reporter)

Comment 5

3 years ago
Created attachment 8558758 [details]
Test report

Alice's changes were merged into: https://github.com/mozilla/test-informant/tree/structured_log

I ran this for awhile on Friday, and then again today so I could generate a report (attached). It looks like it is working, with a few issues:

1) tests under the marionette suite are not the full path, rather the file (or file + subtest). For some reason other suites display the full path, I imagine this has something to do with a slightly different definition of a "test" under marionette. We'll need to special case it so file paths are stored in the database.

2) there are only half as many browser-chrome tests now. It's possible that they are being filtered out at runtime (and the old reports have been wrong all this time), but this should be investigated.

3) no more mochitest-a11y/chrome. This is a known issue and they should show up under "m-oth" when that suite is added.

Happily mochitest-plain and xpcshell report very similar numbers using this branch. Overall I trust the results from the structured logs more than I do from the old method.
(Reporter)

Comment 6

3 years ago
Created attachment 8559354 [details]
Test report

Here's a new report with all of the new suites Alice added.

Note that the test id's for these suites are not always a file path relative to the gecko source. Notably the following suites differ:

* gaia-ui-tests: just the file name (these live outside m-c so we'll need to update the report generator as well)

* marionette: file name + subtest, it's not currently possible to get the path out of the logs (see bug 1129493)

* web-platform-tests: relative file paths, but not to the root of gecko source. They still live in m-c, so this is likely just a matter of normalizing them for that suite.

I think we'll need a per-suite method that can look at a log and extract the test path based on the test id and/or the 'path' attribute proposed in bug 1129495.
Attachment #8558758 - Attachment is obsolete: true
(In reply to Andrew Halberstadt [:ahal] from comment #6)
 
> * web-platform-tests: relative file paths, but not to the root of gecko
> source. They still live in m-c, so this is likely just a matter of
> normalizing them for that suite.

This isn't quite true; they are urls which are normally, but not always, equivalent to relative paths to the test file. In some cases the path can be different (e.g. tets urls ending .worker are derived from filenames ending .worker.js).

I will update wptrunner to log the path directly as planned for marionette.
(Reporter)

Comment 8

3 years ago
Comment on attachment 8559354 [details]
Test report

Erm, seems I accidentally uploaded the same report, but the new report is too big for a bugzilla attachment anyway. So here it is:
http://people.mozilla.org/~ahalberstadt/temp-report.html
Attachment #8559354 - Attachment is obsolete: true
(Reporter)

Comment 9

2 years ago
A bunch of things happened since this bug was worked on, most importantly ActiveData becamse available, and the VM that was running the test-informant service got shut down. I am re-writing test-informant as a static web page that queries ActiveData, wip here:
http://people.mozilla.org/~ahalberstadt/test-informant/

Closing this bug WONTFIX.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
(Reporter)

Comment 10

2 years ago
Jgraham pointed out that this was fixed, just never got deployed. Sorry Alice!
Resolution: WONTFIX → FIXED
You need to log in before you can comment on or make changes to this bug.