Closed Bug 1124689 Opened 9 years ago Closed 8 years ago

Test informant should use structured logs instead of parsing manifests

Categories

(Testing Graveyard :: Test Informant, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ahal, Assigned: adusca, Mentored)

References

Details

User Story

Hello contributor!

This is a mentored project for Test Informant. For more information on Test Informant see:
https://wiki.mozilla.org/Auto-tools/Projects/Test-Informant

This project spans multiple bugs. It requires a basic understanding of python, version control (+github) and a fair amount of time.

Please comment below or contact ahal in #ateam on irc.mozilla.org if you are interested in taking on this project or if you have any questions.

Attachments

(2 obsolete files)

Currently test-informant does the following steps:

1) Listens for build messages on pulse
2) Downloads and unpacks the associated tests.zip
3) Downloads the associated mozinfo.json file
4) Tries to parse as many manifests as it knows about using the mozinfo data
5) Stores results in the database

This system has three really big limitations:

1) It is not possible to use on suites outside of the tree (like all the gaia tests)
2) It is not possible to use on suites that don't have manifestparser manifests (like reftest)
3) It is not possible to know about configuration that harnesses set at runtime. If this happens, the results could actually be inaccurate.

For these reasons, test-informant should be re-written to use structured logs from the test jobs instead of parsing manifest files itself.

This will be a little bit tricky, since now we need to listen to all test jobs on pulse. We'll need to correlate which test jobs belong to which builds, and sometimes (in the case of chunking) correlate which test jobs belong to which suites.

Ideally we wouldn't need to change the schema of the database, but that might be hard to do in practice. We'll see.

The first step is to run a Test-Informant instance locally. To do that, see:
https://wiki.mozilla.org/Auto-tools/Projects/Test-Informant#Contribute
Depends on: 1124720
[0] is a first attempt to use structured logs. I just adapted code from [1]. It doesn't deal with chunking yet.

[0] https://github.com/adusca/test-informant/blob/master/informant/worker.py#L98-L114
[1] http://ahal.ca/blog/2014/consume-structured-test-results/
I am interested in working on this.
Sorry Ankit, Alice is already working on this, I forgot to change the assignee. If you are interested in the Test Informant project, you could look into bug 1125863. Please note, I don't consider that a "good first bug", so if you prefer, we can also help you find a better bug somewhere else.
Assignee: nobody → alicescarpa
Status: NEW → ASSIGNED
Summary: [Tracking] test informant should use structured logs instead of parsing manifests → Test informant should use structured logs instead of parsing manifests
(In reply to Alice Scarpa[:adusca] from comment #1)
> [0] is a first attempt to use structured logs. I just adapted code from [1].
> It doesn't deal with chunking yet.
> 
> [0]
> https://github.com/adusca/test-informant/blob/master/informant/worker.py#L98-
> L114
> [1] http://ahal.ca/blog/2014/consume-structured-test-results/

This looks great! I just noticed that you knew it doesn't deal with chunking, so ignore my comment about it in the PR (as usual, one step ahead of me :p).
Attached file Test report (obsolete) —
Alice's changes were merged into: https://github.com/mozilla/test-informant/tree/structured_log

I ran this for awhile on Friday, and then again today so I could generate a report (attached). It looks like it is working, with a few issues:

1) tests under the marionette suite are not the full path, rather the file (or file + subtest). For some reason other suites display the full path, I imagine this has something to do with a slightly different definition of a "test" under marionette. We'll need to special case it so file paths are stored in the database.

2) there are only half as many browser-chrome tests now. It's possible that they are being filtered out at runtime (and the old reports have been wrong all this time), but this should be investigated.

3) no more mochitest-a11y/chrome. This is a known issue and they should show up under "m-oth" when that suite is added.

Happily mochitest-plain and xpcshell report very similar numbers using this branch. Overall I trust the results from the structured logs more than I do from the old method.
Attached file Test report (obsolete) —
Here's a new report with all of the new suites Alice added.

Note that the test id's for these suites are not always a file path relative to the gecko source. Notably the following suites differ:

* gaia-ui-tests: just the file name (these live outside m-c so we'll need to update the report generator as well)

* marionette: file name + subtest, it's not currently possible to get the path out of the logs (see bug 1129493)

* web-platform-tests: relative file paths, but not to the root of gecko source. They still live in m-c, so this is likely just a matter of normalizing them for that suite.

I think we'll need a per-suite method that can look at a log and extract the test path based on the test id and/or the 'path' attribute proposed in bug 1129495.
Attachment #8558758 - Attachment is obsolete: true
(In reply to Andrew Halberstadt [:ahal] from comment #6)
 
> * web-platform-tests: relative file paths, but not to the root of gecko
> source. They still live in m-c, so this is likely just a matter of
> normalizing them for that suite.

This isn't quite true; they are urls which are normally, but not always, equivalent to relative paths to the test file. In some cases the path can be different (e.g. tets urls ending .worker are derived from filenames ending .worker.js).

I will update wptrunner to log the path directly as planned for marionette.
Comment on attachment 8559354 [details]
Test report

Erm, seems I accidentally uploaded the same report, but the new report is too big for a bugzilla attachment anyway. So here it is:
http://people.mozilla.org/~ahalberstadt/temp-report.html
Attachment #8559354 - Attachment is obsolete: true
A bunch of things happened since this bug was worked on, most importantly ActiveData becamse available, and the VM that was running the test-informant service got shut down. I am re-writing test-informant as a static web page that queries ActiveData, wip here:
http://people.mozilla.org/~ahalberstadt/test-informant/

Closing this bug WONTFIX.
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → WONTFIX
Jgraham pointed out that this was fixed, just never got deployed. Sorry Alice!
Resolution: WONTFIX → FIXED
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.