794901 - Add support for the new Talos TinderboxPrint json results format

Status: RESOLVED FIXED

(Tree Management Graveyard :: TBPL, defect)

Description

[Ed Morley [:emorley]]

Current Talos output:
{
TinderboxPrint:<a href = "http://hg.mozilla.org/integration/mozilla-inbound/rev/157f3efdee37">rev:157f3efdee37</a>

TinderboxPrint: cycle time: 00:18:15<br>

TinderboxPrint:<br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[[220,131,1]]'>tdhtmlr_paint: 619.03</a><br>

TinderboxPrint:<a href='http://graphs.mozilla.org/graph.html#tests=[[254,131,1]]'>tresize: 13.0</a><br>

TinderboxPrint:<p style="font-size:smaller;">Details:<br>| <a href='http://graphs.mozilla.org/graph.html#tests=[[220,131,1]]'>tdhtmlr_paint</a> | <a href='http://graphs.mozilla.org/graph.html#tests=[[254,131,1]]'>tresize</a> |</p>
}

After bug 794799:
{
TinderboxPrint: <a href='http://graphs.mozilla.org/graph.html#tests=[[220,131,1]]'>tdhtmlr_paint: 619.03</a>
TinderboxPrint: <a href='http://graphs.mozilla.org/graph.html#tests=[[254,131,1]]'>tresize: 13.0</a>
}

Comment 1

[Ed Morley [:emorley]]

Fixing dependency:
TBPL still works with the output produced with the patch in bug 794799, so this isn't a prerequisite and instead just a cleanup.

Comment 2

[Ed Morley [:emorley]]

Unassigning, since it's going to be a while until bug 794799 works its way through the trains (and also onto ESR17), so we can't remove the cruft until then.

Comment 3

[Ed Morley [:emorley]]

Post bug 812513 we can remove even more cruft :-)

Comment 4

[Ed Morley [:emorley]]

I've done some preliminary cleanup in bug 817032 to reduce the time this bug will take.

However, until I know in what format we're going to output the results in bug  	816634, I can't really proceed here.

Alternatively, we could scrap the "nice"(r) handling of TinderboxPrint (trying to match against test names and results) and just make everything use the fallback that already exists (for non-talos), which is just to show the entire line, markup and all. (Given that TBPLv2 will no doubt be doing something a lot nicer than TinderboxPrint anyway, so perhaps not worth over-engineering this).

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I think we can come up with a good resulting print/format/datatransfer from talos->log->tbpl.

Comment 6

[Ed Morley [:emorley]]

The new log output is like:

TinderboxPrint: TalosResult: 
{
    "datazilla": {
        "tp5n": {
            "url": "https://datazilla.mozilla.org/talos/summary/Try/74794a471859?product=Firefox&branch_version=21.0a1"
        }
    },
    "graphserver": {
        "tp5n_shutdown_paint": {
            "url": "http://graphs.mozilla.org/graph.html#tests=[[215,23,21]]",
            "result": "1167.00"
        },
        "tp5n_pbytes_paint": {
            "url": "http://graphs.mozilla.org/graph.html#tests=[[207,23,21]]",
            "result": "3.6GB"
        },
        "tp5n_main_rss_paint": {
            "url": "http://graphs.mozilla.org/graph.html#tests=[[209,23,21]]",
            "result": "198.6MB"
        },
        "tp5n_paint": {
            "url": "http://graphs.mozilla.org/graph.html#tests=[[206,23,21]]",
            "result": "277.81"
        },
        "tp5n_responsiveness_paint": {
            "url": "http://graphs.mozilla.org/graph.html#tests=[[216,23,21]]",
            "result": "120.00"
        }
    }
}

TinderboxPrint: <a href='http://graphs.mozilla.org/graph.html#tests=[[206,23,21]]'>tp5n_paint: 277.81</a>
TinderboxPrint: <a href='http://graphs.mozilla.org/graph.html#tests=[[207,23,21]]'>tp5n_pbytes_paint: 3.6GB</a>
TinderboxPrint: <a href='http://graphs.mozilla.org/graph.html#tests=[[209,23,21]]'>tp5n_main_rss_paint: 198.6MB</a>
TinderboxPrint: <a href='http://graphs.mozilla.org/graph.html#tests=[[216,23,21]]'>tp5n_responsiveness_paint: 120.00</a>
TinderboxPrint: <a href='http://graphs.mozilla.org/graph.html#tests=[[215,23,21]]'>tp5n_shutdown_paint: 1167.00</a>

(Have pretty formated the json)

What do we want in the output? Both datazilla and graphserver to start with?

Ideally we could remove the redundant old style lines from talos now too.

Comment 7

[Ed Morley [:emorley]]

Attachment: Patch v1

* Adds support for the new format added by bug 816634
* If the new format isn't found, allows the appending of the old style results. (In the new style, talosResult always appears before the others, so this works fine).
* If a TinderboxPrint line is neither a talosResult, nor a legacy graphs line, it is added to the scrape output (eg revision we built from). We don't have any of these at the moment, but may do in the future (and non-talos supports this already).
* Once Talos no longer outputs both talosResult and the legacy lines, we can remove all the |newStyleFound| nonsense.

Comment 8

[Ed Morley [:emorley]]

One question: 
Will there ever be more than one Datazilla testname?

Comment 9

[Ed Morley [:emorley]]

Attachment: Screenshot

Comment 10

[Jeff Hammel]

(In reply to Ed Morley [:edmorley UTC+0] from comment #8)
> One question: 
> Will there ever be more than one Datazilla testname?

So that is a tricky question in two ways ;)

1. Currently all of the URLs are the same.  This should probably be fixed.  But until it is fixed, it won't matter which url is used for linking.

2. We accumulate datazilla results according to how specified via `--activeTests`; so for `tpn` as tested on try, there is only one test: http://k0s.org:8080/?show=active#tpn . However, for other buildbot suites, more than one `--activeTest` is specified.

Comment 11

[Jeff Hammel]

e.g. for the following command:

(talos)│talos -n -d --activeTests tsvgr:tsvgr_opacity --develop -e `which firefox` --authfile /home/jhammel/mozilla/talos/passwords.py --datazilla-url https://datazilla.mozilla.org/test --branchName foo

we get this:

NOISE: Datazilla results at https://datazilla.mozilla.org/test/summary/foo/712eca11a04e?product=Firefox&branch_version=21.0a1
TinderboxPrint: TalosResult: {"datazilla": {"tsvgr": {"url": "https://datazilla.mozilla.org/test/summary/foo/712eca11a04e?product=Firefox&branch_version=21.0a1"}, "tsvgr_opacity": {"url": "https://datazilla.mozilla.org/test/summary/foo/712eca11a04e?product=Firefox&branch_version=21.0a1"}}}

Comment 12

[Jeff Hammel]

Comment on attachment 703555 [details] [diff] [review]
Patch v1

lgtm. Thanks!

Comment 13

[Ed Morley [:emorley]]

Landing blocked on me having a working machine + ssh key.

Also, don't know if you spotted my inclusion of "whilst" specially for you :-)

Comment 14

[Ed Morley [:emorley]]

Ha, tried applying the local copy of the patch and it failed with conflicts (even though no one else has landed on the repo since). Scratching my head I took a look at the diff & the decrypt of my corrupted FDE drive had turned the second half of it into what looked like a binary file. Yey for bugzilla attachments!

https://hg.mozilla.org/webtools/tbpl/rev/17c474782b50

Comment 15

[Ed Morley [:emorley]]

In production.

Comment 16

[Jeff Hammel]

And lookin' good :)  We should definitely note this at the next SfN meeting