Closed Bug 1229379 Opened 4 years ago Closed 4 years ago

use mozlog in talos

Categories

(Testing :: Talos, defect)

defect
Not set

Tracking

(firefox46 fixed)

RESOLVED FIXED
mozilla46
Tracking Status
firefox46 --- fixed

People

(Reporter: parkouss, Assigned: parkouss)

References

Details

Attachments

(3 files)

mozlog seems a prerequisite for auto-starring and probably that would be good for active data. So let's start by using mozlog in talos.
Depends on: 1229380
now that we are in-tree, can we just use the mozlog from the testing/mozbase/ directory?
No longer depends on: 1229380
(In reply to Joel Maher (:jmaher) from comment #1)
> now that we are in-tree, can we just use the mozlog from the
> testing/mozbase/ directory?

Oh, I forgot about that. :)

We are already doing this: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/talos.py#292

And I'm pretty sure I made the work for that! Thanks Joel for the comment.
looking at mochitest-other which runs mochitest-chrome and mochitest-a11y, we upload two sets of logs, one for chrome_*.log and a11y_*.log (where * = errorsummary|raw)

I suspect we could do this for each talos suite (i.e. tart_raw.log, cart_raw.log, svg_raw.log, etc.)
Attached file use mozlog in talos
I started to work on this:

https://hg.mozilla.org/try/rev/ee589b9a6ba3
https://hg.mozilla.org/try/rev/19abee1211da

on treeherder, https://treeherder.mozilla.org/#/jobs?repo=try&revision=f6ec672f6dc8.

So, this is a basic support of mozlog in talos. It only report tests for now, maybe we could add a subtest support for each cycle of one talos test. but I guess this is good enough, at least for now - tell me what do you think.
Attachment #8698142 - Flags: feedback?(jmaher)
Attachment #8698142 - Flags: feedback?(james)
Comment on attachment 8698142 [details]
use mozlog in talos

this looks very safe in general.  One thing I would like to see is the final patches use suite_start/suite_end for each test, not the entire job.  so for the chromez job, we would have tresize_raw.log and tcanvasmark_raw.log instead of chromez_raw.log.

I am not too familiar with the rest of the mozlog api or output, so I will assume this is working well.
Attachment #8698142 - Flags: feedback?(jmaher) → feedback+
(In reply to Joel Maher (:jmaher) from comment #6)
> Comment on attachment 8698142 [details]
> One thing I would like to see is the final
> patches use suite_start/suite_end for each test, not the entire job.  so for
> the chromez job, we would have tresize_raw.log and tcanvasmark_raw.log

hm, but where do you put the logs that are outside of one test run (setup, finalizers, ...) ?

I believe mozlog is not designed for that. I would guess that we should run talos for one test only instead if you want that kind of behavior. Pseudo code in mozharness:

> for $test in $suite:
>     python talos/run_tests.py -a $test --log...

instead of what we have, e.g:

>     python talos/run_tests.py --suite $suite --log...

If someone has another idea, please shout. :)
I want to see what an error looks like before agreeing to this.  Maybe we can get a good error for linux64 svgr job (which is failing often in the TART suite) and then determine what autostar would be able to do with it.  If it looks reasonable, then I will assume this approach is just fine.  The above mentioned try push has been extended to run the linux64 's' job.
As we saw in https://treeherder.mozilla.org/#/jobs?repo=try&revision=0b4297e0161e&selectedJob=14613772, failures looks good.

I only updated the second patch to include the stack trace into the stack property, not mixed up with the message.

I guess it won't hurt to push a final try after the review, so we can see that everything is fine (even the svg job failures since they are so easy to reproduce :)).
Comment on attachment 8698939 [details]
MozReview Request: Bug 1229379 - use mozlog in talos. r=jmaher

https://reviewboard.mozilla.org/r/28133/#review25191

::: testing/talos/talos/config.py:458
(Diff revision 1)
> +    setup_logging("mozregression", cli_opts, {"mach": sys.stdout})

why do we have mozregression here?

just one small nit here
Attachment #8698939 - Flags: review?(jmaher) → review+
hum, eh. Yeah, that is a copy pasted line. :) I'll ch
Comment on attachment 8698940 [details]
MozReview Request: Bug 1229379 - use mozlog test report protocol in talos. r=jmaher

https://reviewboard.mozilla.org/r/28135/#review25195

looks great
Attachment #8698940 - Flags: review?(jmaher) → review+
(In reply to Julien Pagès (:parkouss) from comment #13)
> hum, eh. Yeah, that is a copy pasted line. :) I'll ch

So, I'll change this to 'talos' before pushing to try.
https://reviewboard.mozilla.org/r/28135/#review25233

::: testing/talos/talos/run_tests.py:194
(Diff revision 1)
>              LOG.info("Starting test %s" % testname)

This seems a little superflous now.

::: testing/talos/talos/run_tests.py:201
(Diff revision 1)
>              LOG.info("Completed test %s (%s)"

This seems a little superflous now.
https://reviewboard.mozilla.org/r/28133/#review25235

::: testing/mozharness/mozharness/mozilla/testing/talos.py:386
(Diff revision 1)
> +        mozlog_opts = ['--log-mach-level=debug']

How sure are you you want to use mach output here rather than tbpl output?

::: testing/talos/talos/ffsetup.py:118
(Diff revision 1)
> -            logging.debug('BROWSER_OUTPUT: %s', line)
> +            LOG.debug('BROWSER_OUTPUT: %s' % line)

This seems like what LOG.process_output is intended for.

::: testing/talos/talos/output.py:14
(Diff revision 1)
> +from StringIO import StringIO

(strictly speaking, I would expect that under the stdlib imports).

::: testing/talos/talos/run_tests.py:211
(Diff revision 1)
>          traceback.print_exception(*sys.exc_info())

Seems like this could be logged too?

::: testing/talos/talos/sps_profile.py:194
(Diff revision 1)
> -                    logging.exception(
> +                    LOG.exception(

`exception` isn't a log level.

::: testing/talos/talos/talos_process.py:61
(Diff revision 1)
> -            logging.debug('BROWSER_OUTPUT: %s', line)
> +            LOG.debug('BROWSER_OUTPUT: %s' % line)

`process_output`
Attachment #8698142 - Flags: feedback?(james) → feedback+
Comment on attachment 8698939 [details]
MozReview Request: Bug 1229379 - use mozlog in talos. r=jmaher

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/28133/diff/1-2/
Comment on attachment 8698940 [details]
MozReview Request: Bug 1229379 - use mozlog test report protocol in talos. r=jmaher

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/28135/diff/1-2/
Yeah - strangely enough, this did not appear in my push try (https://treeherder.mozilla.org/#/jobs?repo=try&revision=14706caafff0), and now it is a really frequent intermittent :(.

Anyway, sorry for the mess - I'm looking into it to fix it in the best way possible. Will do a more complete push try next time.

Note this has been reported in bug 1233664, and we should close it when the issue is resolved.
See Also: → 1233664
https://hg.mozilla.org/mozilla-central/rev/713801f5c627
https://hg.mozilla.org/mozilla-central/rev/4776a775d587
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla46
You need to log in before you can comment on or make changes to this bug.