1034267 - Structured Logging for robocop tests

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Chris Manchester (limited bugmail, email directly)]

Bug 886570 gets us most much of the way there, but we need to emit structured messages from java (bug 1034236).

Comment 1

[Chris Manchester (limited bugmail, email directly)]

Robocop generates unstructured log messages here: http://hg.mozilla.org/mozilla-central/annotate/0dd44135ac32/build/mobile/robocop/FennecMochitestAssert.java#l85

Comment 2

[Ahmed Kachkach [:akachkach]]

Attachment: 0003-FennecMochitest.patch

WIP.

(relies on bug 1034236)

Comment 3

[Ahmed Kachkach [:akachkach]]

Attachment: 0001-Bug-1034267-Structured-Logging-for-robocop.patch

Updated, seems to be working (https://tbpl.mozilla.org/?tree=Try&rev=097966857df3).

Wondering how I should handle dumpLog with a Throwable? I just concatenated the message with the String representation of the error for now.

Comment 4

[Chris Manchester (limited bugmail, email directly)]

The logs in the try run result in multiple suite_end messages. This is ok with respect to the issue in bug 1043485 but will not be compatible with consumers of the structured log.

Comment 5

[Geoff Brown [:gbrown]]

The logs in the try run of Comment 3 look wrong to me. As an example, compare testAddonManager in https://tbpl.mozilla.org/php/getParsedLog.php?id=44545888&tree=Try&full=1 to a current log like https://tbpl.mozilla.org/php/getParsedLog.php?id=44565603&tree=Mozilla-Central&full=1.

Comment 6

[Ahmed Kachkach [:akachkach]]

Attachment: 0001-Bug-1034267-Structured-Logging-for-robocop-r-gbrown.patch

Deactivating buffering seem to have fixed this issue (see this try run: https://tbpl.mozilla.org/?rev=d74d7bb327fc&tree=Try)

I'm also trying to get rid of the multiple SUITE-END messages, waiting for a try run with the submitted patch (that should hopefully fix it).

If the logs are valid, we should try to land this before the workaround in bug 1043485.

Comment 7

[Ahmed Kachkach [:akachkach]]

Attachment: 0001-Bug-1034267-Structured-Logging-for-robocop-r-gbrown.patch

Small change to the patch (to remove multiple suite_end messages).
Try run: https://tbpl.mozilla.org/?tree=Try&rev=18f40cf430b8

Comment 8

[Geoff Brown [:gbrown]]

Comment on attachment 8462917 [details] [diff] [review]
0001-Bug-1034267-Structured-Logging-for-robocop-r-gbrown.patch

Review of attachment 8462917 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/mobile/robocop/FennecMochitestAssert.java
@@ +89,5 @@
>          }
>  
>      }
>  
> +    private void _logMochitestResult(testInfo test, String passStatus, String passExpected, String failStatus, String failExpected) {

I would like to see a comment here explaining what the parameters mean.

@@ +117,5 @@
>              junit.framework.Assert.fail(message);
>          }
>      }
>  
> +

There is an extra line here - remove it.

Comment 9

[Ahmed Kachkach [:akachkach]]

Attachment: 0001-Bug-1034267-Structured-Logging-for-robocop-r-gbrown.patch

fixed nits; carry r+ forward

Comment 10

[Ahmed Kachkach [:akachkach]]

New try push: https://tbpl.mozilla.org/?tree=Try&rev=c98b4306f905

Comment 11

[Carsten Book [:Tomcat]]

hi, do you know if the repeated mochitest 1 error on b2g are related to this push ?

Comment 12

[Ahmed Kachkach [:akachkach]]

I'll rebase and do another push to make sure!

Comment 13

[Ahmed Kachkach [:akachkach]]

Attachment: 0001-Bug-1034267-Structured-Logging-for-robocop-r-gbrown.patch

Carry r+ forward

The B2G-desktop error was caused by a missing call to suite_end and buffer messages dumping. Fixed now!

Try run: https://tbpl.mozilla.org/?tree=Try&rev=c2b0a8fbed01

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/f3a5a084d90e

Comment 15

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/f3a5a084d90e

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/1e06f748a672