1056804 - runner-service : Make passing reporter not obscure terrible thing

Status: RESOLVED INCOMPLETE

(Testing Graveyard :: JSMarionette, defect)

Description

[Zac C (:zac)]

The runner-service is passing in "mocha-socket-reporter" but this causes pretty output to be hidden and makes it difficult to debug test failures.

If we change the default reporter to 'spec' but also allow the reporter to be overridden by a variable then we'll have nicer output.



NB:
lightsofapollo> zac: So change your bug to "Make passing reporter not obscure terrible thing"

Comment 1

[Zac C (:zac)]

I will have a go at this.

Comment 2

[Andrew Halberstadt [:ahal]]

Doing this is fine, especially for devs running locally.. but I just wanted to explain what my intentions had been.

The goal of the mocha-socket-reporter was to use structured logging like we are starting to do with the rest of our harnesses. This means it just logs json, which we can then format with formatters (e.g builtin mozlog formatters [1] or our own custom formatters) however we see fit. We can even stream different formats to different files (e.g upload the raw json logs, and print pretty-formatted ones to stdout).

So it would be possible to create our own formatter that mimics the 'spec' reporter look, while still maintaining access to the raw structured logs (and the benefits that come with it). Obviously writing a new formatter is more work than just switching the reporter to 'spec', so there is a strong argument for going that route. Just be aware that if we do that, we won't be using structured logs anymore.

[1] http://mozbase.readthedocs.org/en/latest/mozlog_structured.html#formatters

Comment 3

[Andrew Halberstadt [:ahal]]

Side note: The format that the runner-service currently prints was just a placeholder that I was intending to replace at a later date.

Comment 4

[Zac C (:zac)]

Ahal, that makes sense, I understand CI's requirement.
I can just flip it around the other way and keep the current default but allow it to be overridden by the arg, so in the meantime I can debug failures locally.

Comment 5

[Andrew Halberstadt [:ahal]]

That also makes sense, glad we're on the same page!

Comment 6

[James Lal [:lightsofapollo]]

>Ahal, that makes sense, I understand CI's requirement.
>I can just flip it around the other way and keep the current default but allow it to be overridden by the >arg, so in the meantime I can debug failures locally.

I don't really have strong opinions on what the reporter does but the defaults should be designed for humans... We can easily configure CI.

Comment 7

[Firefox Bug Husbandry Bot]

Bulk closed as part of https://bugzilla.mozilla.org/show_bug.cgi?id=1422750