573853 - Mozmill cmd-line should output INFO/DEBUG/ERROR in human/editor-readable format

Status: RESOLVED FIXED

(Testing Graveyard :: Mozmill, enhancement)

Description

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Mozmill currently dumps its verbose output in pseudo-JSON (see an example below).  Unfortunately, this is pretty non-standard, so it's not straightforward to parse or to reformat into something human-readable.  

The quoting, in particular (single quotes on the outside, doubles or escaped singles on the inside) isn't JSON-legal, even if it is JavaScript legal, and can be problematic for JSON parsers without a relaxed decoding option.

It'd be awfully nice if Mozmill could do verbose output in a standard human-readable way (formatted as a stack with no embedded \n's, etc).  It'd be even better if it did that output in enough of a standardized way that editors like Vim can parse the call stack for post-run error highlighting.  

Failing all that, outputting valid JSON that can be run through a filter easily could work too.

Sample output:

ERROR:mozmill:Test Failure: {u'exception': {u'message': u"Cc['@mozilla.org/toolkit/crash-reporter;1'] is undefined", u'lineNumber': 60, u'stack': u'([object Object])@resource://mozmill/modules/frame.js -> file:///Users/geo/hg/mozmill/firefox/testInstallation/testBreakpadInstalled.js:60\n((function (module) {controller = mozmill.getBrowserController();var dirService = Cc[\'@mozilla.org/file/directory_service;1\'].getService(Ci.nsIProperties);module.appDir = dirService.get("XCurProcD", Ci.nsILocalFile);module.crashReporter = Cc[\'@mozilla.org/toolkit/crash-reporter;1\'].getService(Ci.nsICrashReporter);}),[object Object])@resource://mozmill/modules/frame.js:463\n([object Object])@resource://mozmill/modules/frame.js:498\n([object Object])@resource://mozmill/modules/frame.js:562\n("/Users/geo/hg/mozmill/firefox/testInstallation/testBreakpadInstalled.js")@resource://mozmill/modules/frame.js:420\n("/Users/geo/hg/mozmill/firefox/testInstallation/testBreakpadInstalled.js")@resource://mozmill/modules/frame.js:574\n((function (filename, invokedFromIDE) {var runner = new Runner(new Collector, invokedFromIDE);runner.runTestFile(filename);runner.end();return true;}),[object Array])@resource://jsbridge/modules/server.js:164\n("90e1dade-7e4c-11df-85a0-58b035ffc0aa",(function (filename, invokedFromIDE) {var runner = new Runner(new Collector, invokedFromIDE);runner.runTestFile(filename);runner.end();return true;}),[object Array])@resource://jsbridge/modules/server.js:168\n@resource://jsbridge/modules/server.js:244\n', u'fileName': u'resource://mozmill/modules/frame.js -> file:///Users/geo/hg/mozmill/firefox/testInstallation/testBreakpadInstalled.js'}}

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Jeff, that's similar to what you will have for buildbot.

Comment 2

[Jeff Hammel]

(In reply to comment #1)
> Jeff, that's similar to what you will have for buildbot.

Indeed: see https://bugzilla.mozilla.org/show_bug.cgi?id=573589 (though to be clear, buildbot is only beneficiary of unification)

Comment 3

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Yeah, pretty similar aims.  In fact, I'm cool with almost any output format as long as I can read a sane formatted callstack from the command line, whether straight from MozMill or via a filter we include.

If formatting for buildbot would accomplish that, that works for me too.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

A filter would be fantastic due to the following reasons:

* JSON: Applications will be able to parse the stdout output and process the JSON string to build up a visual tree of results. Similar what we wanna have on Brasstacks. One example would be my Mozmill Crowd extension which connects via a pipe to stdout.

* Formatted: Perfect for users who wanna run tests from the command line and are irritated by that huge amount of visual unstructured output they get at the moment.

For the filter we could use another command line option (i.e. --filter=json|formatted), which will enable the JSON output. If not set we will use the formatted output.

Comment 5

[Jeff Hammel]

As of the closing of bug 573589 , the format is now real JSON.

Comment 6

[Jeff Hammel]

Is there anything actionable to be done for this bug? I can't figure out what should be done.

The entire logging infrastructure should be redone for mozmill 2.0 . Logging and reporting will become pluggable.  The existing (python) logging will be unified a bit more.

Comment 7

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Let me doublecheck the current --show-errors/--showall output formatting.  If it's real JSON (and thus parsable), think we're in pretty good shape for now, and can probably close this.  

We may want more options for 2.0, but seems like we can handle that in primary design.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #7)
> Let me doublecheck the current --show-errors/--showall output formatting.  If
> it's real JSON (and thus parsable), think we're in pretty good shape for now,
> and can probably close this.  

It's not easily parsable because of mixed output. But that shouldn't matter for my extension. The current output would be perfect for me for 1.4.2.

Geo, any results from your testing?

Comment 9

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Yeah, looks fine.  It's not pure JSON, obviously, but it's highly regular and the objects are JSON.  That's awesome, thanks much!

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, so lets call it fixed by bug 573589.