1065381 - Add result_callbacks to include test debug in HTML reports for failures and errors

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚GMT]

The HTML formatter in mozlog supports extra data such as screenshots to be included. As I understand it we would need to add result callbacks in the test harness for this to start appearing in the HTML.

To be able to replace the HTML report generated by Marionette's test runner, we would need the following extra data:

* Screenshot
* Settings
* Source

See http://hg.mozilla.org/mozilla-central/file/46c3b9de467a/testing/marionette/client/marionette/runner/mixins/reporting.py#l198 for examples of gathering this data.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚GMT]

James: Would you be able to provide additional information of these results callbacks, perhaps with an example?

Comment 2

[James Graham [:jgraham]]

So you basically need to pass a result_callbacks argument when the StructuredTestRunner class is created. These must return a dictionary of extra data to be included in the result message. So in this case it would presumably return something like {"screenshot": "base64 encoded screenshot as string", "source": "source"}, although note that "source" is not currently present in the HTML report.

Comment 3

[James Graham [:jgraham]]

So, looking at this a bit more, you probably need to arrange for BaseMarionetteTestRunner to take a results_callback argument that gets passed to textrunnerclass. This would default to the empty list (via None ofc). Then, when you initialise the BaseMarionetteTestRunner, you would have something like:

def gather_debug(test, status):
    rv = {}
    marionette = test._marionette_weakref()
    rv['screenshot'] = marionette.screenshot()
    rv['source'] = marionette.page_source

    marionette.switch_to_frame()
    rv['settings'] = json.dumps(self.marionette.execute_async_script("""
SpecialPowers.addPermission('settings-read', true, document);
SpecialPowers.addPermission('settings-api-read', true, document);
var req = window.navigator.mozSettings.createLock().get('*');
req.onsuccess = function() {
  marionetteScriptFinished(req.result);
}""", special_powers=True), sort_keys=True, indent=4, separators=(',', ': '))
    return rv


result_callbacks = [gather_debug]

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚GMT]

Attachment: Gather basic debug for HTML log formatter. r=jgraham

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚GMT]

Attachment: Gather Gaia debug for HTML log formatter. v1.0

Comment 6

[James Graham [:jgraham]]

Comment on attachment 8509513 [details] [diff] [review]
Gather basic debug for HTML log formatter. r=jgraham

Review of attachment 8509513 [details] [diff] [review]:
-----------------------------------------------------------------

Just a couple of small fixups.

::: testing/marionette/client/marionette/runner/base.py
@@ +483,5 @@
>          self.gecko_log = gecko_log
>          self.mixin_run_tests = []
>          self.manifest_skipped_tests = []
>          self.tests = []
> +        self.result_callbacks = result_callbacks or []

I tend to prefer the more explicit 'result_callbacks if result_callbacks is not None else []'

@@ +491,5 @@
> +            marionette = test._marionette_weakref()
> +            try:
> +                marionette.set_context(marionette.CONTEXT_CHROME)
> +                rv['screenshot'] = marionette.screenshot()
> +                marionette.set_context(marionette.CONTEXT_CONTENT)

I guess this is safe if we are confident that we're always in CONTENT to begin with. I don't think you can do better at least.

@@ +796,5 @@
>                  break
>  
>          if suite.countTestCases():
> +            runner = self.textrunnerclass(
> +                logger=self.logger,

Revert to the old alignment, please.

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚GMT]

Attachment: Gather basic debug for HTML log formatter. v1.1 r=jgraham

Applied suggested changes. Carrying r+

Comment 8

[Carsten Book [:Tomcat]]

Hi Dave,

seems this patch didn't apply cleanly:

applying bug1065381.patch
patching file testing/marionette/client/marionette/runner/base.py
Hunk #2 FAILED at 214
Hunk #3 FAILED at 437
Hunk #5 FAILED at 790
3 out of 5 hunks FAILED -- saving rejects to file testing/marionette/client/marionette/runner/base.py.rej
patch failed, unable to continue (try -v)
patch failed, rejects left in working dir
errors during apply, please fix and refresh bug1065381.patch

could you take a look. Thanks!

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚GMT]

Attachment: Gather basic debug for HTML log formatter. v1.2

This was caused by bug 1084565 landing first. I've updated the patch and carried the r+ after testing due to the minimal changes.

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚GMT]

Comment on attachment 8509516 [details] [review]
Gather Gaia debug for HTML log formatter. v1.0

Thanks for the review, James. I've updated based on your comments - could you take another look?

Comment 11

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c410756cb386

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚GMT]

Comment on attachment 8509516 [details] [review]
Gather Gaia debug for HTML log formatter. v1.0

Applying r+ from github.

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚GMT]

(In reply to Dave Hunt (:davehunt) from comment #12)
> Comment on attachment 8509516 [details] [review]
> Gather Gaia debug for HTML log formatter. v1.0
> 
> Applying r+ from github.

Landed in:
https://github.com/mozilla-b2g/gaia/commit/5f65d527e7c9566486ddfe5d308ef7fe28b95b1f

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/c410756cb386