1093045 - Generate and upload HTML logs for Marionette based tests

Status: RESOLVED DUPLICATE of bug 1275243

(Release Engineering :: General, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We now have a HTML log option available through mozlog, which is very similar to GaiaTest's own HTML report. We should enable this for all Marionette based tests to assist with diagnosing failures - the report contains screenshots and page source immediately following any failed tests.

To enable this we need the py [1] dependency installed and to add the --log-html command line argument. We should also upload this file so that it's available to view via Treeherder.

This will create a duplicate report for Gaia UI tests, however once bug 1065410 is resolved we can remove the --html-output command line argument and switch over to the new report.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Jonathan: I'm still not entirely familiar with mozharness. Could you point me in the right direction for this? Alternatively, can you suggest someone who might be able to assist? I think it would be very useful for helping us to diagnose intermittent failures.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Looks like I missed the needinfo here.

Comment 3

[Jonathan Griffin (:jgriffin)]

Hey Dave,

There are three parts to this.

1) you need to file a bug to get the py package installed on our internal pypi.  You can use bug 1028386 as a template.

2) you need to modify https://dxr.mozilla.org/build:mozharness/source/scripts/marionette.py#262 to install it; you can just call self.register_virtualenv_module(module_name) and it will be downloaded from our internal pypi

3) you need to modify https://dxr.mozilla.org/mozilla-central/source/testing/config/mozharness/marionette.py to replace --html-output with --log-html, or add the latter if you need both args.  You also need to modify https://dxr.mozilla.org/build:mozharness/source/scripts/marionette.py#348 in the same way

If you use the filename https://dxr.mozilla.org/build:mozharness/source/scripts/marionette.py#348 for this, it should automatically be uploaded by the script.  If you use a different filename, you just need to make sure it gets written to dirs['abs_blob_upload_dir']; everything in that directory is uploaded at the end of the run.

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We no longer need to depend on py since bug 1016929 was fixed. I'm happy to mentor the mozharness changes to get HTML reports using mozlog uploaded for all Marionette based tests.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Henrik: I recall seeing these HTML reports in Treeherder. Can this be closed?

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Correct. This has actually been done in bug 1275243.