1275243 - Marionette jobs should upload reports (screenshots) for failures

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[:Gijs (he/him)]

Per bug 1272652 comment 22: it seems firefox-ui-test takes screenshots on failure but marionette does not (or doesn't stick them in the upload dir so we can't see them on treeherder). It would be helpful if it did.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This could easily be done when we run our Marionette testes with the --log-html option, and instruct taskcluster to also upload that file. This might most likely be a change in mozharness in the marionette entry script.

Comment 2

[Andreas Tolfsen ❲:ato❳]

We do take screenshots on skips, failures, and errors.  I’m happy for us to provide those to TC.  If the --log-html flag already does this, that is great!

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Here those two references as what fx-ui-tests are doing to get HTML reports uploaded via mozharness:

http://mxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/firefox_ui_tests.py#314
http://mxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/firefox_ui_tests.py#273

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Let me actually get this added quickly. It would have been useful for me too various times in the past weeks. Especially now with our Firefox UI tests moving into the Marionette test jobs.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 1275243 - Marionette test jobs should create HTML reports with screenshots.

Review commit: https://reviewboard.mozilla.org/r/98824/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/98824/

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 1275243 - Add some failures to show failure reporting - REMOVE for final landing.

Review commit: https://reviewboard.mozilla.org/r/98826/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/98826/

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Try build works just fine:
https://public-artifacts.taskcluster.net/TGUA6_m-RJyIc_0ACg2_Ug/0/public/test_info//report.html

Andreas, do you know why we take screenshots for skipped tests? I don't see a reason for it.

Comment 8

[Andreas Tolfsen ❲:ato❳]

(In reply to Henrik Skupin (:whimboo) from comment #7)
> Try build works just fine:
> https://public-artifacts.taskcluster.net/TGUA6_m-RJyIc_0ACg2_Ug/0/public/
> test_info//report.html

This is great!  Thanks for doing this.

> Andreas, do you know why we take screenshots for skipped tests? I don't see
> a reason for it.

I don’t see any screenshots for skipped tests in that report?

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Scroll down to tests like: test_import_script.py TestImportScriptChrome.test_imports_apply_globally

I think that this is related to our own defined skip decorators. Looks like those haven't been implemented that well enough. I will file a new bug for that, which should also allow us to add a message to the skip call.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I filed bug 1323770 for that issue.

Comment 11

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8818869 [details]
Bug 1275243 - Marionette test jobs should create HTML reports with screenshots.

https://reviewboard.mozilla.org/r/98824/#review99508

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9de6c2ae8512
Marionette test jobs should create HTML reports with screenshots. r=maja_zf

Comment 13

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/9de6c2ae8512

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Test-only change which would be nice to have for the next ESR release. Can you please uplift to aurora? Thanks.

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/71a43018dae1