Closed Bug 1275243 Opened 4 years ago Closed 3 years ago
Marionette jobs should upload reports (screenshots) for failures
58 bytes, text/x-review-board-request
Per bug 1272652 comment 22: it seems firefox-ui-test takes screenshots on failure but marionette does not (or doesn't stick them in the upload dir so we can't see them on treeherder). It would be helpful if it did.
This could easily be done when we run our Marionette testes with the --log-html option, and instruct taskcluster to also upload that file. This might most likely be a change in mozharness in the marionette entry script.
We do take screenshots on skips, failures, and errors. I’m happy for us to provide those to TC. If the --log-html flag already does this, that is great!
Here those two references as what fx-ui-tests are doing to get HTML reports uploaded via mozharness: http://mxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/firefox_ui_tests.py#314 http://mxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/firefox_ui_tests.py#273
Summary: Marionette should take screenshots on failure → Marionette jobs should upload reports (screenshots) for failures
Let me actually get this added quickly. It would have been useful for me too various times in the past weeks. Especially now with our Firefox UI tests moving into the Marionette test jobs.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Try build works just fine: https://public-artifacts.taskcluster.net/TGUA6_m-RJyIc_0ACg2_Ug/0/public/test_info//report.html Andreas, do you know why we take screenshots for skipped tests? I don't see a reason for it.
(In reply to Henrik Skupin (:whimboo) from comment #7) > Try build works just fine: > https://public-artifacts.taskcluster.net/TGUA6_m-RJyIc_0ACg2_Ug/0/public/ > test_info//report.html This is great! Thanks for doing this. > Andreas, do you know why we take screenshots for skipped tests? I don't see > a reason for it. I don’t see any screenshots for skipped tests in that report?
Scroll down to tests like: test_import_script.py TestImportScriptChrome.test_imports_apply_globally I think that this is related to our own defined skip decorators. Looks like those haven't been implemented that well enough. I will file a new bug for that, which should also allow us to add a message to the skip call.
I filed bug 1323770 for that issue.
Attachment #8818869 - Flags: review?(mjzffr)
Attachment #8818870 - Attachment is obsolete: true
Comment on attachment 8818869 [details] Bug 1275243 - Marionette test jobs should create HTML reports with screenshots. https://reviewboard.mozilla.org/r/98824/#review99508
Attachment #8818869 - Flags: review?(mjzffr) → review+
Pushed by email@example.com: https://hg.mozilla.org/integration/autoland/rev/9de6c2ae8512 Marionette test jobs should create HTML reports with screenshots. r=maja_zf
Test-only change which would be nice to have for the next ESR release. Can you please uplift to aurora? Thanks.
You need to log in before you can comment on or make changes to this bug.