Closed Bug 1275243 Opened 4 years ago Closed 3 years ago

Marionette jobs should upload reports (screenshots) for failures

Categories

(Testing :: Marionette, defect)

defect
Not set

Tracking

(firefox52 fixed, firefox53 fixed)

RESOLVED FIXED
mozilla53
Tracking Status
firefox52 --- fixed
firefox53 --- fixed

People

(Reporter: Gijs, Assigned: whimboo)

References

Details

Attachments

(1 file, 1 obsolete file)

Per bug 1272652 comment 22: it seems firefox-ui-test takes screenshots on failure but marionette does not (or doesn't stick them in the upload dir so we can't see them on treeherder). It would be helpful if it did.
This could easily be done when we run our Marionette testes with the --log-html option, and instruct taskcluster to also upload that file. This might most likely be a change in mozharness in the marionette entry script.
We do take screenshots on skips, failures, and errors.  I’m happy for us to provide those to TC.  If the --log-html flag already does this, that is great!
Summary: Marionette should take screenshots on failure → Marionette jobs should upload reports (screenshots) for failures
Let me actually get this added quickly. It would have been useful for me too various times in the past weeks. Especially now with our Firefox UI tests moving into the Marionette test jobs.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Try build works just fine:
https://public-artifacts.taskcluster.net/TGUA6_m-RJyIc_0ACg2_Ug/0/public/test_info//report.html

Andreas, do you know why we take screenshots for skipped tests? I don't see a reason for it.
Flags: needinfo?(ato)
(In reply to Henrik Skupin (:whimboo) from comment #7)
> Try build works just fine:
> https://public-artifacts.taskcluster.net/TGUA6_m-RJyIc_0ACg2_Ug/0/public/
> test_info//report.html

This is great!  Thanks for doing this.

> Andreas, do you know why we take screenshots for skipped tests? I don't see
> a reason for it.

I don’t see any screenshots for skipped tests in that report?
Flags: needinfo?(ato)
Scroll down to tests like: test_import_script.py TestImportScriptChrome.test_imports_apply_globally

I think that this is related to our own defined skip decorators. Looks like those haven't been implemented that well enough. I will file a new bug for that, which should also allow us to add a message to the skip call.
I filed bug 1323770 for that issue.
Attachment #8818869 - Flags: review?(mjzffr)
Attachment #8818870 - Attachment is obsolete: true
Comment on attachment 8818869 [details]
Bug 1275243 - Marionette test jobs should create HTML reports with screenshots.

https://reviewboard.mozilla.org/r/98824/#review99508
Attachment #8818869 - Flags: review?(mjzffr) → review+
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9de6c2ae8512
Marionette test jobs should create HTML reports with screenshots. r=maja_zf
https://hg.mozilla.org/mozilla-central/rev/9de6c2ae8512
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla53
Test-only change which would be nice to have for the next ESR release. Can you please uplift to aurora? Thanks.
Whiteboard: [checkin-needed-aurora]
Duplicate of this bug: 1093045
You need to log in before you can comment on or make changes to this bug.