Marionette jobs should upload reports (screenshots) for failures

RESOLVED FIXED in Firefox 52

Status

Testing
Marionette
RESOLVED FIXED
2 years ago
11 months ago

People

(Reporter: Gijs, Assigned: whimboo)

Tracking

Trunk
mozilla53
Points:
---
Bug Flags:
in-testsuite -

Firefox Tracking Flags

(firefox52 fixed, firefox53 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

2 years ago
Per bug 1272652 comment 22: it seems firefox-ui-test takes screenshots on failure but marionette does not (or doesn't stick them in the upload dir so we can't see them on treeherder). It would be helpful if it did.
(Assignee)

Comment 1

2 years ago
This could easily be done when we run our Marionette testes with the --log-html option, and instruct taskcluster to also upload that file. This might most likely be a change in mozharness in the marionette entry script.
We do take screenshots on skips, failures, and errors.  I’m happy for us to provide those to TC.  If the --log-html flag already does this, that is great!
Keywords: ateam-marionette-runner
(Assignee)

Updated

a year ago
Summary: Marionette should take screenshots on failure → Marionette jobs should upload reports (screenshots) for failures
(Assignee)

Updated

a year ago
Keywords: ateam-marionette-runner
(Assignee)

Comment 4

a year ago
Let me actually get this added quickly. It would have been useful for me too various times in the past weeks. Especially now with our Firefox UI tests moving into the Marionette test jobs.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
(Assignee)

Comment 7

a year ago
Try build works just fine:
https://public-artifacts.taskcluster.net/TGUA6_m-RJyIc_0ACg2_Ug/0/public/test_info//report.html

Andreas, do you know why we take screenshots for skipped tests? I don't see a reason for it.
Flags: needinfo?(ato)
(In reply to Henrik Skupin (:whimboo) from comment #7)
> Try build works just fine:
> https://public-artifacts.taskcluster.net/TGUA6_m-RJyIc_0ACg2_Ug/0/public/
> test_info//report.html

This is great!  Thanks for doing this.

> Andreas, do you know why we take screenshots for skipped tests? I don't see
> a reason for it.

I don’t see any screenshots for skipped tests in that report?
Flags: needinfo?(ato)
(Assignee)

Comment 9

a year ago
Scroll down to tests like: test_import_script.py TestImportScriptChrome.test_imports_apply_globally

I think that this is related to our own defined skip decorators. Looks like those haven't been implemented that well enough. I will file a new bug for that, which should also allow us to add a message to the skip call.
(Assignee)

Comment 10

a year ago
I filed bug 1323770 for that issue.
(Assignee)

Updated

a year ago
Attachment #8818869 - Flags: review?(mjzffr)
(Assignee)

Updated

a year ago
Attachment #8818870 - Attachment is obsolete: true

Comment 11

a year ago
mozreview-review
Comment on attachment 8818869 [details]
Bug 1275243 - Marionette test jobs should create HTML reports with screenshots.

https://reviewboard.mozilla.org/r/98824/#review99508
Attachment #8818869 - Flags: review?(mjzffr) → review+

Comment 12

a year ago
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9de6c2ae8512
Marionette test jobs should create HTML reports with screenshots. r=maja_zf

Comment 13

a year ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/9de6c2ae8512
Status: ASSIGNED → RESOLVED
Last Resolved: a year ago
status-firefox53: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla53
(Assignee)

Comment 14

a year ago
Test-only change which would be nice to have for the next ESR release. Can you please uplift to aurora? Thanks.
status-firefox52: --- → affected
Whiteboard: [checkin-needed-aurora]

Comment 15

a year ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-aurora/rev/71a43018dae1
status-firefox52: affected → fixed
Flags: in-testsuite-
Whiteboard: [checkin-needed-aurora]
(Assignee)

Updated

11 months ago
Duplicate of this bug: 1093045
You need to log in before you can comment on or make changes to this bug.