Marionette should list all failed tests at the end of the testrun

RESOLVED FIXED in Firefox 25, Firefox OS v1.1hd

Status

Testing
Marionette
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: whimboo, Assigned: Bebe)

Tracking

unspecified
mozilla25
Points:
---

Firefox Tracking Flags

(firefox23 wontfix, firefox24 wontfix, firefox25 fixed, b2g18 fixed, b2g18-v1.0.0 wontfix, b2g18-v1.0.1 wontfix, b2g-v1.1hd fixed)

Details

(Whiteboard: [good first bug][lang=python][mentor=mdas])

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

5 years ago
Right now you only get a very brief summary for the test results at the end of a testrun. If there are failing tests it's very hard to figure out which of those are failing. You would have to scrape the whole console output:

SUMMARY
-------
passed: 108
failed: 2
todo: 0


It would be nice if Marionette could print a list of those failed tests at the end.
For anyone interested in this bug, please follow https://developer.mozilla.org/en-US/docs/Marionette/Developer_setup first to get your dev environment set up. You'll need to get it set up to work with the client, not the server.

Generally speaking, to solve this bug, we'll need to modify (mozilla-central)/testing/marionette/client/marionette/runtests.py so that when the MarionetteTestRunner runs each test, it will gather all failed tests, and when it is done the suite, it should print out the failed tests. MarionetteTestRunner creates MarionetteTextTestRunner objects to run the tests, so you'll need to modify MarionetteTextTestRunner to gather the information we need.
Whiteboard: [good first bug][lang=python][mentor=mdas]

Comment 2

4 years ago
Hello,

The bug seems very insteresting and I would like to fix it,This'll be my first bug, so I'd need some guidance.Thanks:)
Bebe wants to work on this, and asked if this should be default behaviour or triggered by an option. I think having an option, say --list-failed or maybe --verbose, would be ideal. It would list the failed tests.

I'd like this to list the failure messages as well, so instead of having them interlaced with the test output, it can instead be printed at the end. This could be in addition to listing the failed tests if this is too difficult for a first bug.
(Assignee)

Updated

4 years ago
Assignee: nobody → florin.strugariu
Did anyone respond to Mike Ling from comment 2, who also expressed an interest in working on this bug?
(Assignee)

Comment 5

4 years ago
(In reply to Dave Hunt (:davehunt) from comment #4)
> Did anyone respond to Mike Ling from comment 2, who also expressed an
> interest in working on this bug?
As I talked with mdas last night MikeLing is woking on some other bugs and this issue is free.
MikeLing if you you are still working on this I'm more than happy to leave to you
(Assignee)

Comment 6

4 years ago
Created attachment 771580 [details] [diff] [review]
Bug 802742 - Marionette should list all failed tests at the end of the testrun

This is what I did for now. 
mdas can you review it and check if I'm on the right track? After that I can add the other options
Attachment #771580 - Flags: review?(mdas)
(In reply to Florin Strugariu [:Bebe] from comment #6)
> Created attachment 771580 [details] [diff] [review]
> Bug 802742 - Marionette should list all failed tests at the end of the
> testrun
> 
> This is what I did for now. 
> mdas can you review it and check if I'm on the right track? After that I can
> add the other options

This looks good, and actually, I spoke with jgriffin and we can just have the failed tests be listed by default. We can ignore the command line requirement, and ignore the error message printing. As such, your patch's complexity is greatly reduced:)
(Assignee)

Comment 8

4 years ago
Created attachment 773179 [details] [diff] [review]
Bug 802742 - Marionette should list all failed tests at the end of the testrun
Attachment #771580 - Attachment is obsolete: true
Attachment #771580 - Flags: review?(mdas)
Attachment #773179 - Flags: review?(mdas)
Comment on attachment 773179 [details] [diff] [review]
Bug 802742 - Marionette should list all failed tests at the end of the testrun

Review of attachment 773179 [details] [diff] [review]:
-----------------------------------------------------------------

great, thanks! tested with python2.6 as well. I'll push this out to m-i soon.
Attachment #773179 - Flags: review?(mdas) → review+
pushed to inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/69e7468ac58f
https://hg.mozilla.org/mozilla-central/rev/69e7468ac58f
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla25
Keywords: checkin-needed
https://hg.mozilla.org/releases/mozilla-b2g18/rev/8b39fe30acb8
status-b2g18: --- → fixed
status-b2g18-v1.0.0: --- → wontfix
status-b2g18-v1.0.1: --- → wontfix
status-b2g-v1.1hd: --- → affected
status-firefox23: --- → wontfix
status-firefox24: --- → wontfix
status-firefox25: --- → fixed
Keywords: checkin-needed
https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/8b39fe30acb8
status-b2g-v1.1hd: affected → fixed
You need to log in before you can comment on or make changes to this bug.