802742 - Marionette should list all failed tests at the end of the testrun

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Right now you only get a very brief summary for the test results at the end of a testrun. If there are failing tests it's very hard to figure out which of those are failing. You would have to scrape the whole console output:

SUMMARY
-------
passed: 108
failed: 2
todo: 0


It would be nice if Marionette could print a list of those failed tests at the end.

Comment 1

[Malini Das [:mdas] - Away, not checking bugmail]

For anyone interested in this bug, please follow https://developer.mozilla.org/en-US/docs/Marionette/Developer_setup first to get your dev environment set up. You'll need to get it set up to work with the client, not the server.

Generally speaking, to solve this bug, we'll need to modify (mozilla-central)/testing/marionette/client/marionette/runtests.py so that when the MarionetteTestRunner runs each test, it will gather all failed tests, and when it is done the suite, it should print out the failed tests. MarionetteTestRunner creates MarionetteTextTestRunner objects to run the tests, so you'll need to modify MarionetteTextTestRunner to gather the information we need.

Comment 2

[MikeLing]

Hello,

The bug seems very insteresting and I would like to fix it,This'll be my first bug, so I'd need some guidance.Thanks:)

Comment 3

[Malini Das [:mdas] - Away, not checking bugmail]

Bebe wants to work on this, and asked if this should be default behaviour or triggered by an option. I think having an option, say --list-failed or maybe --verbose, would be ideal. It would list the failed tests.

I'd like this to list the failure messages as well, so instead of having them interlaced with the test output, it can instead be printed at the end. This could be in addition to listing the failed tests if this is too difficult for a first bug.

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Did anyone respond to Mike Ling from comment 2, who also expressed an interest in working on this bug?

Comment 5

[Florin Strugariu [:Bebe]]

(In reply to Dave Hunt (:davehunt) from comment #4)
> Did anyone respond to Mike Ling from comment 2, who also expressed an
> interest in working on this bug?
As I talked with mdas last night MikeLing is woking on some other bugs and this issue is free.
MikeLing if you you are still working on this I'm more than happy to leave to you

Comment 6

[Florin Strugariu [:Bebe]]

Attachment: Bug 802742 - Marionette should list all failed tests at the end of the testrun

This is what I did for now. 
mdas can you review it and check if I'm on the right track? After that I can add the other options

Comment 7

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Florin Strugariu [:Bebe] from comment #6)
> Created attachment 771580 [details] [diff] [review]
> Bug 802742 - Marionette should list all failed tests at the end of the
> testrun
> 
> This is what I did for now. 
> mdas can you review it and check if I'm on the right track? After that I can
> add the other options

This looks good, and actually, I spoke with jgriffin and we can just have the failed tests be listed by default. We can ignore the command line requirement, and ignore the error message printing. As such, your patch's complexity is greatly reduced:)

Comment 8

[Florin Strugariu [:Bebe]]

Attachment: Bug 802742 - Marionette should list all failed tests at the end of the testrun

Comment 9

[Malini Das [:mdas] - Away, not checking bugmail]

Comment on attachment 773179 [details] [diff] [review]
Bug 802742 - Marionette should list all failed tests at the end of the testrun

Review of attachment 773179 [details] [diff] [review]:
-----------------------------------------------------------------

great, thanks! tested with python2.6 as well. I'll push this out to m-i soon.

Comment 10

[Malini Das [:mdas] - Away, not checking bugmail]

pushed to inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/69e7468ac58f

Comment 11

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/69e7468ac58f

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/8b39fe30acb8

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/8b39fe30acb8