Last Comment Bug 802742 - Marionette should list all failed tests at the end of the testrun
: Marionette should list all failed tests at the end of the testrun
Status: RESOLVED FIXED
[good first bug][lang=python][mentor=...
:
Product: Testing
Classification: Components
Component: Marionette (show other bugs)
: unspecified
: All All
: -- normal (vote)
: mozilla25
Assigned To: Florin Strugariu [:Bebe]
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2012-10-17 11:53 PDT by Henrik Skupin (:whimboo)
Modified: 2013-11-16 14:13 PST (History)
9 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
wontfix
wontfix
fixed
fixed
wontfix
wontfix
fixed


Attachments
Bug 802742 - Marionette should list all failed tests at the end of the testrun (4.94 KB, patch)
2013-07-05 02:31 PDT, Florin Strugariu [:Bebe]
no flags Details | Diff | Review
Bug 802742 - Marionette should list all failed tests at the end of the testrun (2.11 KB, patch)
2013-07-10 03:06 PDT, Florin Strugariu [:Bebe]
malini: review+
Details | Diff | Review

Description Henrik Skupin (:whimboo) 2012-10-17 11:53:21 PDT
Right now you only get a very brief summary for the test results at the end of a testrun. If there are failing tests it's very hard to figure out which of those are failing. You would have to scrape the whole console output:

SUMMARY
-------
passed: 108
failed: 2
todo: 0


It would be nice if Marionette could print a list of those failed tests at the end.
Comment 1 Malini Das [:mdas] - Away, not checking bugmail 2013-05-28 09:47:59 PDT
For anyone interested in this bug, please follow https://developer.mozilla.org/en-US/docs/Marionette/Developer_setup first to get your dev environment set up. You'll need to get it set up to work with the client, not the server.

Generally speaking, to solve this bug, we'll need to modify (mozilla-central)/testing/marionette/client/marionette/runtests.py so that when the MarionetteTestRunner runs each test, it will gather all failed tests, and when it is done the suite, it should print out the failed tests. MarionetteTestRunner creates MarionetteTextTestRunner objects to run the tests, so you'll need to modify MarionetteTextTestRunner to gather the information we need.
Comment 2 MikeLing 2013-06-01 22:11:43 PDT
Hello,

The bug seems very insteresting and I would like to fix it,This'll be my first bug, so I'd need some guidance.Thanks:)
Comment 3 Malini Das [:mdas] - Away, not checking bugmail 2013-07-04 14:11:13 PDT
Bebe wants to work on this, and asked if this should be default behaviour or triggered by an option. I think having an option, say --list-failed or maybe --verbose, would be ideal. It would list the failed tests.

I'd like this to list the failure messages as well, so instead of having them interlaced with the test output, it can instead be printed at the end. This could be in addition to listing the failed tests if this is too difficult for a first bug.
Comment 4 Dave Hunt (:davehunt) 2013-07-04 14:29:41 PDT
Did anyone respond to Mike Ling from comment 2, who also expressed an interest in working on this bug?
Comment 5 Florin Strugariu [:Bebe] 2013-07-05 01:57:24 PDT
(In reply to Dave Hunt (:davehunt) from comment #4)
> Did anyone respond to Mike Ling from comment 2, who also expressed an
> interest in working on this bug?
As I talked with mdas last night MikeLing is woking on some other bugs and this issue is free.
MikeLing if you you are still working on this I'm more than happy to leave to you
Comment 6 Florin Strugariu [:Bebe] 2013-07-05 02:31:49 PDT
Created attachment 771580 [details] [diff] [review]
Bug 802742 - Marionette should list all failed tests at the end of the testrun

This is what I did for now. 
mdas can you review it and check if I'm on the right track? After that I can add the other options
Comment 7 Malini Das [:mdas] - Away, not checking bugmail 2013-07-08 10:04:28 PDT
(In reply to Florin Strugariu [:Bebe] from comment #6)
> Created attachment 771580 [details] [diff] [review]
> Bug 802742 - Marionette should list all failed tests at the end of the
> testrun
> 
> This is what I did for now. 
> mdas can you review it and check if I'm on the right track? After that I can
> add the other options

This looks good, and actually, I spoke with jgriffin and we can just have the failed tests be listed by default. We can ignore the command line requirement, and ignore the error message printing. As such, your patch's complexity is greatly reduced:)
Comment 8 Florin Strugariu [:Bebe] 2013-07-10 03:06:29 PDT
Created attachment 773179 [details] [diff] [review]
Bug 802742 - Marionette should list all failed tests at the end of the testrun
Comment 9 Malini Das [:mdas] - Away, not checking bugmail 2013-07-10 19:22:20 PDT
Comment on attachment 773179 [details] [diff] [review]
Bug 802742 - Marionette should list all failed tests at the end of the testrun

Review of attachment 773179 [details] [diff] [review]:
-----------------------------------------------------------------

great, thanks! tested with python2.6 as well. I'll push this out to m-i soon.
Comment 10 Malini Das [:mdas] - Away, not checking bugmail 2013-07-10 19:25:24 PDT
pushed to inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/69e7468ac58f
Comment 11 Ed Morley [:emorley] 2013-07-11 03:00:28 PDT
https://hg.mozilla.org/mozilla-central/rev/69e7468ac58f
Comment 12 Ryan VanderMeulen [:RyanVM] 2013-07-12 06:52:19 PDT
https://hg.mozilla.org/releases/mozilla-b2g18/rev/8b39fe30acb8

Note You need to log in before you can comment on or make changes to this bug.