859746 - [mutt] Mutt javascript tests do not print the RESULTS block

Status: RESOLVED FIXED

(Testing Graveyard :: Mozmill, defect)

Description

[Andrei Eftimie]

Running
> mutt testjs

Does not print the RESULTS block at all.
It should show something like this:

> RESULTS | Passed: 1
> RESULTS | Failed: 0
> RESULTS | Skipped: 0

Comment 1

[Andrei Eftimie]

Attachment: Patch v1

This patch enables final results for mutt js tests.

Desired output:
> RESULTS | Passed: 46
> RESULTS | Failed: 1
> RESULTS | Skipped: 0

I'm asking for needinfo since I've noticed weird behaviour on python tests aswell (and we might need to change the summary and fix that aswell since we're here).

Basically only 1 python test reports final RESULTS, and it does that right after running the test. More tests are run afterwards.

This does not appear to behave as it should.

The test that does report is:
> testscreenshotpath.py

Which in turns runs:
> test_dummy.js

And only this js test logs results.

My question are:
1. Should python tests also log final RESULTS? (I assume yes)
2. test_dummy.js should probably *not* log RESULTS

Please let me know if my assumptions are correct or not, so I can safely proceed.
Thanks

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Yes, lets do both in the same test if that is the case. But not sure what you mean with python results are not shown. Those will not accumulate the pass/fail/skip count of the js tests they are exercising. So what's the final output of the py tests?

Comment 3

[Andrei Eftimie]

Running:
> mutt testpy -b %path%
Yields no final RESULTS block.

Halfway through you get the following:
> TEST-START | /private/var/folders/9l/sn_p3bw914s360j602z20jsc0000gq/T/tmp_Hxq1W/test_dummy.js | test
> TEST-PASS | /private/var/folders/9l/sn_p3bw914s360j602z20jsc0000gq/T/tmp_Hxq1W/test_dummy.js | test
> TEST-END | /private/var/folders/9l/sn_p3bw914s360j602z20jsc0000gq/T/tmp_Hxq1W/test_dummy.js | finished in 1ms
> RESULTS | Passed: 1
> RESULTS | Failed: 0
> RESULTS | Skipped: 0
> .Screenshots saved in /var/folders/9l/sn_p3bw914s360j602z20jsc0000gq/T/tmpUdjnhH
> Screenshots saved in /var/folders/9l/sn_p3bw914s360j602z20jsc0000gq/T/tmpUdjnhH
> .Screenshots saved in /var/folders/9l/sn_p3bw914s360j602z20jsc0000gq/T/tmptWfkeD
> Screenshots saved in /var/folders/9l/sn_p3bw914s360j602z20jsc0000gq/T/tmptWfkeD
> ..

Then some more python tests are run.

At the end:
> Ran 14 tests in 127.119s
> 
> FAILED (failures=3, errors=1)
> 
> Some tests were unsuccessful.
> 
> ===========================================================================
> Python Failures:
> [...actual python logged errors...]
> 
> ===========================================================================
> No Javascript Failures

No final RESULTS block for python tests is shown.

Comment 4

[Andrei Eftimie]

Starting to understand more of mozmill/mutt code.

The RESULTS block is generated for Mozmill, yet the python tests are just python unittests (or just run trough the unittest library).

If I understand this correctly, mutt works correctly ATM with logging the RESULTS block only for JS tests (since they are the only ones that pass through mozmill).

Just to make things clear:
Should we amend the code to log the RESULTS block for python tests?

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, so for the python tests we show the correct output. There is not such a final results block as for js tests. Both type of tests are different and can't be compared. The only fact is that python tests have to make use of some js tests to check the proper interaction between the python and js framework. Result blocks you can see in the middle should usually not be shown. Looks like that for those tests the logger needs to be updated. If there is no bug on file for that yes, please create one.

Reverting the summary to javascript tests only given that this is the issue we have to fix here.

Comment 6

[Andrei Eftimie]

Comment on attachment 736731 [details] [diff] [review]
Patch v1

Raised bug 861766 for js RESULTS block in python tests.

Asked for review on this patch.
We correctly see the RESULTS block when running:
> mutt testjs -b %path%

Comment 7

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 736731 [details] [diff] [review]
Patch v1

Review of attachment 736731 [details] [diff] [review]:
-----------------------------------------------------------------

Works for me. Leaving review requests since is a Mozmill bug and I don't have commit access.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 736731 [details] [diff] [review]
Patch v1

Review of attachment 736731 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good. Beside that I see a dozen of failures but only some are marked as failed. This may be related to bug 846351:

RESULTS | Passed: 44
RESULTS | Failed: 3
RESULTS | Skipped: 0

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

https://github.com/mozilla/mozmill/commit/f6b1cfc11506d03dc446cba87b2426787b8cad48

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As seen by Vlad over on bug 861766 we now call finish() multiple times. Once in the API itself and later in the client code (tests). That's not the right way to go and we indeed should get this fixed correctly.

Clint, we only allow a single run with mozmill, so calling .finish() at the end of the run() method should be fine, and we should also update the tests.

https://github.com/mozilla/mozmill/blob/master/mozmill/mozmill/__init__.py#L424

But I would also like to get your feedback. What do you think would be best?

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This patch is indeed wrong. So i backed it out on master:
https://github.com/mozilla/mozmill/commit/46d3190bef3c1c6eb4cf90f0e20eb5d1cb37d3c2

The problem here is not in Mozmill itself but inside Mutt, which does not call m.finish() after the js tests have been finished.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

this only affects how mutt works and so it doesn't block the 2.0 release.

Comment 13

[Andrei Eftimie]

Attachment: patch v2

We're now calling the finish method from mutt directly after mutt finishes js tests.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 757549 [details] [diff] [review]
patch v2

Review of attachment 757549 [details] [diff] [review]:
-----------------------------------------------------------------

::: mutt/mutt/mutt.py
@@ +198,1 @@
>      return m.results

finish() already returns the results from the run. So please return that instead. Otherwise it looks like the correct fix now.

Comment 15

[Andrei Eftimie]

Attachment: patch v3

Good catch!
Updated the patch.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Landed on master:
https://github.com/mozilla/mozmill/commit/370a3e6b1c607c4d10874716108ed99514aac1ae