test.py doesn't output details about skipped/disabled tests via the manifest

NEW
Assigned to

Status

Testing
Mozbase
4 years ago
2 years ago

People

(Reporter: whimboo, Assigned: jotes)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
When you run the unit tests for mozbase via test.py you will not see output for skipped tests. This can cause trouble if we have skipped tests or those are not run due to conditional skip-if/run-if entries. We should show the final skip count at the end.
(Reporter)

Comment 1

4 years ago
To clarify this is not about the skipped tests in the unittests themselves, but if you mark tests as disabled or skipped in the manifest.
Summary: test.py doesn't output details about skipped tests → test.py doesn't output details about skipped/disabled tests via the manifest
(Assignee)

Updated

4 years ago
Assignee: nobody → jot
(Reporter)

Updated

4 years ago
Whiteboard: [mentor=whimboo][lang=py]
(Assignee)

Comment 2

4 years ago
As we discussed on irc, here's a list of skipped tests on windows:
skipped 'test_convert_symlinks.TestSymlinkConversion.test_relpath filter: https://bugzilla.mozilla.org/show_bug.cgi?id=920938'
skipped 'test_convert_symlinks.TestSymlinkConversion.test_relpath_symlink filter: https://bugzilla.mozilla.org/show_bug.cgi?id=920938'
skipped "sut_app.TestApp.test_getAppRoot filter: skip-if: os == 'win'"
skipped "sut_basic.BasicTest.test_init filter: skip-if: os == 'win'"
skipped "sut_basic.BasicTest.test_init_err filter: skip-if: os == 'win'"
skipped "sut_basic.BasicTest.test_shell filter: skip-if: os == 'win'"
skipped "sut_basic.BasicTest.test_timeout_normal filter: skip-if: os == 'win'"
skipped "sut_basic.BasicTest.test_timeout_timeout filter: skip-if: os == 'win'"
skipped "sut_chmod.TestChmod.test_chmod filter: skip-if: os == 'win'"
skipped "sut_copytree.CopyTreeTest.test_copyDir filter: skip-if: os == 'win'"
skipped "sut_copytree.CopyTreeTest.test_copyFile filter: skip-if: os == 'win'"
skipped "sut_copytree.CopyTreeTest.test_copyNonEmptyDir filter: skip-if: os == 'win'"
skipped "sut_fileExists.FileExistsTest.test_onNonexistent filter: skip-if: os == 'win'"
skipped "sut_fileExists.FileExistsTest.test_onRegularFile filter: skip-if: os == 'win'"
skipped "sut_fileExists.FileExistsTest.test_onRoot filter: skip-if: os == 'win'"
skipped "sut_fileMethods.TestFileMethods.test_getDirectory filter: skip-if: os == 'win'"
skipped "sut_fileMethods.TestFileMethods.test_getFile filter: skip-if: os == 'win'"
skipped "sut_fileMethods.TestFileMethods.test_validateFile filter: skip-if: os == 'win'"
skipped "sut_info.TestGetInfo.test_getInfo filter: skip-if: os == 'win'"
skipped "sut_ip.TestGetIP.test_getIP_error filter: skip-if: os == 'win'"
skipped "sut_ip.TestGetIP.test_getIP_eth0 filter: skip-if: os == 'win'"
skipped "sut_ip.TestGetIP.test_getIP_wlan0 filter: skip-if: os == 'win'"
skipped "sut_kill.TestKill.test_killprocess filter: skip-if: os == 'win'"
skipped "sut_list.TestListFiles.test_listFiles filter: skip-if: os == 'win'"
skipped "sut_logcat.TestLogCat.test_getLogcat filter: skip-if: os == 'win'"
skipped "sut_logcat.TestLogCat.test_recordLogcat filter: skip-if: os == 'win'"
skipped "sut_mkdir.MkDirsTest.test_mkdirs filter: skip-if: os == 'win'"
skipped "sut_mkdir.MkDirsTest.test_mkdirs_on_root filter: skip-if: os == 'win'"
skipped "sut_mkdir.MkDirsTest.test_repeated_path_part filter: skip-if: os == 'win'"
skipped "sut_movetree.MoveTreeTest.test_moveDir filter: skip-if: os == 'win'"
skipped "sut_movetree.MoveTreeTest.test_moveFile filter: skip-if: os == 'win'"
skipped "sut_movetree.MoveTreeTest.test_moveNonEmptyDir filter: skip-if: os == 'win'"
skipped "sut_ps.PsTest.test_badProcessList filter: skip-if: os == 'win'"
skipped "sut_ps.PsTest.test_processExist filter: skip-if: os == 'win'"
skipped "sut_ps.PsTest.test_processList filter: skip-if: os == 'win'"
skipped "sut_push.PushTest.test_push filter: skip-if: os == 'win'"
skipped "sut_push.PushTest.test_push_dir filter: skip-if: os == 'win'"
skipped "sut_pull.PullTest.test_pull_failure filter: skip-if: os == 'win'"
skipped "sut_pull.PullTest.test_pull_success filter: skip-if: os == 'win'"
skipped "sut_remove.TestRemove.test_removeDir filter: skip-if: os == 'win'"
skipped "sut_time.TestGetCurrentTime.test_getCurrentTime filter: skip-if: os == 'win'"
skipped "sut_unpackfile.TestUnpack.test_unpackFile filter: skip-if: os == 'win'"
skipped "droidsut_launch.LaunchTest.test_nouserserial filter: skip-if: os == 'win'"
skipped "droidsut_launch.LaunchTest.test_userserial filter: skip-if: os == 'win'"
skipped 'test_mozprocess.ProcTest.test_commandline_from_args filter: bug 877864'
skipped 'test_mozprocess.ProcTest.test_commandline_from_list filter: bug 877864'
skipped 'test_mozprocess.ProcTest.test_commandline_no_args filter: bug 877864'
skipped 'test_mozprocess.ProcTest.test_commandline_over_specified filter: bug 877864'
skipped 'test_mozprocess.ProcTest.test_commandline_overspecified filter: bug 877864'
skipped 'test_mozprocess.ProcTest.test_process_kill filter: bug 877864'
skipped 'test_mozprocess.ProcTest.test_process_normal_finish filter: bug 877864'
skipped 'test_mozprocess.ProcTest.test_process_output_twice filter: bug 877864'
skipped 'test_mozprocess.ProcTest.test_process_timeout filter: bug 877864'
skipped 'test_mozprocess.ProcTest.test_process_timeout_no_kill filter: bug 877864'
skipped 'test_mozprocess.ProcTest.test_process_wait filter: bug 877864'
skipped 'test_mozprocess.ProcTest.test_process_waitnotimeout filter: bug 877864'
skipped 'test_mozprocess.ProcTest.test_process_waittimeout filter: bug 877864'
skipped 'test_mozprocess_kill.ProcTestKill.test_process_kill filter: skip-if: (os == "win")'
skipped 'test_mozprocess_kill.ProcTestKill.test_process_kill_broad filter: skip-if: (os == "win")'
skipped 'test_mozprocess_kill.ProcTestKill.test_process_kill_deep filter: skip-if: (os == "win")'
skipped 'test_mozprocess_kill.ProcTestKill.test_process_kill_deep_wait filter: skip-if: (os == "win")'
skipped 'test_mozprocess_kill_broad_wait.ProcTestKill.test_process_kill_broad_wait filter: bug 921632'
skipped 'test_mozprocess_misc.ProcTestMisc.test_process_output_twice filter: skip-if: (os == "win")'
skipped 'test_mozprocess_wait.ProcTestWait.test_process_normal_finish filter: skip-if: (os == "win")'
skipped 'test_mozprocess_wait.ProcTestWait.test_process_timeout filter: skip-if: (os == "win")'
skipped 'test_mozprocess_wait.ProcTestWait.test_process_wait filter: skip-if: (os == "win")'
skipped 'test_mozprocess_wait.ProcTestWait.test_process_waitnotimeout filter: skip-if: (os == "win")'
skipped 'test_mozprocess_wait.ProcTestWait.test_process_waittimeout filter: skip-if: (os == "win")'
skipped 'test_mozprocess_nonewline.ProcTestMisc.test_process_output_nonewline filter: skip-if: (os == "win")'

I'll post patch in next few minutes.
(Assignee)

Comment 3

4 years ago
Created attachment 8365061 [details] [diff] [review]
0001-Bug-962476-extended-test-results-for-manifest-suppor.patch

First version of patch which outputs skipped tests.
Attachment #8365061 - Flags: review?(hskupin)
Whiteboard: [mentor=whimboo][lang=py] → [mentor=whimboo][lang=py][good first bug]
(Reporter)

Comment 4

4 years ago
Comment on attachment 8365061 [details] [diff] [review]
0001-Bug-962476-extended-test-results-for-manifest-suppor.patch

Review of attachment 8365061 [details] [diff] [review]:
-----------------------------------------------------------------

Jarek, sorry but I missed this review. Meanwhile mozbase has been moved to mozilla-central, so I cannot test this patch. Would you mind updating it please to the new location /testing/mozbase? Also I think you should better ask jgriffin, wlach, or ahal for review on this. I might not be the right person to mentor this bug.
Attachment #8365061 - Flags: review?(hskupin)
(Reporter)

Updated

4 years ago
Whiteboard: [mentor=whimboo][lang=py][good first bug]
(Assignee)

Comment 5

2 years ago
Hi :whimboo,
Is this ticket still valid or we can close it?
Flags: needinfo?(hskupin)
(Reporter)

Comment 6

2 years ago
Oh, a forgotten bug! Thanks for the reminder Jarek. Given that it is still valid, do you want to continue working on the patch?
Flags: needinfo?(hskupin) → needinfo?(jot)
(Assignee)

Comment 7

2 years ago
:whimboo
A pretty long time no see :-D
I can try to look into this bug -> could you tell me where is the current code? I mean which repository/docs etc. That would help a lot.

Thanks in advance!
Flags: needinfo?(jot) → needinfo?(hskupin)
(Reporter)

Comment 8

2 years ago
OMG now I actually see who you are! I didn't notice before due to the nick change. :D It's great to see that you are still around! And I'm more than happy to give you more details.

So meanwhile all of the mozbase code can be found in mozilla-central:
https://dxr.mozilla.org/mozilla-central/source/testing/mozbase

The way how tests are run hasn't changed, so you can still find the test.py file directly under this folder. 

If you haven't worked with mozilla-central yet, here some guidelines for you:
http://ateam-bootcamp.readthedocs.io/en/latest/

Oh, and great to see you want to pick this up!
Flags: needinfo?(hskupin)
(Assignee)

Comment 9

2 years ago
For some reason I didn't receive any notification about your response. I'll try to look into this issue during next week.

Thx for your help.
You need to log in before you can comment on or make changes to this bug.