1256682 - "mach marionette-test" does not obey tags in manifest files and cause lots of test failures

Status: RESOLVED WORKSFORME

(Remote Protocol :: Marionette, defect, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

While testing my patch on bug 1256439 I was using mach to run the marionette unit tests like "mach marionette-test". As result lots of tests are failing because they were run with Firefox desktop while they shouldn't as given by the tags in the manifest file.

Here an example command:

mach marionette-test testing/marionette/harness/marionette/tests/unit-tests.ini

ERROR test_element_touch.py testElementTouch.test_scrolling
ERROR test_element_touch.py testElementTouch.test_touch
ERROR test_screen_orientation.py TestScreenOrientation.test_set_invalid_orientation
ERROR test_screen_orientation.py TestScreenOrientation.test_set_null_orientation
ERROR test_screen_orientation.py TestScreenOrientation.test_set_orientation_to_landscape_primary
ERROR test_screen_orientation.py TestScreenOrientation.test_set_orientation_to_landscape_secondary
ERROR test_screen_orientation.py TestScreenOrientation.test_set_orientation_to_portrait_primary
ERROR test_screen_orientation.py TestScreenOrientation.test_set_orientation_to_portrait_secondary
ERROR test_screen_orientation.py TestScreenOrientation.test_set_orientation_to_shorthand_landscape
ERROR test_screen_orientation.py TestScreenOrientation.test_set_orientation_to_shorthand_portrait
ERROR test_screen_orientation.py TestScreenOrientation.test_set_orientation_with_mixed_casing
FAIL test_accessiblecaret_cursor_mode.py AccessibleCaretCursorModeTestCase.test_caret_not_appear_when_typing_in_scrollable_content
FAIL test_shared_all.py TestSharedUnits.test_units
FAIL test_desktop_all.py TestDesktopUnits.test_units
FAIL test_standards.py TestStandardsUnits.test_units
FAIL test_modules.py TestModulesUnits.test_units
ERROR test_interface.py TestInterfaceUnits.test_units

As you can see tests only suited for B2G are executed on Desktop. Not sure if that happens because I use build faster? David, do you have an idea?

Comment 1

[Andreas Tolfsen ❲:ato❳]

chmanchester appears to know things about the manifest parser the Marionette harness uses.

Comment 2

[Chris Manchester (limited bugmail, email directly)]

I think the marionette runner wants "--type=browser" passed in to filter these correctly. Mach should probably provide this argument by default under some circumstances.

Comment 3

[Andreas Tolfsen ❲:ato❳]

Is this still a problem for you?

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I'm sure I fixed it with a patch on a different bug.