Add the ability to skip tests in the manifest based on the platform

RESOLVED FIXED in Firefox 24

Status

defect
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: davehunt, Assigned: jgriffin)

Tracking

unspecified
mozilla24
ARM
Gonk (Firefox OS)
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox24 fixed, b2g18 fixed, b2g18-v1.0.0 wontfix, b2g18-v1.0.1 wontfix, b2g-v1.1hd fixed)

Details

Attachments

(1 attachment, 2 obsolete attachments)

We would like to be able to skip or fail tests based on the platform they're running on. For example, there may be tests that fail on B2G desktop builds but not on devices. Rather than maintaining separate manifest files, we could take use ManifestDestiny's skip-if/fail-if.

For example, the manifest might look like this:

[test_something_that_fails_on_desktop.py]
fail-if = platform == 'b2gdesktop'

[test_something_that_uses_camera.py]
skip-if = platform == 'b2gdesktop'
This is a WIP.  It adds support for skip-if, fail-if for specific devices, e.g.,

skip-if = device == panda

The possible values of device are qemu, desktop, and then any actual device name (panda, unagi, etc).

Skipped tests appear in the log as:

TEST-SKIP | test_name.py | device = desktop

Skipped tests get added to the 'todo' count at the end of the run.

Dave, does this seem like good approach?
FYI, I don't think fail-if works yet, but I haven't tried.
fail-if doesn't work at all, and will require quite a bit more surgery to support.  In order to avoid stepping on bug 874599, I'm going to split that work off into a separate bug.
Summary: Add the ability to skip/fail tests in the manifest based on the platform → Add the ability to skip tests in the manifest based on the platform
Blocks: 884528
Attachment #764251 - Attachment is obsolete: true
Comment on attachment 764367 [details] [diff] [review]
Add ability to skip tests based on device

Review of attachment 764367 [details] [diff] [review]:
-----------------------------------------------------------------

This changes the "old" self.device to self.device_serial, so we can use self.device to refer to the device that Marionette returns in session_capabilities.

This also fixes some bad whitespace in some (mostly unrelated) code (tabs instead of spaces).
Attachment #764367 - Flags: review?(dave.hunt)
Comment on attachment 764367 [details] [diff] [review]
Add ability to skip tests based on device

Review of attachment 764367 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, just a couple of questions.

::: testing/marionette/client/marionette/runtests.py
@@ +611,4 @@
>                          action='store',
>                          help='host:port of running Gecko instance to connect to')
>          self.add_option('--device',
> +                        dest='device_serial',

Where is this value read? I don't see it read anywhere in this patch.

::: testing/marionette/marionette-server.js
@@ +579,4 @@
>            'javascriptEnabled': true,
>            'nativeEvents': false,
>            'platform': Services.appinfo.OS,
> +          'device': qemu == "1" ? "qemu" : (!device ? "desktop" : device),

Is there (do we need?) a way to distinguish between Firefox desktop and B2G desktop?
Attachment #764367 - Flags: review?(dave.hunt) → review-
Good catches!  self.device_serial is only used by the update tests, so this fixes the only consumer of that.

Re: needing to detect b2g desktop vs firefox, currently, we have no need for that, but it's something that we certainly may need.  Therefore, I've added an 'app' variable to the manifests:

skip-if = app == B2G && device == desktop (for example)

or:

skip-if = app == Firefox && device == desktop

For the gaia-ui-tests, we don't ever intend to run them on firefox, so just using:

skip-if = device == desktop

would be sufficient, I think.
Attachment #764399 - Flags: review?(dave.hunt)
Attachment #764367 - Attachment is obsolete: true
Oops, those should be:

skip-if = app == "B2G" && device == "desktop" (for example)

or:

skip-if = app == "Firefox" && device == "desktop"

(The quotes were missing above.)
Comment on attachment 764399 [details] [diff] [review]
Add ability to skip tests based on device

Review of attachment 764399 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome, thanks! Looking forward to updating the manifest files! :)
Attachment #764399 - Flags: review?(dave.hunt) → review+
Will file another bug to bump and release to pypi after this lands.
https://hg.mozilla.org/mozilla-central/rev/f69a75b67d1f
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.