Add the ability to skip tests in the manifest based on the platform

RESOLVED FIXED in Firefox 24, Firefox OS v1.1hd

Status

RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: davehunt, Assigned: jgriffin)

Tracking

unspecified
mozilla24
ARM
Gonk (Firefox OS)
Points:
---

Firefox Tracking Flags

(firefox24 fixed, b2g18 fixed, b2g18-v1.0.0 wontfix, b2g18-v1.0.1 wontfix, b2g-v1.1hd fixed)

Details

Attachments

(1 attachment, 2 obsolete attachments)

We would like to be able to skip or fail tests based on the platform they're running on. For example, there may be tests that fail on B2G desktop builds but not on devices. Rather than maintaining separate manifest files, we could take use ManifestDestiny's skip-if/fail-if.

For example, the manifest might look like this:

[test_something_that_fails_on_desktop.py]
fail-if = platform == 'b2gdesktop'

[test_something_that_uses_camera.py]
skip-if = platform == 'b2gdesktop'
(Assignee)

Comment 1

5 years ago
Created attachment 764251 [details] [diff] [review]
Add ability to skip tests based on device

This is a WIP.  It adds support for skip-if, fail-if for specific devices, e.g.,

skip-if = device == panda

The possible values of device are qemu, desktop, and then any actual device name (panda, unagi, etc).

Skipped tests appear in the log as:

TEST-SKIP | test_name.py | device = desktop

Skipped tests get added to the 'todo' count at the end of the run.

Dave, does this seem like good approach?
(Assignee)

Comment 2

5 years ago
FYI, I don't think fail-if works yet, but I haven't tried.
(Assignee)

Comment 3

5 years ago
fail-if doesn't work at all, and will require quite a bit more surgery to support.  In order to avoid stepping on bug 874599, I'm going to split that work off into a separate bug.
Summary: Add the ability to skip/fail tests in the manifest based on the platform → Add the ability to skip tests in the manifest based on the platform
(Assignee)

Updated

5 years ago
Blocks: 884528
(Assignee)

Comment 4

5 years ago
Created attachment 764367 [details] [diff] [review]
Add ability to skip tests based on device

Pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=388764ab7626
(Assignee)

Updated

5 years ago
Attachment #764251 - Attachment is obsolete: true
(Assignee)

Comment 5

5 years ago
Comment on attachment 764367 [details] [diff] [review]
Add ability to skip tests based on device

Review of attachment 764367 [details] [diff] [review]:
-----------------------------------------------------------------

This changes the "old" self.device to self.device_serial, so we can use self.device to refer to the device that Marionette returns in session_capabilities.

This also fixes some bad whitespace in some (mostly unrelated) code (tabs instead of spaces).
Attachment #764367 - Flags: review?(dave.hunt)
Comment on attachment 764367 [details] [diff] [review]
Add ability to skip tests based on device

Review of attachment 764367 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, just a couple of questions.

::: testing/marionette/client/marionette/runtests.py
@@ +611,4 @@
>                          action='store',
>                          help='host:port of running Gecko instance to connect to')
>          self.add_option('--device',
> +                        dest='device_serial',

Where is this value read? I don't see it read anywhere in this patch.

::: testing/marionette/marionette-server.js
@@ +579,4 @@
>            'javascriptEnabled': true,
>            'nativeEvents': false,
>            'platform': Services.appinfo.OS,
> +          'device': qemu == "1" ? "qemu" : (!device ? "desktop" : device),

Is there (do we need?) a way to distinguish between Firefox desktop and B2G desktop?
Attachment #764367 - Flags: review?(dave.hunt) → review-
(Assignee)

Comment 7

5 years ago
Created attachment 764399 [details] [diff] [review]
Add ability to skip tests based on device

Good catches!  self.device_serial is only used by the update tests, so this fixes the only consumer of that.

Re: needing to detect b2g desktop vs firefox, currently, we have no need for that, but it's something that we certainly may need.  Therefore, I've added an 'app' variable to the manifests:

skip-if = app == B2G && device == desktop (for example)

or:

skip-if = app == Firefox && device == desktop

For the gaia-ui-tests, we don't ever intend to run them on firefox, so just using:

skip-if = device == desktop

would be sufficient, I think.
Attachment #764399 - Flags: review?(dave.hunt)
(Assignee)

Updated

5 years ago
Attachment #764367 - Attachment is obsolete: true
(Assignee)

Comment 8

5 years ago
Oops, those should be:

skip-if = app == "B2G" && device == "desktop" (for example)

or:

skip-if = app == "Firefox" && device == "desktop"

(The quotes were missing above.)
Comment on attachment 764399 [details] [diff] [review]
Add ability to skip tests based on device

Review of attachment 764399 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome, thanks! Looking forward to updating the manifest files! :)
Attachment #764399 - Flags: review?(dave.hunt) → review+
(Assignee)

Comment 11

5 years ago
Will file another bug to bump and release to pypi after this lands.
https://hg.mozilla.org/mozilla-central/rev/f69a75b67d1f
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Assignee)

Comment 13

5 years ago
http://hg.mozilla.org/releases/mozilla-b2g18/rev/cd86de40cd5c
status-b2g18: --- → fixed
status-b2g18-v1.0.0: --- → wontfix
status-b2g18-v1.0.1: --- → wontfix
status-firefox24: --- → fixed
status-firefox25: --- → fixed
https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/cd86de40cd5c
status-b2g-v1.1hd: --- → fixed
status-firefox25: fixed → ---
You need to log in before you can comment on or make changes to this bug.