Closed
Bug 883281
Opened 12 years ago
Closed 12 years ago
Add the ability to skip tests in the manifest based on the platform
Categories
(Remote Protocol :: Marionette, defect)
Tracking
(firefox24 fixed, b2g18 fixed, b2g18-v1.0.0 wontfix, b2g18-v1.0.1 wontfix, b2g-v1.1hd fixed)
RESOLVED
FIXED
mozilla24
People
(Reporter: davehunt, Assigned: jgriffin)
References
Details
Attachments
(1 file, 2 obsolete files)
6.61 KB,
patch
|
davehunt
:
review+
|
Details | Diff | Splinter Review |
We would like to be able to skip or fail tests based on the platform they're running on. For example, there may be tests that fail on B2G desktop builds but not on devices. Rather than maintaining separate manifest files, we could take use ManifestDestiny's skip-if/fail-if.
For example, the manifest might look like this:
[test_something_that_fails_on_desktop.py]
fail-if = platform == 'b2gdesktop'
[test_something_that_uses_camera.py]
skip-if = platform == 'b2gdesktop'
Assignee | ||
Comment 1•12 years ago
|
||
This is a WIP. It adds support for skip-if, fail-if for specific devices, e.g.,
skip-if = device == panda
The possible values of device are qemu, desktop, and then any actual device name (panda, unagi, etc).
Skipped tests appear in the log as:
TEST-SKIP | test_name.py | device = desktop
Skipped tests get added to the 'todo' count at the end of the run.
Dave, does this seem like good approach?
Assignee | ||
Comment 2•12 years ago
|
||
FYI, I don't think fail-if works yet, but I haven't tried.
Assignee | ||
Comment 3•12 years ago
|
||
fail-if doesn't work at all, and will require quite a bit more surgery to support. In order to avoid stepping on bug 874599, I'm going to split that work off into a separate bug.
Summary: Add the ability to skip/fail tests in the manifest based on the platform → Add the ability to skip tests in the manifest based on the platform
Assignee | ||
Comment 4•12 years ago
|
||
Pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=388764ab7626
Assignee | ||
Updated•12 years ago
|
Attachment #764251 -
Attachment is obsolete: true
Assignee | ||
Comment 5•12 years ago
|
||
Comment on attachment 764367 [details] [diff] [review]
Add ability to skip tests based on device
Review of attachment 764367 [details] [diff] [review]:
-----------------------------------------------------------------
This changes the "old" self.device to self.device_serial, so we can use self.device to refer to the device that Marionette returns in session_capabilities.
This also fixes some bad whitespace in some (mostly unrelated) code (tabs instead of spaces).
Attachment #764367 -
Flags: review?(dave.hunt)
Reporter | ||
Comment 6•12 years ago
|
||
Comment on attachment 764367 [details] [diff] [review]
Add ability to skip tests based on device
Review of attachment 764367 [details] [diff] [review]:
-----------------------------------------------------------------
Looks good, just a couple of questions.
::: testing/marionette/client/marionette/runtests.py
@@ +611,4 @@
> action='store',
> help='host:port of running Gecko instance to connect to')
> self.add_option('--device',
> + dest='device_serial',
Where is this value read? I don't see it read anywhere in this patch.
::: testing/marionette/marionette-server.js
@@ +579,4 @@
> 'javascriptEnabled': true,
> 'nativeEvents': false,
> 'platform': Services.appinfo.OS,
> + 'device': qemu == "1" ? "qemu" : (!device ? "desktop" : device),
Is there (do we need?) a way to distinguish between Firefox desktop and B2G desktop?
Attachment #764367 -
Flags: review?(dave.hunt) → review-
Assignee | ||
Comment 7•12 years ago
|
||
Good catches! self.device_serial is only used by the update tests, so this fixes the only consumer of that.
Re: needing to detect b2g desktop vs firefox, currently, we have no need for that, but it's something that we certainly may need. Therefore, I've added an 'app' variable to the manifests:
skip-if = app == B2G && device == desktop (for example)
or:
skip-if = app == Firefox && device == desktop
For the gaia-ui-tests, we don't ever intend to run them on firefox, so just using:
skip-if = device == desktop
would be sufficient, I think.
Attachment #764399 -
Flags: review?(dave.hunt)
Assignee | ||
Updated•12 years ago
|
Attachment #764367 -
Attachment is obsolete: true
Assignee | ||
Comment 8•12 years ago
|
||
Oops, those should be:
skip-if = app == "B2G" && device == "desktop" (for example)
or:
skip-if = app == "Firefox" && device == "desktop"
(The quotes were missing above.)
Reporter | ||
Comment 9•12 years ago
|
||
Comment on attachment 764399 [details] [diff] [review]
Add ability to skip tests based on device
Review of attachment 764399 [details] [diff] [review]:
-----------------------------------------------------------------
Awesome, thanks! Looking forward to updating the manifest files! :)
Attachment #764399 -
Flags: review?(dave.hunt) → review+
Assignee | ||
Comment 10•12 years ago
|
||
Target Milestone: --- → mozilla24
Assignee | ||
Comment 11•12 years ago
|
||
Will file another bug to bump and release to pypi after this lands.
Comment 12•12 years ago
|
||
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 13•12 years ago
|
||
status-b2g18:
--- → fixed
status-b2g18-v1.0.0:
--- → wontfix
status-b2g18-v1.0.1:
--- → wontfix
status-firefox24:
--- → fixed
status-firefox25:
--- → fixed
Comment 14•12 years ago
|
||
status-b2g-v1.1hd:
--- → fixed
status-firefox25:
fixed → ---
Updated•2 years ago
|
Product: Testing → Remote Protocol
You need to log in
before you can comment on or make changes to this bug.
Description
•