873644 - Fix our test manifests

Status: RESOLVED DUPLICATE of bug 1272414

(Remote Protocol :: Marionette, defect)

Description

[Malini Das [:mdas] - Away, not checking bugmail]

We've been using the wrong assumptions when using manifestparser. 

We assumed that if our [DEFAULT] values say one thing, and the flag under the test says another, then the test won't run. This is wrong. The flag under the test sets the key/value pair for that test, it doesn't bar you from running the test. You want the 'run-if' flag in that case.

We used flags under our tests in our .ini files as conditionals, when really, those flags were just values we set on the test. For example:

[test_findelement.py]
b2g = false

If we pass --tests=b2g to runtests.py, then this test won't run because we want the 'b2g' key on a test to be true. BUT it doesn't stop a test from running if we set b2g = true in DEFAULTS and run without any --tests parameter. This is because we don't explicitly say what we're looking for, so any test will be run.

We'll need to either start using the run-if flag, or set our expected key/value pairs explicitly in testargs, before calling .get() on our TestManifest: http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/runtests.py#416

Comment 1

[Jonathan Griffin (:jgriffin)]

I think in order to avoid disruption for existing users, that passing our default key/value pairs in testargs is probably a nicer way to go.

Comment 2

[Jeff Hammel]

I agree; I think that a new argument, say `query` should be added that will be used to call ManifestParser.get for a match.