985764 - Support skip-if/fail-if based on a "platform" attribute defined at run-time

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

I have disabled a lot of tests on Android over the last few months, primarily using skip-if, based on 'os'/'toolkit' and sometimes on the Android version. On some occasions, people have expressed concern that those skip-ifs are inappropriate, in that those skip-ifs declare that a test should not be run on, say Android 2.3, when all that we really know is that the test hangs on the Android 2.3 emulator when run on an ec2 m1.medium instance: The manifest entry is overly general, excluding tests from running in test configurations where they may work, and suggesting that the cause of the failure is tied to a specific factor (like the os and/or os version) when in fact no such determination has been made.

Of course the same concern applies to fail-if, etc.

We could perhaps do better by using a "platform identifier" (for lack of a better term) which could be passed to the test harness (and on to the manifest parser) at test run-time. The platform identifier might ultimately come from a mozharness or buildbot configuration for tbpl jobs. Then we could state something like:

  skip-if = platform_id == 'android 2.3 emulator, ec2 m1.medium'

I think there is value here, but I don't know how difficult it would be to implement, or how it interacts with other manifest concerns or current manifest plans.

Do we want to do something like this? I'm afraid I haven't kept up with all the recent manifest work: Are we moving toward this, or away from it?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

doing something like this requires us to have better ways to identify the platform.  how from python can we determine the ec2 type?  My understanding is we would need to pass that in via the commandline (or read some settings file generated by the mozharness script or scheduler.)

I can see a use case where somebody wants to run this on a local phone vs tbpl based automation.  In my experience in the past, running on a one off device has so many other issues that it takes weeks of effort just to get something stable (usually a lot of harness, test, and manifest hacking.)  has that changed in recent times?

I like this idea in general.  How do we solve this with reftests?  Is this generic to both reftest, xpcshell, and mochitest?

Comment 2

[Geoff Brown [:gbrown]]

(In reply to Joel Maher (:jmaher) from comment #1)
> doing something like this requires us to have better ways to identify the
> platform.  how from python can we determine the ec2 type?  My understanding
> is we would need to pass that in via the commandline (or read some settings
> file generated by the mozharness script or scheduler.)

I think the way to do this is to have mozharness or buildbot pass the whole platform id (including any distinction made for ec2 type or whatever) to the test harnesses on the command line (or in a file).
 
> I can see a use case where somebody wants to run this on a local phone vs
> tbpl based automation.  In my experience in the past, running on a one off
> device has so many other issues that it takes weeks of effort just to get
> something stable (usually a lot of harness, test, and manifest hacking.) 
> has that changed in recent times?

We have made a few minor changes to make things easier for testing on a local device, but I don't think that has made a great difference: It is still error prone and seems too difficult for some developers. On the other hand, I have talked with some developers who say they run many tests on their local devices all the time.

> I like this idea in general.  How do we solve this with reftests?  Is this
> generic to both reftest, xpcshell, and mochitest?

I think the idea could be generic to reftest/xpcshell/mochitest. Individual implementations might be slightly different.

Comment 3

[Geoff Brown [:gbrown]]

Interesting idea, but we seem to be getting along okay without it.