1217097 - Investigate how to run tests on Gecko for iOS

Status: NEW

(Testing :: General, defect)

Description

[James Willcox (:snorp) ([e-mail]) (he/him)]

We need to figure out the entire story here. I think at a minimum, we would want to be able to run mochitests and reftests. Many of the strategies we use on Android are not applicable here (there is no 'adb shell' or 'adb push'). It's also not clear if we should try to run on attached devices, simulators, or some kind of cloud device farm.

Comment 1

[(not currently active) Ted Mielczarek]

I think simulators are the most feasible way to run tests, running anything on device is a huge headache. Facebook has an SDK for launching multiple simulators at a time:
https://github.com/facebook/FBSimulatorControl

I did have *some* success using ios-deploy to run tests on a local device:
https://github.com/phonegap/ios-deploy

That's what the half-baked mozdevice patch in the gecko-ios repo uses (although it depends on local patches that I didn't upstream as well). Running tests on device would be valuable for JIT tests, at least, which I did get running (albeit not very well).

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

:ted, do the simulators need to run on mac minis or related apple hardware?

It sounds like we want to run mochitest-plain and reftest to get started.  Both of these install custom preferences, other profile files, and addons.  On desktop we launch the browser with cli flags to open the specific profile, we could fix that by putting the custom profile into the specific location we care about on IOS- assuming we have read/write access to that location.

In addition to profile stuff, we need to launch the browser, somehow get the results from a log file of some type, and then close the browser (either upon end, or upon timeout/crash/error).  I am not sure if there are restrictions on file i/o and security to concern ourselves with on a simulator or real phone.

If we can run on a simulator, that would be easier- although it would require us to order tons of mac minis and rack them up.  We could also consider amazon device farm as they support IOS- we would just need to hack our way into a harness.  I would like to avoid real devices as much as possible, but for real testing something needs to happen on a real device.

Ted, do you know if profile creation can be done in our automated way on a simulator?

Comment 3

[(not currently active) Ted Mielczarek]

Simulators need to run on Mac hardware AFAIK, so yeah, you still have the "buy a lot of macs" problem, but that's more tractable than "buy a lot of phones".

The simulator just sticks files on the host system disk. I don't know if Facebook's SDK gives you an easy way to access that, but even with ios-deploy I was able to push files to the device filesystem so I suspect it's doable.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

One thing to figure out is how much testing can be done on simulator vs real device.  If we think 90%+ can realistically be done on a simulator, then we would be looking at either manual testing for the other <10% or some small scale automation with a custom harness (i.e. autophone or amazon device farm).

snorp/ted, any idea what specific tests need to be done on real phones?  Maybe gl tests, perf, some reftests or jittests?

Comment 5

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Joel Maher (:jmaher) from comment #4)
> 
> snorp/ted, any idea what specific tests need to be done on real phones? 
> Maybe gl tests, perf, some reftests or jittests?

Yeah, the usual suspects there. GFX/GL, media, performance.

Comment 6

[Jonathan Griffin (:jgriffin)]

I believe the effort to do this well will be fairly significant; what are the shipping plans for iOS gecko?

Comment 7

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> I believe the effort to do this well will be fairly significant; what are
> the shipping plans for iOS gecko?

We're still in the early stages, but I think we're planning for H1 next year.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

I would like to have a set of things to investigate, and ensure that if things work out as we think they do that we are willing to move forward with the cost, etc.

Also we will most likely need to allocate a fulltime person to making this happen at scale and working on the reliability- this has been a fulltime job for android.

right now lets figure out a realistic plan going forward and determine what we need to investigate.