Open Bug 1217097 Opened 4 years ago Updated 4 years ago

Investigate how to run tests on Gecko for iOS

Categories

(Testing :: General, defect)

Unspecified
iOS
defect
Not set

Tracking

(Not tracked)

People

(Reporter: snorp, Unassigned)

Details

We need to figure out the entire story here. I think at a minimum, we would want to be able to run mochitests and reftests. Many of the strategies we use on Android are not applicable here (there is no 'adb shell' or 'adb push'). It's also not clear if we should try to run on attached devices, simulators, or some kind of cloud device farm.
I think simulators are the most feasible way to run tests, running anything on device is a huge headache. Facebook has an SDK for launching multiple simulators at a time:
https://github.com/facebook/FBSimulatorControl

I did have *some* success using ios-deploy to run tests on a local device:
https://github.com/phonegap/ios-deploy

That's what the half-baked mozdevice patch in the gecko-ios repo uses (although it depends on local patches that I didn't upstream as well). Running tests on device would be valuable for JIT tests, at least, which I did get running (albeit not very well).
:ted, do the simulators need to run on mac minis or related apple hardware?

It sounds like we want to run mochitest-plain and reftest to get started.  Both of these install custom preferences, other profile files, and addons.  On desktop we launch the browser with cli flags to open the specific profile, we could fix that by putting the custom profile into the specific location we care about on IOS- assuming we have read/write access to that location.

In addition to profile stuff, we need to launch the browser, somehow get the results from a log file of some type, and then close the browser (either upon end, or upon timeout/crash/error).  I am not sure if there are restrictions on file i/o and security to concern ourselves with on a simulator or real phone.

If we can run on a simulator, that would be easier- although it would require us to order tons of mac minis and rack them up.  We could also consider amazon device farm as they support IOS- we would just need to hack our way into a harness.  I would like to avoid real devices as much as possible, but for real testing something needs to happen on a real device.

Ted, do you know if profile creation can be done in our automated way on a simulator?
Simulators need to run on Mac hardware AFAIK, so yeah, you still have the "buy a lot of macs" problem, but that's more tractable than "buy a lot of phones".

The simulator just sticks files on the host system disk. I don't know if Facebook's SDK gives you an easy way to access that, but even with ios-deploy I was able to push files to the device filesystem so I suspect it's doable.
One thing to figure out is how much testing can be done on simulator vs real device.  If we think 90%+ can realistically be done on a simulator, then we would be looking at either manual testing for the other <10% or some small scale automation with a custom harness (i.e. autophone or amazon device farm).

snorp/ted, any idea what specific tests need to be done on real phones?  Maybe gl tests, perf, some reftests or jittests?
(In reply to Joel Maher (:jmaher) from comment #4)
> 
> snorp/ted, any idea what specific tests need to be done on real phones? 
> Maybe gl tests, perf, some reftests or jittests?

Yeah, the usual suspects there. GFX/GL, media, performance.
I believe the effort to do this well will be fairly significant; what are the shipping plans for iOS gecko?
(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> I believe the effort to do this well will be fairly significant; what are
> the shipping plans for iOS gecko?

We're still in the early stages, but I think we're planning for H1 next year.
I would like to have a set of things to investigate, and ensure that if things work out as we think they do that we are willing to move forward with the cost, etc.

Also we will most likely need to allocate a fulltime person to making this happen at scale and working on the reliability- this has been a fulltime job for android.

right now lets figure out a realistic plan going forward and determine what we need to investigate.
You need to log in before you can comment on or make changes to this bug.