1130228 - Support UI automation tests

Status: RESOLVED FIXED

(Firefox for iOS :: Build & Test, defect)

Description

[Brian Nicholson (:bnicholson)]

Right now, we have XCTest unit tests, which we use to build "white box" tests for individual components. We need a way to run "black box" UI automation tests that can run browser tests as a user would.

The standard Automation tool, part of the Instruments application, uses JS scripts to build these tests. These tests use accessibility labels as a generic way to hook into views (as opposed to findViewById on Android). They are documented extensively at [1].

Here's my take on Automation:
* No built-in Xcode Server integration. You can, however, start the scripts from the command line, so this might still work. I haven't really played around with this much.
* The editor is awful. No code completion or IDE tools, and running the test annoyingly switches between the script and log views. Most people probably choose to edit their scripts externally, but it's unfortunate that we don't get the same IDE advantages as writing normal code.
* It's not clear how we would set up a mock server for testing. We'll need a way to serve mock pages to test things like navigation, session restore, and all other sorts of browser features. We might be able to piggyback off of the existing built-in web server if we can figure out the port it's running on.

There's a whole suite of other tools in the Instruments panel besides Automation for all sorts of perf-related tests: CPU/disk/network usage, memory allocation, and even energy usage. It would be really nice if there was a way to get all of these automated, but it looks like they're meant only for local profiling.

I've read good things about KIF (made by Square), so I decided to give that a try. Like Automation, KIF uses accessibility labels as hooks. Some initial observations:
* Unlike Automation, KIF tests are written in Obj C/Swift, and KIFTestCase even subclasses XCTestCase. This means we should have Xcode Server integration out of the box.
* Since it's written as a regular XCTest, that means we get the same IDE support. Debugging, code completion, etc.
* Since it's written as a regular XCTest, that also gives us more control of the actual test environment, enabling us to do things like run a test web server.
* Accessibility labels even works with the content pages of UIWebViews, which would be pretty awesome for testing any mock content. Unfortunately, WKWebView doesn't currently work [2]. This shouldn't be a deal breaker, though, since we can reimplement some of this functionality with JS injection.

[1] https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html
[2] https://github.com/kif-framework/KIF/issues/460

Comment 1

[Brian Nicholson (:bnicholson)]

Attachment: Pull request

Writing a navigation test with KIF was pretty nice, so I'd like to propose KIF as our framework of choice for automation. Granted, this is a pretty basic test, so there may be some bigger limitations with KIF that I haven't run into yet. Happy to hear everyone else's opinion.

Comment 2

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

Comment on attachment 8560187 [details] [review]
Pull request

LGTM