1169862 - Prevent saving app state during tests

Status: RESOLVED FIXED

(Firefox for iOS :: Build & Test, defect)

Description

[Brian Nicholson (:bnicholson)]

This currently causes tests to fail since they start up with the last tabs that were open.

Comment 1

[Brian Nicholson (:bnicholson)]

Attachment: Link to Github pull-request: https://github.com/mozilla/firefox-ios/pull/527

Comment 2

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

Comment on attachment 8613162 [details] [review]
Link to Github pull-request: https://github.com/mozilla/firefox-ios/pull/527

Yes, but .. my gut feeling says that having a special TestAppDelegate, or any test specific code in the app, is wrong.

I think this specifically is really a 'make sure tests start with a clean app state' problem, which should be a test or test runner issue. And probably test specific.

Because what happens if we want to unit test the tab state code? Add more special test login in the app?

Is there no 'wipe simulator state before each test' option?

Comment 3

[Brian Nicholson (:bnicholson)]

(In reply to Stefan Arentz [:st3fan] from comment #2)
> I think this specifically is really a 'make sure tests start with a clean
> app state' problem, which should be a test or test runner issue. And
> probably test specific.

Yes, it should be, but it's a flaw with the way iOS tests are run. iOS tests work by first launching the app, then attaching the test framework to the app, and the same app instance is used across all tests [1]. That means we can't clear the simulator, or even restart the app, and we're required to reset the state on our own between each test (hence the reason for our resetToAboutHome() helper function at [2]). If a single UI test fails, it can cause all following UI tests to fail since the app won't be in the "default" state when the tests start. It's all very fragile.

> Because what happens if we want to unit test the tab state code? Add more
> special test login in the app?

The we'd probably want to test something more componentized that doesn't directly interact with NSCoding. We could test that the session data written matches the expected data, and in a separate test we could feed in mock data and test that the post-restore state matches the expected state (this is what we do on Android). Testing actual restoration would be difficult since logic would have to span across multiple tests, breaking the "I" (isolated) condition in the FIRST unit test principles.

> Is there no 'wipe simulator state before each test' option?

Unlikely, given [1]. But that link also mentions running tests from the command line, giving us more control over how they're run, so that may be worth investigating.

[1] https://groups.google.com/forum/?fromgroups#!starred/kif-framework/dLBTlTjqTcg
[2] https://github.com/mozilla/firefox-ios/blob/master/UITests/Global.swift#L73

Comment 4

[Brian Nicholson (:bnicholson)]

https://github.com/mozilla/firefox-ios/commit/a73c80d668f40d3b8d693313e22a17f1ad15a236