Right now, Tests rely on too much state. If the sate of the app isn't exactly right tests can fail sometimes. If you run tests in a different order they can also fail. Lets create a method that can reset firefox to a fresh state. That way between tests we can be sure we have a consistent app state.
Created attachment 8770248 [details] [review] Pull Request Emily you've been working on tests. I'd love your feedback on this. I've also added a XCUI target.
Attachment #8770248 - Flags: review?(etoop)
Comment on attachment 8770248 [details] [review] Pull Request This looks good. I've left a comment in the PR about adding the resetApplication code to TestAppDelegate instead of the production code. Resetting state is one of the things I have been trying to sort out in UITests too - KIF makes this really hard and our tests rely too heavily on state created by other tests already for it to be an effective blanket fix. Starting afresh with new UITesting tests has always felt like a good idea. Do you think it might be useful to reset some of the UserDefaults/Preferences too? Not all of them should be reset - we don't want the intro tour to play every time we start a new test - but ensuring that we don't have to reset at the end of tests settings that can be toggled for things like search preferences, home pages and new tab displays. It might be nice to create a base XCTestCase class that has the RESET_FIREFOX flag attached in setup that all future tests will inherit from to ensure that every single test case we have will reset on start. Something I've been toying with as an idea is also providing a initialised, but completely empty, database that we could install with tests. So not just wiping all of our documents, cookie and caches, but ensuring that we start with no existing history, bookmarks or logins too. This will resolve a lot of the existing slowness in the UITests where we are resetting the data that is contained withing the app by following long winded UI actions.
Attachment #8770248 - Flags: feedback+
Now that I've moved my resetApplication() into the TestAppDelegate. There is already a UserDefaults/Profile reset that happens per test https://github.com/mozilla/firefox-ios/blob/5238e873e77e9ad3e699f926d12f61ccafabdc11/Client/Application/TestAppDelegate.swift#L17 I think it would be better if this wasn't implicit reset of the Profile. Right now if I add the "FIREFOX_TESTS" or "FASTLANE_SNAPSHOT" flag the profile will be reset. I would like to change the flags to not reset the state at all. This will be useful for writing tests that can check for persistence. I'll have to change the current tests that use the "FIREFOX_TESTS" and "FASTLANE_SNAPSHOT" to also contain the new reset flag. This also creates a change for the snapshot script. Is it alright for the DB to be reset between snapshots Stefan?
Comment on attachment 8770248 [details] [review] Pull Request LGTM
Attachment #8770248 - Flags: review?(etoop) → review+
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.