1448815 - XCUITest: Investigate if using preloaded DBs decreases the BB tests execution time

Status: RESOLVED FIXED

(Firefox for iOS :: Build & Test, enhancement)

Description

[Isabel Rios[:isabel_rios]]

It is possible to use pre-populated DBs for History and Bookmarks.
And configuring the setUp now is not necessary to restart the app in each test to see and use the data.

These improvements could make some tests to run faster since it is not needed to follow certain steps to have the data available and then check what the test is aimed for. Also, tests could be more robust after removing those steps where tests could fail when for example loading an url or so.

So, let's investigate if we actually see some improvements in BB test execution time and if that's the case we can refactor all tests that use the pre-loaded data.

Comment 1

[Isabel Rios[:isabel_rios]]

Attachment: Pull Request

Please let me know your thoughts about this PR. Thanks!

Comment 2

[No-Jun Park [:njpark]]

Comment on attachment 8963330 [details] [review]
Pull Request

I think it looks good, we might want to comment in the script that mentions about the preloaded db just for clarity.

Comment 3

[Isabel Rios[:isabel_rios]]

Thanks for the review! You mean clarify what the DB contains and why we use it?

Also, before merging this, we have to take into account that the pre-loaded DB patch did not land in v11.x, so this is only for master and since several tests are modified, they are going to diverge from v11.x. Knowing this, should we give it a try?

Thanks!

Comment 4

[No-Jun Park [:njpark]]

Yes, a quick comment about what that DB is about.  And I think it's reasonable that it diverges from 11.x.  Let's make sure we don't do any partial uplifts that fails the test suite.

Comment 5

[Isabel Rios[:isabel_rios]]

After sometime I'm working again on this bug. It looks like now the time we save per build is not as big as before, we save around 2-3 mins per build, even though I think it would be interesting to land this on master and v12.x since it will make some tests more stable and we could try to increase the time saved using these or more DDBBs in the future.

Please No-Jun, let me know your opinion on this...
Thanks!

Comment 6

[No-Jun Park [:njpark]]

I think having it stable is a big plus, let's go ahead and see what happens.  If it causes problems, we can always revert it.

Comment 7

[Isabel Rios[:isabel_rios]]

Merged on master: https://github.com/mozilla-mobile/firefox-ios/commit/de55229cdff72ebf44a0f676fa23c3b275c630ee

If we see it working on master, lets uplift to v12.x

Comment 8

[Isabel Rios[:isabel_rios]]

Although it is working, lets add this starting on v13 when the branch is cut from master, no need to have it on 12 now