2073095 - Make sure the ui tests can run on the nightly build

Status: RESOLVED FIXED

(Firefox for Android :: General, task)

Description

[Polly [:polly]]

We want the ui tests to be runnable against the nightly or release builds as well as the debug build.
There are places where we hardcode the package name but we should infer those.

Comment 1

[Segun Famisa [:segun]]

Expand for the findings and list of affected tests (copied from bug 2071826#c1)

Tests that are not compatible with being run against a minified release build type

Tests that use mockk

These tests rely on reflection and the library is not available in a release build type, and so we need to exclude them from whatever runs are happening - or better still, we rewrite the tests

1. OnboardingMapperTest
2. CrashPullDelegateTest
3. NimbusEventTest
4. UnsubmittedCrashDialogTest

Tests & robots that hardcode package name

Some tests hardcode org.mozilla.fenix.debug package id for finding the resource id. When running a release variant, the package name is different, so our selectors will not find those UI elements.

1. SyncIntegrationTest.kt
2. DownloadRobot.kt
3. FindInPageRobot.kt

Tests that duplicate production classes in the androdiTest source set

Tests that provide duplicate classes in alternative source sets. This causes duplicate class problems, because both the test APK and the target APK contain this class, and get obfuscated, but they may end up with different names, and that makes it crash when the instrumentation test is running.

For these classes, we need to move from duplicate classes into using some form of dependency injection, or delegation to contribute additional behaviours specific for tests.

1. AppRequestInterceptor
2. FxaServer

Robots that use specific selection indices and were flaky in my experiments

1. SettingsTranslationsRobot

Tests that behave differently when run against debug build type, vs release build type

top sites, sponsored lists & DefaultTopSitesBinding

1. TopSitesTest
2. SponsoredShortcutsTest
3. HomeScreenTest#homeScreenItemsTest
4. SettingsHomepageTest#verifyShortcutOptionTest
5. SettingsAddonsTest#noCrashWithAddonInstalledTest

Extensions

We bundle a debug list of extensions that we test against, and those extensions are not in the release build, so we need to either ignore them when testing in release, or implement some code intervention to make it extensible for release

1. ExtensionPrivateBrowsingTest
2. ExtensionOptionsFilePickerTest

“about settings” UI options

1. SettingsAdvancedTest#verifyAdvancedSettingsSectionItemsTest - leak canary does not exist in the release build
2. SettingsAboutTest#verifyAboutFirefoxMenuAppDetailsItemTest - VCS hash is empty on debug, but non-empty on a release build.

Comment 2

[BMO Github Automation]

Attachment: [mozilla-firefox/firefox] Bug 2073095 - locate sponsored shortcuts by their sponsored label (#367)

Comment 3

[BMO Github Automation]

Attachment: [mozilla-firefox/firefox] Bug 2073095 - remove mockk from ui tests to unlock running on nightly (#368)

(via GitHub webhook, authenticated as shared-secret)

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a82944e1685e
https://hg.mozilla.org/integration/autoland/rev/461c8c1e88ed
remove mockk from ui tests to unlock running on nightly r=mcarare

Comment 5

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/461c8c1e88ed

Comment 6

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/461c8c1e88ed