2071826 - Figure out a set of keep rules that allows us to run automated tests with obfuscation on

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, task)

Description

[Polly [:polly]]

Automated tests don't play nicely with obfuscated builds.
A lot of the code required by the test infrastructure gets optimised away and breaks.
We are hopeful that we can figure out a way round this by using an iterative process to come up with a set of keep rules that enable us to successfully run the tests.
This ticket will enable automated tests against obfuscated code for both Fenix and Focus.

Comment 1

[Segun Famisa [:segun]]

I have concluded the spike here to figure out how to get a successful try run on minified builds.

I have shared my findings in this doc - https://docs.google.com/document/d/1BHoznUPqfRlQ7M5UbtgwTwSHbcQ9_PA9TKHCbSEiuFI/edit?tab=t.ngerhh8tsc47

The spike is now considered done, and the actual work is being tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=2073095
but here's the TL;DR version:

Findings TL;DR

1. It is possible to run UI tests on minified builds
2. We need to fix a couple of tests
3. We cannot produce a single, static proguard rule configuration that is exhaustive for running androidTest on obfuscated builds.
   • The required proguard rules change every time new code is added, either to the app or to the test suite itself
   • We need a gradle plugin to autogenerate the “androidtest” proguard files, in a fashion similar to Keeper.
   • It’s a very low-lift.
4. We need to write a new treeherder task or modify an existing one to run tests against obfuscation builds

Tests that are not compatible with being run against a minified release build type

Tests that use mockk

These tests rely on reflection and the library is not available in a release build type, and so we need to exclude them from whatever runs are happening - or better still, we rewrite the tests

1. OnboardingMapperTest
2. CrashPullDelegateTest
3. NimbusEventTest
4. UnsubmittedCrashDialogTest

Tests & robots that hardcode package name

Some tests hardcode org.mozilla.fenix.debug package id for finding the resource id. When running a release variant, the package name is different, so our selectors will not find those UI elements.

1. SyncIntegrationTest.kt
2. DownloadRobot.kt
3. FindInPageRobot.kt

Tests that duplicate production classes in the androdiTest source set

Tests that provide duplicate classes in alternative source sets. This causes duplicate class problems, because both the test APK and the target APK contain this class, and get obfuscated, but they may end up with different names, and that makes it crash when the instrumentation test is running.

For these classes, we need to move from duplicate classes into using some form of dependency injection, or delegation to contribute additional behaviours specific for tests.

1. AppRequestInterceptor
2. FxaServer

Robots that use specific selection indices and were flaky in my experiments

1. SettingsTranslationsRobot

Tests that behave differently when run against debug build type, vs release build type

top sites, sponsored lists & DefaultTopSitesBinding

1. TopSitesTest
2. SponsoredShortcutsTest
3. HomeScreenTest#homeScreenItemsTest
4. SettingsHomepageTest#verifyShortcutOptionTest
5. SettingsAddonsTest#noCrashWithAddonInstalledTest

Extensions

We bundle a debug list of extensions that we test against, and those extensions are not in the release build, so we need to either ignore them when testing in release, or implement some code intervention to make it extensible for release

1. ExtensionPrivateBrowsingTest
2. ExtensionOptionsFilePickerTest

“about settings” UI options

1. SettingsAdvancedTest#verifyAdvancedSettingsSectionItemsTest - leak canary does not exist in the release build
2. SettingsAboutTest#verifyAboutFirefoxMenuAppDetailsItemTest - VCS hash is empty on debug, but non-empty on a release build.

Comment 2

[Segun Famisa [:segun]]

retroactively assigning story points of what this took