1970031 - Consolidate `mobile/android/*/automation` files

Status: RESOLVED FIXED

(Firefox for Android :: UI Tests, task)

Description

[Ted Campbell [:tcampbell]]

The various flank-*.yml files we use for different test configurations are currently located by running with $CWD of either fenix or focus, rather than being specified enough for test-lab.py to be run from the top-level.

Eg. The fenix-arm64-v8a-nightly-robo-opt task specifies arm-start-test-robo to select the mobile/android/fenix/automation/taskcluster/androidTest/flank-arm-start-test-robo.yml file.

Some options to address this might be:

1. Add a 'fenix' or 'focus-android' argument to test-lab.py calls that is templated into the path within ui-test.py
2. Move the focus + fenix flank.yml to a common directory (in taskcluster/scripts/.. or mobile/android/flank, etc) and add fenix- or focus- prefix to filenames (such as flank-arm-fenix-start-test-robo.yml).
3. ???

Comment 1

[Ted Campbell [:tcampbell]]

Aaron, does your team have any thoughts on how best to organize these flank files?

Comment 2

[Jackie Johnson]

My opinion on this is based on wanting test automation and all it's supporting tooling to be used as Test Automation as a Service that's consumed in CI and is maintained separately from the app-under-test.

With that in mind, I would prefer the solution that moves us closer to CI runtime-configurable test execution. It seems to me it's easier to maintain when it's both centralized and parameterized (toolified).

Now, with that said, it's a legitimate unknown to me how best to do this given Mozilla's build process, expectations, and standards specifically for the Android goings-on.

Correct me if I'm wrong, Aaron, but it seems to me the simplest solution is to first consolidate all the 'things' (technical terminology on-point there) with the idea of moving closer to first toolifying all the scripting and then making it a consumable service, whether live (eg. Microservice) or a static library consumed via CI args -> scripts -> tools -> service/gradlew.

All that to say,

my vote is to move it all into the same directory first

since we'll need that regardless of how our future tooling/service creation looks.

Then we can solve for the actual trigger/ingest for androidTest later, which likely could be helped with some new gradle tasks that hook into the app State Machine, whatever that looks like, to make the app settings also configurable. Whatever that new gradle task is, it should accept runtime configuration parameters the same way we would want it to accept the parameters for this as well.

This all hints at a larger conversation of enabling App Testability, which I believe should definitely happen, but isn't needed specifically here for this bug.

Comment 3

[Aaron Train [:aaronmt]]

Note: Also to consider the androidTest/robo-scripts for Robo and androidTest/geckoview-configs. The Flank configurations would need adjustment to absolute reference the files used in their corresponding configuration at their new absolute path.

I guess I'm indifferent here. They currently live in androidTest as we didn't need test-infra shipping in app code originally (would mobile/android/* ship in package?)

Perhaps option 2 with a structure like:

mobile/android/test_infra/
│
├── flank-configs/
│ ├── flank-arm-fenix-debug.yml
│ ├── flank-arm-focus-debug.yml
│ └── ... (other configs)
│
├── robo-scripts/
│ ├── default-browser-dismissal.json
│
└── geckoview-configs/
├── org.mozilla.fenix.debug-geckoview-config.yaml
└── org.mozilla.focus.debug-geckoview-config.yaml

With that I'm indifferent as well if adding a product name prefix to each config in a rename or creating subdirectories for each and passing a --product to test-lab.py is preferred (e.,g flank_path = f"test-infra/flank-configs/{args.product}/flank-{args.device}.yml" (still need to pass --product and --device explicitly)

Comment 4

[Ted Campbell [:tcampbell]]

Good point to fix robo-scripts, etc at the same time. I'm also now considering (for discoverability) of use fenix/arm-debug.yml instead of arm-debug for the command line in the taskcluster file. Then we have flexibility in the future for organizing the hierarchy.

mobile/android/test_infra/
│
├── flank-configs/
│ ├── fenix/
│ │ ├── arm-debug.yml
│ ├── focus/
│ │ ├── arm-debug.yml
│ └── ... (other configs)
│
├── robo-scripts/
│ ├── default-browser-dismissal.json
│
└── geckoview-configs/
├── org.mozilla.fenix.debug-geckoview-config.yaml
└── org.mozilla.focus.debug-geckoview-config.yaml

Comment 5

[Ted Campbell [:tcampbell]]

Attachment: Bug 1970031 - Remove deprecated automation scripts from Fenix. r?jonalmeida

Comment 6

[Ted Campbell [:tcampbell]]

Attachment: Bug 1970031 - Consolidate 'mobile/android/**/automation' into 'mobile/android/test_infra'. r?aaronmt

Move the various config files we use for Flank into a common location to make it
easier to share test automation configuration and simplify the taskcluster
definitions later.

As a result, we change test-lab.py to run Flank from the mobile/android/test_infra
directory such that additional config files are relative to that.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/af9bbaa652af
https://hg.mozilla.org/integration/autoland/rev/99098b9346a3
Consolidate 'mobile/android/**/automation' into 'mobile/android/test_infra'. r=aaronmt,geckoview-reviewers,android-reviewers,nalexander

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/9ff458add219
https://hg.mozilla.org/integration/autoland/rev/db4646a8de72
Remove deprecated automation scripts from Fenix. r=jonalmeida,android-reviewers

Comment 10

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/99098b9346a3
https://hg.mozilla.org/mozilla-central/rev/db4646a8de72