1280440 - Run Android x86 tests in taskcluster, tier 2

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Geoff Brown [:gbrown]]

Possible complications here:
 - x86 tests run on ix hardware currently, for emulator performance (hardware acceleration)
 - test "sets"

Comment 1

[Geoff Brown [:gbrown]]

An attempt at running x86 xpcshell on aws, in 8 chunks: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c231e022caef

Comment 2

[Geoff Brown [:gbrown]]

I initially had trouble with intermittent crashes, because I was using the newer (24.0.1) Android SDK. When I switch to the Android 18 SDK (as used for buildbot Android x86 tests), the crashes stop, and tests run faster - yeah!

Comment 3

[Geoff Brown [:gbrown]]

Attachment: add taskcluster definitions for Android x86 tests (xpcshell)

We only run xpcshell on Android x86. The "xpcshell-x86" test definition feels a little clumsy, but I need to use different mozharness configuration than the arm "xpcshell" test, so I'm not sure how else to handy this (suggestions welcome).

Comment 4

[Geoff Brown [:gbrown]]

Attachment: add mozharness configuration for Android x86 xpcshell on taskcluster

The new configuration file is similar to the buildbot "androidx86.py", but has a few differences because for taskcluster I am switching to the android_emulator_unittest.py script rather than androidx86_emulator_unittest.py...plus, as usual, we need to handle some different paths.

I also needed to make a small change to android_emulator_unittest.py to support earlier versions of the Android sdk.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8776026 [details] [diff] [review]
add taskcluster definitions for Android x86 tests (xpcshell)

Review of attachment 8776026 [details] [diff] [review]:
-----------------------------------------------------------------

This would be r+ if you're OK with this job having a different name in try.

::: taskcluster/ci/android-test/test-platforms.yml
@@ +16,5 @@
>      test-set: debug-tests
>  android-4.3-arm7-api-15/opt:
>      build-platform: android-api-15/opt
>      test-set: opt-tests
> +android-4-2-x86/opt:

Should this be 4.2 like the arm7?

::: taskcluster/ci/android-test/test-sets.yml
@@ +36,5 @@
>      - robocop
>      - xpcshell
> +
> +x86-tests:
> +    - xpcshell-x86

One downside to this is, the try syntax for this job will be -u xpcshell-x86, rather than -u xpcshell.

::: taskcluster/ci/android-test/tests.yml
@@ +263,5 @@
> +    mozharness:
> +        script: mozharness/scripts/android_emulator_unittest.py
> +        no-read-buildbot-config: true
> +        config:
> +            - mozharness/configs/android/androidx86-tc.py

It looks like it's really only the config that differs here.  You could allow `by-test-platform` here, by updating the schema and then adding to this transform:
  https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/transforms/tests/all_kinds.py#72
then write

  config:
    by-test-platform:
      android-4-2-x86/opt:
        - mozharness/configs/android/androidx86-tc.py
      default:
        - mozharness/configs/android/androidarm_4_3.py
        - mozharness/configs/remove_executables.py	
        - mozharness/configs/android/androidarm_4_3-tc.py

Comment 6

[Geoff Brown [:gbrown]]

Attachment: add taskcluster definitions for Android x86 tests (xpcshell)

Thanks Dustin -- I'm happier with this.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=92bf8983d9e2

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8776147 [details] [diff] [review]
add taskcluster definitions for Android x86 tests (xpcshell)

Review of attachment 8776147 [details] [diff] [review]:
-----------------------------------------------------------------

Looks great, thanks!

::: taskcluster/taskgraph/transforms/base.py
@@ +70,5 @@
>              msg.append(str(error))
>          raise Exception('\n'.join(msg))
>  
>  
> +def get_keyed_by(item, field, item_name, subfield=None):

Can you add a sentence or two to the docstring to indicate what the subfield means?

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ab9098363159
Add mozharness config for Android x86 on taskcluster; r=kmoir
https://hg.mozilla.org/integration/mozilla-inbound/rev/73fe336b6d39
Add TC Android x86 tests, tier 2; r=dustin

Comment 9

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/ab9098363159
https://hg.mozilla.org/mozilla-central/rev/73fe336b6d39