1238678 - (androidunittest) [meta] Run Android unit tests in automation

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect)

Description

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

This ticket tracks running the existing Android JUnit 4 test suites in automation.

This is like Bug 1064004 but for Android unit tests instead of Android instrumentation tests.  It used to be that the former were JUnit 4 and the latter JUnit 3, although now instrumentation tests can use JUnit 4 if the choose too.

Comment 1

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

This is not so hard these days.  The try push at https://treeherder.mozilla.org/#/jobs?repo=try&revision=d2c501f0c22d runs a --disable-compile-environment API 11+ build, and then runs |mach gradle lint| in some form.  I think making this work might be as easy as just bumping the command we execute.  I'm going to try it out.

Comment 2

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=dafecb90537a

Comment 3

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=af657aca2cd6

Comment 4

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=89e71ca59371

Comment 5

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

Attachment: MozReview Request: Bug 1238678 - Add job running Android unit tests in automation. r?dustin

This job looks like a frontend (--disable-compile-environment) build,
followed by some Java commands (that produce build artifacts).  If the
Java commands fail, we should fail the job (after copying artifacts,
etc).  I'll add some post-processor that interprets the outputs
eventually, but this pass/fail serves as a good starting point.

I'd like to extract some kind of android_api_11_base.yml and re-use
the configuration, but I don't really know if that's a good approach.
Guidance appreciated.

Review commit: https://reviewboard.mozilla.org/r/30459/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/30459/

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8706717 [details]
MozReview Request: Bug 1238678 - Add job running Android unit tests in automation. r?dustin

https://reviewboard.mozilla.org/r/30459/#review27945

Brief comment since I think we discussed this at some length.  If I'm imagining that, let me know and I'll go into more detail.

::: testing/taskcluster/tasks/builds/android_api_11_frontend.yml:50
(Diff revision 1)
> -          bin/build.sh
> +          bin/build.sh &&

I think I commented on this in irc, but apparently not here.  Since build.sh is running mozharness, I think that these unit tests should also be run via mozharness, rather than via a script wedged into the task description.

Comment 7

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Nick, now that I've written tests in bug 1244295, I have some incentive to help get this working. :)

What's the status here? Is there anywhere I can help?

Comment 8

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

(In reply to Michael Comella (:mcomella) from comment #7)
> Nick, now that I've written tests in bug 1244295, I have some incentive to
> help get this working. :)
> 
> What's the status here? Is there anywhere I can help?

We're close.  We need to get the Android API 15 front-end build from Bug 1247375 running in automation, using gps's just landed Bug 1245953.  This should be easy; see the eslint example.

Then we need to add some mozharness-fu to allow specifying post-build Gradle tasks (or general mach commands); and some additional mozharness-fu to allow specifying some directories to upload as TC artifacts.  Those options aren't complicated but I haven't had time to do it.  Talk to jlund and have him point you to where the flag goes in the mozharness build script.  Then you add the options to https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/builds/releng_sub_android_configs/64_api_15_frontend.py and profit.

Comment 9

[Geoff Brown [:gbrown]]

Let me know if I can help with the mozharness-fu.

Comment 10

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

Worth nothing that |hg push-to-try -m "try: -b o -p android-api-15-frontend"| runs a Gradle-based --disable-compile-environment, or will after Bug 1248698.

Comment 11

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

Comment on attachment 8706717 [details]
MozReview Request: Bug 1238678 - Add job running Android unit tests in automation. r?dustin

This landed in Bug 1249166 \o/

Comment 12

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

We are totally doing this now!  |mach android unit| for the win!