Open Bug 1064004 (instrumentation) Opened 5 years ago Updated Last year

[meta] Run Android instrumentation tests in automation

Categories

(Firefox for Android :: Testing, defect)

All
Android
defect
Not set

Tracking

()

People

(Reporter: nalexander, Assigned: nalexander)

References

(Depends on 1 open bug)

Details

(Keywords: meta)

Attachments

(3 files)

At the moment, we run one special instrumentation suite in automation: Robocop.  We have two others building but not being run: Browser JUnit 3 [1] and Background Services JUnit 3.  This meta bug tracks getting those running in automation.

There are a lot of pieces to this ticket, but I think broadly we need to:

1) get a new mozharness (ScriptFactory) job running on cedar.
2) write enough mozharness glue to prepare the test environment and execute an in-tree harness.
3) write said in-tree harness.
(In reply to Nick Alexander :nalexander from comment #0)
> At the moment, we run one special instrumentation suite in automation:
> Robocop.  We have two others building but not being run: Browser JUnit 3 [1]
> and Background Services JUnit 3.  This meta bug tracks getting those running
> in automation.
> 
> There are a lot of pieces to this ticket, but I think broadly we need to:
> 
> 1) get a new mozharness (ScriptFactory) job running on cedar.
> 2) write enough mozharness glue to prepare the test environment and execute
> an in-tree harness.
> 3) write said in-tree harness.

Bug 903537 tracks writing an in-tree harness for running instrumentation tests.  This is the part that I can personally move on most easily.
> 1) get a new mozharness (ScriptFactory) job running on cedar.

I've filed Bug 1064010 to track getting something running on cedar.

> 2) write enough mozharness glue to prepare the test environment and execute
> an in-tree harness.

I've filed Bug 1064012 to track getting a mozharness script up and running.
Alias: instrumentation
This is like Bug 1238678 but for Android instrumentation tests instead of Android unit tests.

It used to be that the former were JUnit 3 and the latter JUnit 4, although now instrumentation tests can use JUnit 4 if the choose too.
Comment on attachment 8728697 [details]
MozReview Request: Bug 1064004 - Add Espresso instrumentation test support to Gradle. r?mcomella

mcomella: this isn't really ready for review, but the other bits are, so I wanted to pre-flight the groundwork.

You can see this working in https://treeherder.mozilla.org/#/jobs?repo=try&revision=0f51765c4ab2&selectedJob=17847321, which just runs one (non-Espresso) instrumentation test.

This is performing a --disable-compile-environment build *without Gecko libraries*, so it's not sensible yet; but it leads the way to running DB-only instrumentation tests, at least.
Attachment #8728697 - Flags: review?(michael.l.comella) → feedback?(michael.l.comella)
Comment on attachment 8728696 [details]
MozReview Request: Bug 1064004 - Add TaskCluster job to run Android instrumentation tests in automation. r?dustin

https://reviewboard.mozilla.org/r/39041/#review35877

::: testing/taskcluster/tasks/builds/android_api_15_androidTest.yml:18
(Diff revision 1)
> +    - 'docker-worker:cache:level-{{level}}-{{project}}-build-android-api-15-frontend-workspace'

This will share the workspace with `testing/taskcluster/tasks/builds/android_api_15_frontend.yml`.  Is that intentional?
Attachment #8728696 - Flags: review?(dustin) → review+
Comment on attachment 8728695 [details]
MozReview Request: Bug 1064004 - Pre: Allow running Android emulators in headless mode. r?gbrown

https://reviewboard.mozilla.org/r/39039/#review35891

Can I get some context? Why run with no audio, window, or gpu? Obviously you don't need to for this case, but wouldn't it be better to have audio and gpu available for browser tests? And window for debugging?

On another note, I came to the conclusion that different emulator startup logic is needed for interactive (mach) and automated (mozharness) environments. In our existing tier 1 emulator tests, mozharness (android_emulator_unittest.py) starts the emulator, tries to verify, automatically kills and retries and logs diagnostics, etc. A lot of that seemed an inappropriate bother for the mach command (where if it fails 1 time in a 100, you can just run it again) -- something to think about.
Attachment #8728695 - Flags: review?(gbrown)
Comment on attachment 8728697 [details]
MozReview Request: Bug 1064004 - Add Espresso instrumentation test support to Gradle. r?mcomella

I still don't have much context on how gradle works but these changes seem reasonable.
Attachment #8728697 - Flags: feedback?(michael.l.comella) → feedback+
Assignee: nobody → nalexander
You need to log in before you can comment on or make changes to this bug.