903537 - Write python test script for running Fennec instrumentation test APK

Status: RESOLVED WONTFIX

(Firefox Build System :: Android Studio and Gradle Integration, defect)

Description

[Nick Alexander :nalexander [he/him]]

This is the conclusion of Bug 903531 and Bug 903534.  Once we have the test suite in m-c, and the build system building the Fennec instrumentation APK, we need a script to run it -- both locally and on TBPL.

This script probably looks like a cleaner version of

http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/remotexpcshelltests.py

and

http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtestsremote.py

This will require outreach with the ateam, so that we follow there best practices.  At some point, jhamel suggested to me that this part was pretty free-form and not difficult to make happen on TBPL.

One consideration: this script needs to translate instrumentation test errors into TBPL errors.  This could be done by the script, or by writing a custom JUnit test runner that gets built into the instrumentation testing APK.

Comment 1

[Nick Alexander :nalexander [he/him]]

Attachment: Part 1: Define and use common remote options in automationutils.

Hi gbrown, I'm starting to write the script for running these
background instrumentation tests on TBPL.  It'll be similar to
runtestsremote.py but simpler (I hope) since it needs to do less.
Here's a little clean-up and unification patch that I'll use.  There
are a few more places that could be unified, but that would mean
changing argument names :(

Comment 2

[Nick Alexander :nalexander [he/him]]

http://tbpl.mozilla.org/?tree=Try&rev=d39894b4e8e8

Comment 3

[Nick Alexander :nalexander [he/him]]

(In reply to Nick Alexander :nalexander from comment #1)
> Created attachment 833125 [details] [diff] [review]
> Part 1: Define and use common remote options in automationutils.
> 
> Hi gbrown, I'm starting to write the script for running these
> background instrumentation tests on TBPL.  It'll be similar to
> runtestsremote.py but simpler (I hope) since it needs to do less.
> Here's a little clean-up and unification patch that I'll use.  There
> are a few more places that could be unified, but that would mean
> changing argument names :(

I missed at least one thing: testing/xpcshell/remotexpcshelltests.py needs to have remoteTestRoot option removed as well.

Comment 4

[Geoff Brown [:gbrown]]

Comment on attachment 833125 [details] [diff] [review]
Part 1: Define and use common remote options in automationutils.

Review of attachment 833125 [details] [diff] [review]:
-----------------------------------------------------------------

It's a simple change, but hits all of the Android unit tests, which have lots of use cases...are you sure you want to do that?

The try run looks awful. Ping me again when that's green and you have verified that the tests still run locally with adb and sut.

::: build/automationutils.py
@@ +223,5 @@
> +
> +  if not deviceIP:
> +    raise devicemanager.DMError("You must provide a device IP to connect to with dm_trans=sut")
> +
> +  return devicemanagerSUT.DeviceManagerSUT(deviceIP, devicePort, deviceRoot=remoteTestRoot)

This won't work, will it? Some clients use DroidSUT instead of DeviceManagerSUT.

Comment 5

[Nick Alexander :nalexander [he/him]]

https://tbpl.mozilla.org/?tree=Try&rev=2ba51362a373

Comment 6

[Nick Alexander :nalexander [he/him]]

https://hg.mozilla.org/projects/ash/rev/be9be327fb02

Comment 7

[Nick Alexander :nalexander [he/him]]

https://hg.mozilla.org/projects/ash/rev/aeda38307736

Comment 8

[Nick Alexander :nalexander [he/him]]

chmanchester: Over at [1], I have used the build API to trigger a job with a hacked up tests.zip.  Said tests.zip has most of a runinstrumentation.py script in place.  It seems to be mostly working, but I have a question.  I'd like to show the entire Java stack trace rather than just the first line in the "summary" view.  How would you suggest I do that?

[1] https://treeherder.mozilla.org/ui/logviewer.html#?job_id=62189&repo=ash

Comment 9

[Chris Manchester (limited bugmail, email directly)]

(In reply to Nick Alexander :nalexander from comment #8)
> chmanchester: Over at [1], I have used the build API to trigger a job with a
> hacked up tests.zip.  Said tests.zip has most of a runinstrumentation.py
> script in place.  It seems to be mostly working, but I have a question.  I'd
> like to show the entire Java stack trace rather than just the first line in
> the "summary" view.  How would you suggest I do that?
> 
> [1] https://treeherder.mozilla.org/ui/logviewer.html#?job_id=62189&repo=ash

The treeherder logviewer predates structured logging and I don't think it's been retrofitted to consume JSON logs. I expect it's still line based and regex based, so we probably don't have a great answer for this. The summary in the upper right of the page appears to be line oriented: I might suggest a way to output something from mozharness or format the stack in a way that would trigger the tool's regex, but then I suspect each line would be a separate link in the summary view, and that might be misleading in its own right.

I would drop in to #treeherder for more definitive answers on the logviewer tool.

Comment 10

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Nick, still relevant?

Comment 11

[Nick Alexander :nalexander [he/him]]

(In reply to Michael Comella (:mcomella) from comment #10)
> Nick, still relevant?

Not really -- any new instrumentation test suite would run through GeckoView and hopefully through Gradle's androidTest rather than a bespoke instrumentation runner in Python.