1169476 - Add |mach robocop --serve| option to help run Robocop tests from IDEs

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect)

Description

[Nick Alexander :nalexander [he/him]]

There's a good deal of preparation needed to run a Robocop test on device.  Off the top of my head, we:
* push the Robocop APK;
* start the mochi.test server;
* prepare a PAC file;
* push a profile to the device;
* push a robotium.config to the device;
* launch instrumentation with specific configuration options;
* and after the test is complete, arrange to kill the process.

It's pretty hard to do that from within an IDE.  However, IDEs are really good at pushing APKs only when needed and launching instrumentation tests.

This ticket tracks adding a flag to |mach robocop| that does everything above up to "launch instrumentation".  Instead of launching any tests, |mach robocop --serve| gets everything started, launches Fennec with a test profile, and then just waits.

The user can then launch instrumentation tests into this testing environment from the IDE.  It's not smooth yet but it's helping me locally.

Comment 1

[Nick Alexander :nalexander [he/him]]

For context, this is a rehashed version of Part 8 of Bug 929654.  That got r- from jmaher, largely of concerns with --no-autorun: see https://bugzilla.mozilla.org/show_bug.cgi?id=929654#c21.  That flag got added by another ticket, I wager because doing the right thing is just too painful :(  Also from that ticket, I wrote:

"""
I'm sorry, I didn't give this good context.  The purpose of this is to let you "run Fennec in testing mode" and keep the mochi.test web server up, while you rebuild and redeploy the Robocop APK from an IDE.  That is, in a terminal, I run |mach robocop --serve| and then I can develop my tests in Eclipse.  Killing the mochi.test server after 10 minutes doesn't serve this goal.
"""

I hope the ticket description elaborates on this.

Comment 2

[Nick Alexander :nalexander [he/him]]

Attachment: MozReview Request: Bug 1169476 -- Implement |mach robocop --serve|. r?gbrown

Bug 1169476 -- Implement |mach robocop --serve|. r?gbrown

This adds a flag to |mach robocop| that does everything to run a
Robocop test except launch the actual test.  Instead of launching the
test, it starts the mochi.test server and launches Fennec with a test
profile; then it sits and waits forever.

This allows regular Java IDEs (IntelliJ, but previously Eclipse) to
run Robocop tests like regular instrumentation tests, "injecting" them
into the prepared testing environment.  It's quite nice!

Comment 3

[Nick Alexander :nalexander [he/him]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f226a0dcfb5f

Comment 4

[Geoff Brown [:gbrown]]

Comment on attachment 8612636 [details]
MozReview Request: Bug 1169476 -- Implement |mach robocop --serve|. r?gbrown

https://reviewboard.mozilla.org/r/9603/#review8485

::: mobile/android/tests/browser/robocop/UITest.java
(Diff revision 1)
> -        // Don't show the first run experience.
> -        intent.putExtra(BrowserApp.EXTRA_SKIP_STARTPANE, true);
> -        intent.putExtra("args", "-no-remote -profile " + mProfile);

It looks like the EXTRA_SKIP_STARTPANE bit is lost in the move. I'm not sure of the history here, but worth checking.

::: mobile/android/tests/browser/robocop/BaseRobocopTest.java:212
(Diff revision 1)
> +                Log.i(LOGTAG, "Trying to keep started activity alive.");
> +                setActivity(null);

Seems fragile, but I don't have a better suggestion.

::: build/mobile/robocop/LaunchFennecWithConfigurationActivity.java:31
(Diff revision 1)
> +        final String configFile = FennecNativeDriver.getFile(BaseRobocopTest.DEFAULT_ROOT_PATH + "/robotium.config");

DEFAULT_ROOT_PATH might not work. Is there no way to get the proper root?

::: testing/mochitest/runtestsremote.py:584
(Diff revision 1)
> +                    "-e", "quit", "1",
> +                    "-e", "finish", "1",

I would consolidate "quit" and "finish" into one "quit": If --serve, there is no quit-or-finish; if no --serve, quit-and-finish.

::: testing/mochitest/runtestsremote.py:601
(Diff revision 1)
> +                dm.default_timeout = 7 * 24 * 60 * 60 # One week, in seconds.

Something like sys.maxint might be more informative.

::: testing/mochitest/runtestsremote.py:598
(Diff revision 1)
> +                options.browserArgs = ["start",
> +                                       "-n", "org.mozilla.roboexample.test/org.mozilla.gecko.LaunchFennecWithConfigurationActivity",
> +                                       "&&", "cat"]

You might not need "&& cat" if you set options.timeout to something huge.

Comment 5

[Nick Alexander :nalexander [he/him]]

https://reviewboard.mozilla.org/r/9603/#review8511

> DEFAULT_ROOT_PATH might not work. Is there no way to get the proper root?

Sadly, no.  We could make it more robust by also trying /data/local, I suppose.  This is strictly for local testers, and even more it's just for testers running from IntelliJ.  99% of these testers will be fine with /mnt/sdcard.

> It looks like the EXTRA_SKIP_STARTPANE bit is lost in the move. I'm not sure of the history here, but worth checking.

Great catch!

> I would consolidate "quit" and "finish" into one "quit": If --serve, there is no quit-or-finish; if no --serve, quit-and-finish.

Sure, I'll do this.

> You might not need "&& cat" if you set options.timeout to something huge.

The slave process will exit, though, and the test harness will exit after it.  I arrived at && cat after trying to be more forgiving about the process ending and found this easier.  We'll see how it holds up in the wild.

> Something like sys.maxint might be more informative.

Sounds reasonable, I'll do this.

Comment 6

[Nick Alexander :nalexander [he/him]]

https://hg.mozilla.org/integration/fx-team/rev/e91ac9a35f86

Comment 7

[Nick Alexander :nalexander [he/him]]

gbrown: I tried to push the updated commit back to RB, so you could see the changes with the nice interdiff slider, but all my rebasing and amending confused the system :(

Comment 8

[Nick Alexander :nalexander [he/him]]

https://hg.mozilla.org/integration/fx-team/rev/4fd1a129d197

Comment 9

[Carsten Book [:Tomcat]]

sorry Nick, had to back this out for test failures/crashes like https://treeherder.mozilla.org/logviewer.html#?job_id=3275457&repo=fx-team

Comment 10

[Pulsebot]

Backout:
https://hg.mozilla.org/integration/fx-team/rev/55c42e800046
https://hg.mozilla.org/integration/fx-team/rev/049b2fdb0711

Comment 11

[Nick Alexander :nalexander [he/him]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=65344bef3692

Comment 12

[Nick Alexander :nalexander [he/him]]

https://hg.mozilla.org/integration/fx-team/rev/2feb31e63a45

Comment 13

[Nick Alexander :nalexander [he/him]]

Fixed and re-landed.  Looked healthy on try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6c3fae69b336 -- the rc5 looks like a legit issue.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/2feb31e63a45