Closed Bug 1169476 Opened 4 years ago Closed 4 years ago

Add |mach robocop --serve| option to help run Robocop tests from IDEs

Categories

(Firefox for Android :: Testing, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
Firefox 41
Tracking Status
firefox41 --- fixed

People

(Reporter: nalexander, Assigned: nalexander)

References

Details

Attachments

(1 file)

There's a good deal of preparation needed to run a Robocop test on device.  Off the top of my head, we:
* push the Robocop APK;
* start the mochi.test server;
* prepare a PAC file;
* push a profile to the device;
* push a robotium.config to the device;
* launch instrumentation with specific configuration options;
* and after the test is complete, arrange to kill the process.

It's pretty hard to do that from within an IDE.  However, IDEs are really good at pushing APKs only when needed and launching instrumentation tests.

This ticket tracks adding a flag to |mach robocop| that does everything above up to "launch instrumentation".  Instead of launching any tests, |mach robocop --serve| gets everything started, launches Fennec with a test profile, and then just waits.

The user can then launch instrumentation tests into this testing environment from the IDE.  It's not smooth yet but it's helping me locally.
For context, this is a rehashed version of Part 8 of Bug 929654.  That got r- from jmaher, largely of concerns with --no-autorun: see https://bugzilla.mozilla.org/show_bug.cgi?id=929654#c21.  That flag got added by another ticket, I wager because doing the right thing is just too painful :(  Also from that ticket, I wrote:

"""
I'm sorry, I didn't give this good context.  The purpose of this is to let you "run Fennec in testing mode" and keep the mochi.test web server up, while you rebuild and redeploy the Robocop APK from an IDE.  That is, in a terminal, I run |mach robocop --serve| and then I can develop my tests in Eclipse.  Killing the mochi.test server after 10 minutes doesn't serve this goal.
"""

I hope the ticket description elaborates on this.
Bug 1169476 -- Implement |mach robocop --serve|. r?gbrown

This adds a flag to |mach robocop| that does everything to run a
Robocop test except launch the actual test.  Instead of launching the
test, it starts the mochi.test server and launches Fennec with a test
profile; then it sits and waits forever.

This allows regular Java IDEs (IntelliJ, but previously Eclipse) to
run Robocop tests like regular instrumentation tests, "injecting" them
into the prepared testing environment.  It's quite nice!
Attachment #8612636 - Flags: review?(gbrown)
Comment on attachment 8612636 [details]
MozReview Request: Bug 1169476 -- Implement |mach robocop --serve|. r?gbrown

https://reviewboard.mozilla.org/r/9603/#review8485

::: mobile/android/tests/browser/robocop/UITest.java
(Diff revision 1)
> -        // Don't show the first run experience.
> -        intent.putExtra(BrowserApp.EXTRA_SKIP_STARTPANE, true);
> -        intent.putExtra("args", "-no-remote -profile " + mProfile);

It looks like the EXTRA_SKIP_STARTPANE bit is lost in the move. I'm not sure of the history here, but worth checking.

::: mobile/android/tests/browser/robocop/BaseRobocopTest.java:212
(Diff revision 1)
> +                Log.i(LOGTAG, "Trying to keep started activity alive.");
> +                setActivity(null);

Seems fragile, but I don't have a better suggestion.

::: build/mobile/robocop/LaunchFennecWithConfigurationActivity.java:31
(Diff revision 1)
> +        final String configFile = FennecNativeDriver.getFile(BaseRobocopTest.DEFAULT_ROOT_PATH + "/robotium.config");

DEFAULT_ROOT_PATH might not work. Is there no way to get the proper root?

::: testing/mochitest/runtestsremote.py:584
(Diff revision 1)
> +                    "-e", "quit", "1",
> +                    "-e", "finish", "1",

I would consolidate "quit" and "finish" into one "quit": If --serve, there is no quit-or-finish; if no --serve, quit-and-finish.

::: testing/mochitest/runtestsremote.py:601
(Diff revision 1)
> +                dm.default_timeout = 7 * 24 * 60 * 60 # One week, in seconds.

Something like sys.maxint might be more informative.

::: testing/mochitest/runtestsremote.py:598
(Diff revision 1)
> +                options.browserArgs = ["start",
> +                                       "-n", "org.mozilla.roboexample.test/org.mozilla.gecko.LaunchFennecWithConfigurationActivity",
> +                                       "&&", "cat"]

You might not need "&& cat" if you set options.timeout to something huge.
Attachment #8612636 - Flags: review?(gbrown) → review+
https://reviewboard.mozilla.org/r/9603/#review8511

> DEFAULT_ROOT_PATH might not work. Is there no way to get the proper root?

Sadly, no.  We could make it more robust by also trying /data/local, I suppose.  This is strictly for local testers, and even more it's just for testers running from IntelliJ.  99% of these testers will be fine with /mnt/sdcard.

> It looks like the EXTRA_SKIP_STARTPANE bit is lost in the move. I'm not sure of the history here, but worth checking.

Great catch!

> I would consolidate "quit" and "finish" into one "quit": If --serve, there is no quit-or-finish; if no --serve, quit-and-finish.

Sure, I'll do this.

> You might not need "&& cat" if you set options.timeout to something huge.

The slave process will exit, though, and the test harness will exit after it.  I arrived at && cat after trying to be more forgiving about the process ending and found this easier.  We'll see how it holds up in the wild.

> Something like sys.maxint might be more informative.

Sounds reasonable, I'll do this.
gbrown: I tried to push the updated commit back to RB, so you could see the changes with the nice interdiff slider, but all my rebasing and amending confused the system :(
sorry Nick, had to back this out for test failures/crashes like https://treeherder.mozilla.org/logviewer.html#?job_id=3275457&repo=fx-team
Flags: needinfo?(nalexander)
Fixed and re-landed.  Looked healthy on try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6c3fae69b336 -- the rc5 looks like a legit issue.
Flags: needinfo?(nalexander)
https://hg.mozilla.org/mozilla-central/rev/2feb31e63a45
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 41
You need to log in before you can comment on or make changes to this bug.