601153 - Automate JS tests for any Android phone or emulator

Status: RESOLVED DUPLICATE of bug 858622

(Testing :: General, defect)

Comment 1

[Robert Sayre]

Attachment: python script to start and stop emulator

this will find the emulator, start it, wait for it to boot, and unlock the homescreen. Then it checks that the required apks for python are installed.

Next step is to make the emulator step optional and switch to an OptionsParser. That way, the tests can run on an attached device in automated fashion as well. I have tried this by commenting out the emulator stuff.

Now, one crucial broken step here is that the sl4a Android scripting environment is currently broken when running commandline intents. From perusing the stack traces in logcat, I think it will be easy to fix, but will require diving into Eclipse.

Comment 2

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

We absolutely must stop finding brokenness in the field that our JS tests would have caught.  This needs to block fennec-beta3.

Comment 3

[cmtalbert]

I looked at moving the js shell tests over week before last.  I'll add it to my list and see if I can make these remoteable to work with the system we've developed for Android testing.  I don't think it should be too much work here, the JS tests are pretty straight forward.

I'll report back with an ETA once I look at the code in more detail.

Comment 4

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

https://wiki.mozilla.org/Mobile/Fennec/Android#JS_trace-tests may or may not be relevant, depending on your remoting system.

Comment 5

[Benjamin Smedberg]

Who should own this? It's currently marked as a blocker, and doesn't sound straightforward.

Comment 6

[cmtalbert]

(In reply to comment #5)
> Who should own this? It's currently marked as a blocker, and doesn't sound
> straightforward.

I'm not sure I'm the best person for it, but I am trying to remote these tests so that we can run them the same way that we run everything else on android via buildbot.  

The first hurdle we had to overcome was getting environment variables to work reliably in the remoting system, which we were able to do in november.

Currently, I'm stuck trying to get a debug build of android to compile, it's breaking in breakpad, and I'm working with ted on that issue.  With an opt build I could not get the js trace test to run at all using the method proposed on the wiki page in comment 4 on a tegra.

Note, that I can get them to run using that method on my Droid.  But, I still can't get them to work on tegra and I was hoping by getting a debug build to function I'd get further along that path.  

The actual remoting code is pretty straightforward and simple, once I can get the js shell to run reliably on the tegra device.  

I'll take this for now as I am working on it, but I do apologize for not realizing this had been made a blocker.  I'll see if I can make some progress here this week.

Comment 7

[cmtalbert]

Attachment: jit-test-whitespace-refactor

So, I've been banging my head against getting this automated so that we can run this on tegras via buildbot and by developers on their phones.  I've all but given up on getting it to run on the tegras.  As my last ditch effort on tegras, I'm trying to build a debug android os to see if I can determine why the js shell refuses to start on that platform.

Since the js shell does work fine on any other phone (galaxy, nexus, droid etc) I'm moving ahead with the necessary refactoring to enable these tests to run via the sutagent & buildbot as well as for developers.  When this refactoring is complete, there will be no functional changes to jit-test.py (it will work the same as it always has) and there will be a jit-test-remote.py that will handle running jit-tests against a device running the SUTagent.

I'll attach a series of WIP patches here to start getting some feedback. Apply them in order:
1. jit-test-whitespace-refactor.diff (whitespace and class changes only)
2. jit-options-refactor.diff
3. jit-test-remote.diff

Comment 8

[cmtalbert]

Attachment: jit-options-refactor

Comment 9

[cmtalbert]

Attachment: jit-test-remote WIP

This isn't finished (obviously).  It's attached so that you can see why we refactor into classes for remoted tests and how those classes get used.

Comment 10

[Andreas Gal :gal]

I am a huge fan of automated tests but would we hold the release for this?

Comment 11

[Mike Beltzner [:beltzner, not reading bugmail]]

** PRODUCT DRIVERS PLEASE NOTE **

This bug is one of 7 automatically changed from blocking2.0:final+ to blocking2.0:.x during the endgame of Firefox 4 for the following reasons:

 - it was marked as a soft blocking issue without a requirement for beta coverage

Comment 12

[(not currently active) Ted Mielczarek]

Hey, we finally fixed this. bug 858621 covers getting the jit-tests running from the test package on tbpl (they're running on Cedar currently).