Closed Bug 1066665 Opened 10 years ago Closed 10 years ago

Make it possible to run mozbench on Android

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Platform:
All
Other
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dminor, Assigned: mcc.ricardo, Mentored)

References

Details

(Whiteboard: [good next bug][lang=python]) 

Mozbench is a cross-browser benchmark suite. Our initial target is to compare firefox and chrome on windows and linux platforms. For more information, see [1] and [2]. The mozbench repository is located at [3] and instructions on setting it up and running the benchmarks can be found in the readme.md file at that location.

We're interested in being able to run benchmarks on android as well. This will involve adding a separate code path to install and run a nightly build of firefox on an android phone.

We have a library called mozdevice [4] which abstracts away some of the details of interacting with the phone over adb and should be sufficient to handle installing the firefox apk.

The mozrunner library [5] (which we are already using) has support for running on remote devices, but this is mostly aimed towards firefox os. We should be able to derive a class from one of the base classes there which allows us to run a browser on android. Ideally we would be able to support running both firefox and chrome (if installed) on the device, to allow for cross-browser comparisons.

Please let me know if you have any questions.

[1] https://wiki.mozilla.org/Game_Benchmark_Automation
[2] https://wiki.mozilla.org/Auto-tools/Projects/Mozbench
[3] https://github.com/dminor/mozbench
[4] http://mozbase.readthedocs.org/en/latest/mozdevice.html
[5] http://mozbase.readthedocs.org/en/latest/mozrunner.html
After the work on Bug 1056094 is completed I think this would be fun to work on, if no one shows interest. Even if someone does, I would be more than happy to help out anyway I can.
Thanks! I was hoping you would take this on.
Assignee: nobody → mcc.ricardo
Status: NEW → ASSIGNED
I would not suggest using mozrunner at this time for Android. It has several built-in assumptions (e.g. the idea that you are launching a firefox process which you can control) which are not a good match for this platform. We might fix that someday, but this is a large project.

What we've done in other Android projects is create a small amount of custom automation to setup a profile and launch the browser. Eideticker has some simple code designed for launching/running multiple browsers which you could use with probably only very minimal modification:

https://github.com/mozilla/eideticker/blob/master/src/eideticker/eideticker/runner.py
Excellent! Thank you for the input. You might have just saved us some precious time.

I'll give it a look. Sound promising!
Hey Dan,

I started writing some code for this on a separate module. Giving a looking ate you initial implementation using marionette, I thought it might be preferable to include the android support into mozbench.py.

May, we should include a new argument (maybe --firefox-apk) and develop there?

What do you think?
(In reply to mcc.ricardo from comment #5)
> Hey Dan,
> 
> I started writing some code for this on a separate module. Giving a looking
> ate you initial implementation using marionette, I thought it might be
> preferable to include the android support into mozbench.py.
> 
> May, we should include a new argument (maybe --firefox-apk) and develop
> there?
> 
> What do you think?

That sounds good to me. If you make a class that has 'start', 'stop' and 'wait' methods then it shouldn't be much work to integrate it with the rest of the code.

I'm not sure if it will end up being cleaner to make 'install_firefox' handle .apk files, or just put it into the 'start' method of your class.
Cool. I'll move the code over to mozbench.py and enhance the command line arguments. 

Since mozinstall does not take care of apk files, it's probably better to deal with it in the start method.

What are the purposes for each method? Should there be any return value for any of them?
If you are running a process, the start() method should start the process, the stop() method should stop the process and the wait() method should wait for the process to end. The start() method would return the process id, the stop() and wait() methods return the process exit codes.

The start() method should install the fennec apk and launch the browser with the appropriate url. If you choose to install the apk in the start method, you should probably uninstall it in the stop() method. I think the wait() method can do nothing in this case. I think for our purposes it is ok not to return anything from these methods.
Those were also my thoughts on it. Thanks for the clarification.
After messing around with it, I think it's better not to install in the start method. If we do that we would be installing each time a benchmark is run.

I'll move it out and extend our current 'install_firefox'.
Hi Dan,

Just to keep you updated on progress.

I've managed to install, launch Firefox and Chrome and run the benchmarks. I'll be handling the cleaning up next. 

Ricardo
(In reply to mcc.ricardo from comment #11)
> Hi Dan,
> 
> Just to keep you updated on progress.
> 
> I've managed to install, launch Firefox and Chrome and run the benchmarks.
> I'll be handling the cleaning up next. 
> 
> Ricardo

Wow, nice work! Thanks for taking this on.
Hey Dan,

Sorry this is took longer that I anticipated. I've just been swamped with work.

I'll be on holiday for the next two weeks. Nonetheless, I think I complete the main flow.

I ran some preliminary tests, and it seems to be working. I still want to run some more rigorous tests as well as tidy up some of the code and I think needs some improving.

You can have a look at the progress so far: https://github.com/mccricardo/mozbench/compare/mozbench-android

Ricardo
(In reply to Ricardo Castro from comment #13)
> Hey Dan,
> 
> Sorry this is took longer that I anticipated. I've just been swamped with
> work.
> 
> I'll be on holiday for the next two weeks. Nonetheless, I think I complete
> the main flow.
> 
> I ran some preliminary tests, and it seems to be working. I still want to
> run some more rigorous tests as well as tidy up some of the code and I think
> needs some improving.
> 
> You can have a look at the progress so far:
> https://github.com/mccricardo/mozbench/compare/mozbench-android
> 
> Ricardo

Thanks, looking good so far!
Hi Dan,

I'm back from holiday. I'll be resuming work on this ticket.

Ricardo
(In reply to Ricardo Castro from comment #15)
> Hi Dan,
> 
> I'm back from holiday. I'll be resuming work on this ticket.
> 
> Ricardo

Great!
Comment moved from wrong ticket. Should have been in this one.

*Ricardo*

Hey Dan,

Currently I'm connecting to the device using: mozdevice.ADBAndroid(...)

I can either do:

1) mozdevice.ADBAndroid(None) - If None is passed and there is exactly one device attached to the host, that device is used. https://github.com/mozilla/geckodev/blob/master/testing/mozbase/mozdevice/mozdevice/adb.py#L434

2) mozdevice.ADBAndroid(some_device_id) - When a string is passed, it is interpreted as the device serial number. We can get the device id but I believe the None option is more useful since we can, that way, connect any device we want.

Both of them work, but the issue I'm having is that before connection is successful I get some errors like:


ERROR:adb:ADBDevice.__init__: id
Traceback (most recent call last):
  File "/home/rcastro/.envs/mozbench/local/lib/python2.7/site-packages/mozdevice/adb.py", line 485, in __init__
    if self.shell_output("su -c '%s'" % cmd_id).find(uid) != -1:
  File "/home/rcastro/.envs/mozbench/local/lib/python2.7/site-packages/mozdevice/adb.py", line 860, in shell_output
    raise ADBError("%s" % adb_process)
ADBError: args: adb -s 0019d516170f8e wait-for-device shell su -c 'LD_LIBRARY_PATH=/vendor/lib:/system/lib id'; echo rc=$?, exitcode: 127, stdout: /system/bin/sh: su: not found, stderr: 
ERROR:adb:ADBDevice.__init__: id
Traceback (most recent call last):
  File "/home/rcastro/.envs/mozbench/local/lib/python2.7/site-packages/mozdevice/adb.py", line 491, in __init__
    if self.shell_output("su 0 id").find(uid) != -1:
  File "/home/rcastro/.envs/mozbench/local/lib/python2.7/site-packages/mozdevice/adb.py", line 860, in shell_output
    raise ADBError("%s" % adb_process)
ADBError: args: adb -s 0019d516170f8e wait-for-device shell su 0 id; echo rc=$?, exitcode: 127, stdout: /system/bin/sh: su: not found, stderr: 


Like I said, connection is successful but I would like to avoid those errors showing up on the console if possible. 

*Dan*

Hi Ricardo,

How about adding a command line option to specify the device id, which defaults to None?

This might be useful we end up having a single test machine with both firefox os and Android devices attached. I would have to modify the marionette runner to support this as well.

It looks like those errors come mozdevice attempting to determine whether 'su' is available. When I look at the current version of mozdevice, it looks like those exceptions should be caught and logged. Is it possible you're using an older version of mozdevice?

*Ricardo*

The command line option sounds great.

mozdevie, in the requirements file, is 0.40. According to PyPi, current version (as of 2014-10-06) is 0.41. I'll update and check if those go away.
Updated mozdevice to 0.41 and those exceptions get logged properly.
Hi Dan,

Just submitted the pull request. 

I'm keen to receive your input and improve code.

Ricardo
Fixed in https://github.com/dminor/mozbench/commit/2901eab3997f68298327e54d3f447432fad7a237

Thanks!
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.