Closed Bug 1056094 Opened 10 years ago Closed 10 years ago

Make it possible to run mozbench on OS X

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Platform:
All
Other
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dminor, Assigned: mcc.ricardo, Mentored)

References

Details

(Keywords: meta, Whiteboard: [good next bug][lang=python]) 

Mozbench is a cross-browser benchmark suite. Our initial target is to compare firefox and chrome on windows and linux platforms. For more information, see [1] and [2]. The mozbench repository is located at [3] and instructions on setting it up and running the benchmarks can be found in the readme.md file at that location.

We're interested in being able to run benchmarks on OS X as well. The first step is to make changes (if any are required) to get mozbench/mozbench.py running on OS X. The second step is to modify the linux run script (jenkins/run.sh) to work for both linux and OS X.

[1] https://wiki.mozilla.org/Game_Benchmark_Automation
[2] https://wiki.mozilla.org/Auto-tools/Projects/Mozbench
[3] https://github.com/dminor/mozbench
I gave a quick look and it shouldn't be difficult to achieve this. The only issue I'm getting is when 'mozinstall' is called. It doesn't mount the '.dmg'.

One way around this is to write a simple script that uses 'hdiutil' to mount the '.dmg'. We would identify that the system is 'mac', override the 'mozinstall', mount the '.dmg' and then 'path' would point to that location.

After that it should be fine. I tried with an already mounted version and it worked.
Did some initial coding and ran a full test successfully. Still needs some work, as well as the jenkins script.

Diff: https://github.com/mccricardo/mozbench/compare/mozbench-osx
New diffs for the work: https://github.com/mccricardo/mozbench/compare/mozbench-osx.

Just realized something: if the nightly build will be mounted, the folder where it will reside will be 'FirefoxNightly.app' and not 'Firefox.app/'. Am I correct?
Sorry I didn't respond earlier, I away last week. It looks like you are on the right track, thanks for looking at this!
Assignee: nobody → mcc.ricardo
Status: NEW → ASSIGNED
No problem Dan. I had a flu for in the last few days, so I had to take it easy.

As I mentioned above, when mounted, the nightly builds of Firefox will reside in 'FirefoxNightly.app'. If this code will only be used for the nightly builds, we can specify that folder.

If not, we can either rename the folder to always be the same or we can 'detect' the folder name.
(In reply to mcc.ricardo from comment #5)
> No problem Dan. I had a flu for in the last few days, so I had to take it
> easy.
> 
> As I mentioned above, when mounted, the nightly builds of Firefox will
> reside in 'FirefoxNightly.app'. If this code will only be used for the
> nightly builds, we can specify that folder.
> 
> If not, we can either rename the folder to always be the same or we can
> 'detect' the folder name.

The plan is to only test using nightly builds so 'FirefoxNightly.app' should be fine. We might want to test using Aurora nightly builds in the future, but I'm guessing the naming will be the same there as well.

Thanks!
No problem Dan. 

I'll update the code later today and test with both.
Well, nightly builds mount to 'FirefoxNightly.app' and Aurora to 'FirefoxAurora.app'. We can easily identify based on the URL.
Added some code the deal with different folders.

Diff: https://github.com/mccricardo/mozbench/compare/mozbench-osx?expand=1
What do you think Dan?
Thanks! This looks good to me, feel free to submit a pull request.

Sorry I missed your comment the first time around. I usually catch email from mentored bugs, but you can always needinfo me if I'm not responding :)
mozinstall claims to support dmg files, does that not work?
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #12)
> mozinstall claims to support dmg files, does that not work?

I tried using and it get stuck for a long time and eventually gives an error.
(In reply to Dan Minor [:dminor] from comment #11)
> Thanks! This looks good to me, feel free to submit a pull request.
> 
> Sorry I missed your comment the first time around. I usually catch email
> from mentored bugs, but you can always needinfo me if I'm not responding :)

No problem Dan. Just submitted the pull request.
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #12)
> mozinstall claims to support dmg files, does that not work?

I spent some time this morning on this and I think the issue was that we were downloading the installer to a file called 'firefox.exe' - changing the extension to be 'firefox.dmg' on OS X seems to fix things.
I've taken a look at you comments and I think I got it. I'll tackle it this weekend.
(In reply to mcc.ricardo from comment #16)
> I've taken a look at you comments and I think I got it. I'll tackle it this
> weekend.

Thanks again for looking at this!
Just updated the code Dan.
Thanks again!
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Hi Ricardo, did the 'Massive' benchmark run successfully for you? It is timing out on our test machine and I'm trying to determine if it is problem with Massive or a test machine problem. Thanks!
Flags: needinfo?(mcc.ricardo)
Hi Dan,

It did run. I don't have access to my Mac at the moment but I can test it again and double check when I get home later today.

If you send me the error it's giving and can try and look more thoroughly for it.

Ricardo
Flags: needinfo?(mcc.ricardo)
(In reply to mcc.ricardo from comment #21)
> Hi Dan,
> 
> It did run. I don't have access to my Mac at the moment but I can test it
> again and double check when I get home later today.
> 
> If you send me the error it's giving and can try and look more thoroughly
> for it.
> 
> Ricardo

Thanks! If it was working for you, it is probably something with the way the test machine is configured. Don't worry about following up on this, I'll take a look at it today. Dan
No problem Dan.

If you find some problem with it, just give me a shout.

Ricardo
Hey Dan,

Currently I'm connecting to the device using: mozdevice.ADBAndroid(...)

I can either do:

1) mozdevice.ADBAndroid(None) - If None is passed and there is exactly one device attached to the host, that device is used. https://github.com/mozilla/geckodev/blob/master/testing/mozbase/mozdevice/mozdevice/adb.py#L434

2) mozdevice.ADBAndroid(some_device_id) - When a string is passed, it is interpreted as the device serial number. We can get the device id but I believe the None option is more useful since we can, that way, connect any device we want.

Both of them work, but the issue I'm having is that before connection is successful I get some errors like:


ERROR:adb:ADBDevice.__init__: id
Traceback (most recent call last):
  File "/home/rcastro/.envs/mozbench/local/lib/python2.7/site-packages/mozdevice/adb.py", line 485, in __init__
    if self.shell_output("su -c '%s'" % cmd_id).find(uid) != -1:
  File "/home/rcastro/.envs/mozbench/local/lib/python2.7/site-packages/mozdevice/adb.py", line 860, in shell_output
    raise ADBError("%s" % adb_process)
ADBError: args: adb -s 0019d516170f8e wait-for-device shell su -c 'LD_LIBRARY_PATH=/vendor/lib:/system/lib id'; echo rc=$?, exitcode: 127, stdout: /system/bin/sh: su: not found, stderr: 
ERROR:adb:ADBDevice.__init__: id
Traceback (most recent call last):
  File "/home/rcastro/.envs/mozbench/local/lib/python2.7/site-packages/mozdevice/adb.py", line 491, in __init__
    if self.shell_output("su 0 id").find(uid) != -1:
  File "/home/rcastro/.envs/mozbench/local/lib/python2.7/site-packages/mozdevice/adb.py", line 860, in shell_output
    raise ADBError("%s" % adb_process)
ADBError: args: adb -s 0019d516170f8e wait-for-device shell su 0 id; echo rc=$?, exitcode: 127, stdout: /system/bin/sh: su: not found, stderr: 


Like I said, connection is successful but I would like to avoid those errors showing up on the console if possible. 

Ricardo
Hi Ricardo,

How about adding a command line option to specify the device id, which defaults to None?

This might be useful we end up having a single test machine with both firefox os and Android devices attached. I would have to modify the marionette runner to support this as well.

It looks like those errors come mozdevice attempting to determine whether 'su' is available. When I look at the current version of mozdevice, it looks like those exceptions should be caught and logged. Is it possible you're using an older version of mozdevice?
The command line option sounds great.

mozdevie, in the requirements file, is 0.40. According to PyPi, current version (as of 2014-10-06) is 0.41. I'll update and check if those go away.

Ricardo
Just noticed that I pasted the comment in the wring ticket. I'll move it to the correct one for reference.
You need to log in before you can comment on or make changes to this bug.