Make it possible to run mozbench on OS X

RESOLVED FIXED

Status

RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: dminor, Assigned: mcc.ricardo, Mentored)

Tracking

({meta})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [good next bug][lang=python])

(Reporter)

Description

4 years ago
Mozbench is a cross-browser benchmark suite. Our initial target is to compare firefox and chrome on windows and linux platforms. For more information, see [1] and [2]. The mozbench repository is located at [3] and instructions on setting it up and running the benchmarks can be found in the readme.md file at that location.

We're interested in being able to run benchmarks on OS X as well. The first step is to make changes (if any are required) to get mozbench/mozbench.py running on OS X. The second step is to modify the linux run script (jenkins/run.sh) to work for both linux and OS X.

[1] https://wiki.mozilla.org/Game_Benchmark_Automation
[2] https://wiki.mozilla.org/Auto-tools/Projects/Mozbench
[3] https://github.com/dminor/mozbench
(Assignee)

Comment 1

4 years ago
I gave a quick look and it shouldn't be difficult to achieve this. The only issue I'm getting is when 'mozinstall' is called. It doesn't mount the '.dmg'.

One way around this is to write a simple script that uses 'hdiutil' to mount the '.dmg'. We would identify that the system is 'mac', override the 'mozinstall', mount the '.dmg' and then 'path' would point to that location.

After that it should be fine. I tried with an already mounted version and it worked.
(Assignee)

Comment 2

4 years ago
Did some initial coding and ran a full test successfully. Still needs some work, as well as the jenkins script.

Diff: https://github.com/mccricardo/mozbench/compare/mozbench-osx
(Assignee)

Comment 3

4 years ago
New diffs for the work: https://github.com/mccricardo/mozbench/compare/mozbench-osx.

Just realized something: if the nightly build will be mounted, the folder where it will reside will be 'FirefoxNightly.app' and not 'Firefox.app/'. Am I correct?
(Reporter)

Comment 4

4 years ago
Sorry I didn't respond earlier, I away last week. It looks like you are on the right track, thanks for looking at this!
Assignee: nobody → mcc.ricardo
Status: NEW → ASSIGNED
(Assignee)

Comment 5

4 years ago
No problem Dan. I had a flu for in the last few days, so I had to take it easy.

As I mentioned above, when mounted, the nightly builds of Firefox will reside in 'FirefoxNightly.app'. If this code will only be used for the nightly builds, we can specify that folder.

If not, we can either rename the folder to always be the same or we can 'detect' the folder name.
(Reporter)

Comment 6

4 years ago
(In reply to mcc.ricardo from comment #5)
> No problem Dan. I had a flu for in the last few days, so I had to take it
> easy.
> 
> As I mentioned above, when mounted, the nightly builds of Firefox will
> reside in 'FirefoxNightly.app'. If this code will only be used for the
> nightly builds, we can specify that folder.
> 
> If not, we can either rename the folder to always be the same or we can
> 'detect' the folder name.

The plan is to only test using nightly builds so 'FirefoxNightly.app' should be fine. We might want to test using Aurora nightly builds in the future, but I'm guessing the naming will be the same there as well.

Thanks!
(Assignee)

Comment 7

4 years ago
No problem Dan. 

I'll update the code later today and test with both.
(Assignee)

Comment 8

4 years ago
Well, nightly builds mount to 'FirefoxNightly.app' and Aurora to 'FirefoxAurora.app'. We can easily identify based on the URL.
(Assignee)

Comment 9

4 years ago
Added some code the deal with different folders.

Diff: https://github.com/mccricardo/mozbench/compare/mozbench-osx?expand=1
(Assignee)

Comment 10

4 years ago
What do you think Dan?
(Reporter)

Comment 11

4 years ago
Thanks! This looks good to me, feel free to submit a pull request.

Sorry I missed your comment the first time around. I usually catch email from mentored bugs, but you can always needinfo me if I'm not responding :)
mozinstall claims to support dmg files, does that not work?
(Assignee)

Comment 13

4 years ago
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #12)
> mozinstall claims to support dmg files, does that not work?

I tried using and it get stuck for a long time and eventually gives an error.
(Assignee)

Comment 14

4 years ago
(In reply to Dan Minor [:dminor] from comment #11)
> Thanks! This looks good to me, feel free to submit a pull request.
> 
> Sorry I missed your comment the first time around. I usually catch email
> from mentored bugs, but you can always needinfo me if I'm not responding :)

No problem Dan. Just submitted the pull request.
(Reporter)

Comment 15

4 years ago
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #12)
> mozinstall claims to support dmg files, does that not work?

I spent some time this morning on this and I think the issue was that we were downloading the installer to a file called 'firefox.exe' - changing the extension to be 'firefox.dmg' on OS X seems to fix things.
(Assignee)

Comment 16

4 years ago
I've taken a look at you comments and I think I got it. I'll tackle it this weekend.
(Reporter)

Comment 17

4 years ago
(In reply to mcc.ricardo from comment #16)
> I've taken a look at you comments and I think I got it. I'll tackle it this
> weekend.

Thanks again for looking at this!
(Assignee)

Comment 18

4 years ago
Just updated the code Dan.
(Reporter)

Comment 19

4 years ago
Thanks again!
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(Reporter)

Comment 20

4 years ago
Hi Ricardo, did the 'Massive' benchmark run successfully for you? It is timing out on our test machine and I'm trying to determine if it is problem with Massive or a test machine problem. Thanks!
Flags: needinfo?(mcc.ricardo)
(Assignee)

Comment 21

4 years ago
Hi Dan,

It did run. I don't have access to my Mac at the moment but I can test it again and double check when I get home later today.

If you send me the error it's giving and can try and look more thoroughly for it.

Ricardo
Flags: needinfo?(mcc.ricardo)
(Reporter)

Comment 22

4 years ago
(In reply to mcc.ricardo from comment #21)
> Hi Dan,
> 
> It did run. I don't have access to my Mac at the moment but I can test it
> again and double check when I get home later today.
> 
> If you send me the error it's giving and can try and look more thoroughly
> for it.
> 
> Ricardo

Thanks! If it was working for you, it is probably something with the way the test machine is configured. Don't worry about following up on this, I'll take a look at it today. Dan
(Assignee)

Comment 23

4 years ago
No problem Dan.

If you find some problem with it, just give me a shout.

Ricardo
(Assignee)

Comment 24

4 years ago
Hey Dan,

Currently I'm connecting to the device using: mozdevice.ADBAndroid(...)

I can either do:

1) mozdevice.ADBAndroid(None) - If None is passed and there is exactly one device attached to the host, that device is used. https://github.com/mozilla/geckodev/blob/master/testing/mozbase/mozdevice/mozdevice/adb.py#L434

2) mozdevice.ADBAndroid(some_device_id) - When a string is passed, it is interpreted as the device serial number. We can get the device id but I believe the None option is more useful since we can, that way, connect any device we want.

Both of them work, but the issue I'm having is that before connection is successful I get some errors like:


ERROR:adb:ADBDevice.__init__: id
Traceback (most recent call last):
  File "/home/rcastro/.envs/mozbench/local/lib/python2.7/site-packages/mozdevice/adb.py", line 485, in __init__
    if self.shell_output("su -c '%s'" % cmd_id).find(uid) != -1:
  File "/home/rcastro/.envs/mozbench/local/lib/python2.7/site-packages/mozdevice/adb.py", line 860, in shell_output
    raise ADBError("%s" % adb_process)
ADBError: args: adb -s 0019d516170f8e wait-for-device shell su -c 'LD_LIBRARY_PATH=/vendor/lib:/system/lib id'; echo rc=$?, exitcode: 127, stdout: /system/bin/sh: su: not found, stderr: 
ERROR:adb:ADBDevice.__init__: id
Traceback (most recent call last):
  File "/home/rcastro/.envs/mozbench/local/lib/python2.7/site-packages/mozdevice/adb.py", line 491, in __init__
    if self.shell_output("su 0 id").find(uid) != -1:
  File "/home/rcastro/.envs/mozbench/local/lib/python2.7/site-packages/mozdevice/adb.py", line 860, in shell_output
    raise ADBError("%s" % adb_process)
ADBError: args: adb -s 0019d516170f8e wait-for-device shell su 0 id; echo rc=$?, exitcode: 127, stdout: /system/bin/sh: su: not found, stderr: 


Like I said, connection is successful but I would like to avoid those errors showing up on the console if possible. 

Ricardo
(Reporter)

Comment 25

4 years ago
Hi Ricardo,

How about adding a command line option to specify the device id, which defaults to None?

This might be useful we end up having a single test machine with both firefox os and Android devices attached. I would have to modify the marionette runner to support this as well.

It looks like those errors come mozdevice attempting to determine whether 'su' is available. When I look at the current version of mozdevice, it looks like those exceptions should be caught and logged. Is it possible you're using an older version of mozdevice?
(Assignee)

Comment 26

4 years ago
The command line option sounds great.

mozdevie, in the requirements file, is 0.40. According to PyPi, current version (as of 2014-10-06) is 0.41. I'll update and check if those go away.

Ricardo
(Assignee)

Comment 27

4 years ago
Just noticed that I pasted the comment in the wring ticket. I'll move it to the correct one for reference.
You need to log in before you can comment on or make changes to this bug.