add unity3d benchmark to raptor/android

RESOLVED FIXED in Firefox 64

Status

enhancement
RESOLVED FIXED
Last year
26 days ago

People

(Reporter: jmaher, Assigned: rwood)

Tracking

(Blocks 1 bug)

unspecified
mozilla64
Unspecified
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox64 fixed)

Details

(Whiteboard: [PI:July][geckoview:p2])

Attachments

(2 attachments, 1 obsolete attachment)

https://beta.unity3d.com/jonas/benchmark2015/

we can figure out which subscores are most important and how to run this properly.
Depends on: 1472800
once we get this added for desktop, we can use this bug then to get the test ported to android.
Whiteboard: [PI:June] → [PI:July]
Assignee

Updated

9 months ago
Depends on: 1491424
Reporter

Comment 2

9 months ago
I am having a hard time getting this working on android.  Running it locally the test just loops forever- there is a massive 2MB+ minified .js file that appears to be the main benchmark.

If there is a way to hack this up or send in parameters to get the test to finish, that would help make this a success.  Even running locally on desktop yields some interesting results as this seems to run forever (15+ minutes in a loop).

:jgilbert- do you know much about unity-webgl and how we can control how often it runs?
Flags: needinfo?(jgilbert)
Reporter

Comment 3

9 months ago
current status:
* hacked on benchmark to not popup message about mobile- unable to verify I can update it, but here are the bits: https://hg.mozilla.org/try/rev/79631253d06ec885aae817734f61bcdf2c6690d1
* addition for unity webgl for android (not working)- https://hg.mozilla.org/try/rev/81786987b8110c094a6b4a529a4532fee7bf6122

my try push that fails:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=200499269&revision=2bbe5e2caeceee52fbf92dab6aa4bdae6e9c7378


:rwood, can you pick this up next week
Flags: needinfo?(rwood)
OS: Unspecified → Android
Whiteboard: [PI:July] → [PI:July][geckoview:p2]
Assignee

Comment 4

9 months ago
When I run the existing raptor-unity-webgl with my local Firefox desktop build (OSX) it works fine and finishes.

First I'll see if it runs on my Google Pixel 2 manually (i.e. via a localhost sharing out the benchmark source from our perf-automation github repo) and grab your patches from try (thanks for the WIP).
Assignee: nobody → rwood
Status: NEW → ASSIGNED
Flags: needinfo?(rwood)
Assignee

Comment 5

9 months ago
I served out the benchmark source from [1] locally, and did adb reverse so my google pixel 2 can see it. On the GP2 started the geckoview example app and then browsed to my localhost and selected the benchmark URL.

The unity webgl test starts running in the geckoview example app but then crashes - a pop dialog appears "uncaught exception abort at jsStackTrace@http://localhost:8081/unity-webgl/Data/WebGLBenchmarks.js:1:18533"

[1] https://github.com/mozilla/perf-automation/tree/master/benchmarks/unity-webgl
Assignee

Comment 6

9 months ago
The next time I tried it, the pop-up didn't appear (seems intermittent), but the benchmark seems to run forever as :jmaher mentioned in comment 2
Assignee

Comment 8

9 months ago
(In reply to Robert Wood [:rwood] from comment #7)
> Created attachment 9012627 [details]
> Bug 1466648 - add unity3d benchmark to raptor/android (wip)

^ took :jmaher's patches so we can run the benchmark on a local android device via raptor, i.e.:

./mach raptor-test --test raptor-unity-webgl --app=geckoview --binary="org.mozilla.geckoview_example"
Assignee

Comment 9

9 months ago
With the patch applied in comment 7, running locally does work sometimes - the benchmark finishes and raptor gets the results from the device. It seems the problem where the benchmark runs forever is intermittent.
Assignee

Comment 11

9 months ago
(In reply to Robert Wood [:rwood] from comment #10)
> Created attachment 9012662 [details]
> Bug 1466648 - add unity3d benchmark to raptor/android; r?jmaher

I have had 10 runs in a row locally on my google pixel 2, that all completed successfully - without any changes to the benchmark source except for what :jmaher previously did to get rid of the 'unsupported on android' message).

:bc, if you see similar with this patch and agree, I think we should land this support now and file a separate issue to look into the intermittent failure (endless loop).
Flags: needinfo?(bob)

Comment 12

9 months ago
I agree that with the changes we discussed this is good to land. I haven't seen any issues so for in 5 runs locally on my p2.

Let's do a try run with --rebuild 5 for both desktop and android-hw so we can see how it runs in production especially on the g5s.
Flags: needinfo?(bob)
Assignee

Comment 16

9 months ago
(In reply to Robert Wood [:rwood] from comment #15)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=22aaa8ba9c8a08168f6e67f0a51e8b6b9e54732b

Updating to the new release of the benchmark (new fetch task) in production breaks the benchmark on production desktop. Running locally with the latest perf-auto source works fine.

This is a try push reverting back to the original release / fetch task of unity webgl in production to verify it is actually the fetch task / new perf-auto github release causing the issue and not something on the test machines - since it is an OS error creating a folder that is being reported when upgrading to the new benchmark fetch task release.

Comment 17

9 months ago
reverts back to v2 and still fails desktop but *does* pass on android-hw at least on the first run. Doing two to get data on how consistent the values are:

<https://treeherder.mozilla.org/#/jobs?repo=try&tier=1,2,3&author=bclary@mozilla.com&group_state=expanded&fromchange=59411ce68bde31266aaeb2035c9fc9a3d05b02bb&tochange=8fc1fc87a2f50838b2fbfc117dbe6625e5b285ba>

I'll look into the desktop stuff later this weekend.

Comment 18

9 months ago
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1,2,3&group_state=expanded&selectedJob=202385817&revision=9ffc43224c2ec9a2001a7b36639b4cb93b7ebddd

extract_zip: src /home/cltbld/fetches/unity-webgl-9a5e18d2f687.zip dest /home/cltbld/fetches
extract_zip: filename /home/cltbld/fetches/unity-webgl
extract_zip: name unity-webgl/ ends with /
extract_zip: name unity-webgl/ filename /home/cltbld/fetches/unity-webgl mode 0

Command '['ls', '-laR', '/home/cltbld/fetches']' returned non-zero exit status 1
/home/cltbld/fetches:
total 26504
drwxrwxr-x  5 cltbld cltbld     4096 Sep 29 06:58 .
drwxr-xr-x 26 cltbld cltbld     4096 Sep 28 05:42 ..
drwxrwxr-x  2 cltbld cltbld     4096 Aug 14 10:04 d8
-rwxr-xr-x  1 cltbld cltbld    27376 Jan  1  2010 libplc4.so
-rwxr-xr-x  1 cltbld cltbld    22272 Jan  1  2010 libplds4.so
d---------  2 cltbld cltbld     4096 Sep 20 04:00 unity-webgl
-rw-rw-r--  1 cltbld cltbld 27064654 Sep 29 06:59 unity-webgl-9a5e18d2f687.zip
drwxrwxrwx  2 cltbld cltbld     4096 Aug  3 05:16 web-tooling-benchmark

unity-webgl was created with no permissions. Foul up in the creation of the zip?

jmaher, rwood: can we create a new version of this zip file with the correct permissions? Perhaps jmaher created it Windows which caused the problem?
Flags: needinfo?(rwood)
Flags: needinfo?(jmaher)
Reporter

Comment 19

9 months ago
I did create this on windows- lets see if rwood can clone the repo and upload a new zip on a OS that doesn't break things.
Flags: needinfo?(jmaher)

Comment 20

9 months ago
I worked around the issue with bundle.getinfo(name).external_attr by checking if name is a directory or not and adjusting the mode if it was zero. Zipfile.ZipInfo.external_attr is OS specific and not well documented. We might want to change mozfile to be more robust.

For what its worth I did 20 runs:

<https://treeherder.mozilla.org/#/jobs?repo=try&tier=1,2,3&group_state=expanded&revision=86180a5783dcd25905171cdd165e017f734d7364>

5% runs on Pixel2 32bit and 5% on Pixel2 64bit encountered ADBTimeoutError.
30% runs on Moto G5 failed due to Geckoview being killed.

Pixel2 32bit and Moto G5 reported reasonable results.

Pixel2 64bit did not report reasonable results. The Logcats may hold a clue. See https://queue.taskcluster.net/v1/task/aDB8P05wTR-bnE9cNdDIKw/runs/0/artifacts/public/test_info/logcat-HT85S1A02559.log for example. Snorp, any ideas?
Flags: needinfo?(snorp)
Reporter

Comment 21

9 months ago
this is excellent news- I think we should get patches in progress as the adbtimeout is unrelated to the tests/tooling and more of an issue installing geckoview.

That is odd that we get a 90% drop in perf on aarch64- hopefully :snorp can find a culprit :)
Flags: needinfo?(jgilbert)
Assignee

Comment 22

9 months ago
(In reply to Bob Clary [:bc:] from comment #18)
> 
> unity-webgl was created with no permissions. Foul up in the creation of the
> zip?
> 
> jmaher, rwood: can we create a new version of this zip file with the correct
> permissions? Perhaps jmaher created it Windows which caused the problem?

Thanks :bc! Ok I'll try creating a new release
Flags: needinfo?(rwood)
Assignee

Comment 23

9 months ago
Made new benchmark release:

https://github.com/mozilla/perf-automation/releases/tag/unity-webgl-v3

Updated my patch to have the fetch task use that new release. Pushing to try to see if it fixes raptor-unity-webgl on desktop.
Reporter

Comment 25

9 months ago
Comment on attachment 9012662 [details]
Bug 1466648 - add unity3d benchmark to raptor/android; r?jmaher

Joel Maher ( :jmaher ) (UTC-4) has approved the revision.
Attachment #9012662 - Flags: review+
There's no asm.js/wasm on aarch64, so this is probably the expected result for now.
Flags: needinfo?(snorp)
Reporter

Comment 27

9 months ago
to save headaches, I would say lets not schedule this on p2/arm64- only on g5/arm7 and p2/arm7
Assignee

Comment 28

9 months ago
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #27)
> to save headaches, I would say lets not schedule this on p2/arm64- only on
> g5/arm7 and p2/arm7

Done, thanks, and next I will integrate some code from :bc
Assignee

Comment 30

9 months ago
(In reply to Robert Wood [:rwood] from comment #29)

> https://treeherder.mozilla.org/#/jobs?repo=try&revision=f714f3aafdcac8de21032536a708dce2db35a1ff

^ patch updated with :bc's additions (thanks!)

Comment 31

9 months ago
Comment on attachment 9012662 [details]
Bug 1466648 - add unity3d benchmark to raptor/android; r?jmaher

Bob Clary [:bc:] has approved the revision.
Attachment #9012662 - Flags: review+

Comment 32

9 months ago
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3cc5501b9866
add unity3d benchmark to raptor/android; r=bc,jmaher

Comment 33

9 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/3cc5501b9866
Status: ASSIGNED → RESOLVED
Closed: 9 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Assignee

Updated

3 months ago
Blocks: 1506865
Attachment #9012627 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.