1466648 - add unity3d benchmark to raptor/android

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

https://beta.unity3d.com/jonas/benchmark2015/

we can figure out which subscores are most important and how to run this properly.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

once we get this added for desktop, we can use this bug then to get the test ported to android.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I am having a hard time getting this working on android.  Running it locally the test just loops forever- there is a massive 2MB+ minified .js file that appears to be the main benchmark.

If there is a way to hack this up or send in parameters to get the test to finish, that would help make this a success.  Even running locally on desktop yields some interesting results as this seems to run forever (15+ minutes in a loop).

:jgilbert- do you know much about unity-webgl and how we can control how often it runs?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

current status:
* hacked on benchmark to not popup message about mobile- unable to verify I can update it, but here are the bits: https://hg.mozilla.org/try/rev/79631253d06ec885aae817734f61bcdf2c6690d1
* addition for unity webgl for android (not working)- https://hg.mozilla.org/try/rev/81786987b8110c094a6b4a529a4532fee7bf6122

my try push that fails:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=200499269&revision=2bbe5e2caeceee52fbf92dab6aa4bdae6e9c7378


:rwood, can you pick this up next week

Comment 4

[Robert Wood [:rwood]]

When I run the existing raptor-unity-webgl with my local Firefox desktop build (OSX) it works fine and finishes.

First I'll see if it runs on my Google Pixel 2 manually (i.e. via a localhost sharing out the benchmark source from our perf-automation github repo) and grab your patches from try (thanks for the WIP).

Comment 5

[Robert Wood [:rwood]]

I served out the benchmark source from [1] locally, and did adb reverse so my google pixel 2 can see it. On the GP2 started the geckoview example app and then browsed to my localhost and selected the benchmark URL.

The unity webgl test starts running in the geckoview example app but then crashes - a pop dialog appears "uncaught exception abort at jsStackTrace@http://localhost:8081/unity-webgl/Data/WebGLBenchmarks.js:1:18533"

[1] https://github.com/mozilla/perf-automation/tree/master/benchmarks/unity-webgl

Comment 6

[Robert Wood [:rwood]]

The next time I tried it, the pop-up didn't appear (seems intermittent), but the benchmark seems to run forever as :jmaher mentioned in comment 2

Comment 7

[Robert Wood [:rwood]]

Attachment: Bug 1466648 - add unity3d benchmark to raptor/android (wip)

Comment 8

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #7)
> Created attachment 9012627 [details]
> Bug 1466648 - add unity3d benchmark to raptor/android (wip)

^ took :jmaher's patches so we can run the benchmark on a local android device via raptor, i.e.:

./mach raptor-test --test raptor-unity-webgl --app=geckoview --binary="org.mozilla.geckoview_example"

Comment 9

[Robert Wood [:rwood]]

With the patch applied in comment 7, running locally does work sometimes - the benchmark finishes and raptor gets the results from the device. It seems the problem where the benchmark runs forever is intermittent.

Comment 10

[Robert Wood [:rwood]]

Attachment: Bug 1466648 - add unity3d benchmark to raptor/android; r?jmaher

Comment 11

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #10)
> Created attachment 9012662 [details]
> Bug 1466648 - add unity3d benchmark to raptor/android; r?jmaher

I have had 10 runs in a row locally on my google pixel 2, that all completed successfully - without any changes to the benchmark source except for what :jmaher previously did to get rid of the 'unsupported on android' message).

:bc, if you see similar with this patch and agree, I think we should land this support now and file a separate issue to look into the intermittent failure (endless loop).

Comment 12

[Bob Clary [:bc] (inactive)]

I agree that with the changes we discussed this is good to land. I haven't seen any issues so for in 5 runs locally on my p2.

Let's do a try run with --rebuild 5 for both desktop and android-hw so we can see how it runs in production especially on the g5s.

Comment 13

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=08091be98d6b2f67ff7cbe09bd0a43031e6f5a5d

Comment 14

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3829642150f146a956e28002e4375e929f776240

Comment 15

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=22aaa8ba9c8a08168f6e67f0a51e8b6b9e54732b

Comment 16

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #15)
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=22aaa8ba9c8a08168f6e67f0a51e8b6b9e54732b

Updating to the new release of the benchmark (new fetch task) in production breaks the benchmark on production desktop. Running locally with the latest perf-auto source works fine.

This is a try push reverting back to the original release / fetch task of unity webgl in production to verify it is actually the fetch task / new perf-auto github release causing the issue and not something on the test machines - since it is an OS error creating a folder that is being reported when upgrading to the new benchmark fetch task release.

Comment 17

[Bob Clary [:bc] (inactive)]

Attachment: D7114-fetches.patch

reverts back to v2 and still fails desktop but *does* pass on android-hw at least on the first run. Doing two to get data on how consistent the values are:

<https://treeherder.mozilla.org/#/jobs?repo=try&tier=1,2,3&author=[e-mail]&group_state=expanded&fromchange=59411ce68bde31266aaeb2035c9fc9a3d05b02bb&tochange=8fc1fc87a2f50838b2fbfc117dbe6625e5b285ba>

I'll look into the desktop stuff later this weekend.

Comment 18

[Bob Clary [:bc] (inactive)]

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1,2,3&group_state=expanded&selectedJob=202385817&revision=9ffc43224c2ec9a2001a7b36639b4cb93b7ebddd

extract_zip: src /home/cltbld/fetches/unity-webgl-9a5e18d2f687.zip dest /home/cltbld/fetches
extract_zip: filename /home/cltbld/fetches/unity-webgl
extract_zip: name unity-webgl/ ends with /
extract_zip: name unity-webgl/ filename /home/cltbld/fetches/unity-webgl mode 0

Command '['ls', '-laR', '/home/cltbld/fetches']' returned non-zero exit status 1
/home/cltbld/fetches:
total 26504
drwxrwxr-x  5 cltbld cltbld     4096 Sep 29 06:58 .
drwxr-xr-x 26 cltbld cltbld     4096 Sep 28 05:42 ..
drwxrwxr-x  2 cltbld cltbld     4096 Aug 14 10:04 d8
-rwxr-xr-x  1 cltbld cltbld    27376 Jan  1  2010 libplc4.so
-rwxr-xr-x  1 cltbld cltbld    22272 Jan  1  2010 libplds4.so
d---------  2 cltbld cltbld     4096 Sep 20 04:00 unity-webgl
-rw-rw-r--  1 cltbld cltbld 27064654 Sep 29 06:59 unity-webgl-9a5e18d2f687.zip
drwxrwxrwx  2 cltbld cltbld     4096 Aug  3 05:16 web-tooling-benchmark

unity-webgl was created with no permissions. Foul up in the creation of the zip?

jmaher, rwood: can we create a new version of this zip file with the correct permissions? Perhaps jmaher created it Windows which caused the problem?

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

I did create this on windows- lets see if rwood can clone the repo and upload a new zip on a OS that doesn't break things.

Comment 20

[Bob Clary [:bc] (inactive)]

I worked around the issue with bundle.getinfo(name).external_attr by checking if name is a directory or not and adjusting the mode if it was zero. Zipfile.ZipInfo.external_attr is OS specific and not well documented. We might want to change mozfile to be more robust.

For what its worth I did 20 runs:

<https://treeherder.mozilla.org/#/jobs?repo=try&tier=1,2,3&group_state=expanded&revision=86180a5783dcd25905171cdd165e017f734d7364>

5% runs on Pixel2 32bit and 5% on Pixel2 64bit encountered ADBTimeoutError.
30% runs on Moto G5 failed due to Geckoview being killed.

Pixel2 32bit and Moto G5 reported reasonable results.

Pixel2 64bit did not report reasonable results. The Logcats may hold a clue. See https://queue.taskcluster.net/v1/task/aDB8P05wTR-bnE9cNdDIKw/runs/0/artifacts/public/test_info/logcat-HT85S1A02559.log for example. Snorp, any ideas?

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

this is excellent news- I think we should get patches in progress as the adbtimeout is unrelated to the tests/tooling and more of an issue installing geckoview.

That is odd that we get a 90% drop in perf on aarch64- hopefully :snorp can find a culprit :)

Comment 22

[Robert Wood [:rwood]]

(In reply to Bob Clary [:bc:] from comment #18)
> 
> unity-webgl was created with no permissions. Foul up in the creation of the
> zip?
> 
> jmaher, rwood: can we create a new version of this zip file with the correct
> permissions? Perhaps jmaher created it Windows which caused the problem?

Thanks :bc! Ok I'll try creating a new release

Comment 23

[Robert Wood [:rwood]]

Made new benchmark release:

https://github.com/mozilla/perf-automation/releases/tag/unity-webgl-v3

Updated my patch to have the fetch task use that new release. Pushing to try to see if it fixes raptor-unity-webgl on desktop.

Comment 24

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2be04595a63b02e7d0c376062e605745b6cab582

Comment 25

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 9012662 [details]
Bug 1466648 - add unity3d benchmark to raptor/android; r?jmaher

Joel Maher ( :jmaher ) (UTC-4) has approved the revision.

Comment 26

[James Willcox (:snorp) ([e-mail]) (he/him)]

There's no asm.js/wasm on aarch64, so this is probably the expected result for now.

Comment 27

[Joel Maher ( :jmaher ) (UTC -8)]

to save headaches, I would say lets not schedule this on p2/arm64- only on g5/arm7 and p2/arm7

Comment 28

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #27)
> to save headaches, I would say lets not schedule this on p2/arm64- only on
> g5/arm7 and p2/arm7

Done, thanks, and next I will integrate some code from :bc

Comment 29

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f714f3aafdcac8de21032536a708dce2db35a1ff

Comment 30

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #29)

> https://treeherder.mozilla.org/#/jobs?repo=try&revision=f714f3aafdcac8de21032536a708dce2db35a1ff

^ patch updated with :bc's additions (thanks!)

Comment 31

[Bob Clary [:bc] (inactive)]

Comment on attachment 9012662 [details]
Bug 1466648 - add unity3d benchmark to raptor/android; r?jmaher

Bob Clary [:bc:] has approved the revision.

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3cc5501b9866
add unity3d benchmark to raptor/android; r=bc,jmaher

Comment 33

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/3cc5501b9866