Closed Bug 1466648 Opened 2 years ago Closed 2 years ago
add unity3d benchmark to raptor/android
https://beta.unity3d.com/jonas/benchmark2015/ we can figure out which subscores are most important and how to run this properly.
once we get this added for desktop, we can use this bug then to get the test ported to android.
Whiteboard: [PI:June] → [PI:July]
I am having a hard time getting this working on android. Running it locally the test just loops forever- there is a massive 2MB+ minified .js file that appears to be the main benchmark. If there is a way to hack this up or send in parameters to get the test to finish, that would help make this a success. Even running locally on desktop yields some interesting results as this seems to run forever (15+ minutes in a loop). :jgilbert- do you know much about unity-webgl and how we can control how often it runs?
current status: * hacked on benchmark to not popup message about mobile- unable to verify I can update it, but here are the bits: https://hg.mozilla.org/try/rev/79631253d06ec885aae817734f61bcdf2c6690d1 * addition for unity webgl for android (not working)- https://hg.mozilla.org/try/rev/81786987b8110c094a6b4a529a4532fee7bf6122 my try push that fails: https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=200499269&revision=2bbe5e2caeceee52fbf92dab6aa4bdae6e9c7378 :rwood, can you pick this up next week
OS: Unspecified → Android
Whiteboard: [PI:July] → [PI:July][geckoview:p2]
When I run the existing raptor-unity-webgl with my local Firefox desktop build (OSX) it works fine and finishes. First I'll see if it runs on my Google Pixel 2 manually (i.e. via a localhost sharing out the benchmark source from our perf-automation github repo) and grab your patches from try (thanks for the WIP).
Assignee: nobody → rwood
Status: NEW → ASSIGNED
I served out the benchmark source from  locally, and did adb reverse so my google pixel 2 can see it. On the GP2 started the geckoview example app and then browsed to my localhost and selected the benchmark URL. The unity webgl test starts running in the geckoview example app but then crashes - a pop dialog appears "uncaught exception abort at jsStackTrace@http://localhost:8081/unity-webgl/Data/WebGLBenchmarks.js:1:18533"  https://github.com/mozilla/perf-automation/tree/master/benchmarks/unity-webgl
The next time I tried it, the pop-up didn't appear (seems intermittent), but the benchmark seems to run forever as :jmaher mentioned in comment 2
(In reply to Robert Wood [:rwood] from comment #7) > Created attachment 9012627 [details] > Bug 1466648 - add unity3d benchmark to raptor/android (wip) ^ took :jmaher's patches so we can run the benchmark on a local android device via raptor, i.e.: ./mach raptor-test --test raptor-unity-webgl --app=geckoview --binary="org.mozilla.geckoview_example"
With the patch applied in comment 7, running locally does work sometimes - the benchmark finishes and raptor gets the results from the device. It seems the problem where the benchmark runs forever is intermittent.
(In reply to Robert Wood [:rwood] from comment #10) > Created attachment 9012662 [details] > Bug 1466648 - add unity3d benchmark to raptor/android; r?jmaher I have had 10 runs in a row locally on my google pixel 2, that all completed successfully - without any changes to the benchmark source except for what :jmaher previously did to get rid of the 'unsupported on android' message). :bc, if you see similar with this patch and agree, I think we should land this support now and file a separate issue to look into the intermittent failure (endless loop).
I agree that with the changes we discussed this is good to land. I haven't seen any issues so for in 5 runs locally on my p2. Let's do a try run with --rebuild 5 for both desktop and android-hw so we can see how it runs in production especially on the g5s.
(In reply to Robert Wood [:rwood] from comment #15) > https://treeherder.mozilla.org/#/ > jobs?repo=try&revision=22aaa8ba9c8a08168f6e67f0a51e8b6b9e54732b Updating to the new release of the benchmark (new fetch task) in production breaks the benchmark on production desktop. Running locally with the latest perf-auto source works fine. This is a try push reverting back to the original release / fetch task of unity webgl in production to verify it is actually the fetch task / new perf-auto github release causing the issue and not something on the test machines - since it is an OS error creating a folder that is being reported when upgrading to the new benchmark fetch task release.
reverts back to v2 and still fails desktop but *does* pass on android-hw at least on the first run. Doing two to get data on how consistent the values are: <https://treeherder.mozilla.org/#/jobs?repo=try&tier=1,2,firstname.lastname@example.org&group_state=expanded&fromchange=59411ce68bde31266aaeb2035c9fc9a3d05b02bb&tochange=8fc1fc87a2f50838b2fbfc117dbe6625e5b285ba> I'll look into the desktop stuff later this weekend.
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1,2,3&group_state=expanded&selectedJob=202385817&revision=9ffc43224c2ec9a2001a7b36639b4cb93b7ebddd extract_zip: src /home/cltbld/fetches/unity-webgl-9a5e18d2f687.zip dest /home/cltbld/fetches extract_zip: filename /home/cltbld/fetches/unity-webgl extract_zip: name unity-webgl/ ends with / extract_zip: name unity-webgl/ filename /home/cltbld/fetches/unity-webgl mode 0 Command '['ls', '-laR', '/home/cltbld/fetches']' returned non-zero exit status 1 /home/cltbld/fetches: total 26504 drwxrwxr-x 5 cltbld cltbld 4096 Sep 29 06:58 . drwxr-xr-x 26 cltbld cltbld 4096 Sep 28 05:42 .. drwxrwxr-x 2 cltbld cltbld 4096 Aug 14 10:04 d8 -rwxr-xr-x 1 cltbld cltbld 27376 Jan 1 2010 libplc4.so -rwxr-xr-x 1 cltbld cltbld 22272 Jan 1 2010 libplds4.so d--------- 2 cltbld cltbld 4096 Sep 20 04:00 unity-webgl -rw-rw-r-- 1 cltbld cltbld 27064654 Sep 29 06:59 unity-webgl-9a5e18d2f687.zip drwxrwxrwx 2 cltbld cltbld 4096 Aug 3 05:16 web-tooling-benchmark unity-webgl was created with no permissions. Foul up in the creation of the zip? jmaher, rwood: can we create a new version of this zip file with the correct permissions? Perhaps jmaher created it Windows which caused the problem?
I did create this on windows- lets see if rwood can clone the repo and upload a new zip on a OS that doesn't break things.
I worked around the issue with bundle.getinfo(name).external_attr by checking if name is a directory or not and adjusting the mode if it was zero. Zipfile.ZipInfo.external_attr is OS specific and not well documented. We might want to change mozfile to be more robust. For what its worth I did 20 runs: <https://treeherder.mozilla.org/#/jobs?repo=try&tier=1,2,3&group_state=expanded&revision=86180a5783dcd25905171cdd165e017f734d7364> 5% runs on Pixel2 32bit and 5% on Pixel2 64bit encountered ADBTimeoutError. 30% runs on Moto G5 failed due to Geckoview being killed. Pixel2 32bit and Moto G5 reported reasonable results. Pixel2 64bit did not report reasonable results. The Logcats may hold a clue. See https://queue.taskcluster.net/v1/task/aDB8P05wTR-bnE9cNdDIKw/runs/0/artifacts/public/test_info/logcat-HT85S1A02559.log for example. Snorp, any ideas?
this is excellent news- I think we should get patches in progress as the adbtimeout is unrelated to the tests/tooling and more of an issue installing geckoview. That is odd that we get a 90% drop in perf on aarch64- hopefully :snorp can find a culprit :)
(In reply to Bob Clary [:bc:] from comment #18) > > unity-webgl was created with no permissions. Foul up in the creation of the > zip? > > jmaher, rwood: can we create a new version of this zip file with the correct > permissions? Perhaps jmaher created it Windows which caused the problem? Thanks :bc! Ok I'll try creating a new release
Made new benchmark release: https://github.com/mozilla/perf-automation/releases/tag/unity-webgl-v3 Updated my patch to have the fetch task use that new release. Pushing to try to see if it fixes raptor-unity-webgl on desktop.
Comment on attachment 9012662 [details] Bug 1466648 - add unity3d benchmark to raptor/android; r?jmaher Joel Maher ( :jmaher ) (UTC-4) has approved the revision.
There's no asm.js/wasm on aarch64, so this is probably the expected result for now.
to save headaches, I would say lets not schedule this on p2/arm64- only on g5/arm7 and p2/arm7
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #27) > to save headaches, I would say lets not schedule this on p2/arm64- only on > g5/arm7 and p2/arm7 Done, thanks, and next I will integrate some code from :bc
(In reply to Robert Wood [:rwood] from comment #29) > https://treeherder.mozilla.org/#/jobs?repo=try&revision=f714f3aafdcac8de21032536a708dce2db35a1ff ^ patch updated with :bc's additions (thanks!)
Comment on attachment 9012662 [details] Bug 1466648 - add unity3d benchmark to raptor/android; r?jmaher Bob Clary [:bc:] has approved the revision.
Pushed by email@example.com: https://hg.mozilla.org/integration/autoland/rev/3cc5501b9866 add unity3d benchmark to raptor/android; r=bc,jmaher
You need to log in before you can comment on or make changes to this bug.