"Unity Benchmarks has different tests stressing 3D physics, 2D physics, particles, navigation, animation & skinning, object instantiation, scripting, draw calls and GPU pixel throughput." http://blogs.unity3d.com/2014/10/07/benchmarking-unity-performance-in-webgl/ http://beta.unity3d.com/jonas/WebGLBenchmark/
Ricardo, another mozbench bug if you're interested :) This should only require two modifications, one to launch the benchmark automatically after the page loads, and another to post the results (suitably formatted) back to the test framework on completion.
Hi Dan, I'll be glad to pick this up. Does this have a high priority? If not, I'll assign it to me and add it to my queue :)
Thanks for your help, Ricardo! Adding Unity's WebGLBenchmark is not an urgent priority, though it is the next benchmark we'd like to automate.
Hey Chris, Excellent! I'll mark it as ASSIGNED and I'll be picking it up shortly. Dan, thank you for giving me a heads up on this one.
Hey Chris, Quick question. Will have to launch the benchmark as an resource, using this URL: http://beta.unity3d.com/jonas/WebGLBenchmark/, right?
I'm not familiar with the implementation details of our automation framework or Unity's WebGLBenchmark, but we should archive a snapshot of the WebGLBenchmark files so we don't need to worry about network hiccups or Unity revving, moving, or taking down the WebGLBenchmark.
(In reply to Chris Peterson (needinfo? :cpeterson) from comment #6) > I'm not familiar with the implementation details of our automation framework > or Unity's WebGLBenchmark, but we should archive a snapshot of the > WebGLBenchmark files so we don't need to worry about network hiccups or > Unity revving, moving, or taking down the WebGLBenchmark. As Chris said, I've been adding a local copy of the benchmarks under the mozbench/static directory.
Thought so too. In fact, that's well documented in the README. I just didn't easily find any link to Unity's WebGLBenchmark implementation, so I just though of asking :)
From the page, on the comments we can see: Schmosef: "Will this benchmark project be made open source?" -> Jonas Echterhoff: "I’m planning to release it once Unity 5.0 is available." Unity 5.0 is not yet available. Current version is 4.5.5: http://unity3d.com/unity/download/archive. Unity5 is in pre-order: http://unity3d.com/5 Despite that, I questioned it's availability.
You can just grab all the HTML, JS, and CSS files yourself, either manually or via wget -r; they don't appear to be minified, at least from my brief inspection. That's how I grabbed a lot of benchmarks for a previous similar project.
Cool :) I'll look into that!
The Data folder where the JS and other files lived was forbidden, so I had to get the files one by one. Anyway, we now have a copy of the benchmark: https://github.com/mccricardo/mozbench/tree/add-Unity-WebGLBenchmark/mozbench/static/Unity-WebGLBenchmark I believe the file responsible for the benchmark is WebGLBenchmark.js: https://raw.githubusercontent.com/mccricardo/mozbench/add-Unity-WebGLBenchmark/mozbench/static/Unity-WebGLBenchmark/Data/WebGLBenchmark.js
I'm not sure, but I thin the benchmark results come from this binary file: https://github.com/mccricardo/mozbench/blob/add-Unity-WebGLBenchmark/mozbench/static/Unity-WebGLBenchmark/Data/WebGLBenchmark.data
(In reply to Ricardo Castro from comment #13) > I'm not sure, but I thin the benchmark results come from this binary file: > https://github.com/mccricardo/mozbench/blob/add-Unity-WebGLBenchmark/ > mozbench/static/Unity-WebGLBenchmark/Data/WebGLBenchmark.data I'm guessing that file has shaders and 3d models, etc. I'll have a look at that horrendous minified WebGLBenchmark.js file and see if it looks hackable. If this gets too ugly, we might have to wait for the open source version of the benchmark.
I'm also having a look at the file once I get home from work. I noticed that in the console we get a text with the result. For example: > "Unloading 0 Unused Serialized files (Serialized files now loaded: 0)" > "UnloadTime: 0.496004 ms" > > "Unloading 12 unused Assets to reduce memory usage. Loaded Objects now: 289." > "Total: 0.869263 ms (FindLiveObjects: 0.011902 ms CreateObjectMapping: 0.010814 ms MarkObjects: 0.241348 > ms DeleteObjects: 0.602821 ms)" > " > "Clear > " > "Facebook_PostScreenshot 0" > "Mandelbrot GPU: 0" > "Overall: 0" I couldn't any evidence of where that text came from. But, by chance I found some text in the binary file. For example: the string "Overall Score" that appears in the final screen of the benchmark.
I noticed there are 'print' and 'printErr' functions defined in index.html. Maybe these are being used to log to the console? If so, we could redefine these to scrape results from the printed strings.
(In reply to Dan Minor [:dminor] from comment #16) > I noticed there are 'print' and 'printErr' functions defined in index.html. > Maybe these are being used to log to the console? If so, we could redefine > these to scrape results from the printed strings. Yeah. 'print' is used to log that text I mentioned above. I will do some testing on the output of that text (which at first sight seems to be straightforward) and maybe we can scrape it, as you suggested :)
The output test is something like: > Some uninteresting text > Some uninteresting text > ... > Test Name: SomeValue > Test Name: SomeValue > ... > Overall: SomeValue This text is logged from the 'print' function in index.html. We can alter this function so that each time it runs we check for the presence of the 'Overall' string and if it's there we get the individual test results and overall result.
For reference an example of the "interesting" text, after all tests ran: > "Mandelbrot GPU: 100" > "Mandelbrot Script: 74380" > "Instantiate & Destroy: 28355" > "CryptoHash Script: 143576" > "Animation & Skinning: 20" > "Asteroid Field: 105" > "Particles: 21880" > "Physics Meshes: 156" > "Physics Cubes: 192 > "Physics Spheres: 120" > "2D Physics Spheres: 769" > "2D Physics Boxes: 398" > "AI Agents: 490" > "Overall: 19992" Also, I'm not 100% sure, but I think that text is not printed all at once. I think one call for each line is done. So we can check for the individual strings for each test.
Ricardo, looks promising! We can use the 'Overall' string as a trigger to post results back to mozbench.
Hey Dan, I've made some progress on this. You can take a look here: https://github.com/mccricardo/mozbench/tree/add-Unity-WebGLBenchmark I still need to do some testing and figure out to launch the test automatically. But for now, it looks promising.
I can get the results back without a problem :) Problem now is starting the test automatically. I don't see another way other than 'clicking' on a certain part of the page, referring to the 'Restart' button. Problem is that button is not an element. I though about drawing a rectangle over i and clicking it. Problem is that as soon as I get the canvas context I mess up the benchmark.
(In reply to Ricardo Castro from comment #22) > I can get the results back without a problem :) > > Problem now is starting the test automatically. I don't see another way > other than 'clicking' on a certain part of the page, referring to the > 'Restart' button. Problem is that button is not an element. > > I though about drawing a rectangle over i and clicking it. Problem is that > as soon as I get the canvas context I mess up the benchmark. I was requesting the wrong context. That was the problem :) I don't see any other way, but I might be wrong. I know (close to) nothing on WebGL, so I might be heading in the wrong direction and there might be a better approach. What do you think Dan?
It would be better if we can figure out what code is executed when the 'Restart' button is pressed and then try to call the same code on an onload event. Another thing you can try is to create a fake mouse event like in the example here .  https://developer.mozilla.org/en-US/docs/Talk:DOM/event.initMouseEvent
By the way, thanks for the great work so far!
Yeah. That's what I was my first option, but I still have nothing on that. I'll dig a little bit deeper and try to figure it out. I already tried the fake mouse event with *no* success.
Ahhhhhh! There seems to be a an event called 'touchstart', associated '#canvas', with that I was able to trigger. It gets into 'WebGLBenchmark.js' but now complains that 'event.touches' is undefined. Not sure what that means. Still, there's hope :)
Getting there :)
Manged to trigger 'touchstart' and 'touchend' but with no success.
Hi Dan, I'm getting nowhere. Current status is that I can get benchmark results and send them back to mozbench. The only piece missing is how to auto-start the benchmark.
Would it be helpful to connect you to the person who made the benchmark?
I think so. We're pretty much there. We just need a way to start the benchmark automatically.
Ok, will send an email.
Thank you so much. That would be perfect :)
(In reply to Ricardo Castro from comment #34) > Thank you so much. That would be perfect :) Ricardo, thanks for your hard work on this! Hopefully things move along once you've had a chance to talk to the benchmark author, but if you're getting frustrated, please let me know and I can take this one over, we have lots of stuff for you to work on :)
So, Alon asked me to chime in on this. First of all, if you need the Unity source project to this benchmark, you can grab it from: http://files.unity3d.com/jonas/WebGLBenchmark.zip (I have not formally "released" this yet, but that is mostly a matter of doing a cleanup pass on the code, I don't mind sharing it as is). If there is any specific changes you'd like to see to the project to be able to automate it, let me know which, and I can help implementing them. I think it would be great to have this running on Mozilla's automation (AFAIK, the WebKit team already does the same).
Hey Dan, Not at all. I was just in need of some help. Now that Jonas came to the rescue I'm sure we can move things along quicker :)
Hi Jonas, First of tall, thank you very much for help and availability. For this particular project, we're pretty much there. The only thing we're missing is a way to automatically start the benchmark. I don't know if there's already an easy way for achieve that or if you can point me on the way to do it. Basically, mozbench will launch the benchmark and as soon as it is ready, we want it to start. What do you think? And again, thank you :)
That is easy to do, but requires changing the project in Unity. I can set you up with a Unity 5.0 license code if you want to try it, or make a build myself later when I find time, which works better for you?
Excellent news! Either of them is good for me. What is or suggestion?
I'll email you a U5 serial number, and then, if you need any help, just ping me.
Perfect! I'll need some guidance on how to do it but I'll give you a shout then.
I won't be in London this weekend in order to complete the missing bits on this. With that said, Jonas as been more than helpful, pointed me in the right direction and offered to help if I got into any "trouble". I believe this should be ready for review early next week.
Previous link points to a build of the project. Link for he source code: http://files.unity3d.com/jonas/WebGLBenchmarksSource.zip
And finally some progress! The benchmark is now able to automatically start. I will do some more testing. I couldn't have done it without Jonas help. A big "Thank You" to him :)
(In reply to Ricardo Castro from comment #45) > And finally some progress! The benchmark is now able to automatically start. > I will do some more testing. > > I couldn't have done it without Jonas help. A big "Thank You" to him :) Great news!
Created attachment 8530525 [details] [review] Pull request.
excellent, glad to see this being added!
(In reply to Ricardo Castro from comment #47) > Created attachment 8530525 [details] [review] > Pull request. Hi Ricardo, sorry for the delay in looking at this. I'm travelling today so I might not have a chance to review until tomorrow. Thanks!
Hi Dan, No problem. There's no hurry :)
I'm hitting a problem launching this on linux, maybe a timing problem? Here is what I see in the console: "run() called, but dependencies remain, so not running" index.html:65:10 "pre-main prep time: 631 ms" index.html:65:10 "PlayerInitEngineNoGraphics settings: Could..... not preload global game manager #0 i=0" index.html:61:12 "Failed to initialize player" index.html:61:12
Not sure if that "run() called" message is relevant or not, but what it means is that a data file is being loaded asynchronously, and is not ready yet, so it is waiting on that to run main() etc. Before calling any compiled code, an app needs to wait for that to complete. If the setup here does not call any compiled code directly though, then this is not the issue and something else is going wrong.
My bad. I didn't test it on Linux, only OS X. On OS X I also get that "run() called" message, but the benchmark runs successfully. I'll look into it.
The run() called message comes from emscripten and is not something to worry about. However, "PlayerInitEngineNoGraphics settings: Could..... not preload global game manager #0 i=0" does indicate a failure, which sounds like it might be related to loading the data files somehow.
I already found that the instruction: "tar xvzf WebGLBenchmark.data.tar.gz WebGLBenchmark.data" is wrong. It's missging an "s" so it should be: "tar xvzf WebGLBenchmark.data.tar.gz WebGLBenchmarks.data"
That doesn't fix it though. I probably missed something, I didn't commit it or the compression somehow messed up with the files (which I really don't believe). When I get my back home I'll compare OS X and Linux and check what's different.
Comment on attachment 8530525 [details] [review] Pull request. Please flag me for review again once you've had a chance to look at this. Thanks!
Sure thing. I was out of town this weekend. I believe I got it working. But I want to test it properly before sending for review. I should be able to do it later today.
Created attachment 8533683 [details] [review] Pull request. Added new benchmark code.
Created attachment 8534023 [details] [review] Pull request. Updated following review guidelines.
Comment on attachment 8534023 [details] [review] Pull request. Thanks for you hard work and persistence in getting this working :)
Glad it's working. Going to file and proceed with the bug for Jenkins and get this fully integrated.