Add Unity's WebGLBenchmark to mozbench

RESOLVED FIXED

Status

Testing
General
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: cpeterson, Assigned: Ricardo Castro, Mentored)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 2 obsolete attachments)

41 bytes, text/x-github-pull-request
dminor
: review+
Details | Review | Splinter Review
(Reporter)

Description

3 years ago
"Unity Benchmarks has different tests stressing 3D physics, 2D physics, particles, navigation, animation & skinning, object instantiation, scripting, draw calls and GPU pixel throughput."

http://blogs.unity3d.com/2014/10/07/benchmarking-unity-performance-in-webgl/

http://beta.unity3d.com/jonas/WebGLBenchmark/

Comment 1

3 years ago
Ricardo, another mozbench bug if you're interested :)

This should only require two modifications, one to launch the benchmark automatically after the page loads, and another to post the results (suitably formatted) back to the test framework on completion.
Mentor: dminor@mozilla.com
(Assignee)

Comment 2

3 years ago
Hi Dan,

I'll be glad to pick this up. Does this have a high priority? If not, I'll assign it to me and add it to my queue :)
(Reporter)

Comment 3

3 years ago
Thanks for your help, Ricardo! Adding Unity's WebGLBenchmark is not an urgent priority, though it is the next benchmark we'd like to automate.
Assignee: nobody → mcc.ricardo
(Assignee)

Comment 4

3 years ago
Hey Chris,

Excellent! I'll mark it as ASSIGNED and I'll be picking it up shortly. 

Dan, thank you for giving me a heads up on this one.
Status: NEW → ASSIGNED
(Assignee)

Comment 5

3 years ago
Hey Chris,

Quick question. Will have to launch the benchmark as an resource, using this URL: http://beta.unity3d.com/jonas/WebGLBenchmark/, right?
Flags: needinfo?(cpeterson)
(Reporter)

Comment 6

3 years ago
I'm not familiar with the implementation details of our automation framework or Unity's WebGLBenchmark, but we should archive a snapshot of the WebGLBenchmark files so we don't need to worry about network hiccups or Unity revving, moving, or taking down the WebGLBenchmark.
Flags: needinfo?(cpeterson)

Comment 7

3 years ago
(In reply to Chris Peterson (needinfo? :cpeterson) from comment #6)
> I'm not familiar with the implementation details of our automation framework
> or Unity's WebGLBenchmark, but we should archive a snapshot of the
> WebGLBenchmark files so we don't need to worry about network hiccups or
> Unity revving, moving, or taking down the WebGLBenchmark.

As Chris said, I've been adding a local copy of the benchmarks under the mozbench/static directory.
(Assignee)

Comment 8

3 years ago
Thought so too. In fact, that's well documented in the README. 

I just didn't easily find any link to Unity's WebGLBenchmark implementation, so I just though of asking :)
(Assignee)

Comment 9

3 years ago
From the page, on the comments we can see:

Schmosef: "Will this benchmark project be made open source?" -> Jonas Echterhoff: "I’m planning to release it once Unity 5.0 is available."

Unity 5.0 is not yet available. Current version is 4.5.5: http://unity3d.com/unity/download/archive. Unity5 is in pre-order: http://unity3d.com/5

Despite that, I questioned it's availability.
You can just grab all the HTML, JS, and CSS files yourself, either manually or via wget -r; they don't appear to be minified, at least from my brief inspection.  That's how I grabbed a lot of benchmarks for a previous similar project.
(Assignee)

Comment 11

3 years ago
Cool :) I'll look into that!
(Assignee)

Comment 12

3 years ago
The Data folder where the JS and other files lived was forbidden, so I had to get the files one by one. Anyway, we now have a copy of the benchmark: https://github.com/mccricardo/mozbench/tree/add-Unity-WebGLBenchmark/mozbench/static/Unity-WebGLBenchmark

I believe the file responsible for the benchmark is WebGLBenchmark.js: https://raw.githubusercontent.com/mccricardo/mozbench/add-Unity-WebGLBenchmark/mozbench/static/Unity-WebGLBenchmark/Data/WebGLBenchmark.js
(Assignee)

Comment 13

3 years ago
I'm not sure, but I thin the benchmark results come from this binary file: https://github.com/mccricardo/mozbench/blob/add-Unity-WebGLBenchmark/mozbench/static/Unity-WebGLBenchmark/Data/WebGLBenchmark.data

Comment 14

3 years ago
(In reply to Ricardo Castro from comment #13)
> I'm not sure, but I thin the benchmark results come from this binary file:
> https://github.com/mccricardo/mozbench/blob/add-Unity-WebGLBenchmark/
> mozbench/static/Unity-WebGLBenchmark/Data/WebGLBenchmark.data

I'm guessing that file has shaders and 3d models, etc. I'll have a look at that horrendous minified WebGLBenchmark.js file and see if it looks hackable.

If this gets too ugly, we might have to wait for the open source version of the benchmark.
(Assignee)

Comment 15

3 years ago
I'm also having a look at the file once I get home from work.

I noticed that in the console we get a text with the result. For example:

> "Unloading 0 Unused Serialized files (Serialized files now loaded: 0)"
> "UnloadTime: 0.496004 ms"
> 
> "Unloading 12 unused Assets to reduce memory usage. Loaded Objects now: 289."
> "Total: 0.869263 ms (FindLiveObjects: 0.011902 ms CreateObjectMapping: 0.010814 ms MarkObjects: 0.241348 > ms  DeleteObjects: 0.602821 ms)"
> "
> "Clear
> "
> "Facebook_PostScreenshot 0"
> "Mandelbrot GPU: 0"
> "Overall: 0"

I couldn't any evidence of where that text came from. But, by chance I found some text in the binary file. For example: the string "Overall Score" that appears in the final screen of the benchmark.

Comment 16

3 years ago
I noticed there are 'print' and 'printErr' functions defined in index.html. Maybe these are being used to log to the console? If so, we could redefine these to scrape results from the printed strings.
(Assignee)

Comment 17

3 years ago
(In reply to Dan Minor [:dminor] from comment #16)
> I noticed there are 'print' and 'printErr' functions defined in index.html.
> Maybe these are being used to log to the console? If so, we could redefine
> these to scrape results from the printed strings.

Yeah. 'print' is used to log that text I mentioned above. I will do some testing on the output of that text (which at first sight seems to be straightforward) and maybe we can scrape it, as you suggested :)
(Assignee)

Comment 18

3 years ago
The output test is something like:

> Some uninteresting text 
> Some uninteresting text 
> ...
> Test Name: SomeValue
> Test Name: SomeValue
> ...
> Overall: SomeValue

This text is logged from the 'print' function in index.html. We can alter this function so that each time it runs we check for the presence of the 'Overall'  string and if it's there we get the individual test results and overall result.
(Assignee)

Comment 19

3 years ago
For reference an example of the "interesting" text, after all tests ran:

> "Mandelbrot GPU: 100"
> "Mandelbrot Script: 74380"
> "Instantiate & Destroy: 28355"
> "CryptoHash Script: 143576"
> "Animation & Skinning: 20"
> "Asteroid Field: 105"
> "Particles: 21880"
> "Physics Meshes: 156"
> "Physics Cubes: 192
> "Physics Spheres: 120"
> "2D Physics Spheres: 769"
> "2D Physics Boxes: 398"
> "AI Agents: 490"
> "Overall: 19992"

Also, I'm not 100% sure, but I think that text is not printed all at once. I think one call for each line is done. So we can check for the individual strings for each test.

Comment 20

3 years ago
Ricardo, looks promising! We can use the 'Overall' string as a trigger to post results back to mozbench.
(Assignee)

Comment 21

3 years ago
Hey Dan,

I've made some progress on this. You can take a look here: https://github.com/mccricardo/mozbench/tree/add-Unity-WebGLBenchmark

I still need to do some testing and figure out to launch the test automatically. But for now, it looks promising.
(Assignee)

Comment 22

3 years ago
I can get the results back without a problem :)

Problem now is starting the test automatically. I don't see another way other than 'clicking' on a certain part of the page, referring to the 'Restart' button. Problem is that button is not an element.

I though about drawing a rectangle over i and clicking it. Problem is that as soon as I get the canvas context I mess up the benchmark.
(Assignee)

Comment 23

3 years ago
(In reply to Ricardo Castro from comment #22)
> I can get the results back without a problem :)
> 
> Problem now is starting the test automatically. I don't see another way
> other than 'clicking' on a certain part of the page, referring to the
> 'Restart' button. Problem is that button is not an element.
> 
> I though about drawing a rectangle over i and clicking it. Problem is that
> as soon as I get the canvas context I mess up the benchmark.

I was requesting the wrong context. That was the problem :)

I don't see any other way, but I might be wrong. I know (close to) nothing on WebGL, so I might be heading in the wrong direction and there might be a better approach.

What do you think Dan?

Comment 24

3 years ago
It would be better if we can figure out what code is executed when the 'Restart' button is pressed and then try to call the same code on an onload event.

Another thing you can try is to create a fake mouse event like in the example here [1].

[1] https://developer.mozilla.org/en-US/docs/Talk:DOM/event.initMouseEvent

Comment 25

3 years ago
By the way, thanks for the great work so far!
(Assignee)

Comment 26

3 years ago
Yeah. That's what I was my first option, but I still have nothing on that. I'll dig a little bit deeper and try to figure it out.

I already tried the fake mouse event with *no* success.
(Assignee)

Comment 27

3 years ago
Ahhhhhh! There seems to be a an event called 'touchstart', associated '#canvas', with that I was able to trigger. 

It gets into 'WebGLBenchmark.js' but now complains that 'event.touches' is undefined. Not sure what that means. Still, there's hope :)

Comment 28

3 years ago
Getting there :)
(Assignee)

Comment 29

3 years ago
Manged to trigger 'touchstart' and 'touchend' but with no success.
(Assignee)

Comment 30

3 years ago
Hi Dan,

I'm getting nowhere. Current status is that I can get benchmark results and send them back to mozbench. The only piece missing is how to auto-start the benchmark.
Would it be helpful to connect you to the person who made the benchmark?
(Assignee)

Comment 32

3 years ago
I think so. We're pretty much there. We just need a way to start the benchmark automatically.
Ok, will send an email.
(Assignee)

Comment 34

3 years ago
Thank you so much. That would be perfect :)

Comment 35

3 years ago
(In reply to Ricardo Castro from comment #34)
> Thank you so much. That would be perfect :)

Ricardo, thanks for your hard work on this! Hopefully things move along once you've had a chance to talk to the benchmark author, but if you're getting frustrated, please let me know and I can take this one over, we have lots of stuff for you to work on :)

Comment 36

3 years ago
So, Alon asked me to chime in on this. First of all, if you need the Unity source project to this benchmark, you can grab it from: http://files.unity3d.com/jonas/WebGLBenchmark.zip

(I have not formally "released" this yet, but that is mostly a matter of doing a cleanup pass on the code, I don't mind sharing it as is).

If there is any specific changes you'd like to see to the project to be able to automate it, let me know which, and I can help implementing them.

I think it would be great to have this running on Mozilla's automation (AFAIK, the WebKit team already does the same).
(Assignee)

Comment 37

3 years ago
Hey Dan,

Not at all. I was just in need of some help. Now that Jonas came to the rescue I'm sure we can move things along quicker :)
(Assignee)

Comment 38

3 years ago
Hi Jonas,

First of tall, thank you very much for help and availability.

For this particular project, we're pretty much there. The only thing we're missing is a way to automatically start the benchmark. 

I don't know if there's already an easy way for achieve that or if you can point me on the way to do it.

Basically, mozbench will launch the benchmark and as soon as it is ready, we want it to start.

What do you think?

And again, thank you :)
Flags: needinfo?(jonas)

Comment 39

3 years ago
That is easy to do, but requires changing the project in Unity. I can set you up with a Unity 5.0 license code if you want to try it, or make a build myself later when I find time, which works better for you?
Flags: needinfo?(jonas)
(Assignee)

Comment 40

3 years ago
Excellent news! Either of them is good for me. What is or suggestion?

Comment 41

3 years ago
I'll email you a U5 serial number, and then, if you need any help, just ping me.
(Assignee)

Comment 42

3 years ago
Perfect! I'll need some guidance on how to do it but I'll give you a shout then.
(Assignee)

Comment 43

3 years ago
I won't be in London this weekend in order to complete the missing bits on this. 

With that said, Jonas as been more than helpful, pointed me in the right direction and offered to help if I got into any "trouble". I believe this should be ready for review early next week.
(Assignee)

Comment 44

3 years ago
Previous link points to a build of the project.

Link for he source code: http://files.unity3d.com/jonas/WebGLBenchmarksSource.zip
(Assignee)

Comment 45

3 years ago
And finally some progress! The benchmark is now able to automatically start. I will do some more testing.

I couldn't have done it without Jonas help. A big "Thank You" to him :)

Comment 46

3 years ago
(In reply to Ricardo Castro from comment #45)
> And finally some progress! The benchmark is now able to automatically start.
> I will do some more testing.
> 
> I couldn't have done it without Jonas help. A big "Thank You" to him :)

Great news!
(Assignee)

Comment 47

3 years ago
Created attachment 8530525 [details] [review]
Pull request.
Attachment #8530525 - Flags: review?(dminor)

Comment 48

3 years ago
excellent, glad to see this being added!

Comment 49

3 years ago
(In reply to Ricardo Castro from comment #47)
> Created attachment 8530525 [details] [review]
> Pull request.

Hi Ricardo, sorry for the delay in looking at this. I'm travelling today so I might not have a chance to review until tomorrow.

Thanks!
(Assignee)

Comment 50

3 years ago
Hi Dan,

No problem. There's no hurry :)

Comment 51

3 years ago
I'm hitting a problem launching this on linux, maybe a timing problem?
Here is what I see in the console:

"run() called, but dependencies remain, so not running" index.html:65:10

"pre-main prep time: 631 ms" index.html:65:10

"PlayerInitEngineNoGraphics settings: Could..... not preload global game manager #0   i=0" index.html:61:12
"Failed to initialize player" index.html:61:12
Not sure if that "run() called" message is relevant or not, but what it means is that a data file is being loaded asynchronously, and is not ready yet, so it is waiting on that to run main() etc. Before calling any compiled code, an app needs to wait for that to complete.

If the setup here does not call any compiled code directly though, then this is not the issue and something else is going wrong.
(Assignee)

Comment 53

3 years ago
My bad. I didn't test it on Linux, only OS X.

On OS X I also get that "run() called"  message, but the benchmark runs successfully.

I'll look into it.

Comment 54

3 years ago
The run() called message comes from emscripten and is not something to worry about. However, 
"PlayerInitEngineNoGraphics settings: Could..... not preload global game manager #0   i=0"

does indicate a failure, which sounds like it might be related to loading the data files somehow.
(Assignee)

Comment 55

3 years ago
I already found that the instruction:

"tar xvzf WebGLBenchmark.data.tar.gz WebGLBenchmark.data" is wrong. It's missging an "s" so it should be:

"tar xvzf WebGLBenchmark.data.tar.gz WebGLBenchmarks.data"
(Assignee)

Comment 56

3 years ago
That doesn't fix it though. 

I probably missed something, I didn't commit it or the compression somehow messed up with the files (which I really don't believe).

When I get my back home I'll compare OS X and Linux and check what's different.

Comment 57

3 years ago
Comment on attachment 8530525 [details] [review]
Pull request.

Please flag me for review again once you've had a chance to look at this. Thanks!
Attachment #8530525 - Flags: review?(dminor)
(Assignee)

Comment 58

3 years ago
Sure thing. I was out of town this weekend. 

I believe I got it working. But I want to test it properly before sending for review. I should be able to do it later today.
(Assignee)

Comment 59

3 years ago
Created attachment 8533683 [details] [review]
Pull request.

Added new benchmark code.
Attachment #8530525 - Attachment is obsolete: true
Attachment #8533683 - Flags: review?(dminor)
(Assignee)

Comment 60

3 years ago
Created attachment 8534023 [details] [review]
Pull request.

Updated following review guidelines.
Attachment #8533683 - Attachment is obsolete: true
Attachment #8533683 - Flags: review?(dminor)
Attachment #8534023 - Flags: review?(dminor)

Comment 61

3 years ago
Comment on attachment 8534023 [details] [review]
Pull request.

Thanks for you hard work and persistence in getting this working :)
Attachment #8534023 - Flags: review?(dminor) → review+

Updated

3 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Assignee)

Comment 62

3 years ago
Glad it's working. Going to file and proceed with the bug for Jenkins and get this fully integrated.
You need to log in before you can comment on or make changes to this bug.