1472800 - add unity3d benchmark to raptor for windows browser runs

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we currently run unity3d benchmark in AWFY for windows 8/10:
https://github.com/mozilla/arewefastyet/tree/2696cbea7fbbe50c3a8cbf70a0ac3c70f135852e/benchmarks/unity-webgl

For android we want to run this benchmark as well.  Lets get it ported and running on raptor desktop first:
https://beta.unity3d.com/jonas/benchmark2015/

looking at what is in the arewefastyet repo, I think we can just use that and do any modifications we need to run/report.

just like bug 1472782, there are probably some similar details for running the test and reporting details.  The data from arewefastyet.com appears to automatically run the test, we can probably use that.  I suspect mozbench.js helps out a bit.


another thing to watch out for is that unity source is large (29MB in a .zip format, 89MB raw).  I would like to find a method for keeping this source code out of mozilla-central.  A couple of methods:
1) copy .zip to tooltool and manage downloads from there
2) use a fetch task or something similar to get a .zip which we can use as a taskcluster artifact

:ahal, given this is a test for raptor (not jsshell-bench), what method would you suggest for obtaining the artifact?  This will also need to be accessed from bitbar (that might make the fetch task more desirable)

Comment 1

[Andrew Halberstadt [:ahal]]

Yeah, I like the idea of using a fetch task, I can look into this if you like. Though we'll still need said .zip to be hosted somewhere, do you know if there are official distributions of this benchmark for download?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I am not sure if there is a normal distributed .zip file, we could pull from awfy:
https://github.com/mozilla/arewefastyet/tree/2696cbea7fbbe50c3a8cbf70a0ac3c70f135852e/benchmarks/unity-webgl

otherwise, upload it to a benchmarks repo that we can use better for the fetch task.

:ahal, if you could work on the fetch task portion of this, I would appreciate it.  That would be good prior art for other tests that we are considering a fetch task for.

Comment 3

[Andrew Halberstadt [:ahal]]

I got the mechanics for a raptor task to depend on a fetch task working here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=01400859d51f48592e61c15c1e448218c4f48a49&filter-tier=1&filter-tier=2&filter-tier=3

If you click the "unity-webgl" fetch task, you'll see the zip is exposed as an artifact. A few things to note:

1. The "ugl" raptor tasks are just dummy tp6 tasks that I copied and changed the label. So the work to actually get the unity-webgl benchmarks running in raptor remains to be done.

2. These raptor tasks will still need to download and extract the "unity-webgl.zip" artifact at some point. This will probably need to be done in mozharness. This is kind of what the "use-artifacts" thing I implemented does already. I might be able to solve this piece as part of bug 1468812.

3. We'll need to host this .zip some place better than an hg user repo. When talking to gps at the work week, he was opposed to using hgweb's built-in .zip directory bundling for some reason. Alternative places to host include github (and making official releases) or tooltool.

4. We'll need to decide whether or not to have separate artifacts for each benchmark, or have a single large artifact that contains every benchmark. Either way, we should have all benchmarks work via these fetch tasks (having some checked into the tree, and some downloaded on demand seems like it'll be confusing).

Comment 4

[Andrew Halberstadt [:ahal]]

Also worth noting that the work to support 'fetch' tasks in raptor will likely be useful for downloading Chrome as well.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

So I think we should wait for bug 1468812 and then figure out a more formal benchmark repo to pull from.  We have:
https://github.com/rwood-moz/talos-pagesets/

Maybe we turn that into both pagesets and benchmarks.  :rwood, do you think we could extend your repo, or should we create a new one that has all benchmarks and general pagesets?

Comment 6

[Robert Wood [:rwood]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #5)

> So I think we should wait for bug 1468812 and then figure out a more formal
> benchmark repo to pull from.  We have:
> https://github.com/rwood-moz/talos-pagesets/
> 
> Maybe we turn that into both pagesets and benchmarks.  :rwood, do you think
> we could extend your repo, or should we create a new one that has all
> benchmarks and general pagesets?

Maybe we should create a new repo that could be put in the mozilla github group (?) or at least in our own team's group (?) so that everyone on our team could have access / able to merge pull requests etc.

I need to update the tp6 pages soon anyway so now would be a good time if we switch to a new repo...

Comment 7

[Andrew Halberstadt [:ahal]]

I have the fetch aspects working:
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&filter-tier=1&filter-tier=2&filter-tier=3

Those 'ugl' tasks are still just tp6 in disguise, so what's left here is the actual work of porting the unity-webgl benchmark to raptor.

Comment 8

[Andrew Halberstadt [:ahal]]

I managed to get unity-webgl running locally with raptor. So the last piece of the puzzle is where do we store it? There are two separate problems to solve, local and CI.

Local
=====

For the local case, we don't want to incur a ~100MB download penalty every time we |mach raptor-test|. So I propose we clone the repo containing the benchmarks into ~/.mozbuild and copy the files from there to the objdir when needed. We'd have to `git/hg pull` to update them, and would want to make sure we have the proper revision of the repo checked out. This way there is a one time clone cost, and then incremental pull costs to update.

Alternatively we could download the whole repo as a zip and re-download when we update something (which would presumably be rare).


CI
==

We'll want to use fetch tasks to expose the benchmarks as artifacts. We could either expose benchmarks individually or do the entire repo in one big zip. Unfortunately github releases are capped at 10MB, which is not big enough for the unity-webgl benchmark (and possibly others as well). This means if we want to use github to store our repo, we'd have to expose the entire repo in one big zip.

Personally I'd prefer to use mercurial and its ability to automatically bundle a directory from a URL like this:
https://hg.mozilla.org/users/ahalberstadt_mozilla.com/webkit-benchmarks/archive/bb5a5e3fee0f.zip/unity-webgl/

Gps, you previously told me that you'd prefer we use github+releases for fetch tasks, but this isn't an option due to the file size. Would you mind clarifying your reservations around using an hg.m.o repo? AIUI, the fetch task should only be downloading the bundle when something changes, so we shouldn't be hammering CI with them.

Comment 9

[Andrew Halberstadt [:ahal]]

s/CI/hg.m.o

Comment 10

[Gregory Szorc [:gps]]

https://phabricator.services.mozilla.com/D1609 and patches around it in the stack implement support for `fetch` tasks that obtain data from VCS. I would create a task that turns a specific Git commit into an archive artifact and have CI and local builds consume that.

Comment 11

[Gregory Szorc [:gps]]

Also, I have no immediate plans to resume my work on fetch tasks. Feel free to commandeer my patches in Phabricator, disassociate them from others in the series, and get them landed yourself - even under your name if you want.

Comment 12

[Andrew Halberstadt [:ahal]]

Thanks! This stuff is on a tight deadline (needs to be finished before scl3 is decommissioned), so while your patch series does look like the right way to do it, I won't have time to finish it up before it's needed.

With that in mind, using github with manual release archives (wrapped in normal fetch tasks) will be good enough for now. We can look into a more automated approach as a follow-up.

Comment 13

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1472800 - [raptor] Bootstrap webkit-benchmarks to the user's state dir

Currently the benchmarks raptor runs are checked into the tree at:
third_party/webkit/PerformanceTests

However, some benchmarks are too big to vendor (200+MB). This patch will
clone a repo containing the benchmarks that *aren't* checked into the tree,
into the user's ~/.mozbuild directory. If the repo already exists, an update
will be run.

Comment 14

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1472800 - [raptor] Add the unity-webgl benchmark

This will run the benchmark in taskcluster as well as allow us to run it locally with:
./mach raptor-test -t raptor-unity-webgl

Depends on D2306.

Comment 15

[Andrew Halberstadt [:ahal]]

With these patches I'm getting close, though they're still WIP. Things left to do:

1) Use the perf-automation repo with github releases (currently it's using an hg.m.o user repo)
2) Get result parsing hooked up (this is why the ugl tasks are failing)

Comment 16

[Andrew Halberstadt [:ahal]]

3) Pin the local benchmark repository to a revision (defined in the raptor config)

Comment 17

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1472800 - [ci] Add the raptor unity-webgl benchmark to taskcluster

Enables the benchmark in CI, uses a fetch task to download the benchmark.

Depends on D2307.

Comment 18

[Phabricator Automation]

Comment on attachment 8994338 [details]
Bug 1472800 - [raptor] Bootstrap webkit-benchmarks to the user's state dir

Robert Wood [:rwood] has approved the revision.

https://phabricator.services.mozilla.com/D2306

Comment 19

[Robert Wood [:rwood]]

(In reply to Phabricator Automation from comment #18)
> Comment on attachment 8994338 [details]
> Bug 1472800 - [raptor] Bootstrap webkit-benchmarks to the user's state dir
> 
> Robert Wood [:rwood] has approved the revision.
> 
> https://phabricator.services.mozilla.com/D2306

FYI there are a couple of nits, please see phabricator ^

Comment 20

[Phabricator Automation]

Comment on attachment 8994339 [details]
Bug 1472800 - [raptor] Add the unity-webgl benchmark

Robert Wood [:rwood] has approved the revision.

https://phabricator.services.mozilla.com/D2307

Comment 21

[Phabricator Automation]

Comment on attachment 8995538 [details]
Bug 1472800 - [ci] Add the raptor unity-webgl benchmark to taskcluster

Robert Wood [:rwood] has approved the revision.

https://phabricator.services.mozilla.com/D2469

Comment 22

[Robert Wood [:rwood]]

Note: Before landing this ensure both the raptor-unity-webgl job and fetch tasks are green on all the platforms. This will also require a rebase as we have since split the jbos into raptor-firefox and raptor-chrome in taskcluster since we don't want to run the chrome jobs on PGO.

Comment 23

[Andrew Halberstadt [:ahal]]

Thanks, unfortunately just before I left for PTO I discovered there was an intermittent:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1e37a51193d6e3ac79aa1938f02261c1502ac9bc&duplicate_jobs=visible

I'll spend a bit of time trying to fix this before landing.

Comment 24

[Andrew Halberstadt [:ahal]]

I think the intermittent is just a page timeout, I'll bump it up to 15 min. New try run going here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e541296cbcc4614bb225ddfaf13f592ac1067c94

Comment 25

[Andrew Halberstadt [:ahal]]

Turns out increasing the timeout didn't help. The benchmark itself must be hanging at some point. Joel suggested getting it landed as tier 3 (so we at least have data points in perfherder to play around with).

The intermittent is fairly frequent, so hopefully we can solve it before we need to make it tier 2.

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3d2b630fe3c9
[raptor] Bootstrap webkit-benchmarks to the user's state dir r=rwood

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/acf55a4fc359
[raptor] Add the unity-webgl benchmark r=rwood

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ed8b7ca3f5be
[ci] Add the raptor unity-webgl benchmark to taskcluster r=rwood

Comment 29

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/3d2b630fe3c9
https://hg.mozilla.org/mozilla-central/rev/acf55a4fc359
https://hg.mozilla.org/mozilla-central/rev/ed8b7ca3f5be