Closed Bug 1472800 Opened 2 years ago Closed 2 years ago

add unity3d benchmark to raptor for windows browser runs

Categories

(Testing :: Raptor, enhancement)

enhancement
Not set

Tracking

(firefox63 fixed)

RESOLVED FIXED
mozilla63
Tracking Status
firefox63 --- fixed

People

(Reporter: jmaher, Assigned: ahal)

References

Details

(Whiteboard: [PI:July])

Attachments

(3 files)

we currently run unity3d benchmark in AWFY for windows 8/10:
https://github.com/mozilla/arewefastyet/tree/2696cbea7fbbe50c3a8cbf70a0ac3c70f135852e/benchmarks/unity-webgl

For android we want to run this benchmark as well.  Lets get it ported and running on raptor desktop first:
https://beta.unity3d.com/jonas/benchmark2015/

looking at what is in the arewefastyet repo, I think we can just use that and do any modifications we need to run/report.

just like bug 1472782, there are probably some similar details for running the test and reporting details.  The data from arewefastyet.com appears to automatically run the test, we can probably use that.  I suspect mozbench.js helps out a bit.


another thing to watch out for is that unity source is large (29MB in a .zip format, 89MB raw).  I would like to find a method for keeping this source code out of mozilla-central.  A couple of methods:
1) copy .zip to tooltool and manage downloads from there
2) use a fetch task or something similar to get a .zip which we can use as a taskcluster artifact

:ahal, given this is a test for raptor (not jsshell-bench), what method would you suggest for obtaining the artifact?  This will also need to be accessed from bitbar (that might make the fetch task more desirable)
Flags: needinfo?(ahal)
Blocks: 1445952
Yeah, I like the idea of using a fetch task, I can look into this if you like. Though we'll still need said .zip to be hosted somewhere, do you know if there are official distributions of this benchmark for download?
Flags: needinfo?(ahal)
I am not sure if there is a normal distributed .zip file, we could pull from awfy:
https://github.com/mozilla/arewefastyet/tree/2696cbea7fbbe50c3a8cbf70a0ac3c70f135852e/benchmarks/unity-webgl

otherwise, upload it to a benchmarks repo that we can use better for the fetch task.

:ahal, if you could work on the fetch task portion of this, I would appreciate it.  That would be good prior art for other tests that we are considering a fetch task for.
Flags: needinfo?(ahal)
I got the mechanics for a raptor task to depend on a fetch task working here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=01400859d51f48592e61c15c1e448218c4f48a49&filter-tier=1&filter-tier=2&filter-tier=3

If you click the "unity-webgl" fetch task, you'll see the zip is exposed as an artifact. A few things to note:

1. The "ugl" raptor tasks are just dummy tp6 tasks that I copied and changed the label. So the work to actually get the unity-webgl benchmarks running in raptor remains to be done.

2. These raptor tasks will still need to download and extract the "unity-webgl.zip" artifact at some point. This will probably need to be done in mozharness. This is kind of what the "use-artifacts" thing I implemented does already. I might be able to solve this piece as part of bug 1468812.

3. We'll need to host this .zip some place better than an hg user repo. When talking to gps at the work week, he was opposed to using hgweb's built-in .zip directory bundling for some reason. Alternative places to host include github (and making official releases) or tooltool.

4. We'll need to decide whether or not to have separate artifacts for each benchmark, or have a single large artifact that contains every benchmark. Either way, we should have all benchmarks work via these fetch tasks (having some checked into the tree, and some downloaded on demand seems like it'll be confusing).
Flags: needinfo?(ahal)
Also worth noting that the work to support 'fetch' tasks in raptor will likely be useful for downloading Chrome as well.
So I think we should wait for bug 1468812 and then figure out a more formal benchmark repo to pull from.  We have:
https://github.com/rwood-moz/talos-pagesets/

Maybe we turn that into both pagesets and benchmarks.  :rwood, do you think we could extend your repo, or should we create a new one that has all benchmarks and general pagesets?
Flags: needinfo?(rwood)
Depends on: 1468812
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #5)

> So I think we should wait for bug 1468812 and then figure out a more formal
> benchmark repo to pull from.  We have:
> https://github.com/rwood-moz/talos-pagesets/
> 
> Maybe we turn that into both pagesets and benchmarks.  :rwood, do you think
> we could extend your repo, or should we create a new one that has all
> benchmarks and general pagesets?

Maybe we should create a new repo that could be put in the mozilla github group (?) or at least in our own team's group (?) so that everyone on our team could have access / able to merge pull requests etc.

I need to update the tp6 pages soon anyway so now would be a good time if we switch to a new repo...
Flags: needinfo?(rwood)
Assignee: nobody → ahal
Status: NEW → ASSIGNED
I have the fetch aspects working:
https://treeherder.mozilla.org/#/jobs?repo=try&author=ahalberstadt@mozilla.com&filter-tier=1&filter-tier=2&filter-tier=3

Those 'ugl' tasks are still just tp6 in disguise, so what's left here is the actual work of porting the unity-webgl benchmark to raptor.
I managed to get unity-webgl running locally with raptor. So the last piece of the puzzle is where do we store it? There are two separate problems to solve, local and CI.

Local
=====

For the local case, we don't want to incur a ~100MB download penalty every time we |mach raptor-test|. So I propose we clone the repo containing the benchmarks into ~/.mozbuild and copy the files from there to the objdir when needed. We'd have to `git/hg pull` to update them, and would want to make sure we have the proper revision of the repo checked out. This way there is a one time clone cost, and then incremental pull costs to update.

Alternatively we could download the whole repo as a zip and re-download when we update something (which would presumably be rare).


CI
==

We'll want to use fetch tasks to expose the benchmarks as artifacts. We could either expose benchmarks individually or do the entire repo in one big zip. Unfortunately github releases are capped at 10MB, which is not big enough for the unity-webgl benchmark (and possibly others as well). This means if we want to use github to store our repo, we'd have to expose the entire repo in one big zip.

Personally I'd prefer to use mercurial and its ability to automatically bundle a directory from a URL like this:
https://hg.mozilla.org/users/ahalberstadt_mozilla.com/webkit-benchmarks/archive/bb5a5e3fee0f.zip/unity-webgl/

Gps, you previously told me that you'd prefer we use github+releases for fetch tasks, but this isn't an option due to the file size. Would you mind clarifying your reservations around using an hg.m.o repo? AIUI, the fetch task should only be downloading the bundle when something changes, so we shouldn't be hammering CI with them.
Flags: needinfo?(gps)
s/CI/hg.m.o
https://phabricator.services.mozilla.com/D1609 and patches around it in the stack implement support for `fetch` tasks that obtain data from VCS. I would create a task that turns a specific Git commit into an archive artifact and have CI and local builds consume that.
Flags: needinfo?(gps)
Also, I have no immediate plans to resume my work on fetch tasks. Feel free to commandeer my patches in Phabricator, disassociate them from others in the series, and get them landed yourself - even under your name if you want.
Thanks! This stuff is on a tight deadline (needs to be finished before scl3 is decommissioned), so while your patch series does look like the right way to do it, I won't have time to finish it up before it's needed.

With that in mind, using github with manual release archives (wrapped in normal fetch tasks) will be good enough for now. We can look into a more automated approach as a follow-up.
Currently the benchmarks raptor runs are checked into the tree at:
third_party/webkit/PerformanceTests

However, some benchmarks are too big to vendor (200+MB). This patch will
clone a repo containing the benchmarks that *aren't* checked into the tree,
into the user's ~/.mozbuild directory. If the repo already exists, an update
will be run.
This will run the benchmark in taskcluster as well as allow us to run it locally with:
./mach raptor-test -t raptor-unity-webgl

Depends on D2306.
With these patches I'm getting close, though they're still WIP. Things left to do:

1) Use the perf-automation repo with github releases (currently it's using an hg.m.o user repo)
2) Get result parsing hooked up (this is why the ugl tasks are failing)
3) Pin the local benchmark repository to a revision (defined in the raptor config)
Enables the benchmark in CI, uses a fetch task to download the benchmark.

Depends on D2307.
Attachment #8994339 - Flags: review?(rwood)
Attachment #8995538 - Flags: review?(rwood)
Attachment #8994338 - Flags: review?(rwood)
Comment on attachment 8994338 [details]
Bug 1472800 - [raptor] Bootstrap webkit-benchmarks to the user's state dir

Robert Wood [:rwood] has approved the revision.

https://phabricator.services.mozilla.com/D2306
Attachment #8994338 - Flags: review+
(In reply to Phabricator Automation from comment #18)
> Comment on attachment 8994338 [details]
> Bug 1472800 - [raptor] Bootstrap webkit-benchmarks to the user's state dir
> 
> Robert Wood [:rwood] has approved the revision.
> 
> https://phabricator.services.mozilla.com/D2306

FYI there are a couple of nits, please see phabricator ^
Attachment #8994338 - Flags: review?(rwood)
Comment on attachment 8994339 [details]
Bug 1472800 - [raptor] Add the unity-webgl benchmark

Robert Wood [:rwood] has approved the revision.

https://phabricator.services.mozilla.com/D2307
Attachment #8994339 - Flags: review+
Attachment #8994339 - Flags: review?(rwood)
Attachment #8995538 - Flags: review?(rwood)
Comment on attachment 8995538 [details]
Bug 1472800 - [ci] Add the raptor unity-webgl benchmark to taskcluster

Robert Wood [:rwood] has approved the revision.

https://phabricator.services.mozilla.com/D2469
Attachment #8995538 - Flags: review+
Note: Before landing this ensure both the raptor-unity-webgl job and fetch tasks are green on all the platforms. This will also require a rebase as we have since split the jbos into raptor-firefox and raptor-chrome in taskcluster since we don't want to run the chrome jobs on PGO.
Thanks, unfortunately just before I left for PTO I discovered there was an intermittent:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1e37a51193d6e3ac79aa1938f02261c1502ac9bc&duplicate_jobs=visible

I'll spend a bit of time trying to fix this before landing.
I think the intermittent is just a page timeout, I'll bump it up to 15 min. New try run going here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e541296cbcc4614bb225ddfaf13f592ac1067c94
Turns out increasing the timeout didn't help. The benchmark itself must be hanging at some point. Joel suggested getting it landed as tier 3 (so we at least have data points in perfherder to play around with).

The intermittent is fairly frequent, so hopefully we can solve it before we need to make it tier 2.
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3d2b630fe3c9
[raptor] Bootstrap webkit-benchmarks to the user's state dir r=rwood
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/acf55a4fc359
[raptor] Add the unity-webgl benchmark r=rwood
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ed8b7ca3f5be
[ci] Add the raptor unity-webgl benchmark to taskcluster r=rwood
Depends on: 1481938
You need to log in before you can comment on or make changes to this bug.