1050645 - Make it possible to run Web Audio API performance regression testing on CI

Status: RESOLVED FIXED

(Core :: Web Audio, defect, P1)

Description

[Chris Peterson [:cpeterson]]

mozbench running padenot's webaudio-benchmark [1] shows that Nightly 34 is about 2–3x slower than Chrome 36 on most of the webaudio test cases (on Windows, Linux, and OS X).

Take the "Simple gain test without resampling" test case (on OS X, for example):

 Firefox 31 = 165 ms
 Beta 32    = 161 ms
 Aurora 33  = 182 ms
 Nightly 34 = 230 ms
 Chrome 36  =  55 ms (3x faster than Firefox 31)
 Safari 7   =  66 ms

Or the "Simple mixing (different buffers)" test case:

 Firefox 31 = 3333 ms
 Beta 32    = 3326 ms
 Aurora 33  = 3445 ms
 Nightly 34 = 3931 ms
 Chrome 36  = 1364 ms (2.4x faster than Firefox 31)
 Safari 7   = 1186 ms

It looks like Firefox performance difference is exaggerated in the Nightly channel builds, probably related to Nightly's profiling/debugging code. Nightly 34 looks slower than Aurora 33, but Nightly 33 is not faster than Nightly 34. Bug 974464 tracks the work to create no-profiling mozconfigs and builds.


[1] https://github.com/padenot/webaudio-benchmark

[2] https://datazilla.mozilla.org/?start=1406911655&stop=1407412800&product=firefox&repository=nightly&os=linux&os_version=Ubuntu%2012.04&test=webaudio-benchmark&graph_search=34.0,37.0.2062.58,38.0.2115.0&tr_id=176&graph=Downmix%20without%20resampling%20%28Mono%20-%3E%20Stereo%29&x86=false&x86_64=true&compare_product=chrome&compare_repository=canary&project=mozbench

Comment 1

[Benoit Girard (:BenWa)]

That's quite the performance drop. The major disadvantage to profiling build is typically losing a register. Since you measuring on mac 64 bit losing a register there shouldn't be that big of a deal IMO. Since we're 3x simpler that's indicative of a problem. I would probably focus on understanding the 3x before looking into the why profiling is adding a ~.4x drop in performance, it might just lower the profiling overhead once you've fixed that bottleneck.

Comment 2

[Paul Adenot (:padenot)]

This bug was kind of a meta bug for Web Audio API optimization, but we're using the "perf" tag to track optimizations bugs. Since it's already about the benchmark suite, let's make it about performance regression testing.

When profiling and optimizing an application that makes use of the Web Audio API, we should try to come up with a synthetic benchmark that has the same work profile as the application, to make the optimization observable so we know we're not regressing over time.

Then, we should be able to run those benchmarks on a CI setup, and get graphs out of it. We already have this, but we should be able to add new benchmarks, etc.

Dan, is the third paragraph something we can do with the current mozbench/graphana setupe or something ? Could you show me how it's done, or give me credentials or whatever is needed to set this up?

Comment 3

[Dan Minor [:dminor]]

Paul this sounds like a good candidate for mozbench/grafana. If you want to add your benchmark, you can submit a pullrequest to [1]. Once merged, the test machines will pick up the new benchmarks automatically. There are some instructions in the readme about adding new benchmarks, please let me know if something is unclear or misleading. For the grafana side, you can edit the graph through the web user interface and export the "Dashboard JSON" and I will merge it for you.

You can also file a bug like Bug 1103064 and I (or maybe a contributor) will take care of part or all of this for you.

Mozbench runs on nightlies and gives performance comparisons to other browsers. If you are interested in finding per-commit regressions, you probably want to add your benchmark to Talos instead (or as well). Jmaher is the contact for Talos.

[1] https://github.com/dminor/mozbench

Comment 4

[Maire Reavy [:mreavy]]

I think this is going to be really valuable to have as soon as we can. This was originally a meta (so we didn't set a priority), but now that it's about getting perf regression tests into CI, I'd like to make this a P1.

Comment 5

[Dan Minor [:dminor]]

Paul, I've placed your webaudio.json grafana file up at [1]. Please let me know if anything seems unexpected.

[1] http://ouija.allizom.org/grafana/index.html#/dashboard/file/webaudio.json

Comment 6

[Paul Adenot (:padenot)]

(In reply to Dan Minor [:dminor] from comment #5)
> Paul, I've placed your webaudio.json grafana file up at [1]. Please let me
> know if anything seems unexpected.
> 
> [1] http://ouija.allizom.org/grafana/index.html#/dashboard/file/webaudio.json

Looks good. Has the new version of mozbench (that includes my PR) been pushed to production yet? I don't see results for the two benchmarks I added (the last two in the dashboard).

Comment 7

[Martin Best (:mbest)]

Whatever benchmarks we finalize, please update http://ouija.allizom.org/grafana/index.html#/dashboard/file/mozbench.json to include them as well, or at least the agrigate for regression monitoring.  

Also let's be sure to measure Windows, Android, and FirefoxOS.  Linux is a developer environment but not one that has a big impact at the the market level unfortunately.  The true test of how we are doing is the performance on these other platforms.  There is hardware in the Toronto office that continuously runs tests on these platforms.

Comment 8

[Paul Adenot (:padenot)]

(In reply to Martin Best (:mbest) from comment #7)
> Whatever benchmarks we finalize, please update
> http://ouija.allizom.org/grafana/index.html#/dashboard/file/mozbench.json to
> include them as well, or at least the agrigate for regression monitoring.

Yes, the aggregate is on mozbench and has the new benchmarks, detailed results are on the other dashboard (that is web audio api specific).
 
> Also let's be sure to measure Windows, Android, and FirefoxOS.  Linux is a
> developer environment but not one that has a big impact at the the market
> level unfortunately.  The true test of how we are doing is the performance
> on these other platforms.  There is hardware in the Toronto office that
> continuously runs tests on these platforms.

Noted, I'll update the dashboard accordingly.

Comment 9

[Dan Minor [:dminor]]

Paul, it looks like the webaudio benchmarks.js is missing from your commit, so none of the webaudio benchmarks are running. Sorry I didn't catch that in review.

Comment 10

[Paul Adenot (:padenot)]

We can close this, now, it's running fine.