1256408 - Add graphics microbenchmarking

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[Benoit Girard (:BenWa)]

Based on my conversation with wlach, we are now in a state where we should be able to trivially add in-tree microbenchmarking tests, using the GTest framework, and report this data to perfherder.

There's a lot of special consideration that come with microbenchmarking. Most importantly is a sheriffing policy. Since microbenchmarking are very sensitive to small change that are not user visible regressions they should have a more relaxed back out policy. An appropriate sheriff policy will be setup as these test go to production.

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] wlach:1256408 > mozilla:master

Comment 2

[William Lachance (:wlach)]

Comment on attachment 8730335 [details] [review]
[treeherder] wlach:1256408 > mozilla:master

This will need to be enabled on treeherder stage before we can start testing this.

:BenWa: Does platform_microbench sound ok as a grouping name?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8730335 [details] [review]
[treeherder] wlach:1256408 > mozilla:master

just make sure we deploy this :)

Comment 4

[Benoit Girard (:BenWa)]

(In reply to William Lachance (:wlach) from comment #2)
> Comment on attachment 8730335 [details] [review]
> [treeherder] wlach:1256408 > mozilla:master
> 
> This will need to be enabled on treeherder stage before we can start testing
> this.
> 
> :BenWa: Does platform_microbench sound ok as a grouping name?

Yes, but it might be odd if non platform stuff starts using it.

Comment 5

[Benoit Girard (:BenWa)]

Perfherder schema:

08:16:34     INFO - PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"subtests": [{"name": "libxul.so", "value": 93013840}], "name": "installer size", "value": 60343406}]}

http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-linux64-debug/1457962948/mozilla-beta-linux64-debug-bm74-build1-build5.txt.gz

https://github.com/mozilla/treeherder/blob/master/schemas/performance-artifact.json

Comment 6

[Benoit Girard (:BenWa)]

Attachment: Smple output

How does this look?

Comment 7

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/04d61215c5a40cac4b0b1fe196068dfc2f5db543
Bug 1256408 - Add a platform microbench perf framework

Comment 8

[Benoit Girard (:BenWa)]

Attachment: MozReview Request: Bug 1256408 - Add graphics microbenchmarking. r=mstange

Review commit: https://reviewboard.mozilla.org/r/39831/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/39831/

Comment 9

[Benoit Girard (:BenWa)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=fb105671b6df

Comment 10

[Benoit Girard (:BenWa)]

Looks like the logs are hanging on parse? Or maybe just a backlog?

Comment 11

[William Lachance (:wlach)]

(In reply to Benoit Girard (:BenWa) from comment #10)
> Looks like the logs are hanging on parse? Or maybe just a backlog?

It took me a while to figure out what's wrong. It's actually pretty silly. :) There's a stray ':' in your json under the "name" property for the suite.

{"framework": {"name": "platform_microbench"}, "suites": [{"name:": "GfxBench", ...
                                                            ^^^^^^

Fix that and everything should be good.

Comment 12

[Benoit Girard (:BenWa)]

Comment on attachment 8730386 [details]
MozReview Request: Bug 1256408 - Add graphics microbenchmarking. r=mstange

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/39831/diff/1-2/

Comment 13

[Benoit Girard (:BenWa)]

Comment on attachment 8730386 [details]
MozReview Request: Bug 1256408 - Add graphics microbenchmarking. r=mstange

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/39831/diff/2-3/

Comment 14

[Benoit Girard (:BenWa)]

Results look good. Doing a few re-triggers to see how stable the results are:
https://treeherder.allizom.org/#/jobs?repo=try&revision=11c4b8adf0dd&selectedJob=18676584

Comment 15

[Benoit Girard (:BenWa)]

Comment on attachment 8730386 [details]
MozReview Request: Bug 1256408 - Add graphics microbenchmarking. r=mstange

Since Milan is away I'm asking mstange. No need to wait for a small patch.

Comment 16

[Markus Stange [:mstange]]

Comment on attachment 8730386 [details]
MozReview Request: Bug 1256408 - Add graphics microbenchmarking. r=mstange

https://reviewboard.mozilla.org/r/39831/#review36997

The specific benchmarks you're adding don't look all that useful to me, but at least it gives us something to start playing with.

::: gfx/tests/gtest/TestCompositor.cpp:232
(Diff revision 3)
>  {
>    auto layerManagers = GetLayerManagers(GetPlatformBackends());
>  }
>  
> -TEST(Gfx, CompositorSimpleTree)
> -{
> +static void CompositorSimpleTree() {
> +  const int size = 30;

I'd call this variable "benchmarkRepeatCount" or something that conveys that this is not a geometric size.

Comment 17

[Benoit Girard (:BenWa)]

I've added some wiki information:
https://wiki.mozilla.org/Buildbot/Talos/Tests#Microbench
https://wiki.mozilla.org/Buildbot/Talos/Sheriffing#Microbench_Policy

(I'll probably split these off shortly)

(In reply to Markus Stange [:mstange] from comment #16)
> Comment on attachment 8730386 [details]
> MozReview Request: Bug 1256408 - Add graphics microbenchmarking. r?milan
> 
> https://reviewboard.mozilla.org/r/39831/#review36997
> 
> The specific benchmarks you're adding don't look all that useful to me, but
> at least it gives us something to start playing with.

They're not great, but they're not awful either. Having good microbenchmark test is hard(tm), but we can mitigate this by having many of them. We have occasionally modified the Region code. The compositor test is a bit better. It will construct a lot of layers and composite them. A regression in that area should be picked up by this test.

But it will be more useful to add them once we have a particular performance regressions that we are interested in watching.

Comment 18

[Benoit Girard (:BenWa)]

Comment on attachment 8730386 [details]
MozReview Request: Bug 1256408 - Add graphics microbenchmarking. r=mstange

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/39831/diff/3-4/

Comment 19

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/396639988f51

Comment 20

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/396639988f51