1329815 - 41.5% of CPU time in a WebGL 2 demo is lost on gl.drawElements(), gl.clearBufferfv() and gl.blitFramebuffer() checking framebuffer completeness.

Status: RESOLVED FIXED

(Core :: Graphics: CanvasWebGL, defect, P3)

Description

[Jukka Jylänki]

Attachment: checkframebufferstatus.png

Profiling a WebGL 2 demo from one of our partners on Windows GTX980Ti & a 16-core i7 5960X, with the prefs

webgl.disable-angle;true
webgl.enable-draft-extensions;true
webgl.enable-privileged-extensions;true
webgl.force-enabled;true

set, the demo spends 41.5% of its total CPU time inside glCheckFramebufferStatus() validation calls in the WebGL backend. See the attached screenshot for an illustration.

The framebuffer completeness checks seem to be present both in webgl.force-enabled = false and true runs, and presumably take up the same amount of absolute time, but they show up with much smaller proportional overhead when ANGLE is enabled, because of other added ANGLE CPU overhead that piles up in that case.

The motivation to these prefs is to test how much bypassing ANGLE improves performance on Windows, since Jeff Gilbert has dreamed about the possibly in the future that we could do away with various ANGLE validation, making this a sensible setup to benchmark. The demo can be natively compiled as a Windows application, or Emscripten compiled to run in browser. Measuring FPS, CPU and GPU utilization in different cases:

Native Win64:
 - 120-160fps
 - 197% CPU utilization, i.e. ~two full CPU cores worth of work
 - 100% GPU utilization (TDP throttled)

FF Nightly with ANGLE enabled:
 - 11-20 fps
 - 92% CPU utilization, i.e. roughly one core fully utilized
 - 10% GPU utilization, not even enough to spin up the GPU fan

FF Nightly with ANGLE disabled:
 - 22-30 fps
 - 76% CPU utilization, not even one core fully utilized
 - 18% GPU utilization, a bit better

Which looks like when running in a browser, the system is neither fully CPU or GPU bound, but both are regularly idle throughout an application frame, which indicates per-frame pipeline sync bubbles: GPU is fast enough to occassionally starve for work, but CPU also needs to stall to wait for GPU in the middle of a frame while it is working.

The function glCheckFramebufferStatus() would ideally only ever be called at FBO creation time ("load" time), and never at render time. What kind of measures could we do to optimize this? Losing about half of CPU performance to FBO validation is too much. Can we e.g. assume that an FBO configuration that has already once been tested to be complete to be always complete?

Contact me for a live testcase if needed, it is unfortunately not public.

Comment 1

[Jukka Jylänki]

Doing a custom build of Firefox where I commented out gl->fCheckFramebufferStatus(LOCAL_GL_FRAMEBUFFER); check in WebGLFramebuffer::CheckFramebufferStatus() and faked it to always return LOCAL_GL_FRAMEBUFFER_COMPLETE for testing.

This gives the following numbers (ANGLE disabled as well):
 - 38-45 fps (+50% - 72% improvement)
 - 92% CPU utilization (+21% more saturated)
 - 26% GPU utilization (+44% more saturated)

Comment 2

[Jukka Jylänki]

Running on OS X Mac Pro (Late 2013) in stock FF Nightly, 3.5 GHz 6-core Intel Xeon E5 with AMD FirePro D500, the results are
 - 20-22 fps
 - 130% CPU utilization (1.3 cores)
 - 23% GPU utilization

Looking at geckoprofiler, on OS X the FBO completeness checks are a much smaller overhead, around 5% of total execution time.

Comment 3

[Kelsey Gilbert [:jgilbert]]

I'm confident this is the app using the single-FBO antipattern.

I can add a pref warning for this, and we can check.

Comment 4

[Kelsey Gilbert [:jgilbert]]

This will spew invalidations to FB completeness:
 https://treeherder.mozilla.org/#/jobs?repo=try&revision=ec5728ae08b44a77ca51c95b3092c30697b783b7

There are two prefs:
pref("webgl.max-perf-warnings", -1);
pref("webgl.max-acceptable-fb-status-invals", 0);

It only triggers when the status is already cached as complete, but something invalidates our cache.
It will spew to console at after the number of invalidations for a specific FB exceed uint32_t(webgl.max-acceptable-fb-status-invals), and will spew at most uint32_t(webgl.max-perf-warnings) warnings. (-1 => UINT32_MAX, or 'practically forever')

Running with this build (assuming it builds on Try, and not just locally!) should make it obvious if our cache is being invalidated.

Comment 5

[Jukka Jylänki]

Thanks Jeff, this is excellent!

Seeing a running counter of "WebGL perf warning: framebufferTexture2D: FB was invalidated after being complete 25672 times.". We'll give this a look at using multiple static FBOs and comparing performance to that.

Comment 6

[Kelsey Gilbert [:jgilbert]]

Attachment: Bug 1329815 - GeneratePerfWarning and warn on completed-FB invalidation. -

Review commit: https://reviewboard.mozilla.org/r/104686/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/104686/

Comment 7

[Dzmitry Malyshau [:kvark]]

Comment on attachment 8826803 [details]
Bug 1329815 - GeneratePerfWarning and warn on completed-FB invalidation. -

https://reviewboard.mozilla.org/r/104686/#review105414

::: dom/canvas/WebGLContext.h:2038
(Diff revision 1)
>  public:
>      // console logging helpers
>      void GenerateWarning(const char* fmt, ...);
>      void GenerateWarning(const char* fmt, va_list ap);
>  
> +    void GeneratePerfWarning(const char* fmt, ...) const;

I don't think this needs to be `const`. Consequently, `mNumPerfWarnings` would not need to be `mutable`.

::: dom/canvas/WebGLTexture.h
(Diff revision 1)
>  
> -        void Clear();
> +        void Clear(const char* funcName);
>  
>          ~ImageInfo() {
> -            if (!IsDefined())
> +            MOZ_ASSERT(!mAttachPoints.size());
> -                Clear();

What's the reason for this `Clear()` to be removed?

::: gfx/thebes/gfxPrefs.h:619
(Diff revision 1)
>    DECL_GFX_PREF(Live, "webgl.prefer-16bpp",                    WebGLPrefer16bpp, bool, false);
>    DECL_GFX_PREF(Live, "webgl.restore-context-when-visible",    WebGLRestoreWhenVisible, bool, true);
>    DECL_GFX_PREF(Live, "webgl.allow-immediate-queries",         WebGLImmediateQueries, bool, false);
>    DECL_GFX_PREF(Live, "webgl.allow-fb-invalidation",           WebGLFBInvalidation, bool, false);
>  
> +  DECL_GFX_PREF(Live, "webgl.max-perf-warnings",               WebGLMaxPerfWarnings, int32_t, 0);

shouldn't there be some sensible defaults?

Comment 8

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8826803 [details]
Bug 1329815 - GeneratePerfWarning and warn on completed-FB invalidation. -

https://reviewboard.mozilla.org/r/104686/#review105414

> I don't think this needs to be `const`. Consequently, `mNumPerfWarnings` would not need to be `mutable`.

Conceptually this should be const, since it doesn't mutate any notable state on the object. (there's no case where we'd want const to protect us from incremeting the number of posted warnings, so we remove it from the set of members that count towards const)

> What's the reason for this `Clear()` to be removed?

Clear is called during WebGLTexture teardown, so these should all be cleared already.
Also notice that Clear was only previously being called if the ImageInfo was /not/ defined. Which was totally a bug anyway.

> shouldn't there be some sensible defaults?

These are sensible defaults:
Most people never want to see these, and they don't technically represent an error.
They should be useful for guiding optimizations for devs. We should dev-doc-needed this.

Comment 9

[Dzmitry Malyshau [:kvark]]

Comment on attachment 8826803 [details]
Bug 1329815 - GeneratePerfWarning and warn on completed-FB invalidation. -

https://reviewboard.mozilla.org/r/104686/#review105690

Comment 10

[Dzmitry Malyshau [:kvark]]

Looks like mozreview bugged out on that bug, it doesn't let me "r+" it.
https://bugzilla.mozilla.org/show_bug.cgi?id=1331392

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/304ab4d298b7
GeneratePerfWarning and warn on completed-FB invalidation. - r=kvark

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/304ab4d298b7

Comment 13

[Dzmitry Malyshau [:kvark]]

Comment on attachment 8826803 [details]
Bug 1329815 - GeneratePerfWarning and warn on completed-FB invalidation. -

https://reviewboard.mozilla.org/r/104686/#review106538