2070629 - [Linux] No zero-copy VideoFrame to Canvas2D path: every hardware-decoded frame is downloaded to CPU, drawImage takes ~89% of main thread at 1080p60

Status: RESOLVED FIXED

(Core :: Graphics: Canvas2D, defect)

Description

[Abhinav Singh]

Attachment: about:support Graphics + Media excerpts (GPU, decision log, codec table); non-default prefs noted in header

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:154.0) Gecko/20100101 Firefox/154.0

Steps to reproduce:

1. Linux (Pop!_OS 24.04), Firefox 154, AMD Cezanne iGPU + NVIDIA RTX 2050, Mesa 25.2.8.
2. Open a 60fps VP9 1080p file in a blank tab via file picker — plays at ~59fps presented.
3. Play the same file inside an editor that decodes via WebCodecs (VideoDecoder, mediabunny CanvasSink) at full-1920 on the main thread and paints each resulting VideoFrame via Canvas2D drawImage.
4. Capture a Gecko profile of the editor tab during playback.
   Actual results:

Actual results:

Editor plays at ~16fps.
Profiler (inverted call tree): ~85-89% of main-thread time inside native drawImage under the per-frame canvas paint; no decode cost;
app composite ~3.6%.
Setting gfx.canvas.accelerated.force-enabled=true gives the identical 89%.
The GPU to CPU readback per hardware-decoded frame is unconditional on Linux.

Expected results:

A zero-copy VideoFrame to Canvas2D path on Linux (as exists in Chrome, which plays the identical code at smooth 60fps): hardware-decoded frames reach the canvas without a per-frame GPU to CPU download.

Comment 1

[Abhinav Singh]

I would like to take this bug, please assign it to me. I intend to fix it and am looking for a mentor.
Proof so far: same 60fps VP9 file plays ~59fps presented in a blank tab vs ~16fps in the editor path
Profile shows ~89% of main-thread time in native drawImage with no decode cost
gfx.canvas.accelerated.force-enabled=true changes nothing.
about:support excerpts attached
(note DMABUF_SURFACE_EXPORT blocklisted, non-default prefs listed in file header).
Happy to provide profiles on request.

Comment 2

[Martin Stránský [:stransky] (ni? me)]

DMABUF_SURFACE_EXPORT is not related here. Please attach complete about:support page.
Thanks.

Comment 3

[Martin Stránský [:stransky] (ni? me)]

We disable dmabuf on NVIDIA due to breakages so that may be the reason you see it.

Comment 4

[Abhinav Singh]

Attachment: Full about:support page (raw data)

Comment 5

[Abhinav Singh]

Understood on both, thank you.
Full about:support is attached.
For the record, my MOZ_LOG run opens only the radeonsi driver (va_openDriver returns 0) with the ffvpx VP9 decoder in RDD, and nvidia-smi stays idle, so everything runs on the AMD chip as you expected.

Comment 6

[Martin Stránský [:stransky] (ni? me)]

Please try to run on clear profile without any modificatons:
https://fedoraproject.org/wiki/How_to_debug_Firefox_problems#Test_Firefox_with_a_new_profile

and if you can reproduce it, run on terminal with MOZ_LOG="Dmabuf:5" env variable, reproduce the issue and attach the log here.
Thanks.

Comment 7

[Abhinav Singh]

Attachment: MOZ_LOG Dmabuf:5 on clean profile while reproducing (lag persists)

Comment 8

[Abhinav Singh]

Reproduced on a clean profile with stock prefs (the lag is still there).
Dmabuf log attached: libva opens only the radeonsi driver, VA-API surfaces are created and used per frame with no errors (pool shows copied 0), nothing touches NVIDIA.
So the download happens later in the Canvas2D path.
Clearing needinfo.

Comment 9

[Martin Stránský [:stransky] (ni? me)]

Yes, looks like DMABufSurfaceYUV::BuildSurfaceDescriptorBuffer() is used. Do you have any exact testcase for it?
Thanks.

Comment 10

[Martin Stránský [:stransky] (ni? me)]

[RDD 139337: MediaSupervisor #4]: D/Dmabuf [79f4a7790c20]: DMABufSurface::ReadIntoBuffer UID 139341

Looks like it's called in RDD process for some reason?

Comment 11

[Abhinav Singh]

Attachment: Minimal standalone testcase: VideoDecoder + drawImage(VideoFrame) fps meter (needs IVF from ffmpeg -i IN.webm -c:v copy OUT.ivf

Comment 12

[Abhinav Singh]

Testcase attached.
On my box (AMD Cezanne, clean profile, HW decode confirmed with va_openDriver() = 0) it prints ~72fps. The editor with the same file does 16fps with 89% of main thread in drawImage. So the testcase shows the path alone, and the app layers add their own cost. I can share the 67MB IVF if needed.

On the RDD ReadIntoBuffer: my log has 435 pairs of DMABufSurfaceYUV::BuildSurfaceDescriptorBuffer directly followed by ReadIntoBuffer, all in RDD. From widget/gtk/DMABufSurface.cpp:2354-2382, this function always flattens each YUV DMABUF frame into an RGB shmem buffer for transport to the content process. So the per-frame GPU to CPU download happens at serialize time in RDD.

For context on the 16fps editor number: the preview renders every tick with background fill, roundRect-clipped and translated/scaled drawImage of the decoded canvas, a second facecam decode blit, text overlays, and a full-canvas destination-in outer rounding, all on one CPU-rasterized 2D canvas at up to 1920 wide. So the editor puts ~6 operations per frame on top of the per-frame download.

Live editor: https://panoptik-studio.vercel.app/editor
Render code: https://github.com/Panoptik-Studio/Panoptik/blob/main/packages/engine/src/render.ts

Comment 13

[Abhinav Singh]

See Also: bug 1844548 (Construct engine, same createImageBitmap/drawImage-video readback pattern, 60fps Chrome vs ~15fps Firefox)

Comment 14

[Martin Stránský [:stransky] (ni? me)]

Yes, I think I can reproduce it.

Comment 15

[Abhinav Singh]

Great, thank you.
Happy to test any Try builds or patches on my Cezanne + RTX 2050 setup.

Comment 16

[Martin Stránský [:stransky] (ni? me)]

Assigned to you. If RTX 2050 is NVIDIA you may hit another issues with https://bugzilla.mozilla.org/show_bug.cgi?id=2012285 where GL context without actual window display connection is created as SW rendering one - so you'd need to make soure you have valid GL context and not dmabuf snapshot (if possible).

Comment 17

[Abhinav Singh]

Two follow-ups.

On the GL context concern: verified on my machine - the Dmabuf log shows zero llvmpipe and all surfaces bind via zero-copy EGLImageTargetTexture2D on radeonsi in both RDD and Renderer. So my snapshot context is hardware; I'll keep Bug 2012285 in mind so any new path does not depend on a HW snapshot context existing on NVIDIA.

On the patch direction, I mapped the full descriptor journey for VideoFrame:
DrawImage never requests a descriptor for videoFrame, ValidSurfaceDescriptorForRemoteCanvas2d rejects DMABuf (default: return false), and even if allowed, SurfaceDescriptorDMABuf carries file descriptors which SerializeToBytesUtil cannot put in the recorded byte stream, it would need handle-based transport like D3D10/MacIOSurface, plus a DMABuf branch in CanvasTranslator::LookupSourceSurfaceFromSurfaceDescriptor.
That is the full remote-canvas path.

The narrower alternative is a same-process import:
Implement DMABuf (EGLImage -> SkImage) in DrawTargetSkia::CreateSourceSurfaceFromNativeSurface for software canvases. No new IPC protocol and no recording/translation machinery though the RDD->content transport of the fd itself still needs verification, since today the shmem flatten bypasses it.

Which shape do you prefer as the first patch?
I have the readback chain mapped and the tree builds.

Comment 18

[Martin Stránský [:stransky] (ni? me)]

ValidSurfaceDescriptorForRemoteCanvas2d() should be what we want here.

Comment 19

[Abhinav Singh]

One quick correction to my previous comment:
After checking the code more carefully, the handle-based transport is actually already built. When hardware decoding runs in RDD, WebCodecs in the content process receives a GPUVideoImage (via RemoteMediaManagerChild::TransferToImage in dom/media/ipc/RemoteMediaManagerChild.cpp). Its GetDesc() in gfx/layers/GPUVideoImage.h returns SurfaceDescriptorGPUVideo wrapping SurfaceDescriptorRemoteDecoder with an integer handle and RemoteDecoderVideoType::DMABuf (from gfx/layers/LayersTypes.h). Because this descriptor contains only integer handles and enums, it has no file descriptor attachments and can safely pass through SerializeToBytesUtil (in ipc/glue/SerializeToBytesUtil.h) in the canvas recording stream without crashing. The file descriptor concern only applies to SurfaceDescriptorDMABuf (from widget/gtk/DMABufSurface.cpp:2050), which is only used when falling back to Image::GetDescFromTexClient() because DMABUFSurfaceImage does not override GetDesc().

I also found why the CPU download gets triggered. In layout/base/nsLayoutUtils.cpp:7102, nsLayoutUtils::SurfaceFromVideoFrame does not have the bool aOptimizeSourceSurface parameter that SurfaceFromElement has for HTMLVideoElement (at line 7502). Because of this, line 7194 calls result.mLayersImage->GetAsSourceSurface() unconditionally whenever aTarget is passed. For GPUVideoImage, that call fires the synchronous RemoteMediaManagerChild::Readback (dom/media/ipc/RemoteMediaManagerChild.cpp:1082) to RDD. Adding bool aOptimizeSourceSurface = true to SurfaceFromVideoFrame and passing false from DrawImage will allow us to query the descriptor without forcing a CPU readback.

So my plan for the prototype is to add aOptimizeSourceSurface = false support to SurfaceFromVideoFrame, query MaybeGetSurfaceDescriptorForRemoteCanvas for VideoFrame in CanvasRenderingContext2D::DrawImage, and allow RemoteDecoderVideoType::DMABuf in both ValidSurfaceDescriptorForRemoteCanvas2d (behind a new pref gfx.canvas.remote.use-draw-image-fast-path-dmabuf) and SDIsSupportedRemoteDecoder. Then in CanvasTranslator::LookupSourceSurfaceFromSurfaceDescriptor (gfx/layers/ipc/CanvasTranslator.cpp), I will implement the DMABuf unpack to expose the DMABufSurface for the GPU DrawTarget.

I will start working on this prototype on my Cezanne machine and update here once I have test results.

Comment 20

[Abhinav Singh]

Update with prototype results on Cezanne (testing the prototype outlined in Comment 19 with the 3687-frame 1080p VP9 IVF):

The handle transport and lookup path works end-to-end:

• MaybeGetSurfaceDescriptorForRemoteCanvas returned a SurfaceDescriptorGPUVideo (RemoteDecoderVideoType::DMABuf) for every single drawImage(VideoFrame) call (3687/3687).
• CanvasTranslator resolved every texture via VideoBridgeParent::LookupTexture with 0 failures.
• Synchronous RDD ReadIntoBuffer was completely eliminated from the content side, unblocking the draw loop (~200 fps vs previously sync-blocked).

However, sustained playback testing revealed two blockers:

1. Watchdog demotion:
   The canvas demotes to software after ~63 draws. DrawTargetWebgl::UsageProfile counts each fresh per-frame CPU DataSourceSurface upload as a cache miss. Once mFailedFrames exceeds the fallback ratio, NotifyRequiresRefresh causes the content process to set mAllowAcceleration = false. Setting gfx.canvas.accelerated.profile-frames = 0 keeps acceleration alive for all 3687 draws.

2. Presentation starvation under sustained load:
   In sustained playback, the CanvasRenderer thread cannot keep up because DMABUFTextureHostOGL::GetAsSurface() performs a synchronous EGL import and glReadPixels readback (DMABufSurface::GetAsSourceSurface) for every frame. Without content main-thread backpressure, the translation queue backs up, resulting in presentation stalls (8-10s freezes and "RemoteTexture ready timeout" warnings).

This confirms that off-thread CPU unpack is not sufficient for sustained 60 fps. We need a direct GPU-to-GPU path where DrawTargetWebgl (or CanvasTranslator) samples the DMABuf/EGLImage texture directly without reading back to CPU RAM and re-uploading. That would also resolve (1) because no CPU texture uploads would take place.

I have the raw logs available if needed. How would you recommend structuring the zero-copy texture sampling in CanvasTranslator / DrawTargetWebgl? Pointers to existing EGLImage or texture-as-source sharing patterns in Gecko would be very helpful.

Comment 21

[Martin Stránský [:stransky] (ni? me)]

Any GetAsSurface() should be obeyed - it's GPU->CPU path. DMABuf/EGLImage is already implemented, for instance as SharedTextureDMABuf which is already used for WebGL/WebGPU:
https://searchfox.org/firefox-main/source/dom/webgpu/SharedTextureDMABuf.h

Comment 22

[Abhinav Singh]

Following up on Bug 2070629 comment 21:

I implemented the true GPU-to-GPU zero-copy path on Linux and verified sustained playback on AMD Cezanne (Radeon Vega / Mesa).

Implementation Architecture

1. In CanvasRenderingContext2D::DrawImage, the patch passes aOptimizeSourceSurface = false to nsLayoutUtils::SurfaceFromVideoFrame, queries MaybeGetSurfaceDescriptorForRemoteCanvas, and gates RemoteDecoderVideoType::DMABuf behind a new pref gfx.canvas.remote.use-draw-image-fast-path-dmabuf (default false).
2. In CanvasTranslator::LookupSourceSurfaceFromSurfaceDescriptor, instead of calling GetAsSurface(), the translator resolves the DMABUFTextureHostOGL, extracts its SurfaceDescriptorGPUVideo, and calls mSharedContext->ImportSurfaceDescriptor(hostSd, size, gfx::SurfaceFormat::R8G8B8X8).
   • This invokes TexUnpackSurface::BlitSd -> GLBlitHelper::Blit(DMABufSurface*) directly in the CanvasRenderer WebGL context, converting YUV planes to RGB in VRAM without CPU readbacks.
   • It returns a SourceSurfaceWebgl, which DrawTargetWebgl recognizes as a texture handle, counting it as a cache hit (mProfile.OnCacheHit()) and completely preventing watchdog demotion.
3. In DrawTargetWebgl, GLBlitHelper outputs true RGBA (SurfaceFormat::R8G8B8X8), whereas the 2D canvas framebuffer expects BGRA bytes. To avoid any CPU/GPU copies, I added a uniform swizzle branch (u_swizzle > 1.5 -> image.bgra) in the mImageProgram fragment shader, setting swizzleVal = 2.0f for R8G8B8A8/R8G8B8X8 inputs.

Verification & Performance Numbers (Cezanne iGPU)

Testing with the 3687-frame 1080p60 VP9 IVF and the live editor:

• Content main thread drawImage time dropped from 89% to 17.8% in the Gecko profiler (the remaining ~17% is standard canvas recording memcpy and IPC shmem transport).
• Synchronous RDD ReadIntoBuffer and glReadPixels: 0 calls.
• Standalone testcase framerate jumped from 72 fps to ~310 fps.
• Complex multi-canvas editor: runs at a solid, vsync-capped 60 fps (previously ~16 fps).
• Presentation stalls / RemoteTexture ready timeout: 0.
• Watchdog demotion: 0 (remains hardware accelerated for the entire duration with default profile-frames = 10).
• Color reproduction: 100% verified (R and B channels render accurately; existing Mochitest 2D Canvas test suites pass 4/4).
• All 9 touched files format cleanly with ./mach format and compile cleanly with ./mach build binaries.

The pref gfx.canvas.remote.use-draw-image-fast-path-dmabuf is currently default false. Would you like to keep it false for the initial landing, or enable it on NIGHTLY_BUILD?

I have the patch ready. Should I submit it to Phabricator for review (r?stransky)?

Comment 23

[Martin Stránský [:stransky] (ni? me)]

Good. Please note that you should not hardcode surface format anywhere (gfx::SurfaceFormat::R8G8B8X8) but use the actual one.

Comment 24

[Abhinav Singh]

Updated the patch to dynamically query the format from the TextureHost (preserving existing RGBA/BGRA formats, and mapping YUV to R8G8B8A8/R8G8B8X8 based on HasAlpha()). Submitting to Phabricator now for review.

Comment 25

[Abhinav Singh]

Attachment: Bug 2070629 - Implement zero-copy DMABuf VideoFrame import to remote Canvas2D on Linux r?stransky

Comment 26

[Abhinav Singh]

Attachment: Bug 2070629 - Implement zero-copy DMABuf VideoFrame import to remote Canvas2D on Linux r?stransky

Comment 27

[Martin Stránský [:stransky] (ni? me)]

Thanks, will look at it. Next time you can re-use the same phabricator revision ID so the revision is just updated and not created a new one.

Comment 28

[Abhinav Singh]

Thanks Martin! Just to clarify, no new revision was created, I updated the existing revision D326510, which is now on Diff 3 with all review feedback addressed and inline replies posted.
The extra attachment stub was automatically generated when attachment 9643322 [details] [diff] [review]'s MIME type was edited.

Comment 29

[Abhinav Singh]

Attachment: Bug 2070629 - Implement zero-copy DMABuf VideoFrame import to remote Canvas2D on Linux r?stransky,lsalzman

Comment 30

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:lsalzman, could you have a look please?

For more information, please visit BugBot documentation.

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a1ce6fdd6513
https://hg.mozilla.org/integration/autoland/rev/f7c591a105d9
Implement zero-copy DMABuf VideoFrame import to remote Canvas2D on Linux r=gfx-reviewers,lsalzman

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/1af0d7a63a86
https://hg.mozilla.org/integration/autoland/rev/4c513dda5aca
Revert "Bug 2070629 - Implement zero-copy DMABuf VideoFrame import to remote Canvas2D on Linux r=gfx-reviewers,lsalzman" for causing bustages on GLBlitHelper.

Comment 33

[Cosmin Sabou [:CosminS]]

Backed out for causing bustages on GLBlitHelper.

Push with failures

Failure log

Backout link

Comment 34

[Abhinav Singh]

Wrapped kFragConvert_ColorMatrixBGR in MOZ_WIDGET_GTK in both GLBlitHelper.h and GLBlitHelper.cpp to fix the non-unified build bustage on macOS, Android, and iOS. Revision D327325 has been updated and is ready to re-land.

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a3c328e4aff3
https://hg.mozilla.org/integration/autoland/rev/5a7ebbc00c27
Implement zero-copy DMABuf VideoFrame import to remote Canvas2D on Linux r=gfx-reviewers,lsalzman

Comment 36

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/5a7ebbc00c27