1581240 - Add async API for fetching composition recording frames

Status: RESOLVED FIXED

(Core :: Graphics: Layers, enhancement)

Description

[Nick Alexander :nalexander [he/him]]

Bug 1536174 allows to capture composition frames in memory, dumping them to disk when capturing is disabled. The dump is intended to be synchronous, but isn't with Web Render enabled. The dump file format also encodes critical timestamp information on the filesystem, which is not particular robust.

I would like an async API that resolves the frame data for JS consumers in some way. My use case is exfiltrating this data into Browsertime. Browsertime serializes JS objects from the browser across a Web Driver channel to a consumer; generally, this means that strings are preferred. I don't know if binary data arrays are possible, but I expect they are. More optimized layouts such as array buffers are probably not supported.

I would like a takeFrames() API that resolves to a Promise of a list of pairs (or objects) like:

Promise<[Promise<[timestamp1, frame1]>,
                Promise<[timestamp2, frame2]>,
                ...
                Promise<[timestamp2, frame2]>>

The only reason for the layers of Promise instances is to allow to discard the potentially large amount of data as it is streamed across the Web Driver boundary. There's no particular need for this specific API: I just want something that gives timestamps and frame data in an async manner such that I know when the frames are available.

Comment 1

[Nick Alexander :nalexander [he/him]]

This probably blocks running Browsertime for Android in automation.

Comment 2

[Nick Alexander :nalexander [he/him]]

See also https://github.com/mozilla/browsertime/issues/12, which appears to not work on my Pixel 2 due to some kind of race condition here. It is possible that the race is not in the composition recorder, however; I'm trying to track that down as we speak.

Comment 3

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1581240 - Return a Promise from windowUtils.setCompositionRecording() r?kvark!,nika!

windowUtils.setCompositionRecording() now returns a promise that is resolved
when the composition recorder is enabled (if given true) or when frames are
written to disk (if given false). To accomplish this, the
WebRenderCompositionRecorder now returns a MozPromise when writing frames
to disk begins that is resolved when that process finishes.

Comment 4

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1581240 - Return collected frames from the composition recorder as data URIs r?mstange

The composition recorder can now either write frames to disk as PNGs or return
the frames as an array of data URIs. This will allow us to send the collected
frames across IPC and hand them over to JS in a later patch.

Depends on D47300

Comment 5

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1581240 - Add an API to retrieve the collected frames from WebRender r?mstange

Depends on D47815

Comment 6

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1581240 - Return collected frames as a promise to JS r?bzbarsky!,mstange!,nika!

The setCompositionRecording API on nsIDOMWindowUtils has been broken up into
two new APIs:

• startCompositionRecording(), which starts the composition recorder; and
• stopCompositionRecording(bool writeToDisk) which stops the composition
  recorder and either returns a Promise that resolves to the collected frames
  or returns a Promise that resolves when the frames have been written to disk.

The collected frames are serialized over IPC as part of a Shmem as to not
approach the IPC data transfer limit.

Depends on D47816

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c0589670c762
Return a Promise from windowUtils.setCompositionRecording() r=nika
https://hg.mozilla.org/integration/autoland/rev/e2fd47ff8a89
Return collected frames from the composition recorder as data URIs r=mstange
https://hg.mozilla.org/integration/autoland/rev/17dd226e21ae
Add an API to retrieve the collected frames from WebRender r=mstange
https://hg.mozilla.org/integration/autoland/rev/b73b86efe266
Return collected frames as a promise to JS r=bzbarsky,mstange,nika

Comment 8

[Daniel Varga [:dvarga]]

Backed out 4 changesets (Bug 1581240) for build bustage with error: implicit instantiation of undefined template 'nsTString<char>.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedJob=274945083&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=b73b86efe266cba006e14a3b181508ab960ee805

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=274948092&repo=autoland&lineNumber=32542

Backout link: https://hg.mozilla.org/integration/autoland/rev/f10b5382734327d525b5008ff61c41b3e2f5bccd

[task 2019-11-06T23:22:10.979Z] 23:22:10     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/image/decoders'
[task 2019-11-06T23:22:12.410Z] 23:22:12     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/layers'
[task 2019-11-06T23:22:12.419Z] 23:22:12     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/gcc/bin/g++ -o Unified_cpp_gfx_layers12.o -c  -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/gfx/layers -I/builds/worker/workspace/build/src/obj-firefox/gfx/layers -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/_ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/docshell/base -I/builds/worker/workspace/build/src/layout/base -I/builds/worker/workspace/build/src/layout/generic -I/builds/worker/workspace/build/src/media/libyuv/libyuv/include -I/builds/worker/workspace/build/src/gfx/skia -I/builds/worker/workspace/build/src/gfx/skia/skia -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wno-invalid-offsetof -Wduplicated-cond -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wformat -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -freorder-blocks -O2 -fno-omit-frame-pointer -funwind-tables -Werror -I/builds/worker/workspace/build/src/obj-firefox/dist/include/cairo -I/builds/worker/workspace/build/src/widget/gtk/compat-gtk3 -pthread -I/usr/include/gtk-3.0 -I/usr/include/pango-1.0 -I/usr/include/gio-unix-2.0/ -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/freetype2 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/libpng12 -I/usr/include/gtk-3.0/unix-print -Wno-error=shadow -Wno-maybe-uninitialized  -MD -MP -MF .deps/Unified_cpp_gfx_layers12.o.pp   Unified_cpp_gfx_layers12.cpp
[task 2019-11-06T23:22:12.420Z] 23:22:12     INFO -  In file included from /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderCompositionRecorder.h:8:0,
[task 2019-11-06T23:22:12.420Z] 23:22:12     INFO -                   from /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderCompositionRecorder.cpp:5,
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -                   from Unified_cpp_gfx_layers12.cpp:2:
[task 2019-11-06T23:22:12.421Z] 23:22:12    ERROR -  /builds/worker/workspace/build/src/gfx/layers/CompositionRecorder.h:51:13: error: field 'mDataUri' has incomplete type 'nsCString {aka nsTString<char>}'
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -     nsCString mDataUri;
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -               ^~~~~~~~
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -  In file included from /builds/worker/workspace/build/src/obj-firefox/dist/include/nsXPCOM.h:38:0,
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -                   from /builds/worker/workspace/build/src/obj-firefox/dist/include/nsDebug.h:13,
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -                   from /builds/worker/workspace/build/src/obj-firefox/dist/include/nsISupportsUtils.h:13,
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -                   from /builds/worker/workspace/build/src/obj-firefox/dist/include/nsISupportsImpl.h:13,
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -                   from /builds/worker/workspace/build/src/gfx/layers/CompositionRecorder.h:12,
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -                   from /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderCompositionRecorder.h:8,
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -                   from /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderCompositionRecorder.cpp:5,
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -                   from Unified_cpp_gfx_layers12.cpp:2:
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -  /builds/worker/workspace/build/src/obj-firefox/dist/include/nsStringFwd.h:33:7: note: declaration of 'using nsCString = class nsTString<char> {aka class nsTString<char>}'
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -   class nsTString;
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -         ^~~~~~~~~
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -  /builds/worker/workspace/build/src/config/rules.mk:787: recipe for target 'Unified_cpp_gfx_layers12.o' failed
[task 2019-11-06T23:22:12.421Z] 23:22:12    ERROR -  make[4]: *** [Unified_cpp_gfx_layers12.o] Error 1
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/layers'
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -  make[4]: *** Waiting for unfinished jobs....
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/image'
[task 2019-11-06T23:22:12.421Z] 23:22:12     INFO -  image/Unified_cpp_image1.o
[task 2019-11-06T23:22:12.423Z] 23:22:12     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/image'
[task 2019-11-06T23:22:12.680Z] 23:22:12     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/image/decoders'
[task 2019-11-06T23:22:12.682Z] 23:22:12     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/gcc/bin/g++ -o 

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d9917b1c5017
Return a Promise from windowUtils.setCompositionRecording() r=nika
https://hg.mozilla.org/integration/autoland/rev/a11afd00e52d
Return collected frames from the composition recorder as data URIs r=mstange
https://hg.mozilla.org/integration/autoland/rev/34d96c88ee69
Add an API to retrieve the collected frames from WebRender r=mstange
https://hg.mozilla.org/integration/autoland/rev/8e822bb904b1
Return collected frames as a promise to JS r=bzbarsky,mstange,nika

Comment 10

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/d9917b1c5017
https://hg.mozilla.org/mozilla-central/rev/a11afd00e52d
https://hg.mozilla.org/mozilla-central/rev/34d96c88ee69
https://hg.mozilla.org/mozilla-central/rev/8e822bb904b1