1551735 - Support composition recording on WebRender

Status: RESOLVED FIXED

(Core :: Graphics, enhancement)

Description

[Barret Rennie [:barret] (they/them)]

Presently WebRender is the only graphics backend that does not support composition recording. It should be possible to leverage its profiler screenshot infrastructure to implement this, similar to how LayerManagerComposite re-uses the Compositor-specific screenshot infrastructure.

Comment 1

[Barret Rennie [:barret] (they/them)]

We can use the composition payload infrastructure that is a side effect from bug 1553261 to determine what render calls correspond to rendering content.

Comment 2

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1551735 - Record content paint times for WebRender r?kvark,jrmuizel

The WebRenderBridgeChild now records whether or not it was painting content
while sending display lists to the WebRenderBridgeParent, allowing for
composition times to be recorded for WebRender.

Depends on D32228

Comment 3

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1551735 - Ref count and document CompositionRecorder r?bas

The CompositionRecorder was being stored as a UniquePtr on the
CompositorBridgeParent, but was then passed to and stored on the LayerManger as
a raw pointer. This has been updated to use a RefPtr.

Depends on D32229

Comment 4

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1551735 - Add a thread-safe composition recorder for WebRender r?bas,kvark

Since WebRender does its rendering on a separate thread from the compositor
thread, we need a composition recorder that can be shared between threads.

Depends on D32230

Comment 5

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1551735 - Add a mode to the AsyncScreenshotGrabber to enable composition recording r?kvark

The AsyncScreenshotGrabber now can operate in two modes:

• ProfilerScreenshots, which does asynchronous scaling of the captured frames
  for inclusion in profiles by the Gecko Profiler; and
• CompositionRecorder, which does not do any scaling and is used for visual
  metrics computations.

The latter mode is exposed by on the Renderer via the record_frame,
map_recorded_frame, and release_composition_recorder_structures methods.

A different handle type (RecordedFrameHandle) is returned and consumed by
these functions, but they translate between RecordedFrameHandle and
AsyncScreenshotHandle when communicating with the underlying
AsyncScreenshotGrabber.

I considered making the AsyncScreenshotGrabber generic over its handle type,
but the extra cost of monomorphization just to change the handle type did not
seem worth it.

Depends on D32231

Comment 6

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1551735 - Expose bindings to the Renderer's composition recorder structures r?kvark

Depends on D32232

Comment 7

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1551735 - Record compositions in Web Render r?kvark

Depends on D32233

Comment 8

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1551735 - Record frames from WebRender and log them to the CompositionRecorder r?bas,kvark

Depends on D32234

Comment 9

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1551735 - Clearly document the case of the RendererOGL receiving a new WebRenderCompositionRecorder while it has one r?bas

Depends on D32235

Comment 10

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1551735 - Split out screenshot grabbing infrasturcture into a new module r?kvark

Depends on D32230

Comment 11

[Phabricator Automation]

Comment on attachment 9066873 [details]
Bug 1551735 - Record content paint times for WebRender r?kvark,jrmuizel

Revision D32229 was moved to bug 1555379. Setting attachment 9066873 [details] to obsolete.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a516d20303e6
Ref count and document CompositionRecorder r=kats
https://hg.mozilla.org/integration/autoland/rev/3b2078f90715
Split out screenshot grabbing infrasturcture into a new module r=kvark
https://hg.mozilla.org/integration/autoland/rev/2e6ca6d6c527
Add a mode to the AsyncScreenshotGrabber to enable composition recording r=kvark
https://hg.mozilla.org/integration/autoland/rev/5891d00fca85
Expose bindings to the Renderer's composition recorder structures r=kvark
https://hg.mozilla.org/integration/autoland/rev/f2a2396a0d4a
Add a thread-safe composition recorder for WebRender r=kvark,kats
https://hg.mozilla.org/integration/autoland/rev/e7b857609786
Record compositions in Web Render r=kats
https://hg.mozilla.org/integration/autoland/rev/aa165d8c181d
Clearly document the case of the RendererOGL receiving a new WebRenderCompositionRecorder while it has one r=kats

Comment 13

[Noemi Erli[:noemi_erli]]

Backed out 7 changesets (Bug 1551735) for build bustages in RenderThread.cpp CLOSED TREE

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&revision=aa165d8c181d94af56e0b93d225bfae77acd4b8b

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=249228129&repo=autoland&lineNumber=21155

Backout: https://hg.mozilla.org/integration/autoland/rev/1b10a1e394f46d08872a163b8117e255f65811b6

Comment 14

[Barret Rennie [:barret] (they/them)]

Latest revisions address this.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/06c8e5f7768d
Ref count and document CompositionRecorder r=kats
https://hg.mozilla.org/integration/autoland/rev/a7868d694fe1
Split out screenshot grabbing infrasturcture into a new module r=kvark
https://hg.mozilla.org/integration/autoland/rev/4a0936bda490
Add a mode to the AsyncScreenshotGrabber to enable composition recording r=kvark
https://hg.mozilla.org/integration/autoland/rev/6154bdfe6fad
Expose bindings to the Renderer's composition recorder structures r=kvark
https://hg.mozilla.org/integration/autoland/rev/9deb5350e244
Add a thread-safe composition recorder for WebRender r=kvark,kats
https://hg.mozilla.org/integration/autoland/rev/152e3a6e5c10
Record compositions in Web Render r=kats
https://hg.mozilla.org/integration/autoland/rev/98e75ac2cf4f
Clearly document the case of the RendererOGL receiving a new WebRenderCompositionRecorder while it has one r=kats

Comment 16

[Cristina Coroiu [:ccoroiu]]

Backed out 7 changesets (bug 1551735) for build bustage at src/gfx/layers/wr/WebRenderCompositionRecorder.h

Backout: https://hg.mozilla.org/integration/autoland/rev/b302d44cf73dc8e903cc2fc1181750461c26e97e

Failure push: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=249214499&revision=98e75ac2cf4fc6a43dea8bd7a77c0625a7456ccf

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=249235250&repo=autoland&lineNumber=17182

[task 2019-05-30T20:52:15.257Z] 20:52:15 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/image/decoders/icon/android'
[task 2019-05-30T20:52:15.263Z] 20:52:15 INFO - /builds/worker/workspace/build/src/sccache/sccache /builds/worker/workspace/build/src/clang/bin/clang++ --target=arm-linux-androideabi -o nsIconChannel.o -c -flto=thin -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/image/decoders/icon/android -I/builds/worker/workspace/build/src/obj-firefox/image/decoders/icon/android -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/sysroot/usr/include/arm-linux-androideabi -isystem /builds/worker/workspace/build/src/android-ndk/sysroot/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API_=16 -fstack-protector-strong -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wc++1z-compat -Wc++2a-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wtautological-overlap-compare -Wtautological-unsigned-enum-zero-compare -Wtautological-unsigned-zero-compare -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=backend-plugin -Wno-error=return-std-move -Wno-error=atomic-alignment -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -mno-unaligned-access -fno-sized-deallocation -fno-short-enums -fno-exceptions -stdlib=libstdc++ -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=neon -mfloat-abi=softfp -fstack-protector-strong -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -Xclang -load -Xclang /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -Oz -fno-omit-frame-pointer -funwind-tables -Werror -MD -MP -MF .deps/nsIconChannel.o.pp /builds/worker/workspace/build/src/image/decoders/icon/android/nsIconChannel.cpp
[task 2019-05-30T20:52:15.263Z] 20:52:15 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/image/decoders/icon/android'
[task 2019-05-30T20:52:15.265Z] 20:52:15 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/base/test/gtest'
[task 2019-05-30T20:52:15.266Z] 20:52:15 INFO - mkdir -p '.deps/'
[task 2019-05-30T20:52:15.267Z] 20:52:15 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/base/test/gtest'
[task 2019-05-30T20:52:15.268Z] 20:52:15 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/dom/base/test/gtest'
[task 2019-05-30T20:52:15.269Z] 20:52:15 INFO - dom/base/test/gtest/Unified_cpp_dom_base_test_gtest0.o
[task 2019-05-30T20:52:15.269Z] 20:52:15 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/dom/base/test/gtest'
[task 2019-05-30T20:52:18.492Z] 20:52:18 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/gfx/layers'
[task 2019-05-30T20:52:18.496Z] 20:52:18 INFO - /builds/worker/workspace/build/src/sccache/sccache /builds/worker/workspace/build/src/clang/bin/clang++ --target=arm-linux-androideabi -o Unified_cpp_gfx_layers12.o -c -flto=thin -I/builds/worker/workspace/build/src/obj-firefox/dist/stl_wrappers -I/builds/worker/workspace/build/src/obj-firefox/dist/system_wrappers -include /builds/worker/workspace/build/src/config/gcc_hidden.h -DNDEBUG=1 -DTRIMMED=1 -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DOS_POSIX=1 -DOS_LINUX=1 -DSTATIC_EXPORTABLE_JS_API -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -I/builds/worker/workspace/build/src/gfx/layers -I/builds/worker/workspace/build/src/obj-firefox/gfx/layers -I/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl/ipdlheaders -I/builds/worker/workspace/build/src/ipc/chromium/src -I/builds/worker/workspace/build/src/ipc/glue -I/builds/worker/workspace/build/src/docshell/base -I/builds/worker/workspace/build/src/layout/base -I/builds/worker/workspace/build/src/layout/generic -I/builds/worker/workspace/build/src/media/libyuv/libyuv/include -I/builds/worker/workspace/build/src/gfx/skia -I/builds/worker/workspace/build/src/gfx/skia/skia/include/config -I/builds/worker/workspace/build/src/gfx/skia/skia/include/core -I/builds/worker/workspace/build/src/gfx/skia/skia/include/docs -I/builds/worker/workspace/build/src/gfx/skia/skia/include/gpu -I/builds/worker/workspace/build/src/gfx/skia/skia/include/utils -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/sysroot/usr/include/arm-linux-androideabi -isystem /builds/worker/workspace/build/src/android-ndk/sysroot/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -D__ANDROID_API_=16 -fstack-protector-strong -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wc++1z-compat -Wc++2a-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wtautological-overlap-compare -Wtautological-unsigned-enum-zero-compare -Wtautological-unsigned-zero-compare -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=backend-plugin -Wno-error=return-std-move -Wno-error=atomic-alignment -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -mno-unaligned-access -fno-sized-deallocation -fno-short-enums -fno-exceptions -stdlib=libstdc++ -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -march=armv7-a -mthumb -mfpu=neon -mfloat-abi=softfp -fstack-protector-strong -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -Xclang -load -Xclang /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -Oz -fno-omit-frame-pointer -funwind-tables -Werror -I/builds/worker/workspace/build/src/obj-firefox/dist/include/cairo -I/builds/worker/workspace/build/src/obj-firefox/dist/include/cairo -Wno-error=shadow -Wno-maybe-uninitialized -MD -MP -MF .deps/Unified_cpp_gfx_layers12.o.pp /builds/worker/workspace/build/src/obj-firefox/gfx/layers/Unified_cpp_gfx_layers12.cpp
[task 2019-05-30T20:52:18.496Z] 20:52:18 INFO - In file included from /builds/worker/workspace/build/src/obj-firefox/gfx/layers/Unified_cpp_gfx_layers12.cpp:2:
[task 2019-05-30T20:52:18.497Z] 20:52:18 INFO - In file included from /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderCompositionRecorder.cpp:5:
[task 2019-05-30T20:52:18.497Z] 20:52:18 ERROR - /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderCompositionRecorder.h:33:45: error: no type named 'WrPipelineId' in namespace 'mozilla::wr'
[task 2019-05-30T20:52:18.498Z] 20:52:18 INFO - wr::WrPipelineId aRootPipelineId)
[task 2019-05-30T20:52:18.498Z] 20:52:18 INFO - ^
[task 2019-05-30T20:52:18.499Z] 20:52:18 ERROR - /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderCompositionRecorder.h:82:29: error: no type named 'Renderer' in namespace 'mozilla::wr'
[task 2019-05-30T20:52:18.500Z] 20:52:18 INFO - bool MaybeRecordFrame(wr::Renderer* aRenderer,
[task 2019-05-30T20:52:18.500Z] 20:52:18 INFO - ^
[task 2019-05-30T20:52:18.501Z] 20:52:18 ERROR - /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderCompositionRecorder.h:115:7: error: no type named 'PipelineId' in namespace 'mozilla::wr'
[task 2019-05-30T20:52:18.502Z] 20:52:18 INFO - wr::PipelineId mRootPipelineId;
[task 2019-05-30T20:52:18.502Z] 20:52:18 INFO - ^
[task 2019-05-30T20:52:18.503Z] 20:52:18 ERROR - /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderCompositionRecorder.h:121:36: error: no member named 'Epoch' in namespace 'mozilla::wr'
[task 2019-05-30T20:52:18.504Z] 20:52:18 INFO - std::unordered_map<uint64_t, wr::Epoch> mContentPipelines;
[task 2019-05-30T20:52:18.505Z] 20:52:18 INFO - ^
[task 2019-05-30T20:52:18.506Z] 20:52:18 INFO - In file included from /builds/worker/workspace/build/src/obj-firefox/gfx/layers/Unified_cpp_gfx_layers12.cpp:2:
[task 2019-05-30T20:52:18.507Z] 20:52:18 ERROR - /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderCompositionRecorder.cpp:54:36: error: out-of-line definition of 'MaybeRecordFrame' does not match any declaration in 'mozilla::layers::WebRenderCompositionRecorder'
[task 2019-05-30T20:52:18.508Z] 20:52:18 INFO - bool WebRenderCompositionRecorder::MaybeRecordFrame(
[task 2019-05-30T20:52:18.509Z] 20:52:18 INFO - ^
[task 2019-05-30T20:52:18.510Z] 20:52:18 ERROR - /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderCompositionRecorder.cpp:91:26: error: cannot initialize object parameter of type 'mozilla::layers::CompositionRecorder' with an expression of type 'mozilla::layers::WebRenderCompositionRecorder'
[task 2019-05-30T20:52:18.511Z] 20:52:18 INFO - CompositionRecorder::RecordFrame(frame);
[task 2019-05-30T20:52:18.512Z] 20:52:18 INFO - ^
[task 2019-05-30T20:52:18.513Z] 20:52:18 ERROR - /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderCompositionRecorder.cpp:105:24: error: cannot initialize object parameter of type 'mozilla::layers::CompositionRecorder' with an expression of type 'mozilla::layers::WebRenderCompositionRecorder'
[task 2019-05-30T20:52:18.513Z] 20:52:18 INFO - CompositionRecorder::WriteCollectedFrames();
[task 2019-05-30T20:52:18.514Z] 20:52:18 INFO - ^
[task 2019-05-30T20:52:18.514Z] 20:52:18 ERROR - /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderCompositionRecorder.cpp:116:3: error: cannot initialize object parameter of type 'mozilla::layers::CompositionRecorder' with an expression of type 'mozilla::layers::WebRenderCompositionRecorder'
[task 2019-05-30T20:52:18.515Z] 20:52:18 INFO - ClearCollectedFrames();
[task 2019-05-30T20:52:18.515Z] 20:52:18 INFO - ^
[task 2019-05-30T20:52:18.515Z] 20:52:18 INFO - 8 errors generated.
[task 2019-05-30T20:52:18.516Z] 20:52:18 INFO - /builds/worker/workspace/build/src/config/rules.mk:810: recipe for target 'Unified_cpp_gfx_layers12.o' failed
[task 2019-05-30T20:52:18.517Z] 20:52:18 ERROR - make[4]: *** [Unified_cpp_gfx_layers12.o] Error 1
[task 2019-05-30T20:52:18.517Z] 20:52:18 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/gfx/layers'
[task 2019-05-30T20:52:18.518Z] 20:52:18 INFO - make[4]: *** Waiting for unfinished jobs...

Comment 17

[Barret Rennie [:barret] (they/them)]

Unified build issue on android. Verified fixed.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0c837987a829
Ref count and document CompositionRecorder r=kats
https://hg.mozilla.org/integration/autoland/rev/caca3a7d4c2f
Split out screenshot grabbing infrasturcture into a new module r=kvark
https://hg.mozilla.org/integration/autoland/rev/b09d27fc1584
Add a mode to the AsyncScreenshotGrabber to enable composition recording r=kvark
https://hg.mozilla.org/integration/autoland/rev/58dd508e43e6
Expose bindings to the Renderer's composition recorder structures r=kvark
https://hg.mozilla.org/integration/autoland/rev/ced6621fcb1b
Add a thread-safe composition recorder for WebRender r=kvark,kats
https://hg.mozilla.org/integration/autoland/rev/0ef19da1d641
Record compositions in Web Render r=kats
https://hg.mozilla.org/integration/autoland/rev/63568b2a8178
Clearly document the case of the RendererOGL receiving a new WebRenderCompositionRecorder while it has one r=kats

Comment 19

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/0c837987a829
https://hg.mozilla.org/mozilla-central/rev/caca3a7d4c2f
https://hg.mozilla.org/mozilla-central/rev/b09d27fc1584
https://hg.mozilla.org/mozilla-central/rev/58dd508e43e6
https://hg.mozilla.org/mozilla-central/rev/ced6621fcb1b
https://hg.mozilla.org/mozilla-central/rev/0ef19da1d641
https://hg.mozilla.org/mozilla-central/rev/63568b2a8178