1934009 - Implement Android encoder and decoder backends with NDK, via ffmpeg

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, task, P2)

Comment 1

[Paul Adenot (:padenot)]

Attachment: Bug 1934009 - Part 1. Add the MediaCodec NDK files to our ffmpeg copy.

Comment 2

[Paul Adenot (:padenot)]

Attachment: Bug 1934009 - Log PDM order on initialization to better understand what is being picked at run time.

Comment 3

[Paul Adenot (:padenot)]

Attachment: Bug 1934009 - Initialize FFmpeg PDM in GPU process on Android, behind a pref.

Comment 4

[Paul Adenot (:padenot)]

Attachment: Bug 1934009 - DO NOT LAND - Comment out remote android PDM to speed up iterating.

Comment 5

[Paul Adenot (:padenot)]

Attachment: Bug 1934009 - Allow running FFmpegDecoderModule in the GPU process on Android.

Comment 6

[Paul Adenot (:padenot)]

Attachment: Bug 1934009 - Part 5. Expose one more MediaCodec NDK related function through FFmpegLibWrapper.

Comment 7

[Paul Adenot (:padenot)]

Attachment: Bug 1934009 - Part 7. Teach FFmpegVideoDecoder to perform hardware decoding on Android.

Comment 8

[Paul Adenot (:padenot)]

Attachment: Bug 1934009 - Add a log module for VideoFrameContainer.

Comment 9

[Paul Adenot (:padenot)]

Attachment: Bug 1934009 - Add a log module for CompositableTransactionParent.

Comment 10

[Paul Adenot (:padenot)]

Attachment: Bug 1934009 - DO NOT LAND - flip some pref to ease testing Android MediaCodec NDK via ffmpeg

Comment 11

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 2. Add the ARM specializations to our ffmpeg copy.

Comment 12

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 4. Update ffmpeg config and build files to support Android.

Comment 13

[Andrew Osmond [:aosmond] (he/him)]

Attachment: WIP: Bug 1934009 - Save video sample durations in table for Android hardware decoding.

Comment 14

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 8. Make rendering pipeline support presenting SurfaceTextureImage from FFmpegVideoDecoder.

Comment 15

[Andrew Osmond [:aosmond] (he/him)]

Attachment: WIP: Bug 1934009 - Add required files from ffmpeg to build H264/HEVC Android HW support.

Comment 16

[Andrew Osmond [:aosmond] (he/him)]

Attachment: WIP: Bug 1934009 - Update ffmpeg config and build files to support Android HW decoding for H264/HEVC.

Comment 18

[Phabricator Automation]

Comment on attachment 9460784 [details]
Bug 1934009 - Log PDM order on initialization to better understand what is being picked at run time.

Revision D235023 was moved to bug 1977399. Setting attachment 9460784 [details] to obsolete.

Comment 19

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 2. Patch ffvpx to avoid importing unnecessary H264/HEVC dependencies.

Comment 20

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 3. Patch ffvpx to expose end-of-stream flag for MediaCodec decoders.

Comment 21

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 6. Allow the RDD process on Android.

Comment 22

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 9. Rework FFmpegDataDecoder draining to allow pausing/resuming.

Comment 23

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 10. Teach FFmpegAudioDecoder to perform hardware decoding on Android.

Comment 24

[Phabricator Automation]

Comment on attachment 9477260 [details]
Bug 1934009 - Part 2. Add the ARM specializations to our ffmpeg copy.

Revision D244497 was moved to bug 1977468. Setting attachment 9477260 [details] to obsolete.

Comment 25

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 5. Expose one more MediaCodec NDK related function through FFmpegLibWrapper.

Comment 26

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 1. Add the MediaCodec NDK files to our ffmpeg copy.

Comment 27

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 2. Patch ffvpx to avoid importing unnecessary H264/HEVC dependencies.

Comment 28

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 3. Patch ffvpx to expose end-of-stream flag for MediaCodec decoders.

Comment 29

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 4. Update ffmpeg config and build files to support Android.

Comment 30

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 5. Expose one more MediaCodec NDK related function through FFmpegLibWrapper.

Comment 31

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 6. Allow the RDD process on Android.

Comment 32

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 7. Teach FFmpegVideoDecoder to perform hardware decoding on Android.

Comment 33

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 8. Make rendering pipeline support presenting SurfaceTextureImage from FFmpegVideoDecoder.

Comment 34

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 9. Rework FFmpegDataDecoder draining to allow pausing/resuming.

Comment 35

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1934009 - Part 10. Teach FFmpegAudioDecoder to perform hardware decoding on Android.

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e3c9a784011a
https://hg.mozilla.org/integration/autoland/rev/73d48ef5571e
Part 1. Add the MediaCodec NDK files to our ffmpeg copy. r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/aac5bad7a131
https://hg.mozilla.org/integration/autoland/rev/35f161e1fa7d
Part 2. Patch ffvpx to avoid importing unnecessary H264/HEVC dependencies. r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/b7ff02f5b401
https://hg.mozilla.org/integration/autoland/rev/9efdb4e152ff
Part 3. Patch ffvpx to expose end-of-stream flag for MediaCodec decoders. r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/9b9cf2c4be7e
https://hg.mozilla.org/integration/autoland/rev/bd3ab9487871
Part 4. Update ffmpeg config and build files to support Android. r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/194a36f59d93
https://hg.mozilla.org/integration/autoland/rev/303d69a39570
Part 5. Expose one more MediaCodec NDK related function through FFmpegLibWrapper. r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/f6148c95eebc
https://hg.mozilla.org/integration/autoland/rev/6a648f21ccd9
Part 6. Allow the RDD process on Android. r=geckoview-reviewers,jolin,nalexander
https://github.com/mozilla-firefox/firefox/commit/ebe516ffc144
https://hg.mozilla.org/integration/autoland/rev/dd768909145d
Part 7. Teach FFmpegVideoDecoder to perform hardware decoding on Android. r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/1484fb0a4b09
https://hg.mozilla.org/integration/autoland/rev/5d78c5be46be
Part 8. Make rendering pipeline support presenting SurfaceTextureImage from FFmpegVideoDecoder. r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/7e9c2b3561df
https://hg.mozilla.org/integration/autoland/rev/df27149c03e3
Part 9. Rework FFmpegDataDecoder draining to allow pausing/resuming. r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/7076e12d217d
https://hg.mozilla.org/integration/autoland/rev/4cee7dd64eca
Part 10. Teach FFmpegAudioDecoder to perform hardware decoding on Android. r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/73f2a7040deb
https://hg.mozilla.org/integration/autoland/rev/13efd6515c88
apply code formatting via Lando

Comment 37

[agoloman]

https://hg.mozilla.org/mozilla-central/rev/73d48ef5571e
https://hg.mozilla.org/mozilla-central/rev/35f161e1fa7d
https://hg.mozilla.org/mozilla-central/rev/9efdb4e152ff
https://hg.mozilla.org/mozilla-central/rev/bd3ab9487871
https://hg.mozilla.org/mozilla-central/rev/303d69a39570
https://hg.mozilla.org/mozilla-central/rev/6a648f21ccd9
https://hg.mozilla.org/mozilla-central/rev/dd768909145d
https://hg.mozilla.org/mozilla-central/rev/5d78c5be46be
https://hg.mozilla.org/mozilla-central/rev/df27149c03e3
https://hg.mozilla.org/mozilla-central/rev/4cee7dd64eca
https://hg.mozilla.org/mozilla-central/rev/13efd6515c88

Comment 38

[Acasandrei Beatrice (needinfo me)]

(In reply to Pulsebot from comment #36)

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e3c9a784011a
https://hg.mozilla.org/integration/autoland/rev/73d48ef5571e
Part 1. Add the MediaCodec NDK files to our ffmpeg copy.
r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/aac5bad7a131
https://hg.mozilla.org/integration/autoland/rev/35f161e1fa7d
Part 2. Patch ffvpx to avoid importing unnecessary H264/HEVC dependencies.
r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/b7ff02f5b401
https://hg.mozilla.org/integration/autoland/rev/9efdb4e152ff
Part 3. Patch ffvpx to expose end-of-stream flag for MediaCodec decoders.
r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/9b9cf2c4be7e
https://hg.mozilla.org/integration/autoland/rev/bd3ab9487871
Part 4. Update ffmpeg config and build files to support Android.
r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/194a36f59d93
https://hg.mozilla.org/integration/autoland/rev/303d69a39570
Part 5. Expose one more MediaCodec NDK related function through
FFmpegLibWrapper. r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/f6148c95eebc
https://hg.mozilla.org/integration/autoland/rev/6a648f21ccd9
Part 6. Allow the RDD process on Android.
r=geckoview-reviewers,jolin,nalexander
https://github.com/mozilla-firefox/firefox/commit/ebe516ffc144
https://hg.mozilla.org/integration/autoland/rev/dd768909145d
Part 7. Teach FFmpegVideoDecoder to perform hardware decoding on Android.
r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/1484fb0a4b09
https://hg.mozilla.org/integration/autoland/rev/5d78c5be46be
Part 8. Make rendering pipeline support presenting SurfaceTextureImage from
FFmpegVideoDecoder. r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/7e9c2b3561df
https://hg.mozilla.org/integration/autoland/rev/df27149c03e3
Part 9. Rework FFmpegDataDecoder draining to allow pausing/resuming.
r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/7076e12d217d
https://hg.mozilla.org/integration/autoland/rev/4cee7dd64eca
Part 10. Teach FFmpegAudioDecoder to perform hardware decoding on Android.
r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/73f2a7040deb
https://hg.mozilla.org/integration/autoland/rev/13efd6515c88
apply code formatting via Lando

Perfherder has detected a browsertime performance change from push 13efd6515c880c58fa37e2ceecce3ef8724fe95a.

If you have any questions, please reach out to a performance sheriff. Alternatively, you can find help on Slack by joining #perf-help, and on Matrix you can find help by joining #perftest.

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
4%	vpl-av1 secondFrame	android-hw-a55-14-0-aarch64-shippable	warm webrender	213.62 -> 204.30	Before/After

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a performance sheriff to do that for you.

You can run all of these tests on try with ./mach try perf --alert 46163

The following documentation link provides more information about this command.