1159171 - Enable ffmpeg on Linux platforms for media mochitests

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect)

Description

[JW Wang [:jwwang] [:jw_wang]]

We want to exercise mp4 reader/demuxer code as much as possible.

Comment 1

[JW Wang [:jwwang] [:jw_wang]]

Attachment: 1159171_enable_ffmpeg_on_linux-v1.patch

Comment 2

[JW Wang [:jwwang] [:jw_wang]]

Got such error messages when playing gizmo.mp4 on Try:

01:51:00     INFO -  [aac @ 0x7f3e2c2cd800] err{or,}_recognition separate: 1; 1
01:51:00     INFO -  [aac @ 0x7f3e2c2cd800] err{or,}_recognition combined: 1; 1
01:51:00     INFO -  [aac @ 0x7f3e2c2cd800] Unsupported bit depth: 0
01:51:00     INFO -  [h264 @ 0x7f3e2c2d0000] err{or,}_recognition separate: 1; 1
01:51:00     INFO -  [h264 @ 0x7f3e2c2d0000] err{or,}_recognition combined: 1; 1
01:51:00     INFO -  [h264 @ 0x7f3e2c2d0000] Unsupported bit depth: 0

Not a single video is decoded where FrameStatistics::GetParsedFrames() and FrameStatistics::GetDecodedFrames() return zeros.

Btw, I can't repro the problem on my computer where LIBAV_VER is 54 while it is 53 on Try.


Hi Edwin,
Do you have any clue about the error messages?

Comment 3

[Edwin Flores [inactive from 2016-12-01] [:eflores] [:edwin]]

(In reply to JW Wang [:jwwang] from comment #2)
> Do you have any clue about the error messages?

Looks like ABI incompatibility. Jean-Yves knows a lot more about this.

Comment 4

[Jean-Yves Avenard [:jya]]

Comment on attachment 8598517 [details] [diff] [review]
1159171_enable_ffmpeg_on_linux-v1.patch

Review of attachment 8598517 [details] [diff] [review]:
-----------------------------------------------------------------

With this patch, making FFmpeg by default will mean that on mac with ffmpeg compilation enabled, the Apple VDA and VT (hardware accelerated) will never be use anymore (as PlatformDecoder::Create() will check for FFmpeg and if found stop trying new decoders)
PlatformDecoderModule::Create() should be modified so the apple decoders are tried first, and still have an option to force the use of ffmpeg.

Comment 5

[Jean-Yves Avenard [:jya]]

(In reply to JW Wang [:jwwang] from comment #2)

> Btw, I can't repro the problem on my computer where LIBAV_VER is 54 while it
> is 53 on Try.

which LIBAV_VER is that? libav or ffmpeg ?

Comment 6

[JW Wang [:jwwang] [:jw_wang]]

Comment on attachment 8598517 [details] [diff] [review]
1159171_enable_ffmpeg_on_linux-v1.patch

Review of attachment 8598517 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/profiles/prefs_general.js
@@ +172,5 @@
>  
> +// Enable fragmented MP4 parser for testing
> +user_pref("media.fragmented-mp4.exposed", true);
> +
> +#if defined(LINUX)

Is LINUX also defined on Mac? If not, this change should affect Linux platforms only and Mac can still prefer Apple VDA and VT.

Comment 7

[Jean-Yves Avenard [:jya]]

(In reply to JW Wang [:jwwang] from comment #6)
> Comment on attachment 8598517 [details] [diff] [review]
> Is LINUX also defined on Mac? If not, this change should affect Linux
> platforms only and Mac can still prefer Apple VDA and VT.

oh, you're right... my bad.

Comment 8

[JW Wang [:jwwang] [:jw_wang]]

(In reply to Jean-Yves Avenard [:jya] from comment #5)
> which LIBAV_VER is that? libav or ffmpeg ?

https://hg.mozilla.org/mozilla-central/file/617dbce26726/dom/media/fmp4/ffmpeg/FFmpegLibs.h#l27

It is the version number of libavformat.

Comment 9

[Jean-Yves Avenard [:jya]]

Attachment: Enable ffmpeg on Linux platforms for media mochitests

Rebasing

Comment 10

[Jean-Yves Avenard [:jya]]

(In reply to JW Wang [:jwwang] from comment #8)
> (In reply to Jean-Yves Avenard [:jya] from comment #5)
> > which LIBAV_VER is that? libav or ffmpeg ?
> 
> https://hg.mozilla.org/mozilla-central/file/617dbce26726/dom/media/fmp4/
> ffmpeg/FFmpegLibs.h#l27
> 
> It is the version number of libavformat.

sure... but which libavformat is it? From FFmpeg (www.ffmpeg.org) or LibAV (www.libav.org). And which version of this is it ?

ABI compatibility is a bit hard to maintain because that while similar they all use their own. Just trying to reproduce the problem locally.

Comment 11

[Jean-Yves Avenard [:jya]]

libavformat doesn't appear to be install on the try machine. Only gstreamer:

from mgerva:
find /builds/mock_mozilla/ -type f | grep -i libav
/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/lib64/libavahi-common.so.3.5.1
/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/lib64/libavahi-glib.so.1.0.1
/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/lib64/libavahi-client.so.3.2.5
/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/lib64/libavahi-core.so.6.0.1
/builds/mock_mozilla/mozilla-centos6-x86_64-android/root/usr/lib64/libavahi-common.so.3.5.1
/builds/mock_mozilla/mozilla-centos6-x86_64-android/root/usr/lib64/libavahi-client.so.3.2.5

[[e-mail] ~]# find /builds/mock_mozilla/ -type f | grep -i ffm
/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/include/gtk-2.0/gtk/gtktearoffmenuitem.h
/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/lib64/gstreamer-0.10/libgstffmpegcolorspace.so
/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/bin/tiffmedian
/builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/share/man/man1/tiffmedian.1.gz
/builds/mock_mozilla/mozilla-centos6-x86_64-android/root/usr/bin/tiffmedian
/builds/mock_mozilla/mozilla-centos6-x86_64-android/root/usr/share/man/man1/tiffmedian.1.gz

[[e-mail] ~]# ls /builds/mock_mozilla/mozilla-centos6-x86_64/root/usr/lib64/gstreamer-0.10
libgstadder.so         libgstaudiorate.so      libgstcoreelements.so  libgstffmpegcolorspace.so  libgstogg.so       libgsttcp.so                libgstvideorate.so     libgstvorbis.so
libgstalsa.so          libgstaudioresample.so  libgstcoreindexers.so  libgstgdp.so               libgstpango.so     libgsttheora.so             libgstvideoscale.so    libgstximagesink.so
libgstapp.so           libgstaudiotestsrc.so   libgstdecodebin2.so    libgstgio.so               libgstplaybin.so   libgsttypefindfunctions.so  libgstvideotestsrc.so  libgstxvimagesink.so
libgstaudioconvert.so  libgstcdparanoia.so     libgstdecodebin.so     libgstlibvisual.so         libgstsubparse.so  libgstvideo4linux.so        libgstvolume.so
[[e-mail] ~]#

maybe that's why it's just not working.
Now, what is bizarre is that if libavformat isn't installed, we shouldn't even reach that part of the code.

Comment 12

[Jean-Yves Avenard [:jya]]

Would you have a link to the try test?

Comment 13

[JW Wang [:jwwang] [:jw_wang]]

(In reply to Jean-Yves Avenard [:jya] from comment #10)
> (In reply to JW Wang [:jwwang] from comment #8)
> > (In reply to Jean-Yves Avenard [:jya] from comment #5)
> > > which LIBAV_VER is that? libav or ffmpeg ?
> > 
> > https://hg.mozilla.org/mozilla-central/file/617dbce26726/dom/media/fmp4/
> > ffmpeg/FFmpegLibs.h#l27
> > 
> > It is the version number of libavformat.
> 
> sure... but which libavformat is it? From FFmpeg (www.ffmpeg.org) or LibAV
> (www.libav.org). And which version of this is it ?
I don't even know both FFmepg and libAV provide libavformat.so. I have /usr/lib/x86_64-linux-gnu/libavformat.so.54.20.3 installed from libav.org.

> ABI compatibility is a bit hard to maintain because that while similar they
> all use their own. Just trying to reproduce the problem locally.
How is ABI compatibility concerned here? Can you detail the problem?

The last link is long gone. I will re-run Try to get a new link.

Comment 14

[Jean-Yves Avenard [:jya]]

(In reply to JW Wang [:jwwang] from comment #13)
> (In reply to Jean-Yves Avenard [:jya] from comment #10)
> > (In reply to JW Wang [:jwwang] from comment #8)
> > > (In reply to Jean-Yves Avenard [:jya] from comment #5)
> > > > which LIBAV_VER is that? libav or ffmpeg ?
> > > 
> > > https://hg.mozilla.org/mozilla-central/file/617dbce26726/dom/media/fmp4/
> > > ffmpeg/FFmpegLibs.h#l27
> > > 
> > > It is the version number of libavformat.
> > 
> > sure... but which libavformat is it? From FFmpeg (www.ffmpeg.org) or LibAV
> > (www.libav.org). And which version of this is it ?
> I don't even know both FFmepg and libAV provide libavformat.so. I have
> /usr/lib/x86_64-linux-gnu/libavformat.so.54.20.3 installed from libav.org.

libAV is a fork of FFmpeg, they both install the same library names: libavcodec, libavformat, libavutil.

> 
> > ABI compatibility is a bit hard to maintain because that while similar they
> > all use their own. Just trying to reproduce the problem locally.
> How is ABI compatibility concerned here? Can you detail the problem?

The problem is that we based on which code we are going to run on the libavformat version we find:
libavformat.53/54/55 from that we assume that libavutil and libavcodec implement a particular ABI which unfortunately is completely wrong depending on which version is installed locally: ffmpeg or libav.
Using libavformat so version was a poor choice, libavcodec would have been better choice but that one changes at a much greater rate than libavformat :(

In particular how a frame is allocated use different set of API:
av_frame_alloc, avcodec_alloc_frame or allocating on the stack and called avcodec_get_frame_defaults on it.
Those are found either in libavcodec or libavutils.

I've done my best to make a universal code and to test all versions: FFmpeg from version 0.9 to 2.4 (every public releases) and LibAV version 0.8 to version 10.0 (since I last tried, version 11 came out).

I've just configured locally libav 0.8.15 (ubuntu 12.04 uses libav 0.8.17) and I can't reproduce the issue you first mentioned.

I ran into segfault with FFmpeg 0.8, but it crashes completely outside the media code, so I'm suspecting some memory corruption somewhere.

> 
> The last link is long gone. I will re-run Try to get a new link.

I'm running one here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=492a35e88e2b

Comment 15

[Jean-Yves Avenard [:jya]]

looking good so far... the failure are due to the tests now working.

do you want me to take over this bug and update all the webref test?

Comment 16

[Jean-Yves Avenard [:jya]]

Ok.. I can reproduce a problem locally with an old libav.

Now, the question remain on if it's a bug in libav and if we can get around it, or just have to keep the ffmpeg code disable until try machines are upgraded with a newer version.

Comment 17

[JW Wang [:jwwang] [:jw_wang]]

Sure. Please take this bug.

Comment 18

[Jean-Yves Avenard [:jya]]

Attachment: Part2. Fix pts calculations in older LibAV version

Get around invalid PTS calculations in buggy LibAV 0.8.5.

Comment 19

[Jean-Yves Avenard [:jya]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=998cef71bc77

Still getting failure with libav 0.8.5. 

I'm afraid it's too buggy to be usable ; locally I can get it to crash rather easily. It frames with the same pts several times in a row too causing some test to fail.

May have to give up trying to get it to work with ubuntu 12.04 / libav 0.8 and wait for a newer version

Comment 20

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/18f7b45c54a5

Comment 21

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/18f7b45c54a5

Comment 22

[Jean-Yves Avenard [:jya]]

Will continue on in bug 1206979 ; I've now identified all issues preventing us to decode with libav 0.8.5 (as used on try machines)

Comment 23

[Jean-Yves Avenard [:jya]]

Actually, no .. will need to push JW's patch here.

Comment 24

[Jean-Yves Avenard [:jya]]

Attachment: [MSE] P3. Update webref results.

Comment 25

[Jean-Yves Avenard [:jya]]

Attachment: [MSE] P4. Changing waiting time value.

When ffmpeg is enabled, it will use the FFmpeg's VPX decoder. FFmpeg appears to always buffer 15 frames before returning one (this is the same with h264) causing the waiting event to be fired much earlier than when using libvpx

Comment 26

[Jean-Yves Avenard [:jya]]

Comment on attachment 8664082 [details] [diff] [review]
[MSE] P3. Update webref results.

got a failure for the same reason as P3 of this bug.

timeout due to ffmpeg buffering much more frames than libvpx.

Comment 27

[Jean-Yves Avenard [:jya]]

Attachment: [MSE] P3. Update webref results.

Disabling test mediasource-append-buffer.html on linux due to TIMEOUT.
FFmpeg doesn't output anything before being fed 15 frames ; and the webm test file first cluster only has 12 frames (they only feed the first cluster).

Comment 28

[Karl Tomlinson (:karlt)]

Comment on attachment 8664104 [details] [diff] [review]
[MSE] P3. Update webref results.

>Bug 1159171: [MSE] P3. Update webref results. r=karlt

Please document in the commit message what change is producing different test
results.

>-      if not debug and (os == "linux") and (version == "Ubuntu 12.04") and (processor == "x86") and (bits == 32): PASS
>-      if debug and (os == "win") and (version == "5.1.2600") and (processor == "x86") and (bits == 32): PASS
>-      if not debug and (os == "linux") and (version == "Ubuntu 12.04") and (processor == "x86_64") and (bits == 64): PASS
>-      if debug and (os == "linux") and (version == "Ubuntu 12.04") and (processor == "x86_64") and (bits == 64): PASS
>-      if debug and (os == "linux") and (version == "Ubuntu 12.04") and (processor == "x86") and (bits == 32): PASS

I assume the 'debug and (os == "win")' change is not intentional.

Comment 29

[Jean-Yves Avenard [:jya]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=75a160984e50

Comment 30

[Jean-Yves Avenard [:jya]]

no idea what this test is doing, but using ffvpx as decoder makes it fail with dom/media/test/test_mediarecorder_bitrate.html | seek.webm encoded@1000000=5317 > encoded@100000=5504

bug 1161276.

mreavy, what could this be the reason?

The only difference between using ffvpx over libvpx for decoding is that ffvpx buffers 15 frames before outputting a frame while libvpx will output one immediately (reading to the end of the file will cause the same amount of frames to be output).

This test is about verifying that the encoder produced data with a lower bitrate ....

I can't reproduce the problem locally, regardless of the version of ffvpx used (including 0.8.5 which is the version used by try machines)

Comment 31

[Jean-Yves Avenard [:jya]]

looks like it's just intermittent results , and my guess is that as ffvpx is significantly faster than libvpx for VP8 (like 10* as fast) it just expose a problem in the encoder (seeing that it's fed data much quicker than it used to be)

Comment 32

[Maire Reavy [:mreavy]]

Hi Jean-Yves -- I don't see how timing could impact this.  Here's what I know: there were no tests in the tree encoding decoded video (only gUM video) before my bug landed.  MediaRecorder only supports a limited set of formats which is being fixed in bug 1182426.  I suggest you try with the patches on that bug and see if that fixes your issues.  You can also dump the resulting blobs and examine the files.  The test should dump out info with bitrates and sizes.

Comment 33

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/f4253f28d4ca
https://hg.mozilla.org/integration/mozilla-inbound/rev/6419850d7cbb
https://hg.mozilla.org/integration/mozilla-inbound/rev/408b38dc976d

Comment 34

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/f4253f28d4ca
https://hg.mozilla.org/mozilla-central/rev/6419850d7cbb
https://hg.mozilla.org/mozilla-central/rev/408b38dc976d

Comment 35

[Jean-Yves Avenard [:jya]]

Comment on attachment 8603984 [details] [diff] [review]
Enable ffmpeg on Linux platforms for media mochitests

This is an approval for all but part 2.

Approval Request Comment
[Feature/regressing bug #]: 1214943
[User impact if declined]: Will cause mochitest to fail.
[Describe test coverage new/current, TreeHerder]: In central for a few weeks.
[Risks and why]: Low, at worse mochitest failure now that we are testing that a component is properly running (it wasn't tested before)
[String/UUID change made/needed]: None

Comment 36

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8603984 [details] [diff] [review]
Enable ffmpeg on Linux platforms for media mochitests

Approved for uplift to aurora. Tests for ffmpeg. jya will do the uplifts.

Comment 37

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/b49459989720
https://hg.mozilla.org/releases/mozilla-aurora/rev/816f830d4b6a
https://hg.mozilla.org/releases/mozilla-aurora/rev/9e3e16bf7718