1196398 - Video file > 3Gb doesn't play

Status: VERIFIED FIXED

(Core :: Audio/Video, defect, P1)

Description

[[e-mail]]

Attachment: ff_video_corrupt.png

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36

Steps to reproduce:


1. open firefox 
2. navigate to local file (file://) or remote url (http://) where you can open (stream) a video mp4 file which is > 3Gb. 

In my case I have a mp4 video file = 3.9 Gb. 

I've tested this on FF version 40 on Mac (2 machines) and virtual Windows 7.

Result is the same and doesn't matter if file is opened via http or locally.

File is valid - it plays in Chrome, Safary, IE 9.

Other files recorded on the same device which are < 1.9Gb works in FF without a problem.



Actual results:

Firefox shows a black page with the "Video can't be played because the file is corrupt" message. See attached file.


Expected results:

Video should be played

Comment 1

[Loic]

Regression range:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=1ad65cbeb2f4&tochange=4b9b12c248dc

Probably:
JW Wang — Bug 1159144 - check (-1, -1) for invalid video size. r=jya.

Comment 2

[Jean-Yves Avenard [:jya]]

I can't see how that bug could influence that behaviour.

but I just created a 3.9GB video and can reproduce the problem locally. So will take it

Comment 3

[Jean-Yves Avenard [:jya]]

Attachment: [mp4] Do not allocate memory spanning across ftyp and moov atom.

A typical non-fragmented mp4 would have the ftyp atom located at the beginning of the mp4 and the moov at the end. We would to calculate the location of the metadata by spanning the byte range of the two atoms.
As such, we would typically allocate an amount of memory equivalent to the size of the mp4.

Instead we now reconstruct the metadata to only have the ftyp and moov atoms contiguously.

Comment 4

[Jean-Yves Avenard [:jya]]

this needs to be uplifted ASAP.

Also very likely to what caused bug 1193094

Comment 5

[Jean-Yves Avenard [:jya]]

The reason it likely caused bug 1193094 is that we were reading the entire content of the mp4 rather than just seeking through it

Comment 6

[Loic]

(In reply to Jean-Yves Avenard [:jya] from comment #2)
> I can't see how that bug could influence that behaviour.

In the regression range, there is one of your bugfix too:  Bug 1156891: Disable use of stagefright::String8::clear

Anyway it regressed in FF40.

Comment 7

[Jean-Yves Avenard [:jya]]

Actually, the regression isn't as bad as I first thought.
And doesn't affect bug 1193094.

Most MP4 , all the one use by streaming site (YouTube, Dailymotion, deezer etc) would use MP4 where the mooov is located at the beginning. 
The only MP4 I knew up to now that had the mooov at the end were videos generated by the iPhone, they wouldn't be typically available via http.

Bug 1156891 certainly isn't the one causing the problem.

It's the one introducing the MP4Metadata object. I don't think that regression range is right

It's bug 1159027 and when MediaFormatReader became the default MP4 player

Comment 8

[Loic]

I tried again, same reg range.
With the 1st bad build (4b9b12c248dc), I get the error "MPEG4Extractor(5032): Error -1007 parsing chuck at offset looking for first track".

But whatever, the important thing is the patch. :)

Comment 9

[Jean-Yves Avenard [:jya]]

I know what you were seeing, it's the consequence of http://hg.mozilla.org/mozilla-central/rev/c99d19bfd6a0

We prevented reading atoms with a size > 2GB. As when capturing the metadata we captured the entire video, which will have an atom > 2GB. So it aborts.

So the patch attached gets around that as stagefright won't get called to parse that huge atom

Comment 12

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/73156610be5f

Comment 13

[rjain2]

BTW, what @dimitry84 says about reproducibility when the file size is > 3GB is NOT true. I just verified two instances of the same video --file size was > 4GB, the only difference was the moov was at the beginning of the file (just like Youtube etc as mentioned by @jya. I wish I could share the video but can't because its proprietary.

Now, one other thing to note, which is puzzling is that the video still plays in other Browsers (Chrome/Safari). Thats probably something for another day as I am not Video expert.

Comment 14

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/f062f4e35769

Comment 15

[Jean-Yves Avenard [:jya]]

(In reply to rjain2 from comment #13)
> Now, one other thing to note, which is puzzling is that the video still
> plays in other Browsers (Chrome/Safari). Thats probably something for
> another day as I am not Video expert.

There's nothing wrong with the video per say. it's perfectly valid.
While it's not optimised for web use it should play.

Comment 16

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7aebd5a8b8a8

Comment 17

[rjain2]

@jya : Maybe I should have clarified earlier. I am talking about playing it via a web server, not local file system. So, net net, Chrome and Safari are able to play the MP4 file containing a H.264 Baseline@3.1 when served from a web server even though the moov is at the end. From what I have been told Firefox used to be able to do that too pre-38. Something went wrong in v39 and above which broke that playback. So, there may be two separate issues: a) One regression in Firefox, b) Something else in the Video file that makes it work even though the moov is at the end. But I did want to give you information that I personally tried playing file sizes greater than 4GB without any problem...the only caveat being the moov atom has to be at the beginning. So, I am not quite sure what is going on because based on my understanding, thats how its supposed to be. Why it played before (with moov at the end) and why its broken now, thats what I am trying to figure out.

Comment 18

[[e-mail]]

Thank you for your efforts.
Our video is h.26
Let me know if you need some info from the video file. (meta properties, etc.)

I'll try to create a demo video > 3.9Gb for you which I would be able to share.

Comment 19

[Jean-Yves Avenard [:jya]]

No one said that a video with the mooov at the end can't be played. There was a bug in Firefox that is currently pending (should make Nightly tomorrow) that caused Firefox to read the entire file and store it in ram: that's bad. This is what caused the file > 2GB (not 3) to not be playing.

The bug was reported and now it's been fixed.

The only problem hosting a MP4 with a mooov at the end is that you must parse the entire file to find the mooov. This will cause lots a little http requests

Comment 20

[deheleri]

Jean, you keep mentioning the MOOV atom in both bug threads.

This has nothing to do with MOOV. Maybe that one guy's example mp4 was wrong, but I have a video site with 80+ MP4 videos that used to play fine in FF 34 and below, and now I get hundreds of HTTP requests when the files are played in FF 35+. They all have the moov at the beginning, and played fine. FF 41+ does not have this problem, and I have been waiting weeks for it to release while 36-40 all have the problem with multiple HTTP requests.

But MOOV is **NOT** the problem.

Comment 21

[Jean-Yves Avenard [:jya]]

Please read the bug description *and* the solution.

this bug is about not being able to play file > 3GB.

The reason for this was *exactly* because of where the moov was stored and as such this particular problem has all to do with where the moov is stored. *THAT* is the bug. And has now been fixed.

You're talking about a different issue where you're seeing small http requests.
In the other bug, the moov was also at the end, and due to the problem described in *this* bug, we would read the entire file content: hence the multiple http requests. That bug has been marked as a duplicate.

You have added comments in multiple bugs about a particular problem you are seeing , that may or may not looked similar to the original problem described.

If you're seeing a problem and you believe it's different, not "I kind of have the same issue", then please open a new bug and it will be investigated.

Comment 22

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/7aebd5a8b8a8

Comment 23

[Jean-Yves Avenard [:jya]]

Comment on attachment 8650317 [details] [diff] [review]
[mp4] Do not allocate memory spanning across ftyp and moov atom.

Approval Request Comment
[Feature/regressing bug #]: 1159027 
[User impact if declined]: Playback not working, out of memory crashes, excessive network requests.
[Describe test coverage new/current, TreeHerder]: in central, manual tests.
[Risks and why]: Low, it is logically what the previous code was doing, but now we just don't copy the data we don't need.
[String/UUID change made/needed]: None

Comment 24

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8650317 [details] [diff] [review]
[mp4] Do not allocate memory spanning across ftyp and moov atom.

Recent regression, taking it.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/f4b2c0381b88

Comment 26

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/32562add4eb6

Comment 27

[Loic]

Is there a reason to not take this bugfix for the FF40 dot release?

Comment 28

[Jean-Yves Avenard [:jya]]

FF40 isn't affected by this bug.

Comment 29

[Jean-Yves Avenard [:jya]]

Sorry, I should rephrase:

FF40 doesn't have the issue where the entire file may be loaded into RAM. It will just not play a file where an mp4 atom is > 2GB.

On 40, we rely on libstagefright to parse the mp4 and it will error on any block > 2GB (due to various security issues where it uses index internally that could overflow).

I can wrap something if you think it's necessary. It shouldn't take long.

Comment 30

[Vasilica Tamas, QA [:vtamas]]

I was unable to reproduce this issue on Firefox 40, Firefox 40.0.2 and Firefox 41.0a1 under Mac OS X 10.9.5 and Windows 7 64-bit.
Tried several times to reproduce it, without any luck. 

Dmitry, since you've reproduced this issue, could you please verify if it's now fixed?

Comment 31

[Loic]

On my side (Win 7 x64), I verified 4-GB or 5-GB video files are played locally by Nightly.
In addition, I verified the online MP4 provided in the duplicate https://bugzilla.mozilla.org/show_bug.cgi?id=1145608#c1 is played correctly. All is fine for me.

Comment 32

[Jean-Yves Avenard [:jya]]

the bug would only affect mp4 where the moov atom was at the end.

You can check with a utility like mp4dump (http://www.bento4.com/downloads/) if you do mp4dump filename.mp4

if you see the mdat atom prior the moov ; then the bug fixed here would apply.

Comment 33

[[e-mail]]

Hi Vasilica.

I can confirm that in FF nightly 43.0a1 (2015-08-26) for MAC the bug is fixed. 
Tested on 3.57Gb, 2.37 Gb and 1.57Gb files.

If someone need files to test, you can use these files. (I'll try to keep then until official version is released)

2.37Gb:
https://m1-osl.rec.vc/getvideofile?name=RGh5VUZ0azd3RWY5SThFZ3o4S2ZNUlBBeGQ2Zks5bEdxN2h3aU8yRGZMbWkrMlJ5V3duVTlnZWRVM0xOTnZHN09lYmVPN3hNL2NzOE40MjQrd2F4V1hsdGt1ak8rVzNteHVPSHBnU3NsUllMUzlvSk1rbGIxK2VqRWpBOUZ3VVpsak9kN2ZsUC8rZHowRW5uekllekFDa1JQeVRIOU1PYlY3TWFMYm5CUFJBPQ==



1.57Gb:

https://m1-osl.rec.vc/getvideofile?name=TzhhVFd0RTROVHUrN1NKTzVKTXRmUEpzNFh1LzNxMlZmaTQ4UXBZUVh5SmFvRUFWbW42UFhvdHBMWUV0bHJTdVIvRFZscXd5MytDR2FYY2kwUzZhYzNHMlIySEloUW8vYVoxcE1NWkp4RVRtUTFxTUxlUTg5ZXZzSEZwbU5iZzJYd2svbWduL2Y4MjRCbTQ5NGRHOEI1cSs1eTdvUEJTcDhUOW5xbUxCMGJzPQ==


I can not publish 3Gb file, but I guess if >2Gb works then the problem is solved.


Huge thanks for everyone involved in this issue. Hope to see this in the official release soon.

Comment 34

[Vasilica Tamas, QA [:vtamas]]

Thanks dmitry and Loic for your help!

Tested also myself on Firefox 43.0a1 (2015-08-28), Firefox 42.0a2 (2015-08-27) and Firefox 41 Beta 5 (20150827142634) under Windows 7 64-bit and Mac OS X 10.9.5.
Verified fixed using the testing files from Comment 33 and the MP4 video from the duplicated bug.

Based on Comment 31, Comment 33 and my testing, I am marking this bug as Verified Fixed.