Resolution changes in vpx streams contained in webm files results in video not respecting original aspect ratio
Categories
(Core :: Audio/Video: Playback, defect, P3)
Tracking
()
People
(Reporter: bryce, Assigned: alwu, NeedInfo)
References
(Blocks 1 open bug, Regression)
Details
(Keywords: parity-chrome, regression)
Attachments
(4 files)
Following bug 1637658 if we encounter a webm container with a video stream that has resolution information that disagrees with the container, we trust the stream. When doing so we no longer honour the aspect ratio set by the container. For some streams, the stream itself can contain this info, but in the case of vp8, it does not.
For vp8 we had previously trusted the container, and Chrome appear to still have some logic that remembers the original ratio and maintains it. This can be seen with the attached file (captured during bug 1695033 -- it will play once that bug lands). The file will look squashed width-wise in Firefox (i.e. too narrow), but Chrome will display it in the intended aspect ratio.
Chrome appear to be specifically storing the aspect ratio based on the display width and height + the pixel width and height in the webm and using it to do stretching (you can inspect the attached webm with something like mkvtoolnix's info tool to see this).
This is pretty niche, and it would be ideal if we were just given correct files, so I don't know how much we want to fix this, but figure it's worth tracking. Following bug 1695033 a simple fix would be adding code here that checks if we're VP8 and then keeps the original display, but we could also get more clever and just store an aspect ratio/stretch value.
|
||
Comment 1•3 years ago
|
||
Set release status flags based on info from the regressing bug 1637658
|
||
Updated•3 years ago
|
This issue affects WebMs with VP9 videos too (however AV1 videos are not affected).
Using FFmpeg it's very easy to create examples, by setting the SAR (Sample Aspect Ratio) to anything but 1:1.
ffmpeg -i input.mkv -c:v libvpx-vp9 -vf scale=960:720,setsar=4/3 out.webm
The real world use case is for videos where the source is known to have non-square pixels (such as a DVD-Video)
|
|
||
Updated•3 years ago
|
I just wanted to say that yes, this bug is still extant in version 87, and how 'niche' it is depends largely on whether or not you are in the habit of streaming e.g. live TV via webm/VP8.
Previous versions of firefox were at least usable, in that they scaled to the pixel ratios extant at the point of invocation (although they did not track live format changes from e.g. 16:9 formats to 4:3) so it is clear that the legacy code at least recognises the correct format at one point in its lifetime :-)
I realise that this is not an issue many users would likely run into, but I just wanted to flag that its still there unfixed, and raise a hand to respectifully request time be devoted to it. :-)
|
Reporter | |
Comment 5•3 years ago
|
||
One of the problems is that there is no specified behaviour as to how to handle these changes in pixel format during stream for vp8. The defacto behaviour that I'd like for consistency with other browsers is pinning the original pixel aspect ratio -- so if a stream starts with non-square pixels and changes resolution we assume that the pixels are still not square. The difficulty is handling this across codecs. For vp8, that information is never sent in stream, only in the container (which is why we revert to square pixels), but this is not the case for all codecs.
What we probably want is a hierarchy of pixel aspect ratio (or similar) information with something like doesn't exit < codec level < container level for webm and where we only update the information if it's of equal or higher precedence than that already existing information.
I hope to look at this in the future, but do not have time in the immediate future to do so.
Well Bryce, I realise I am very unfamiliar with the details of video encodoing, to the extent that I am not sure what te difference between 'container' and 'stream' is in this context.
But in this specification:
https://www.webmproject.org/docs/container/#segment-information
it clearly states that the height and width in pixels, and display height and width, in pixels, are supported by what I conjecture is 'in stream' information.
Are you in fact saying that this information gives no clue as to the 'squareness' or otherwise of the pixels?
But that at some other level, the 'container' specifies the pixel 'squareness ratio' and the solution is to carry this 'pixel aspect ratio' along and apply it to every stream segment' ?
Ex of testing, that does not seem a hugely onerous task.
As you say, if I read you aright:
GLOBAL PIXEL RATIO=SQUARE; // default
IF (CONTAINER specifies pixelAspectRatio) THEN
SET (PIXEL_RATIO TO CONTAINER DATA)
IF (STREAM specifies pixelAspectRatio) THEN
SET (PIXEL_RATIO TO STREAM DATA)
..then subsequently multiply the given display in pixels by the above ratio to get the correct number of 'square' display pixels to scale to..
the key issue being not to reset pixel aspect ratio unless explicitly specified in the stream data , as in general a TV stream will maintain that as constant, only varying the display aspect ratio, in terms of pixels.
|
|
Reporter | |
Comment 7•3 years ago
•
|
||
In this case the WebM is the container and the Vp8 content inside it is the stream.
As you note, the WebM container has multiple dimensions, and indeed we can derive the 'squareness' of the pixels from it. If pixel dimensions and display dimensions are preset and those dimensions do not share the same ratio then we derive that pixels should not be square, and instead stretched in whatever dimension.
The Vp8 stream inside the WebM container has its own info (see https://tools.ietf.org/html/rfc6386). My understanding is that it carries something analogous to pixel dimensions, but not display dimensions -- though further investigating the stream format is part of the time this would take. Assuming it's correct that it only contains pixel dimensions, we would then need to handle resolution changes by remembering the ratio from the container and scaling the in stream changes by that amount (or just ignoring the in stream amount).
There are, however, streams that can be contained in WebMs that do contain more rich in stream info, and in those cases we'd need to be able to handle them changing the ratio. We'd also need to deal with if we had the same information in the container and which should be treated as more important. This presents cases further to the current issue:
- If the container has only pixel dimensions and the stream has something akin to pixel and display dimensions should we trust the stream? Probably.
- If both container and stream has pixel and display dimensions which should we trust? I think the container (if for no other reason than that's what some other browsers do), but it's debatable.
There's also handling the similar case for mp4s and taking into consideration how this interacts with other specifications such as MediaSource Extensions. Part of the reason for the regressing bug changing this behaviour was down to drivers interacting poorly with different dimension data, and that bug didn't land any tests, so we'd want to make sure any changes don't cause issue there.
So at a high level it boils down to something like the if else above, but understanding the different cases and the actual plumbing of those changes through the code base would take time.
I have an almost usable workaround for this bug, which may help others.
Html:
<Video style=" width: 900px; height: 500px; object-fit:fill" id="videoplayer" controls preload="auto" >
does at least ensure that the video obect is stretched to fit a reasonable aspect ratio frame.
Further comment: the above breaks on picture - in - picture, and in fullscreen as it appears that the container size is not carried through...
|
||
Updated•3 years ago
|
|
||
Updated•3 years ago
|
|
Assignee | |
Comment 12•2 years ago
|
||
Already have a patch, and will submit it later.
|
|
Assignee | |
Updated•2 years ago
|
|
|
Assignee | |
Comment 13•2 years ago
|
||
Sometime we found that the DAR (display aspect ration) will be inconsistent between the one described in the WebM container and the one in bytestream, and we want to trust the one in the container more.
For VP8, their frame header doesn't have anything to describe aspect ratio, so to trust the container will definitely a good thing.
For VP9, render_and_frame_size_different [1] can be used to describe DAR, but we found that there are some videos didn't set that attribute correctly in bytestream, which will cause us to calculate an incorrect result that is against the DAR specified in the container.
Therefore, if the DAR in the bytestream is different from the one in the container, we will tend to trust the container more.
|
|
Assignee | |
Comment 14•2 years ago
|
||
Depends on D133559
|
|
Reporter | |
Comment 15•2 years ago
|
||
For reference, this is one of the Chromium bugs I'd looked at while thinking about this previously: https://bugs.chromium.org/p/chromium/issues/detail?id=837337
The first comment there provides some more context for different containers and streams.
|
|
Assignee | |
Comment 16•2 years ago
|
||
Depends on D133560
|
||
Comment 17•2 years ago
|
||
Pushed by alwu@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/62039c0255e0 part1 : for vpx videos, update the display configuration only when the new DAR in byte stream matches the one in the container. r=bryce https://hg.mozilla.org/integration/autoland/rev/2f00990d4ee9 part2 : add reftests to ensure VPX video being displayed in correct DAR. r=bryce https://hg.mozilla.org/integration/autoland/rev/668eb55faa54 part3 : add log for VPXChangeMonitor. r=bryce
|
||
Comment 18•2 years ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/62039c0255e0
https://hg.mozilla.org/mozilla-central/rev/2f00990d4ee9
https://hg.mozilla.org/mozilla-central/rev/668eb55faa54
|
||
Updated•2 years ago
|
|
||
Updated•2 years ago
|
|
||
Comment 19•2 years ago
|
||
First of all, I finally got the fixed code in my normal distro release channel, and yes it is fixed....almost.
It is now back to what it always used to be: Namely selecting a webm TV stream will display the correct aspect ratio. Picture in picture now works beautifully as well...
...until that aspect ratio changes.
In the UK this is typically when an old 4:3 picture stream is interspersed with 16:9 commercial content, or when an old 4:3 stream finishes or begins.
Closing the stream and reopening it restores the correct aspect ratio so it would seem that it is simply (Ha!) a matter of changing a one-time examination of the stream to a polled approach. Actual TVs when faced with the same content will glitch slightly at the transition anyway.
Whether or not this is simply too much work to fix, for too few users, I will leave to the team. But it remains an - albeit small - issue.
But I will flag it here anyway.
|
|
Assignee | |
Comment 20•2 years ago
|
||
Thank for reporting this! Would you mind to file another bug, which depends on this bug, to track that issue?
Thanks!
|
|
Reporter | |
Comment 21•2 years ago
|
||
FWIW, I've crafted a whole lot of other cases where we still do not do the right thing and/or do not have Chrome parity. Sadly it's on a machine that's on a cargo ship somewhere, but once the machine arrives (may be a couple of months yet), I'll chuck them on bug 1746949 (or can recraft them if it's urgent).
Description
•