applying width/height/framerate constraints to getUserMedia/getDisplayMedia track clone affects original track
Categories
(Core :: WebRTC: Audio/Video, defect, P2)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox147 | --- | fixed |
People
(Reporter: avade, Assigned: pehrsons)
References
Details
(Keywords: dev-doc-needed)
Attachments
(23 files, 1 obsolete file)
|
1.38 KB,
text/html
|
Details | |
|
1.25 KB,
text/html
|
Details | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review |
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36
Steps to reproduce:
- Open the attached html file in Firefox
- Click the button to trigger getDisplayMedia API
- Notice that getSettings() info on the original track and cloned track is both incorrect (height and width is 0)
- The original track now has lower framerate of 2, instead of original framerate of 24
Actual results:
- The original track has lower framerate of 2, instead of original framerate of 24
- The height and width has incorrect values
Expected results:
- Applying constraints on cloned track should not affect the original track
- The height and width should have correct values
We just discovered this happens with getUserMedia too. Attaching another html file for the getUserMedia issue with same steps to reproduce the issue.
Comment 3•4 years ago
|
||
Thanks for filing and diagnosing this bug! I have a couple of questions that would help us triage this issue. Arjun, do you believe that this bug is limited to macOS? Is this something that worked in previous versions?
- Seems to be macOS only issue. This seems to be working in windows.
- I'm not really sure if this is a regression. It may be one though.
| Comment hidden (obsolete) |
Comment 6•1 year ago
•
|
||
I've confirmed this is still an issue for both gUM and gDM (and not just for max constraints despite what what comment 5 says).
I think we need to up-prioritize this as it's likely to become more problematic with these upcoming features:
- bug 1286945 resizeMode (currently in Nightly as
media.navigator.video.resize_mode.enabled=truein about:config) - bug 1749543 where it stands to create an unfortunate side-channel between tracks on main thread and clones in a worker
STR:
- Open https://jan-ivar.github.io/dummy/gum_head2head.html
- Click the
Camera Abutton and allow camera - In the selector to the right of the
Camera Bbutton, select "clone of Camera A" - Adjust the size of A to 640x480
Expected result (matching Chrome and Safari):
- Video A is 640x480 and video B is 1280x720 (assuming your camera has this mode)
Actual result (Firefox bug and nonstandard behavior):
- Both Video A and B are 640x480
Workaround
- In step 3, instead of "clone of Camera A", pick the SAME camera by name.
The workaround shows we already have a code path (calling gUM twice) that produces the desired behavior. The task here would be to reuse that path somehow for the clone() method.
Updated•1 year ago
|
Updated•1 year ago
|
Updated•1 year ago
|
| Assignee | ||
Updated•1 year ago
|
| Assignee | ||
Comment 7•1 year ago
|
||
| Assignee | ||
Comment 8•1 year ago
|
||
Note, no sources implement it yet.
| Assignee | ||
Comment 9•1 year ago
|
||
| Assignee | ||
Comment 10•1 year ago
|
||
This is needed because MediaEngineSource::Stop/Deallocate are not idempotent,
and cloning allocates async after creating DeviceState, unlike getUserMedia.
| Assignee | ||
Comment 11•1 year ago
|
||
| Assignee | ||
Comment 12•1 year ago
|
||
Only if the same window is already capturing the same device, like in cloning
cases.
This patch also renames some VideoEngine members to make their purpose clearer.
| Assignee | ||
Comment 13•1 year ago
|
||
| Assignee | ||
Comment 14•1 year ago
|
||
| Assignee | ||
Comment 15•1 year ago
|
||
| Assignee | ||
Comment 16•1 year ago
|
||
Native cloning, which behaves like an independent gUM request, ends up using a
shared camera backend instance that is already live and does not need
reconfiguration.
The v4l2loopback device we set up on linux only issues a single frame during a
session. And since with deep cloning the session is reused, the new clone never
sees a frame, causing test_gUM_mediaStreamTrackClone.html, and intermittently
test_gUM_mediaElementCapture_tracks.html, to time out.
The "real fake" camera behaves like a normal camera in that it issues new frames
as requested, while still using the native capture stack.
| Assignee | ||
Comment 17•1 year ago
|
||
| Assignee | ||
Comment 18•1 year ago
|
||
| Assignee | ||
Comment 19•1 year ago
|
||
| Assignee | ||
Comment 20•1 year ago
|
||
| Assignee | ||
Comment 21•1 year ago
|
||
| Assignee | ||
Comment 22•1 year ago
|
||
| Assignee | ||
Comment 23•1 year ago
|
||
This allows more flexibility with methods explicitly taking an arbitrary item, e.g.:
struct State {
int mId;
bool mSuccess;
};
nsTArray<State> states;
int id{};
(...)
size_t idx = states.IndexOf(
/*aItem=*/id, /*aStart=*/0,
[](const State& aState, const int& aItem) { return aState.mId - aItem; });
| Assignee | ||
Comment 24•1 year ago
|
||
Prior to this commit, a site that made two gUM requests for the same device,
would receive two DeliverFrame IPC calls with distinct shmems containing
identical images. This is unnecessary, and a larger problem when tracks are
deep-cloned, i.e. where cloning a track has the same effect as a separate
request for the same device.
With this commit, only a single shmem is sent for each device on a single
PCameras channel.
| Assignee | ||
Comment 25•1 year ago
|
||
This mitigates the risk of exhausting the ShmemPool when many distinct sources
are live.
| Assignee | ||
Comment 26•1 year ago
|
||
This is safe as the image buffer backing webrtc::VideoFrame is refcounted.
Updated•1 year ago
|
Updated•1 year ago
|
Comment 27•11 months ago
|
||
Comment 28•11 months ago
|
||
Comment 29•11 months ago
•
|
||
Backed out for causing multiple failures
Failure log bc browser_devices_get_user_media_in_xorigin_frame.js
Failure log Shmem.cpp
Failure log Assertions.h
Failure log mda
| Assignee | ||
Comment 30•11 months ago
|
||
Failure to send stop does not mean the capturer is still running. Treat it as
stopped.
| Assignee | ||
Comment 31•11 months ago
|
||
Pernosco helped with fixing three issues that the browser-chrome media tests hit and the regular ones didn't. But I see a TSAN-specific test timeout at 100% error rate that I'm still debugging.
| Assignee | ||
Comment 32•11 months ago
|
||
| dev-doc-info | ||
dev-doc-needed to add a release note for developers. Proposed text:
The
clone()method onMediaStreamTrack, when used on tracks fromgetUserMediaorgetDisplayMedia, will now create a track that has an independent set of constraints and settings from the original track. This allows a track clone to capture a different resolution and/or framerate than that of the track it was cloned from.
| Assignee | ||
Updated•11 months ago
|
Comment 33•11 months ago
|
||
Comment 34•11 months ago
|
||
Comment 35•11 months ago
|
||
Backed out for causing bc failure on browser_devices_get_user_media_paused.js
Backout link: https://hg-edge.mozilla.org/integration/autoland/rev/dc0cb67dadd1cddf7b8604f07e88536be36a2eef
Log link: https://treeherder.mozilla.org/logviewer?job_id=532759975&repo=autoland&task=CYUoMGTCSS2ngcphBCT4JQ.0&lineNumber=4348
| Assignee | ||
Comment 36•11 months ago
|
||
With deep cloning, instead of a single device listener for all tracks
originating from the same source, there is one per track. A device listener
isn't aware of other device listeners when notifying chrome, which means the
number of recording-device-events messages has changed.
Note that setting a track's enabled attribute triggers the chrome notification
in an async task. Stopping a track on the other hand triggers the chrome
notification synchronously. Multiple notifications in a single task are
coalesced into a single chrome event (from stable state).
| Assignee | ||
Comment 37•10 months ago
|
||
This is ready again, but I'll hold until the merge.
Comment 38•10 months ago
|
||
Comment 39•10 months ago
|
||
Comment 40•10 months ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/1bd8b95b92d5
https://hg.mozilla.org/mozilla-central/rev/8dcb159656ce
https://hg.mozilla.org/mozilla-central/rev/468853f413f1
https://hg.mozilla.org/mozilla-central/rev/7c5372a83b4f
https://hg.mozilla.org/mozilla-central/rev/3cf0a1340ec4
https://hg.mozilla.org/mozilla-central/rev/803f1292d1d3
https://hg.mozilla.org/mozilla-central/rev/8583133a99ed
https://hg.mozilla.org/mozilla-central/rev/d17ac92e4bb1
https://hg.mozilla.org/mozilla-central/rev/dbf6b2cbb4ab
https://hg.mozilla.org/mozilla-central/rev/a333081ac05e
https://hg.mozilla.org/mozilla-central/rev/47a9a530cdeb
https://hg.mozilla.org/mozilla-central/rev/5ff03b3ab159
https://hg.mozilla.org/mozilla-central/rev/ba7f8713dfcf
https://hg.mozilla.org/mozilla-central/rev/9ed5a58f520e
https://hg.mozilla.org/mozilla-central/rev/fae4cc35f0e7
https://hg.mozilla.org/mozilla-central/rev/d87b20e0d2c8
https://hg.mozilla.org/mozilla-central/rev/1f6e30c20c6c
https://hg.mozilla.org/mozilla-central/rev/62cdf4cd9fa8
https://hg.mozilla.org/mozilla-central/rev/ed825825e907
https://hg.mozilla.org/mozilla-central/rev/dd5ee0b73c8d
https://hg.mozilla.org/mozilla-central/rev/828e97f9302c
Comment 41•10 months ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/06764ffdd391
https://hg.mozilla.org/mozilla-central/rev/80c2100f2c1c
https://hg.mozilla.org/mozilla-central/rev/ffbc0fdae4bb
https://hg.mozilla.org/mozilla-central/rev/8dbda1488cfb
https://hg.mozilla.org/mozilla-central/rev/5ab89bf2d2ad
https://hg.mozilla.org/mozilla-central/rev/2f9ce82dc126
https://hg.mozilla.org/mozilla-central/rev/2d91152ef032
https://hg.mozilla.org/mozilla-central/rev/d7d2acb93e26
https://hg.mozilla.org/mozilla-central/rev/63306413a489
https://hg.mozilla.org/mozilla-central/rev/1c6dabc4b130
https://hg.mozilla.org/mozilla-central/rev/e418cbdfb1d7
https://hg.mozilla.org/mozilla-central/rev/0067817ee3f7
https://hg.mozilla.org/mozilla-central/rev/a007a8a37635
https://hg.mozilla.org/mozilla-central/rev/d40d20195aa5
https://hg.mozilla.org/mozilla-central/rev/d506f631bd2a
https://hg.mozilla.org/mozilla-central/rev/12cd3fc71c1d
https://hg.mozilla.org/mozilla-central/rev/3f145a6de52d
https://hg.mozilla.org/mozilla-central/rev/8de64b254b78
https://hg.mozilla.org/mozilla-central/rev/384fd1bd965b
https://hg.mozilla.org/mozilla-central/rev/0d7174770027
https://hg.mozilla.org/mozilla-central/rev/3332b9ff2c08
Updated•9 months ago
|
Updated•5 months ago
|
Description
•