1451394 - Update video capture code on OS X

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, enhancement, P2)

Description

[Andreas Pehrson [:pehrsons]]

It looks like the code in [1] is old (pre webrtc.org 57 merge) and does no longer exist upstream. The current upstream for android capture seems to be [2].

We should also check whether we sit on similar outdated implementations for other platforms, even the ones where webrtc.org holds an implementation, it might be outdated.


[1] https://searchfox.org/mozilla-central/rev/a0665934fa05158a5a943d4c8b277465910c029c/media/webrtc/trunk/webrtc/modules/video_capture/android/java/src/org/webrtc/videoengine/VideoCaptureAndroid.java
[2] https://cs.chromium.org/chromium/src/media/capture/video/android/java/src/org/chromium/media/VideoCaptureCamera.java

Comment 1

[Dan Minor [:dminor]]

Another thing to check out are the devicechange event handlers added to media/webrtc/trunk/webrtc/modules/video_capture/linux/device_info_linux.cc

Comment 2

[Dan Minor [:dminor]]

Chromium has stopped using the webrtc.org video capture code and now uses it's own variant (under src/media/capture). We need to decide whether we want to continue using the webrtc.org code, which is not being actively maintained by upstream or do the work to switch to using the Chromium video capture code.

If we stick with the webrtc.org code, we're stuck doing the maintenance and updates, which could be substantial. For instance, the webrtc.org code does not support MediaFoundation on Windows, which caused problems with using the camera on aarch64 windows. It does appear that Safari uses the webrtc.org code, for instance Bug 1530792 reproduces on Safari, but not Chrome.

If we switch to using the Chromium code, we're stuck depending upon another internal API over which we have no control. Offhand, I'd say it's even less likely we'll be able to upstream our local changes to Chromium than to webrtc.org. We would have to evaluate how much internal Chromium code the capture code depends upon. This could end up basically being a fork rather than something we can update from on a regular basis.

I'm changing the summary to emphasize this is not an update, but a complete switchover.

Comment 3

[Dan Minor [:dminor]]

I take back what I said about Bug 1530792, I can't get Safari to give me an option of what webcam to use, and the builtin camera on my Macbook is not capable of 1920x1280.

Comment 6

[Eric Carlson]

It does appear that Safari uses the webrtc.org code, for instance Bug 1530792 reproduces on Safari, but not Chrome.

WebKit does not use libwebrtc for capture. The capture code for the macOS and iOS ports are here, the code for the GTK port is here.

If bug 1530792 reproduced in Safari (it works in TOT), it must have been a conicidence.

Comment 7

[Dan Minor [:dminor]]

(In reply to Eric Carlson from comment #6)

It does appear that Safari uses the webrtc.org code, for instance Bug 1530792 reproduces on Safari, but not Chrome.

WebKit does not use libwebrtc for capture. The capture code for the macOS and iOS ports are here, the code for the GTK port is here.

If bug 1530792 reproduced in Safari (it works in TOT), it must have been a conicidence.

Yes, that was my mistake, I was testing with the builtin camera on my Macbook, which is not capable of the higher resolutions. Thanks for the clarification.

Comment 8

[Dan Minor [:dminor]]

The OS X video capture code which we are using has also been removed from upstream [1]. The Windows and Linux code seems to be maintained, however.

[1] https://webrtc-review.googlesource.com/c/src/+/124828

Comment 9

[Dan Minor [:dminor]]

Looks like Android camera code has resurfaced in webrtc.org: https://cs.chromium.org/chromium/src/third_party/webrtc/sdk/android/api/org/webrtc/CameraVideoCapturer.java

Comment 10

[Dan Minor [:dminor]]

I updated the Android camera code in Bug 1578073, so let's make this one about updating OS X.

Comment 11

[Dan Minor [:dminor]]

There is OS X video capture code in libwebrtc here: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/sdk/objc/components/capturer/RTCCameraVideoCapturer.m. That will probably be easier than switching to Chromium's code.

Comment 12

[Serj Lavrin]

There were no updates for quite a while. Unfortunately, that bug makes Firefox unusable for a good portion of users who want to capture a Full HD webcam.

Are there any plans to update libwebrtc?

Comment 13

[Andreas Pehrson [:pehrsons]]

I plan to integrate the newer libwebrtc camera backend for mac, to see what that gets us.

Comment 14

[Andreas Pehrson [:pehrsons]]

This is quite ready to go up for review, but I want to have it hooked up to the PerformanceRecorder before pushing.

Comment 15

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1451394 - Fix LogCapability's video type logging in MediaEngineRemoteVideoSource. r?#webrtc-reviewers

Comment 16

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1451394 - Expose mac camera capture backend in .gn and switch it to gecko libyuv. r?#webrtc-reviewers

Comment 17

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1451394 - Update generated files. r?#webrtc-reviewers

Comment 18

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1451394 - Integrate with the libwebrtc camera backend for Mac. r?#webrtc-reviewers

Comment 19

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1451394 - Enable new Mac camera backend in Nightly and early Beta. r?#webrtc-reviewers

Comment 20

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1451394 - Record video frame captures with PerformanceRecorder in the new mac camera backend. r?padenot!

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1f6355b0d42c
Fix LogCapability's video type logging in MediaEngineRemoteVideoSource. r=webrtc-reviewers,mjf
https://hg.mozilla.org/integration/autoland/rev/b06588889693
Expose mac camera capture backend in .gn and switch it to gecko libyuv. r=webrtc-reviewers,mjf
https://hg.mozilla.org/integration/autoland/rev/49002523b70d
Update generated files. r=webrtc-reviewers,mjf
https://hg.mozilla.org/integration/autoland/rev/703360f0b861
Integrate with the libwebrtc camera backend for Mac. r=webrtc-reviewers,jib
https://hg.mozilla.org/integration/autoland/rev/04d6fcc72269
Enable new Mac camera backend in Nightly and early Beta. r=webrtc-reviewers,jib
https://hg.mozilla.org/integration/autoland/rev/a7362238c9e6
Record video frame captures with PerformanceRecorder in the new mac camera backend. r=padenot

Comment 22

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/1f6355b0d42c
https://hg.mozilla.org/mozilla-central/rev/b06588889693
https://hg.mozilla.org/mozilla-central/rev/49002523b70d
https://hg.mozilla.org/mozilla-central/rev/703360f0b861
https://hg.mozilla.org/mozilla-central/rev/04d6fcc72269
https://hg.mozilla.org/mozilla-central/rev/a7362238c9e6