1724900 - Use xdg camera portal and pipewire for webcam access

Status: RESOLVED FIXED

(Core :: WebRTC, enhancement)

Description

[Robert Mader [:rmader]]

Accessing webcams through the portal[1] and pipewire has several advantages:

• tighter permission control for flatpaks (unsandboxed clients will get permission to use the camera without user feedback)
• multiplexing / multiple applications can use the webcam at the same time
• potentially simplifications in the code as it works more similarly to screen sharing
• dmabuf import for easier hardware encoding (?)

Context: at the recent GUADEC 2021 there was a BoF about this topic (with members of the wlroots community around as well) and it was agreed to push for making the webcam portal the default. FF should be on board, as for many users it's the main webcam user. There is a gstreamer demo client[2], but I haven't come across an example for a ffmpeg user yet. (edit: a GTK4 demo using the camera portal[4]).

As of today (Aug. 10. 2021), there's no upstream pipewire support for video capturing in WebRTC, so any work here should be upstreamed if possible.

1: https://github.com/flatpak/xdg-desktop-portal/blob/master/data/org.freedesktop.portal.Camera.xml
2: https://gitlab.gnome.org/-/snippets/762
3: https://webrtc.googlesource.com/src/+/refs/heads/main/modules/video_capture/linux/
4: https://gitlab.gnome.org/matthiasc/video-play

---

Development happens in https://github.com/rmader/gecko-dev/tree/bug1724900

Comment 1

[Robert Mader [:rmader]]

Attachment: WIP: Bug 1724900 - WIP: Split V4L2 video capture implementations into their own files

We'll need to support V4L2 as well as Pipewire.

Comment 2

[Robert Mader [:rmader]]

Attachment: WIP: Bug 1724900 - WIP: Add skeletons for Pipewire video capture implementations

TODO: fix (generated) build files

Depends on D122236

Comment 3

[Robert Mader [:rmader]]

Michael, in order to properly test this I'd love to have an easy way to build flatpak builds. Unfortunately this doesn't appear to work via try runs yet[1] and bhearsum from #Firefox CI said it's unlikely to change any time soon. Do you have any suggestion how to best test flatpak related changes? Do you just build things locally? If so, how? Thanks and regards.

1: https://treeherder.mozilla.org/jobs?repo=try&revision=43223b2fc9eea4937e586b2cb0c66b30b16039e1

Comment 4

[Michael Froman [:mjf]]

Robert, unfortunately I have exactly zero experience with flatpack builds. You might try the #build to ask about build issues.

Comment 5

[Robert Mader [:rmader]]

(In reply to Michael Froman [:mjf] from comment #4)

Robert, unfortunately I have exactly zero experience with flatpack builds. You might try the #build to ask about build issues.

Oh, sorry, that makes sense. You are the triage owner of WebRTC, not Flatpak - confused that, sorry!

Comment 6

[Robert Mader [:rmader]]

Mitchell, redirecting the question to you: in order to properly test this I'd love to have an easy way to build flatpak builds. Unfortunately this doesn't appear to work via try runs yet[1] and bhearsum from #Firefox CI said it's unlikely to change any time soon. Do you have any suggestion how to best test flatpak related changes? Do you just build things locally? If so, how? Thanks and regards.

1: https://treeherder.mozilla.org/jobs?repo=try&revision=43223b2fc9eea4937e586b2cb0c66b30b16039e1

---

Edit: also opened bug 1726251 to track try builds.

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

That's a great question, and unfortunately I haven't worked with Firefox + Flatpak yet.
I'm going to redirect the NI to glandium, he's probably familiar with the area.
Also, FWIW, :mtabara did the RelEng work for setting up Flatpak releases, so he might have a workflow for testing with Flatpak builds, but no guarantees.

Comment 8

[Mike Hommey [:glandium]]

I have no knowledge about the build environment for flatpak, although I think those are just repacks.

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

No good way to test this on try indeed. I build them locally two years ago when I added the automation so I'll have to dig up some notes. I'm a bit swamped right now, but I'll do my best to paste it soon. Leaving the NI open until then.

Comment 10

[Robert Mader [:rmader]]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #9)

No good way to test this on try indeed. I build them locally two years ago when I added the automation so I'll have to dig up some notes. I'm a bit swamped right now, but I'll do my best to paste it soon. Leaving the NI open until then.

That would be great, thanks. I'll also give it a go (in bug 1726251), maybe I can get the try build to run.

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Dug a bit into this today. If I wanted to hack locally on flatpaks today, here's what I'd do. I'm fairly certain there are optimisations here and there that can be applied, but roughly this is the logic:

1. In order to generate any flatpaks, we need a good docker image with the pre-requisites. That's https://searchfox.org/mozilla-central/source/taskcluster/docker/firefox-flatpak/Dockerfile file. Looking at the most recent dot release from yesterday, https://firefox-ci-tc.services.mozilla.com/tasks/UdPTilx5Sw2Zk2yhyffkFw#artifacts is the image is being used. In the artifacts there, there's an image.tar.zst within here. Download image, unzst it and load the image in docker, creating a container off of it.

2. All the logic generating the flatpaks lies in the run.me script (from here). That file will be presend in the docker container but is expecting some environment variables. Again, looking at the most recent task in our lattest dot release, I see this task. One can download those env vars and tweak them accordingly to point to an existing beta instead for example.

      "LANG": "C.UTF-8",
      "LC_ALL": "C.UTF-8",
      "VERSION": "91.0.2",
      "BUILD_NUMBER": "1",
      "CANDIDATES_DIR": "https://net-mozaws-prod-delivery-firefox.s3.amazonaws.com/pub/firefox/candidates",
      "FLATPAK_BRANCH": "stable",
      "MOZ_AUTOMATION": "1",
      "L10N_CHANGESETS": "https://hg.mozilla.org/releases/mozilla-release/raw-file/1aa232202866276c3ee7a2748b2bb291bf4fe6fa/browser/locales/l10n-changesets.json",

3. Tweaking these locally to match whatever beta/release version we want to test, I'd throw them into pass locally so that I can load them easier in Docker. (e.g. a file called local-prod) would look like:

VERSION=92.0b1
BUILD_NUMBER=1
...

4. Create the docker container feeding in the env vars e.g. docker run -ti $(pass show local-prod | grep -v ^# | grep -v '^$' | sed 's/^/-e /') image_name /bin/bash

5. Tweak run.me to your preference and run it. The script is downloading the existing tar.gz for the version/build_no specified. Then downloads all langpacks, performs some sanity check and then starts packaging the flatpak, placing it at the end in the right path for Taskcluster to find it. One can uncomment various commands to speed things up (not download all locales for examples but instead only one or two?, skip checksums verification, etc).

If you mount your locale volume somewhere in the docker container, you can extract the local flatpak build at the end to play with it locally on your host machine.

Hope this helps.

Comment 12

[Robert Mader [:rmader]]

@Mihai: thanks a lot, will try that.

@topic: just got my first (entirely green) frame with the Pipewire implementation on https://www.webrtc-experiment.com/RecordRTC/ \o/

An insight gained today (as already said in bug 1658900 comment 2): Pipewire appears to prefer DMAbuf for buffer sharing. That supports point 4 of comment 0, meaning that VAAPI video encoding for camera streams could end up being easier than thought.

Comment 13

[Robert Mader [:rmader]]

Attachment: WIP: Bug 1724900 - WIP: Initial portal/pipewire camera support

Right now this is just enough to get my own camera running - but
it works \o/

Depends on D122470

Comment 14

[Jan Grulich]

Hi Robert, do you plan to upstream your work so all the browsers can benefit from this change? Do you need any help? I can see it has lots of similarities to screensharing code, maybe once you are finished I can help you to upstream this into WebRTC and we can try to de-duplicate the code.

Comment 15

[Robert Mader [:rmader]]

(In reply to grulja from comment #14)

Hi Robert, do you plan to upstream your work so all the browsers can benefit from this change? Do you need any help? I can see it has lots of similarities to screensharing code, maybe once you are finished I can help you to upstream this into WebRTC and we can try to de-duplicate the code.

Hi Jan! I definitely plan to upstream this code once I get back to finish it. However, as I currently can't quite say when that will be (some weeks probably), please feel free to move it upstream / start from scratch / work on the issue! Just let me/us know here ;)

Comment 16

[Jan Grulich]

Hi Robert,

I started slowly working on this. As a first step, I moved implementation for xdg-desktop-portal into a separate class and added support for the Camera portal and opened a WIP review for WebRTC (you should be in CC). We will probably want to move the implementation somewhere else since the webcam implementation is a different kind of module in WebRTC, but I will do that once I have some code for the camera support.

WebRTC review: https://webrtc-review.googlesource.com/c/src/+/236845/

Comment 17

[Robert Mader [:rmader]]

(In reply to grulja from comment #16)

...

Thanks Jan! In case you didn't see: I think it would be great if we help push https://github.com/flatpak/xdg-desktop-portal/pull/659 over the line so we have a common interface for both webcam and microphone access. Unfortunately I'll not be able to work on all this before January :/

Comment 18

[Robert Mader [:rmader]]

Here's some upstream work going on: https://chromium-review.googlesource.com/c/chromium/src/+/3308882

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, so the assignee is being reset.

Comment 20

[Suhaib Mujahid [:suhaib]]

Sorry, there was a problem with the detection of inactive users. I'm reverting the change.

Comment 21

[Robert Mader [:rmader]]

Looks like all webrtc bits have landed to implement this: https://webrtc-review.googlesource.com/c/src/+/261620 and https://webrtc-review.googlesource.com/c/src/+/264553.

Here's the chromium specific part again: https://chromium-review.googlesource.com/c/chromium/src/+/3308882 - something roughly like that is probably what we'll need to do in FF.

Comment 22

[Jan Grulich]

(In reply to Robert Mader [:rmader] from comment #21)

Looks like all webrtc bits have landed to implement this: https://webrtc-review.googlesource.com/c/src/+/261620 and https://webrtc-review.googlesource.com/c/src/+/264553.

Here's the chromium specific part again: https://chromium-review.googlesource.com/c/chromium/src/+/3308882 - something roughly like that is probably what we'll need to do in FF.

For the record, I'm trying to backport these patches with intention to look what is missing on FF side and implement it if there is anything needed. The work is here: https://github.com/grulja/gecko-dev/tree/pipewire-camera.

Comment 23

[Robert Mader [:rmader]]

Uh, awesome! Taking the freedom to assign you then :)

One thing still missing in webrtc is rotation support, which is needed on most phones (the cameras are usually mounted 90 deg rotated) - I hope to have a look into that soonish (upstream).

Comment 24

[Jan Grulich]

Attachment: Bug 1724900: Add PipeWire camera support r=jib,rmader,pehrsons

Adds a PipeWire based camera support that was recently merged into
WebRTC. This should be an experimental feature for now and therefore it
is kept behind a config option.

Comment 25

[Jan Grulich]

Attachment: Bug 1724900: WebRTC backport: PipeWire video capture - set device unique ID during initialization r=jib,rmader,pehrsons

This is a simple backport of an WebRTC upstream change.

Upstream commit: 4beafa38d546ab6c0bb423c12762f0c4568aa5ce

Depends on D176622

Comment 26

[Jan Grulich]

Attachment: Bug 1724900: WebRTC backport: Add pipewire/portal video capture support r=jib,rmader,pehrsons

This is a simple backport of an WebRTC upstream change.

Upstream commit: f0be3bee1fc17d96354ce3c9bf457673b82e7d2d

Comment 27

[Jan Grulich]

Attachment: Bug 1724900: Add PipeWire camera support r=jib,rmader,pehrsons

Adds a PipeWire based camera support that was recently merged into
WebRTC. This should be an experimental feature for now and therefore it
is kept behind a config option.

Depends on D176624

Comment 28

[Jan Grulich]

Attachment: Bug 1724900: WebRTC backport: PipeWire video capture - set device unique ID during initialization r=jib,rmader,pehrsons

This is a simple backport of an WebRTC upstream change.

Upstream commit: 4beafa38d546ab6c0bb423c12762f0c4568aa5ce

Depends on D176625

Comment 29

[Jan Grulich]

Attachment: Bug 1724900: Accomodate FF build to pipewire/portal video capture support r=jib,rmader,pehrsons

With PipeWire addition to video capture we need to include PipeWire into
the build system and add some new functions we will be dlopening.

Depends on D176624

Comment 30

[Jan Grulich]

Attachment: Bug 1724900: WebRTC backport: PipeWire video capture: split portal and PipeWire implementations r=jib,rmader,pehrsons

This is a simple backport of an WebRTC upstream change.

Note: This change has not been merged yet.
WebRTC change: https://webrtc-review.googlesource.com/c/src/+/305222

Depends on D176648

Comment 31

[Jan Grulich]

Attachment: Bug 1724900: Generate libwebrtc moz.build files r=jib,rmader,pehrsons

Generate libwebrtc moz.build files after PipeWire camera backports

Depends on D178937

Comment 32

[Dallas Strouse [:oro]]

What's the current status of this? Anything else that needs to be done, like bug testing? If Flatpak bundles are available anywhere, I'll definitely be down to see how well it works.

Comment 33

[Jan Grulich]

(In reply to Dallas Strouse [:oro] from comment #32)

What's the current status of this? Anything else that needs to be done, like bug testing? If Flatpak bundles are available anywhere, I'll definitely be down to see how well it works.

We are still working on this change. It is basically complete, but there are just some minor issues I need to address before it is good enough to be merged.

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/241d7632685a
WebRTC backport: PipeWire video capture: split portal and PipeWire implementations r=pehrsons,webrtc-reviewers
https://hg.mozilla.org/integration/autoland/rev/8ebb0f9ec0ae
Generate libwebrtc moz.build files r=pehrsons,webrtc-reviewers
https://hg.mozilla.org/integration/autoland/rev/8ac9e05de573
Add PipeWire camera support r=pehrsons,ipc-reviewers,nika
https://hg.mozilla.org/integration/autoland/rev/b94558d76bd2
WebRTC backport: PipeWire video capture - set device unique ID during initialization r=pehrsons,webrtc-reviewers

Comment 35

[Natalia Csoregi [:nataliaCs]]

Backed out for causing crashes on getUserMedia-audio

• backout: https://hg.mozilla.org/integration/autoland/rev/08cb4ca984b3aabd7881a042a28bdae29c0a5095
• push: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=cwwksra9QA651eWRqsDZXQ.0&revision=b94558d76bd2943ba1063d7fade51c4d54d0db35&group_state=expanded
• failure log: https://treeherder.mozilla.org/logviewer?job_id=419400764&repo=autoland&lineNumber=11480

Comment 36

[Natalia Csoregi [:nataliaCs]]

Also failing on Mingw: /builds/worker/checkouts/gecko/ipc/glue/BackgroundParentImpl.cpp:1434:3: error: use of undeclared identifier 'resolver' with failure log

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0472c2c90775
WebRTC backport: PipeWire video capture: split portal and PipeWire implementations r=pehrsons,webrtc-reviewers
https://hg.mozilla.org/integration/autoland/rev/9aa4190e2f2f
Generate libwebrtc moz.build files r=pehrsons,webrtc-reviewers
https://hg.mozilla.org/integration/autoland/rev/a1a0f8998a75
Add PipeWire camera support r=pehrsons,ipc-reviewers,nika
https://hg.mozilla.org/integration/autoland/rev/583ba187d9e5
WebRTC backport: PipeWire video capture - set device unique ID during initialization r=pehrsons,webrtc-reviewers

Comment 38

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/0472c2c90775
https://hg.mozilla.org/mozilla-central/rev/9aa4190e2f2f
https://hg.mozilla.org/mozilla-central/rev/a1a0f8998a75
https://hg.mozilla.org/mozilla-central/rev/583ba187d9e5