Closed Bug 1903688 Opened 1 year ago Closed 1 year ago

nvv4l2dec: sandboxing blocks hardware accelerated nvidia tegra ffmpeg

Categories

(Core :: Security: Process Sandboxing, defect, P3)

Product:
Core
Component:
Security: Process Sandboxing
Version:
Firefox 129
Platform:
ARM64
Linux
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
130 Branch
Tracking Flags:
Tracking Status
firefox130 --- fixed

People

(Reporter: dofficialgman, Assigned: gerard-majax)

References

(Blocks 1 open bug,
URL
)

Details

Attachments

(7 files, 2 obsolete files)

Bug 1903688 - NVIDIA Tegra hardware decoding r?gcp!
48 bytes, text/x-phabricator-request
Details | Review
/sys recursive output
1.14 MB, text/plain
Details
terminal output from https://share.firefox.dev/4c7Exyb
387.65 KB, text/plain
Details
Screenshot 2024-06-21 at 16-25-39 Gestionnaire de processus.png
77.79 KB, image/png
Details
sys_folder_output_v2.txt
4.97 MB, text/plain
Details
audio only decode partial log
10.74 KB, text/plain
Details
firefox-tegra-decode
7.05 MB, image/jpeg
Details

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36

Steps to reproduce:

use custom hardware acclerated ffmpeg (that replaces system ffmpeg) and attempt to use firefox.

Actual results:

all media decoding/encoding fails due to firefox sandboxing

[83530] Sandbox: SandboxBroker: denied op=open rflags=0 perms=1 path=/proc/83860/status for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 00 path /proc/83860/status
[83530] Sandbox: SandboxBroker: denied op=open rflags=2044000 perms=1 path=/sys/devices/system/node for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 02044000 path /sys/devices/system/node
[83860] Sandbox: rewriting /proc/self/status -> /proc/83860/status
[83530] Sandbox: SandboxBroker: denied op=open rflags=0 perms=1 path=/proc/83860/status for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 00 path /proc/83860/status
[83530] Sandbox: SandboxBroker: denied op=open rflags=0 perms=1 path=/sys/module/tegra_fuse/parameters/tegra_chip_id for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 00 path /sys/module/tegra_fuse/parameters/tegra_chip_id
[83530] Sandbox: SandboxBroker: denied op=open rflags=0 perms=1 path=/sys/module/tegra_fuse/parameters/tegra_chip_rev for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 00 path /sys/module/tegra_fuse/parameters/tegra_chip_rev
[83530] Sandbox: SandboxBroker: denied op=open rflags=0 perms=1 path=/sys/module/fuse/parameters/tegra_chip_id for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 00 path /sys/module/fuse/parameters/tegra_chip_id
[83530] Sandbox: SandboxBroker: denied op=open rflags=0 perms=1 path=/sys/module/fuse/parameters/tegra_chip_rev for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 00 path /sys/module/fuse/parameters/tegra_chip_rev
[83530] Sandbox: SandboxBroker: denied op=open rflags=0 perms=1 path=/sys/devices/soc0/soc_id for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 00 path /sys/devices/soc0/soc_id
[83530] Sandbox: SandboxBroker: denied op=open rflags=0 perms=1 path=/sys/devices/soc0/revision for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 00 path /sys/devices/soc0/revision
NvRmPrivGetChipIdLimited: Could not read Tegra chip id/rev
Expected on kernels without fuse support, using Tegra K1
[83530] Sandbox: SandboxBroker: denied op=open rflags=0 perms=1 path=/sys/module/tegra_fuse/parameters/tegra_platform for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 00 path /sys/module/tegra_fuse/parameters/tegra_platform
[83530] Sandbox: SandboxBroker: denied op=open rflags=0 perms=1 path=/sys/module/fuse/parameters/tegra_platform for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 00 path /sys/module/fuse/parameters/tegra_platform
NvRmPrivGetChipPlatform: Could not read platform information
Expected on kernels without fuse support, using silicon
[83530] Sandbox: SandboxBroker: denied op=open rflags=6010002 perms=1 path=/dev/nvhost-ctrl for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 06010002 path /dev/nvhost-ctrl
[83530] Sandbox: SandboxBroker: denied op=open rflags=2044000 perms=1 path=/dev/dri for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 02044000 path /dev/dri
Error: Can't initialize nvrm channel
[83530] Sandbox: SandboxBroker: denied op=open rflags=6010002 perms=1 path=/dev/nvhost-ctrl for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 06010002 path /dev/nvhost-ctrl
[83530] Sandbox: SandboxBroker: denied op=open rflags=2044000 perms=1 path=/dev/dri for pid=83860
[83860] Sandbox: Failed errno -13 op open flags 02044000 path /dev/dri
Error: Can't initialize nvrm channel
Couldn't create ddkvic Session: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession
[83530] Sandbox: EOF from pid 83860
[83579] Sandbox: Failed errno -2 op open flags 00 path /sys/devices/system/cpu/cpu0/cache/index3/size
[83579] Sandbox: Failed errno -2 op open flags 00 path /sys/devices/system/cpu/cpu0/topology/core_cpus
[83579] Sandbox: Failed errno -2 op open flags 00 path /sys/devices/system/cpu/cpu1/topology/core_cpus
[83579] Sandbox: Failed errno -2 op open flags 00 path /sys/devices/system/cpu/cpu2/topology/core_cpus
[83579] Sandbox: Failed errno -2 op open flags 00 path /sys/devices/system/cpu/cpu3/topology/core_cpus
[83530] Sandbox: EOF from pid 83579
[83530] Sandbox: EOF from pid 83816

Expected results:

media decoding succeeds

Before it gets asked. setting security.sandbox.content.level to 0 has no effect on the above output. Also, setting the read_path_whitelist to include those directories does not work as well, which leads me to believe that those configuration parameters are broken

The Bugbug bot thinks this bug should belong to the 'Core::Security: Process Sandboxing' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Security: Process Sandboxing
Product: Firefox → Core

Before it gets asked. setting security.sandbox.content.level to 0 has no effect on the above output. Also, setting the read_path_whitelist to include those directories does not work as well, which leads me to believe that those configuration parameters are broken

That's probably because those failures aren't from content, they're from RDD or a Utility process.

Alex, wdyt, do we try to open up some holes for this? Or add support for whitelisting in RDD/Utility?

Severity: -- → S4
Flags: needinfo?(lissyx+mozillians)
Priority: -- → P3

Could you verify e.g. what process is 83860 and 83579 ? about:processes should help. Also what media are you decoding, audio or video?

Until we can investigate more you should be able to verify if it works with MOZ_DISABLE_RDD_SANDBOX=1 and/or MOZ_DISABLE_UTILITY_SANDBOX=1 (please do it only for testing)

Also, nightly now allows you to collect those errors in the profiler, which can be much easier to share with us: https://firefox-source-docs.mozilla.org/tools/profiler/sandbox.html#recoding-sandbox-violations (it landed a few days ago, literally)

Severity: S4 → --
Flags: needinfo?(dofficialgman)
Priority: P3 → --
Severity: -- → S4
Flags: needinfo?(lissyx+mozillians)
Priority: -- → P3
Assignee: nobody → lissyx+mozillians

(In reply to :gerard-majax from comment #5)

Could you verify e.g. what process is 83860 and 83579 ? about:processes should help.

not sure how about:processes is supposed to help with this.

Also what media are you decoding, audio or video?

video. audio doesn't use any particular hardware blocks.

Until we can investigate more you should be able to verify if it works with MOZ_DISABLE_RDD_SANDBOX=1 and/or MOZ_DISABLE_UTILITY_SANDBOX=1 (please do it only for testing)

yes, setting both of these works. doing only one or the other does not work. both need to be set.

Also, nightly now allows you to collect those errors in the profiler, which can be much easier to share with us: https://firefox-source-docs.mozilla.org/tools/profiler/sandbox.html#recoding-sandbox-violations (it landed a few days ago, literally)

here is a profile where I just attempt to play a local video. https://share.firefox.dev/4bcj59K
I verified that with MOZ_DISABLE_RDD_SANDBOX=1 and MOZ_DISABLE_UTILITY_SANDBOX=1 set it plays (not profiled).

Flags: needinfo?(dofficialgman)

(In reply to dofficialgman from comment #6)

(In reply to :gerard-majax from comment #5)

Could you verify e.g. what process is 83860 and 83579 ? about:processes should help.

not sure how about:processes is supposed to help with this.

To verify the types of processes that reported the failure

Also what media are you decoding, audio or video?

video. audio doesn't use any particular hardware blocks.

Until we can investigate more you should be able to verify if it works with MOZ_DISABLE_RDD_SANDBOX=1 and/or MOZ_DISABLE_UTILITY_SANDBOX=1 (please do it only for testing)

yes, setting both of these works. doing only one or the other does not work. both need to be set.

Also, nightly now allows you to collect those errors in the profiler, which can be much easier to share with us: https://firefox-source-docs.mozilla.org/tools/profiler/sandbox.html#recoding-sandbox-violations (it landed a few days ago, literally)

here is a profile where I just attempt to play a local video. https://share.firefox.dev/4bcj59K
I verified that with MOZ_DISABLE_RDD_SANDBOX=1 and MOZ_DISABLE_UTILITY_SANDBOX=1 set it plays (not profiled).

Thanks, I'll have a look tomorrow. I forgot, but you ran that on a local build or nightly? In both case it needs to be recent for having the profiler bits (if you selected Debug as documented then it should be fine)

sorry bugzilla was down since my last comment

maybe I am missing something but seems to me like no sandbox errors were logged in the profiler in the previous post
this time I enabled MOZ_SANDBOX_LOGGING=1 and then debug profile logged (last one was debug profile logged as well) https://share.firefox.dev/45BjFNc

this is on the nightly build from yesterday

Thanks, there's a lot of noise in the data, but i see a few:

/sys/devices/system/cpu/present

    mozilla::SandboxBrokerClient::Open(char const*, int)security/sandbox/linux/SandboxBrokerClient.cpp
    mozilla::SandboxPolicyCommon::OpenAtTrap(sandbox::arch_seccomp_data const&, void*)security/sandbox/linux/SandboxFilter.cpp
    sandbox::Trap::SigSys(int, siginfo_t*, ucontext_t*)security/sandbox/chromium/sandbox/linux/seccomp-bpf/trap.cc
    mozilla::SigSysHandler(int, siginfo_t*, void*)security/sandbox/linux/Sandbox.cpp
    _rseq_flagsld-linux-aarch64.so.1
    _open64libc.so.6
    IO_file_openlibc.so.6
    IO_file_fopenlibc.so.6
    fun_6fc60libc.so.6
    PR_GetNumberOfProcessorsnsprpub/pr/src/misc/prsystem.c
    mozilla::FFmpegVideoDecoder<60>::InitCodecContext()dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp
    mozilla::FFmpegDataDecoder<60>::InitDecoder(AVDictionary**)dom/media/platforms/ffmpeg/FFmpegDataDecoder.cpp
    mozilla::FFmpegVideoDecoder<60>::Init()dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp
    mozilla::MediaDataDecoderProxy::Init()::$_0::operator()() constdom/media/platforms/wrappers/MediaDataDecoderProxy.cpp
    mozilla::detail::ProxyFunctionRunnable<mozilla::MediaDataDecoderProxy::Init()::$_0, mozilla::MozPromise<mozilla::TrackInfo::TrackType, mozilla::MediaResult, true> >::Run()xpcom/threads/MozPromise.h
    mozilla::TaskQueue::Runner::Run()xpcom/threads/TaskQueue.cpp
    nsThreadPool::Run()xpcom/threads/nsThreadPool.cpp
    nsThread::ProcessNextEvent(bool, bool*)xpcom/threads/nsThread.cpp
    NS_ProcessNextEvent(nsIThread*, bool)xpcom/threads/nsThreadUtils.cpp
    mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*)ipc/glue/MessagePump.cpp
    MessageLoop::RunInternal()ipc/chromium/src/base/message_loop.cc
    MessageLoop::RunHandler()ipc/chromium/src/base/message_loop.cc
    MessageLoop::Run()ipc/chromium/src/base/message_loop.cc
    nsThread::ThreadFunc(void*)xpcom/threads/nsThread.cpp
    _pt_rootnsprpub/pr/src/pthreads/ptthread.c
    0x5590469d0c
    fun_85600libc.so.6
    fun_eb7d0libc.so.6
    0x0
    0x0
    (root)

/dev/nvhost-nvdec

    mozilla::SandboxBrokerClient::Open(char const*, int)security/sandbox/linux/SandboxBrokerClient.cpp
    mozilla::SandboxPolicyCommon::OpenAtTrap(sandbox::arch_seccomp_data const&, void*)security/sandbox/linux/SandboxFilter.cpp
    sandbox::Trap::SigSys(int, siginfo_t*, ucontext_t*)security/sandbox/chromium/sandbox/linux/seccomp-bpf/trap.cc
    mozilla::SigSysHandler(int, siginfo_t*, void*)security/sandbox/linux/Sandbox.cpp
    _rseq_flagsld-linux-aarch64.so.1
    syscalllibc.so.6
    v4l2_openlibv4l2.so.0
    fun_650760libavcodec.so.60
    fun_650df0libavcodec.so.60
    avcodec_open2libavcodec.so.60
    mozilla::FFmpegDataDecoder<60>::InitDecoder(AVDictionary**)dom/media/platforms/ffmpeg/FFmpegDataDecoder.cpp
    mozilla::FFmpegVideoDecoder<60>::Init()dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp
    mozilla::MediaDataDecoderProxy::Init()::$_0::operator()() constdom/media/platforms/wrappers/MediaDataDecoderProxy.cpp
    mozilla::detail::ProxyFunctionRunnable<mozilla::MediaDataDecoderProxy::Init()::$_0, mozilla::MozPromise<mozilla::TrackInfo::TrackType, mozilla::MediaResult, true> >::Run()xpcom/threads/MozPromise.h
    mozilla::TaskQueue::Runner::Run()xpcom/threads/TaskQueue.cpp
    nsThreadPool::Run()xpcom/threads/nsThreadPool.cpp
    nsThread::ProcessNextEvent(bool, bool*)xpcom/threads/nsThread.cpp
    NS_ProcessNextEvent(nsIThread*, bool)xpcom/threads/nsThreadUtils.cpp
    mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*)ipc/glue/MessagePump.cpp
    MessageLoop::RunInternal()ipc/chromium/src/base/message_loop.cc
    MessageLoop::RunHandler()ipc/chromium/src/base/message_loop.cc
    MessageLoop::Run()ipc/chromium/src/base/message_loop.cc
    nsThread::ThreadFunc(void*)xpcom/threads/nsThread.cpp
    _pt_rootnsprpub/pr/src/pthreads/ptthread.c
    0x5590469d0c
    fun_85600libc.so.6
    fun_eb7d0libc.so.6
    0x0
    0x0
    (root)

I'm curious of your hardware, I'm wondering if you might already be dropping syscalls profiler logging because of the size of the queue https://searchfox.org/mozilla-central/rev/e729cdd1537458f9268215c2ff6dddb231c3a6d9/security/sandbox/linux/SandboxProfiler.cpp#283-289

I also dont see those two entries in the provided excerpt of log above, so i assume we still lack information. Can you make sure you provide a complete log from MOZ_SANDBOX_LOGGING=1 ?

The profiler logging should not depend on MOZ_SANDBOX_LOGGING=1 so i dont get why you had no data in your first attempt, maybe you enabled the profiler after the failures happened? On the doc we suggest you might want to run with the profiler from the beggining, that might help in your case, or manually kill RDD/Utility processes after you started the profiler, and reproduce your problem, that should make sure

(assuming we are not running into the queue dropping)

Maybe what we could try is another profiler setting: select maybe Nightly, make sure you uncheck "Unregistered threads" and make sure you input "SandboxProfilerEmitterSyscalls" in the list of threads

We can probaly add /present next to https://searchfox.org/mozilla-central/rev/cb1060f7b4581e6c2d30f1accc84c7d807132d82/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp#914, for RDD

Given we already allow this one on RDD https://searchfox.org/mozilla-central/rev/cb1060f7b4581e6c2d30f1accc84c7d807132d82/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp#915, I suspect PID 83860 was a Utility process?

[83860] Sandbox: Failed errno -13 op open flags 02044000 path /sys/devices/system/node

Given video decoding only happens in RDD, I'm wondering if we would really need to open it on Utility as well,

Would be nice to have a bit more view of your system's /sys tree to know what is actually meaningful in those errors, and what we should allow. Also knowing how we can detect tegra platforms to limit the sandbox holes to that platform?

Can you give a first try to this patch? It should allow some of the reported failure on RDD process, which should be enough. If you need to play around with Utility it would be within https://searchfox.org/mozilla-central/rev/cb1060f7b4581e6c2d30f1accc84c7d807132d82/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp#1026-1081 but without a full log of the errors it's hard to be more definitive that it is enough or other bits might be missing.

Also the /sys entries errors, we can't tell how much those can really be important in how ffmpeg decides to do its hwdecoding, there are multiple attempts but they all suggest from the log to be from a Utility process not a RDD, so hard to be definitive again. We might want to allow /sys/module/tegra_fuse/ completely? (with AddDir) On RDD and/or on Utility ? And also /sys/devices/soc0/ ?

Flags: needinfo?(dofficialgman)

(i assume you are building your own nightly, without knowing more of the target SoC i can't push to try a build)

Attached file /sys recursive output —
Flags: needinfo?(dofficialgman)

(In reply to :gerard-majax from comment #12)

Would be nice to have a bit more view of your system's /sys tree to know what is actually meaningful in those errors, and what we should allow. Also knowing how we can detect tegra platforms to limit the sandbox holes to that platform?

I have attached recursive output for this now

(In reply to :gerard-majax from comment #15)

(i assume you are building your own nightly, without knowing more of the target SoC i can't push to try a build)

I am using the official aarch64 firefox nightly builds. not building myself.

(In reply to :gerard-majax from comment #14)

Also the /sys entries errors, we can't tell how much those can really be important in how ffmpeg decides to do its hwdecoding, there are multiple attempts but they all suggest from the log to be from a Utility process not a RDD, so hard to be definitive again. We might want to allow /sys/module/tegra_fuse/ completely? (with AddDir) On RDD and/or on Utility ? And also /sys/devices/soc0/ ?

from previous experience poking holes in sandboxes for other applications, all of the paths that are mentioned in my first post are required.

OS: Unspecified → Linux
Regressions: egl-linux-v4l
Hardware: Unspecified → ARM64
See Also: → https://docs.nvidia.com/jetson/archives/r35.1/DeveloperGuide/text/SD/Multimedia/AcceleratedDecodeWithFfmpg.html
Summary: sandboxing blocks hardware accelerated nvidia tegra ffmpeg → nvv4l2dec: sandboxing blocks hardware accelerated nvidia tegra ffmpeg
Blocks: egl-linux-v4l
No longer regressions: egl-linux-v4l

(In reply to dofficialgman from comment #19)

(In reply to :gerard-majax from comment #14)

Also the /sys entries errors, we can't tell how much those can really be important in how ffmpeg decides to do its hwdecoding, there are multiple attempts but they all suggest from the log to be from a Utility process not a RDD, so hard to be definitive again. We might want to allow /sys/module/tegra_fuse/ completely? (with AddDir) On RDD and/or on Utility ? And also /sys/devices/soc0/ ?

from previous experience poking holes in sandboxes for other applications, all of the paths that are mentioned in my first post are required.

Of course, but the log shared so far is incomplete, would be nice if you could share a more complete one. Also, NOT having to open those on Utility would be better since it defeats the whole purpose of audio decoding on utility which was introduced explicitely so we could more easily open the sandbox when required on RDD for that kind of usages :)

(In reply to dofficialgman from comment #18)

(In reply to :gerard-majax from comment #15)

(i assume you are building your own nightly, without knowing more of the target SoC i can't push to try a build)

I am using the official aarch64 firefox nightly builds. not building myself.

ok so i'll trigger an aarch64 build with the current patch and we can iterate from that (denial hit by the utility from the log i see, but that would have been hit as well by rdd), with a more complete log (or profiler traces) where we can know which processes was granted/denied what

this log hopefully contains all the paths. I forget that not all media playback triggers the same required sandbox paths.

https://share.firefox.dev/4c7Exyb

Can't find a /sys/devices/soc0 in the shared /sys listing so i assume we dont care

(In reply to dofficialgman from comment #22)

this log hopefully contains all the paths. I forget that not all media playback triggers the same required sandbox paths.

https://share.firefox.dev/4c7Exyb

this one misses the sandbox profiler bits

The attachment in comment #23 is complicated if you dont share from about:processes which PIDs are RDD and Utility processes

[Child 7718, MediaDecoderStateMachine #1] WARNING: Decoder=7f82be7400 Decode error: NS_ERROR_DOM_MEDIA_FATAL_ERR (0x806e0005) - MediaResult mozilla::FFmpegDataDecoder<60>::InitDecoder(AVDictionary **): Couldn't open avcodec: file /builds/worker/checkouts/gecko/dom/media/MediaDecoderStateMachineBase.cpp:167

[7797] Sandbox: Failed errno -2 op open flags 02000000 path /home/gman/Documents/firefox/firefox/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /lib/aarch64-linux-gnu/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /usr/lib/aarch64-linux-gnu/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /lib/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /usr/lib/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /home/gman/Documents/firefox/firefox/libavcodec.so.60

Can you make sure you have some libavcodec somewhere and share with us where it is? error from comment 27 suggetss there's no one usable and there are those errors where it's not found

See Also: → https://docs.nvidia.com/jetson/archives/r35.1/DeveloperGuide/text/SD/Multimedia/AcceleratedDecodeWithFfmpg.html

Just for reference I am not actually using that but something else that accomplishes a similar thing but for older hardware (tegra-x1 and not orin) and older bsp (32.3.1 and not 32.5.1)

all nvv4l2 commits are applied ontop of 6.1.1 tag https://github.com/theofficialgman/FFmpeg/tree/6.1.1-nvv4l2
I don't anticipate the source to be of much help though since this is just an ffmpeg wrapper around nvidia's nvv4l2 decoder/encoder

(and the rest will have to wait next monday)

(In reply to :gerard-majax from comment #26)

The attachment in comment #23 is complicated if you dont share from about:processes which PIDs are RDD and Utility processes

I don't see any information in about:processes for the PID of any process. All I see are "Name" "Memory" and "CPU" columns

(In reply to :gerard-majax from comment #28)

[7797] Sandbox: Failed errno -2 op open flags 02000000 path /home/gman/Documents/firefox/firefox/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /lib/aarch64-linux-gnu/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /usr/lib/aarch64-linux-gnu/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /lib/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /usr/lib/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /home/gman/Documents/firefox/firefox/libavcodec.so.60

Can you make sure you have some libavcodec somewhere and share with us where it is? error from comment 27 suggetss there's no one usable and there are those errors where it's not found

it is usable otherwise it wouldn't work and I wouldn't be getting the debug logs that are added to ffmpeg in terminal output. the reason you see those logs above are because (I assume) firefox is checking multiple paths for libavcodec and failing to find it at the above paths. eventually it finds it at /usr/lib/aarch64-linux-gnu/libavcodec.so.60 or /lib/aarch64-linux-gnu/libavcodec.so.60. You can tell from the output above that that path is the next one that would be checked following the pattern, and it checks it, and doesn't output anything because it succeeds

(In reply to dofficialgman from comment #32)

(In reply to :gerard-majax from comment #26)

The attachment in comment #23 is complicated if you dont share from about:processes which PIDs are RDD and Utility processes

I don't see any information in about:processes for the PID of any process. All I see are "Name" "Memory" and "CPU" columns

Please share a screenshot, there should be the PID next to the process name between parenthesis

(In reply to dofficialgman from comment #33)

(In reply to :gerard-majax from comment #28)

[7797] Sandbox: Failed errno -2 op open flags 02000000 path /home/gman/Documents/firefox/firefox/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /lib/aarch64-linux-gnu/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /usr/lib/aarch64-linux-gnu/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /lib/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /usr/lib/libavcodec.so.61
[7797] Sandbox: Failed errno -2 op open flags 02000000 path /home/gman/Documents/firefox/firefox/libavcodec.so.60

Can you make sure you have some libavcodec somewhere and share with us where it is? error from comment 27 suggetss there's no one usable and there are those errors where it's not found

it is usable otherwise it wouldn't work and I wouldn't be getting the debug logs that are added to ffmpeg in terminal output. the reason you see those logs above are because (I assume) firefox is checking multiple paths for libavcodec and failing to find it at the above paths. eventually it finds it at /usr/lib/aarch64-linux-gnu/libavcodec.so.60 or /lib/aarch64-linux-gnu/libavcodec.so.60. You can tell from the output above that that path is the next one that would be checked following the pattern, and it checks it, and doesn't output anything because it succeeds

Perfect, that would be consistent yes. I was worried your ffmpeg might have been in a location we dont allow as well (happened on NixOS and Snap packages), but if it's at that place we should be fine and the error might just be a false positive, 7718 being a content process

about:processes where we see the PIDs next to the name

Attached file sys_folder_output_v2.txt —

(In reply to :gerard-majax from comment #25)

(In reply to dofficialgman from comment #22)

this log hopefully contains all the paths. I forget that not all media playback triggers the same required sandbox paths.

https://share.firefox.dev/4c7Exyb

this one misses the sandbox profiler bits

^ this log was created with MOZ_PROFILER_STARTUP=1 MOZ_SANDBOX_LOGGING=1 ./firefox
looks like that command doesn't work since it must be using a non-debug profiler mode.

here is a profiler from MOZ_SANDBOX_LOGGING=1 ./firefox where I manually start the debug profiler before attempting playback.
https://share.firefox.dev/3RyYPs3

the terminal output log is also attached

I see no PID info in about:proccess for anything

Can you please share your about:processes? It makes no sense you have no pid

(In reply to :gerard-majax from comment #36)

Created attachment 9408906 [details]
Screenshot 2024-06-21 at 16-25-39 Gestionnaire de processus.png

about:processes where we see the PIDs next to the name

ah I see now. some processes are too short lived that the sandboxbroxer is referencing. I do not them it in the list in about:processes

one that I can confirm is /dev/nvhost-nvdec is Data Decoder since that process continues to run

/dev/nvhost-ctrl, /sys/module/tegra_fuse/parameters/tegra_platform, /sys/module/tegra_fuse/parameters/tegra_chip_rev, etc all use a different PID that does not appear in the about:processes list

(In reply to dofficialgman from comment #37)

^ this log was created with MOZ_PROFILER_STARTUP=1 MOZ_SANDBOX_LOGGING=1 ./firefox
looks like that command doesn't work since it must be using a non-debug profiler mode.

Yes you need extra env var to pass the list of threads to register, but i dont remember which env var it is

Is there a way to print to terminal the PID and process name at time of creation and deletion? it seems like that is the only way that we could determine which process everything is from since they don't live long enough to show (or stay) in about:processes

(In reply to dofficialgman from comment #41)

Is there a way to print to terminal the PID and process name at time of creation and deletion? it seems like that is the only way that we could determine which process everything is from since they don't live long enough to show (or stay) in about:processes

no, that was not possible in the sandbox code that's why i only added the PID, and that's why the profiler is useful :)

so far from your latest profiler there's no more path, i hope it's not because we are dropping from the queue. We'll see what the try i shared in comment 30 gets you, but it's obvious e.g., /dev/nvhost-nvdec will help, cf https://share.firefox.dev/45Ejwsu we can see it's called from avcodec_open2 then funny names

(In reply to :gerard-majax from comment #30)

https://treeherder.mozilla.org/jobs?repo=try&revision=d68df0186e1f1605ecfa0f9442b8754c660acc2f
https://treeherder.mozilla.org/jobs?repo=try&revision=7b9cd0b08b8c4e3542f8e6c12ae2ab8b8ab79c58

As you can see on try the second one has one more sandbox allowance for /sys/module/tegra_fuse

it's likely i'll be asleep when you get those to run, so here is an example of your last profiler with more filtering that makes it more obvious which syscalls gets tried/denied: https://share.firefox.dev/4ezURta

Select Remote Data Decoder's SandboxProfilerEmitterSyscalls track + parent's FSBrokerXXX with XXX matching the PID of the RDD process, and filter with: -val4:/task,-val4:/stat that removes the noisy from /proc/xxx/task and /proc/xxx/stat

(In reply to :gerard-majax from comment #42)

(In reply to dofficialgman from comment #41)

Is there a way to print to terminal the PID and process name at time of creation and deletion? it seems like that is the only way that we could determine which process everything is from since they don't live long enough to show (or stay) in about:processes

no, that was not possible in the sandbox code that's why i only added the PID, and that's why the profiler is useful :)

so far from your latest profiler there's no more path, i hope it's not because we are dropping from the queue. We'll see what the try i shared in comment 30 gets you, but it's obvious e.g., /dev/nvhost-nvdec will help, cf https://share.firefox.dev/45Ejwsu we can see it's called from avcodec_open2 then funny names

I'm 100% certain those other paths are required from experience. The only reason you don't see those other paths in the profiler is because they are coming from a different PID that isn't captured in the profiler and isn't listed in about:processes. it is only visible in the terminal logs....

(In reply to dofficialgman from comment #44)

(In reply to :gerard-majax from comment #42)

(In reply to dofficialgman from comment #41)

Is there a way to print to terminal the PID and process name at time of creation and deletion? it seems like that is the only way that we could determine which process everything is from since they don't live long enough to show (or stay) in about:processes

no, that was not possible in the sandbox code that's why i only added the PID, and that's why the profiler is useful :)

so far from your latest profiler there's no more path, i hope it's not because we are dropping from the queue. We'll see what the try i shared in comment 30 gets you, but it's obvious e.g., /dev/nvhost-nvdec will help, cf https://share.firefox.dev/45Ejwsu we can see it's called from avcodec_open2 then funny names

I'm 100% certain those other paths are required from experience. The only reason you don't see those other paths in the profiler is because they are coming from a different PID that isn't captured in the profiler and isn't listed in about:processes. it is only visible in the terminal logs....

I'm not sure what "other paths" you are referring to

What i see from attachment 9408908 [details] is consistent with what you shared in the last profile: same path being denied on RDD, and that should be allowed in the two try that are pending.

(In reply to dofficialgman from comment #39)

/dev/nvhost-ctrl, /sys/module/tegra_fuse/parameters/tegra_platform, /sys/module/tegra_fuse/parameters/tegra_chip_rev, etc all use a different PID that does not appear in the about:processes list

^ these paths. the ones that you can see in the terminal log sys_folder_output_v2.txt for pid=16003. they don't show in the profiler and that pid does not exist in about:processes

(In reply to dofficialgman from comment #47)

(In reply to dofficialgman from comment #39)

/dev/nvhost-ctrl, /sys/module/tegra_fuse/parameters/tegra_platform, /sys/module/tegra_fuse/parameters/tegra_chip_rev, etc all use a different PID that does not appear in the about:processes list

^ these paths. the ones that you can see in the terminal log sys_folder_output_v2.txt for pid=16003. they don't show in the profiler and that pid does not exist in about:processes

https://share.firefox.dev/45xK7Y7 we can see IPC message StartUtilityAudioDecoderService sent to that PID 16003, so it was trying to start a Utility Audio Decoder

(In reply to :gerard-majax from comment #20)

Also, NOT having to open those on Utility would be better since it defeats the whole purpose of audio decoding on utility which was introduced explicitely so we could more easily open the sandbox when required on RDD for that kind of usages :)

AH I think that functionality is bugged.

video files without audio play fine with only MOZ_DISABLE_RDD_SANDBOX=1 .
video+audio files require MOZ_DISABLE_UTILITY_SANDBOX=1 and MOZ_DISABLE_RDD_SANDBOX=1` .

I think what must be happening is when audio is present, it uses a utility proccess for some sort of probing for BOTH audio and video, this fails due to the sandboxing. The RDD process still gets used for the actual video decoding. When there is no audio, it uses the RDD process for probing and video decoding.

(In reply to :gerard-majax from comment #48)

(In reply to dofficialgman from comment #47)

(In reply to dofficialgman from comment #39)

/dev/nvhost-ctrl, /sys/module/tegra_fuse/parameters/tegra_platform, /sys/module/tegra_fuse/parameters/tegra_chip_rev, etc all use a different PID that does not appear in the about:processes list

^ these paths. the ones that you can see in the terminal log sys_folder_output_v2.txt for pid=16003. they don't show in the profiler and that pid does not exist in about:processes

https://share.firefox.dev/45xK7Y7 we can see IPC message StartUtilityAudioDecoderService sent to that PID 16003, so it was trying to start a Utility Audio Decoder

I think this confirmed my above theory (that I typed up before seeing your message) ^

there's no "used for probing", but it means the hwdecoding gets started by ffmpeg in all cases.

AH I think that functionality is bugged.

What do you mean it's bugged? It's working as intended. The fact that the decoder probes for hwdecoding directly and breaks completely is bad but it's not on our side. Why would ffmpeg fail and not just not perform hw decoding? For audio it's irrelevant anyway

Attached file audio only decode partial log —
audio only files (no video) also do not play unless `MOZ_DISABLE_UTILITY_SANDBOX=1` is set

(In reply to dofficialgman from comment #52)

audio only files (no video) also do not play unless
MOZ_DISABLE_UTILITY_SANDBOX=1 is set

Thanks, but I would say this is not normal, we should not have to open the sandbox for hw decoding support when we dont need it.

(In reply to dofficialgman from comment #52)

audio only files (no video) also do not play unless
MOZ_DISABLE_UTILITY_SANDBOX=1 is set

From that log my guess is that ffmpeg fails to open the hwdec stuff and just completely fail instead of gracefully doing a fallback

Since you seem to own https://github.com/theofficialgman/FFmpeg/tree/6.1.1-nvv4l2, maybe this is a fix to improve on ffmpeg side? I'm not sure how it integrates with the nvdec part.

(In reply to :gerard-majax from comment #53)

(In reply to dofficialgman from comment #52)

audio only files (no video) also do not play unless
MOZ_DISABLE_UTILITY_SANDBOX=1 is set

Thanks, but I would say this is not normal, we should not have to open the sandbox for hw decoding support when we dont need it.

Indeed. Even ffplay of an audio file (which does open a picture of the file in a window during playback) requires access to these sysfs nodes

gman@gman-nob:~$ sudo chmod -r /sys/module/tegra_fuse/parameters/tegra_chip_id
[sudo] password for gman: 
gman@gman-nob:~$ ffplay '/media/gman/Universal/home/garrett/Music/Breath of The Wild/10,000 Year Old Legend.mp3' 
No matching chip spec found for chip Id=0, Revision=3, Platform=0
Note: only internal builds support chips that haven't been announced yet.
NvRm init failed (capability).
Couldn't open NvRm Device: Cannot allocate memory
nvbuf_utils: Could not create Default NvBufferSession

(In reply to :gerard-majax from comment #54)

(In reply to dofficialgman from comment #52)

audio only files (no video) also do not play unless
MOZ_DISABLE_UTILITY_SANDBOX=1 is set

From that log my guess is that ffmpeg fails to open the hwdec stuff and just completely fail instead of gracefully doing a fallback

Since you seem to own https://github.com/theofficialgman/FFmpeg/tree/6.1.1-nvv4l2, maybe this is a fix to improve on ffmpeg side? I'm not sure how it integrates with the nvdec part.

I don't any reasonable way to do that. The codecs are only forced for video playback https://github.com/theofficialgman/FFmpeg/commit/15f49f2c158da26b0900d6bd30ff21b56e6aa39f

What I think is happening is you are still having ffmpeg decode a video stream even for only audio playback. ffplay does this too, it sends a static image video stream to show in the preview window even when I play an mp3

Input #0, mp3, from '/media/gman/Universal/home/garrett/Music/Breath of The Wild/10,000 Year Old Legend.mp3':
  Metadata:
    album           : The Legend of Zelda Breath of the Wild
    title           : 10,000 Year Old Legend
  Duration: 00:03:19.84, start: 0.025056, bitrate: 131 kb/s
  Stream #0:0: Audio: mp3, 44100 Hz, stereo, fltp, 128 kb/s
    Metadata:
      encoder         : LAME3.99r
    Side data:
      replaygain: track gain - 3.000000, track peak - unknown, album gain - unknown, album peak - unknown, 
  Stream #0:1: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 500x500 [SAR 1:1 DAR 1:1], 90k tbr, 90k tbn (attached pic)
    Metadata:
      comment         : Other
[swscaler @ 0x7f6c088a70] deprecated pixel format used, make sure you did set range correctly
    Last message repeated 1 times
[swscaler @ 0x7f6c088a70] deprecated pixel format used, make sure you did set range correctly
    Last message repeated 1 times
   7.15 A-V:    nan fd=   0 aq=   16KB vq=    0KB sq=    0B f=0/0

even if passing -nodisp to ffplay it still has a video stream (it just doesn't show it)

(In reply to dofficialgman from comment #58)

(In reply to :gerard-majax from comment #54)

(In reply to dofficialgman from comment #52)

audio only files (no video) also do not play unless
MOZ_DISABLE_UTILITY_SANDBOX=1 is set

From that log my guess is that ffmpeg fails to open the hwdec stuff and just completely fail instead of gracefully doing a fallback

Since you seem to own https://github.com/theofficialgman/FFmpeg/tree/6.1.1-nvv4l2, maybe this is a fix to improve on ffmpeg side? I'm not sure how it integrates with the nvdec part.

I don't any reasonable way to do that. The codecs are only forced for video playback https://github.com/theofficialgman/FFmpeg/commit/15f49f2c158da26b0900d6bd30ff21b56e6aa39f

What I think is happening is you are still having ffmpeg decode a video stream even for only audio playback. ffplay does this too, it sends a static image video stream to show in the preview window even when I play an mp3

no, our code will not fire a utility to touch anything related to video. Unfortunately, without a stack it's hard to know. There must be something in the init that hits those paths

here is a third try that includes some holes in utility: https://treeherder.mozilla.org/jobs?repo=try&revision=55fb5276613d329b76edd7089a14181f54b6bbc6

but i insist, the ffmpeg code should be able to gracefully fallback in absence of those hwdecoder, which will be fine in utility since we only do software decoding with audio (by design)

I don't think there is any way to do this. You see our ffmpeg links to the binaries responsible for decoding via nvv4l2. loading those (which happens when ffmpeg/libavcodec gets loaded) requires these permissions. these are closed source binaries so that cannot be changed.

Nvidia's implementation for orin previously mentioned here "https://docs.nvidia.com/jetson/archives/r35.1/DeveloperGuide/text/SD/Multimedia/AcceleratedDecodeWithFfmpg.html" does the same thing:

	./configure --prefix=/usr --enable-nvv4l2dec --enable-libv4l2 --enable-shared --extra-libs="-L/usr/lib/aarch64-linux-gnu/tegra -lnvbuf_utils" --extra-cflags="-I /usr/src/jetson_multimedia_api/include/"

CTCaer's does the same thing https://github.com/theofficialgman/FFmpeg/commit/8ea4f19bcb6472f9770d81c98c888731b4ea8e96#diff-90d08e583c4c9c6f391b2ae90f819f600a6326928ea9512c9e0c6d98e9f29ac2R7214-R7216

It looks to me like the other hwaccel decoders in ffmpeg dlopen their separate (potentially proprietary like nvidia's) libraries so they do not link at runtime. Making that change is really beyond me as I'm not a programmer (I'm just stubborn enough to read commit history and documentation to forward port these decoders/encoders written by CTCaer).

also fyi, this was my attempt previously pushing back sandbox in chromium for similar reasons (though there it didn't work because ffmpeg isn't run inside the gpu process where I was applying my sandbox pushbacks)

+void AddArmTegraGpuPermissions(std::vector<BrokerFilePermission>* permissions) {

  • // Device file needed by the Tegra GPU userspace.
  • static const char* const kReadWriteList[] = {
  •  "/dev/nvmap",

  •  "/dev/nvhost-ctrl-gpu",

  •  "/dev/nvhost-ctrl",

  •  "/dev/nvhost-nvdec",

  •  "/dev/nvhost-nvdec1",

  •  "/dev/nvhost-nvjpg",

  •  "/dev/nvhost-msenc",

  •  "/dev/nvhost-msenc1",

  •  "/dev/nvhost-vic",

  •  "/dev/tegra_dc_ctrl",

  •  "/dev/tegra_dc_0",

  •  "/dev/tegra_dc_1",

  •  "/dev/tegra_dc_2",

  •  "/dev/fb0",

  •  "/dev/fb1",

  •  "/dev/fb2"};
  • for (const char* item : kReadWriteList)
  • permissions->push_back(BrokerFilePermission::ReadWrite(item));
  • permissions->push_back(BrokerFilePermission::ReadOnlyRecursive("/proc/"));
  • permissions->push_back(BrokerFilePermission::ReadOnly("/usr/lib/aarch64-linux-gnu/libv4lconvert.so.0"));
  • permissions->push_back(BrokerFilePermission::ReadOnlyRecursive("/usr/lib/aarch64-linux-gnu/libv4l/plugins/"));
    +}

https://treeherder.mozilla.org/jobs?repo=try&revision=7b9cd0b08b8c4e3542f8e6c12ae2ab8b8ab79c58

with this I get new paths that need to be pushed back (all mentioned in the above comment)

profiler and process manager in the browser are useless though because they don't show the pid at all for rdd

(I'm just a user/tester.)
What device (product) are you using (how powerful is it) and for what purpose (desktop computer, industrial device with touchscreen, large info display, etc)?

Workaround:
Please try this, it should prevent RDD video process and Utility audio process crashes.
If the device is not for general web browsing (Twitch), you might not need the AAC audio codec (AV1 or VP9 video with Opus audio are recommended instead).
If it's powerful enough, you might not need hardware video decoding. For example, Raspberry Pi 4 supported H264 hw decoding, but Pi5 removed it (and never supported VP9 and AV1) because its CPU is powerful enough.

Open about:config,
set media.ffmpeg.enabled to false,
media.hardware-video-decoding.enabled to false,
open about:addons > Plugins > Wheel icon > Check updates, you should then see OpenH264 in the list and it should have a version number,
restart Nightly.
AV1,VP9,VP8,Opus,Flac,MP3,Vorbis would then be decoded with Nightly's internal ffmpeg (ffvpx) and patented H264 would be decoded via OpenH264 (bug 1795014), you would just be missing patented AAC audio.

(In reply to Darkspirit from comment #65)

(I'm just a user/tester.)
What device (product) are you using (how powerful is it) and for what purpose (desktop computer, industrial device with touchscreen, large info display, etc)?

a line of computing devices based on nvidia tegra SOCs (eg: Nvidia Jetson Nano, Jetson TX1, Jetson TX2, Jetson Xavier, Jetson Orin, Nintendo Switch, Nvidia Shield, and Pixel C, among others). Some for desktop linux, some industrial, some with touchscreen and some without, some in digital signage, some robotics, etc.

Workaround:
Please try this, it should prevent RDD video process and Utility audio process crashes.
If the device is not for general web browsing (Twitch), you might not need the AAC audio codec (AV1 or VP9 video with Opus audio are recommended instead).
If it's powerful enough, you might not need hardware video decoding. For example, Raspberry Pi 4 supported H264 hw decoding, but Pi5 removed it (and never supported VP9 and AV1) because its CPU is powerful enough.

Most (except maybe Xavier) do not have additional CPU headroom to decode a high resolution and framerates which is desirable for these usecases. Additionally, it appears even if I wanted to I cannot disable media.ffmpeg.enabled as its greyed out and forced enabled.

Thanks for the comment Darkspirit but your assistance isn't necessary here.

(In reply to dofficialgman from comment #63)

also fyi, this was my attempt previously pushing back sandbox in chromium for similar reasons (though there it didn't work because ffmpeg isn't run inside the gpu process where I was applying my sandbox pushbacks)

+void AddArmTegraGpuPermissions(std::vector<BrokerFilePermission>* permissions) {

  • // Device file needed by the Tegra GPU userspace.
  • static const char* const kReadWriteList[] = {
  •  "/dev/nvmap",

  •  "/dev/nvhost-ctrl-gpu",

  •  "/dev/nvhost-ctrl",

  •  "/dev/nvhost-nvdec",

  •  "/dev/nvhost-nvdec1",

  •  "/dev/nvhost-nvjpg",

  •  "/dev/nvhost-msenc",

  •  "/dev/nvhost-msenc1",

  •  "/dev/nvhost-vic",

  •  "/dev/tegra_dc_ctrl",

  •  "/dev/tegra_dc_0",

  •  "/dev/tegra_dc_1",

  •  "/dev/tegra_dc_2",

  •  "/dev/fb0",

  •  "/dev/fb1",

  •  "/dev/fb2"};
  • for (const char* item : kReadWriteList)
  • permissions->push_back(BrokerFilePermission::ReadWrite(item));
  • permissions->push_back(BrokerFilePermission::ReadOnlyRecursive("/proc/"));
  • permissions->push_back(BrokerFilePermission::ReadOnly("/usr/lib/aarch64-linux-gnu/libv4lconvert.so.0"));
  • permissions->push_back(BrokerFilePermission::ReadOnlyRecursive("/usr/lib/aarch64-linux-gnu/libv4l/plugins/"));
    +}

Thanks, I'll look into that tomorrow. Is this list based on your trial/error or just a safe list? like I see framebuffer, jpeg etc., I'm not super happy we open that much the sandbox if we can avoid them, but i dont know the details of the nvidia-specific part (usually they are pretty OK with fallback from our experience)

It looks to me like the other hwaccel decoders in ffmpeg dlopen their separate (potentially proprietary like nvidia's) libraries so they do not link at > runtime. Making that change is really beyond me as I'm not a programmer (I'm just stubborn enough to read commit history and documentation to forward port these decoders/encoders written by CTCaer).

That sounds like this is what is happening, yes. But on the Utility sandbox side, it's obvious that kind of change would be refused. I understand you might not be able to do the change, but switching from link-time to dlopen is not that complicated so likely other can do it? From discussing with media people, that's how all the others are doing it

Attachment #9408940 - Attachment is obsolete: true
Attachment #9408941 - Attachment is obsolete: true

This try should produce a correct RDD sandbox (I've added the list of path you shared above) https://treeherder.mozilla.org/jobs?repo=try&revision=f364c76e120c6e718b6c3728553d10f816a10724

You will still have to disable the Utility sandbox for audio, though, this is not something we will be able to change I worry, ffmpeg needs to properly deal with the loading of the nvidia libs and gracefully fallback.

Flags: needinfo?(dofficialgman)

If we need I've also prepared a build https://treeherder.mozilla.org/jobs?repo=try&revision=668e6887f49a09ad624ba0dfc2d9b60e9a69b6b5 where we activate more debug information from the sandbox profiler code (especially to verify if we might be dropping syscalls)

(In reply to :gerard-majax from comment #68)

This try should produce a correct RDD sandbox (I've added the list of path you shared above) https://treeherder.mozilla.org/jobs?repo=try&revision=f364c76e120c6e718b6c3728553d10f816a10724

now getting this

[6361] Sandbox: using seccomp tsync
[6361] Sandbox: Failed errno -2 op open flags 02000000 path /tmp/firefox/libnvtvmr.so
[6361] Sandbox: Failed errno -2 op open flags 02000000 path /tmp/firefox/libnvid_mapper.so.1.0.0
[6361] Sandbox: Failed errno -2 op open flags 02000000 path /tmp/firefox/libnvmm_utils.so
[6361] Sandbox: seccomp sandbox violation: pid 6361, tid 6371, syscall 29, args 27 18446744072636354048 547173093400 547334402048 547236422528 547576873008. Killing process.
[5982] Sandbox: EOF from pid 6361
[GFX1-]: VideoBridgeParent receives IPC close with reason=AbnormalShutdown
[Child 6274, MediaDecoderStateMachine #1] WARNING: Decoder=7f06869800 Decode error: NS_ERROR_DOM_MEDIA_DECODE_ERR (0x806e0004) - void mozilla::MediaFormatReader::Update(TrackType): Unable to decode: file /builds/worker/checkouts/gecko/dom/media/MediaDecoderStateMachineBase.cpp:167

these are typically located in /usr/lib/aarch64-linux-gnu/tegra which is in this systems dynamic library paths

copying those files temporarily to the firefox folder the above missing files are gone but it still doesn't work but I don't see the reason. the profile for this is -> https://share.firefox.dev/3RJdK2K

You will still have to disable the Utility sandbox for audio, though, this is not something we will be able to change I worry, ffmpeg needs to properly deal with the loading of the nvidia libs and gracefully fallback.

having a friend working on that rn. it will dlopen like the other hwcodecs in ffmpeg so won't dynamically link to the libraries.

Flags: needinfo?(dofficialgman)

(In reply to :gerard-majax from comment #69)

If we need I've also prepared a build https://treeherder.mozilla.org/jobs?repo=try&revision=668e6887f49a09ad624ba0dfc2d9b60e9a69b6b5 where we activate more debug information from the sandbox profiler code (especially to verify if we might be dropping syscalls)

let me know if you need me to get specific logs from this after reviewing the above

(In reply to dofficialgman from comment #70)

(In reply to :gerard-majax from comment #68)

This try should produce a correct RDD sandbox (I've added the list of path you shared above) https://treeherder.mozilla.org/jobs?repo=try&revision=f364c76e120c6e718b6c3728553d10f816a10724

now getting this

[6361] Sandbox: using seccomp tsync
[6361] Sandbox: Failed errno -2 op open flags 02000000 path /tmp/firefox/libnvtvmr.so
[6361] Sandbox: Failed errno -2 op open flags 02000000 path /tmp/firefox/libnvid_mapper.so.1.0.0
[6361] Sandbox: Failed errno -2 op open flags 02000000 path /tmp/firefox/libnvmm_utils.so
[6361] Sandbox: seccomp sandbox violation: pid 6361, tid 6371, syscall 29, args 27 18446744072636354048 547173093400 547334402048 547236422528 547576873008. Killing process.
[5982] Sandbox: EOF from pid 6361
[GFX1-]: VideoBridgeParent receives IPC close with reason=AbnormalShutdown
[Child 6274, MediaDecoderStateMachine #1] WARNING: Decoder=7f06869800 Decode error: NS_ERROR_DOM_MEDIA_DECODE_ERR (0x806e0004) - void mozilla::MediaFormatReader::Update(TrackType): Unable to decode: file /builds/worker/checkouts/gecko/dom/media/MediaDecoderStateMachineBase.cpp:167

these are typically located in /usr/lib/aarch64-linux-gnu/tegra which is in this systems dynamic library paths

Anything under /usr/lib/aarch64-linux-gnu/tegra should be alrady allowed from https://searchfox.org/mozilla-central/rev/9fcc11127fbfbdc88cbf37489dac90542e141c77/security/sandbox/linux/broker/SandboxBrokerPolicyFactory.cpp#957-958

Your problem is now:

[6361] Sandbox: seccomp sandbox violation: pid 6361, tid 6371, syscall 29, args 27 18446744072636354048 547173093400 547334402048 547236422528 547576873008. Killing process.

So assuming our headers are correct: https://searchfox.org/mozilla-central/rev/9fcc11127fbfbdc88cbf37489dac90542e141c77/security/sandbox/chromium/sandbox/linux/system_headers/arm64_linux_syscalls.h#137-139 that's ioctl which we filter at https://searchfox.org/mozilla-central/rev/9fcc11127fbfbdc88cbf37489dac90542e141c77/security/sandbox/linux/SandboxFilter.cpp#1879-1907

Chances are that the args for your case is a bit different

#include <sys/ioctl.h>
#include <stdio.h>

static const unsigned long kIoctlTypeMask = _IOC_TYPEMASK << _IOC_TYPESHIFT;
static constexpr unsigned long kTestType = static_cast<unsigned long>('F') << _IOC_TYPESHIFT;
static constexpr unsigned long kFbDevType = static_cast<unsigned long>('N') << _IOC_TYPESHIFT;
static constexpr unsigned long kVideoType = static_cast<unsigned long>('V') << _IOC_TYPESHIFT;
static constexpr unsigned long kDrmType = static_cast<unsigned long>('d') << _IOC_TYPESHIFT;
static constexpr unsigned long kDmaBufType = static_cast<unsigned long>('b') << _IOC_TYPESHIFT;

int main() {
    unsigned long request = 18446744072636354048;
    auto shifted_type = request & kIoctlTypeMask;

    printf("shifted_type=%lu\n", shifted_type);
    printf("kTestType=%lu\n", kTestType);
    printf("kFbDevType=%lu\n", kFbDevType);
    printf("kVideoType=%lu\n", kVideoType);
    printf("kDrmType=%lu\n", kDrmType);
    printf("kDmaBufType=%lu\n", kDmaBufType);
}

So N, makes sense.

Hopefully builds from https://treeherder.mozilla.org/jobs?repo=try&revision=76f02cfb29dedc62350a19f183ec9fa9a65b0fb4 should allow that ioctl() if I was not mistaken in reading it

(In reply to :gerard-majax from comment #74)

Hopefully builds from https://treeherder.mozilla.org/jobs?repo=try&revision=76f02cfb29dedc62350a19f183ec9fa9a65b0fb4 should allow that ioctl() if I was not mistaken in reading it

still having the same issue on that build I think

[29952] Sandbox: seccomp sandbox violation: pid 29952, tid 29962, syscall 29, args 30 3221768208 547948581304 548110355368 548011798176 0.  Killing process.

That's the same but with a H rather than an N, let's allow also

I just found that recent serie of patch against ffmpeg: http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2024-May/328552.html it does indeed define IOCTLs for Tegra, with those N and H magic

(In reply to :gerard-majax from comment #78)

I just found that recent serie of patch against ffmpeg: http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2024-May/328552.html it does indeed define IOCTLs for Tegra, with those N and H magic

That works. it has been able to decode anything I have tested so far without issue.

Could you try a build with the following removed so I can see if they are required or not:

+  policy->AddPath(rdwr, "/dev/fb0");
+  policy->AddPath(rdwr, "/dev/fb1");
+  policy->AddPath(rdwr, "/dev/fb2");

One thing I have noticed is I am still getting a
nvbuf_utils: Could not get EGL display connection
in the log which I do not get with the RDD sandbox disabled
https://share.firefox.dev/3VIYBjm

I will be out of town over the next week so can't test with the changes to ffmpeg using dlopen instead of dynamic linking so won't be able to verify that it works with the utility sandbox until after I come back.

(In reply to dofficialgman from comment #79)

(In reply to :gerard-majax from comment #78)

I just found that recent serie of patch against ffmpeg: http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2024-May/328552.html it does indeed define IOCTLs for Tegra, with those N and H magic

That works. it has been able to decode anything I have tested so far without issue.

Now you need to show me video or pictures :)

Could you try a build with the following removed so I can see if they are required or not:

+  policy->AddPath(rdwr, "/dev/fb0");
+  policy->AddPath(rdwr, "/dev/fb1");
+  policy->AddPath(rdwr, "/dev/fb2");

Perfect, that is exactly what I was going to suggest

One thing I have noticed is I am still getting a
nvbuf_utils: Could not get EGL display connection
in the log which I do not get with the RDD sandbox disabled
https://share.firefox.dev/3VIYBjm

Maybe harmless? It works right?

I will be out of town over the next week so can't test with the changes to ffmpeg using dlopen instead of dynamic linking so won't be able to verify that it works with the utility sandbox until after I come back.

No problem, I'm going to update the patch and make you a build, should be ready in a couple of hours hopefully in time for you

Attached image firefox-tegra-decode —

(In reply to :gerard-majax from comment #81)

removing fb devices + /usr/lib: https://treeherder.mozilla.org/jobs?repo=try&revision=958513799289ca58739e1b5df238245c4d7f49af

this still works

(In reply to :gerard-majax from comment #80)

Now you need to show me video or pictures :)

see above :) tegrastats is on the right showing NVDEC and VIC usage

(In reply to :gerard-majax from comment #80)

One thing I have noticed is I am still getting a
nvbuf_utils: Could not get EGL display connection
in the log which I do not get with the RDD sandbox disabled
https://share.firefox.dev/3VIYBjm

Maybe harmless? It works right?

Yes it works. I think this library has the ability to do direct framebuffer presentation (which our ffmpeg and thus firefox doesn't use) so that is why it is checking. Should not be a problem.

(In reply to :gerard-majax from comment #80)

I will be out of town over the next week so can't test with the changes to ffmpeg using dlopen instead of dynamic linking so won't be able to verify that it works with the utility sandbox until after I come back.

No problem, I'm going to update the patch and make you a build, should be ready in a couple of hours hopefully in time for you

No I just don't have the time to build system ffmpeg with the new patch using dlopen and dynamic linking. Its not your builds I am waiting on.

(In reply to dofficialgman from comment #83)

(In reply to :gerard-majax from comment #80)

I will be out of town over the next week so can't test with the changes to ffmpeg using dlopen instead of dynamic linking so won't be able to verify that it works with the utility sandbox until after I come back.

No problem, I'm going to update the patch and make you a build, should be ready in a couple of hours hopefully in time for you

No I just don't have the time to build system ffmpeg with the new patch using dlopen and dynamic linking. Its not your builds I am waiting on.

Found a little bit of time... new system ffmpeg that dlopen's the hardware decode libraries works without any sandbox disabling (utility and rdd)

If you get a chance could you also look at this? https://bugzilla.mozilla.org/show_bug.cgi?id=1232960#c15 that way no environment variables will be necessary.

See Also: → 1232960

Found a little bit of time... new system ffmpeg that dlopen's the hardware decode libraries works without any sandbox disabling (utility and rdd)

But you still need the rdd sandbox changes right?

(In reply to :gerard-majax from comment #86)

Found a little bit of time... new system ffmpeg that dlopen's the hardware decode libraries works without any sandbox disabling (utility and rdd)

But you still need the rdd sandbox changes right?

Yes. Sorry I should have been clearer about that.

Flags: needinfo?(dofficialgman)

(In reply to dofficialgman from comment #89)

(In reply to :gerard-majax from comment #86)

Found a little bit of time... new system ffmpeg that dlopen's the hardware decode libraries works without any sandbox disabling (utility and rdd)

But you still need the rdd sandbox changes right?

Yes. Sorry I should have been clearer about that.

That's fine. Let me know when you have a chance to test the new builds. It might also be cool if you can run the profiler but instead of the generic Debug profile, use Custom an in the thread list you set Sandbox,FSBroker, so we can verify what is really used by the driver when it works?

Also which device do you use to explore tegra platform? is it something i could put my hands on?

(In reply to :gerard-majax from comment #91)

Also which device do you use to explore tegra platform? is it something i could put my hands on?

Currently I am using my nintendo switch (a Nvidia Tegra X1 based video game console) with Switchroot Ubuntu 24.04 (a distro based on ubuntu with a customized linux kernel for the hardware and nvidia tegra drivers preinstalled) https://wiki.switchroot.org/wiki/linux/l4t-ubuntu-noble-installation-guide . I am the maintainer for Switchroot Ubuntu 22.04/24.04.
Unpatched RCM exploit switch's can run it would any modchip https://ismyswitchpatched.com/ . It also supports patched switches

Nvidia Jetson/Tegra X1/X2/Xavier/Orin (with NVDEC) would also be suitable hardware but the default OS setup from Nvidia doesn't have the hardware accelerated FFmpeg decoder/encoder on it.

(In reply to :gerard-majax from comment #88)

That works

As expected that does not work

I didn't test this. This looks like the wrong link

Using this build

and the profiler options you requested https://share.firefox.dev/3VZvMQP

(In reply to :gerard-majax from comment #94)

sys only is https://treeherder.mozilla.org/jobs?repo=try&revision=b1ca069521a28d6d5944df09d796a25421c3abb2 sorry

as expected that does not work

(In reply to dofficialgman from comment #93)

(In reply to :gerard-majax from comment #88)

That works

As expected that does not work

I didn't test this. This looks like the wrong link

Using this build

and the profiler options you requested https://share.firefox.dev/3VZvMQP

From that, some filtering and https://share.firefox.dev/3L5ixIh (sorry I'm happy to see that feature so useful so soon)

Attachment #9408827 - Attachment description: WIP: Bug 1903688 - NVIDIA Tegra custom FFmpeg hwdec → WIP: Bug 1903688 - NVIDIA Tegra hardware decoding via V4L2

(In reply to dofficialgman from comment #92)

Unpatched RCM exploit switch's can run it would any modchip https://ismyswitchpatched.com/ . It also supports patched switches

I butchered this sentence. It was supposed to say "Unpatched RCM exploit switch's can run it without any modchip https://ismyswitchpatched.com/ . It also supports patched switches with a modchip"

I'll take the timer later to review the actual list of what is being accessed and see what we can prune.

(In reply to :gerard-majax from comment #96)

From that, some filtering and https://share.firefox.dev/3L5ixIh (sorry I'm happy to see that feature so useful so soon)

just an FYI.

/dev/nvhost-msenc is used for encoding of videos. I don't think firefox supports that out of the box but if an extension or it gets added in the future it could be used.

also the reason /dev/nvhost-msenc and /dev/nvhost-msenc1 and /dev/nvhost-nvdec and /dev/nvhost-nvdec1 exist is because some hardware has two of these engines.

/dev/tegra_dc_ctrl and /dev/tegra_dc_X is unused similar to /dev/fbX because our ffmpeg decoder/encoder (and thus firefox) isn't using the nvidia direct presentation functionality

/dev/nvhost-nvjpg is used for decoding jpegs but we do not have a ffmpeg decoder backend for this implemented

(In reply to dofficialgman from comment #99)

(In reply to :gerard-majax from comment #96)

From that, some filtering and https://share.firefox.dev/3L5ixIh (sorry I'm happy to see that feature so useful so soon)

just an FYI.

/dev/nvhost-msenc is used for encoding of videos. I don't think firefox supports that out of the box but if an extension or it gets added in the future it could be used.

Firefox can encode videos (including with hw on some platform), but you're right that we have no code that uses V4L2 for that, we only have the decoding size. Thanks for mentioning it though.

It is wrong to have the sandbox pushbacks behind the #ifdef MOZ_ENABLE_V4L2. Put them outside of that flag like it was before.

This isn't using firefox's v4l2 backend (that is only for h264_v4l2m2m which this is not. tegra v4l2 is based on upstream v4l2 but it is a distant fork and not compatible) or any of the hwcodec backends. Instead the custom ffmpeg is taking advantage of the typical software decode path and re-purposing it for tegra hwdecode.

(In reply to dofficialgman from comment #101)

It is wrong to have the sandbox pushbacks behind the #ifdef MOZ_ENABLE_V4L2. Put them outside of that flag like it was before.

This isn't using firefox's v4l2 backend (that is only for h264_v4l2m2m which this is not. tegra v4l2 is based on upstream v4l2 but it is a distant fork and not compatible) or any of the hwcodec backends. Instead the custom ffmpeg is taking advantage of the typical software decode path and re-purposing it for tegra hwdecode.

I'm not knowledgeable enough here

this: https://treeherder.mozilla.org/jobs?repo=try&revision=daa890af6b2ce30838ece83fd2d5e9a8daa93df5

should be the current patch on phab, reducing what we granted based on the profiler content and what you shared in comment 99, let us know :)

See also bug 1748460 probably

(In reply to :gerard-majax from comment #103)

this: https://treeherder.mozilla.org/jobs?repo=try&revision=daa890af6b2ce30838ece83fd2d5e9a8daa93df5

should be the current patch on phab, reducing what we granted based on the profiler content and what you shared in comment 99, let us know :)

Sorry for the delay. I missed this earlier.

Yes that works as expected.

(In reply to dofficialgman from comment #105)

(In reply to :gerard-majax from comment #103)

this: https://treeherder.mozilla.org/jobs?repo=try&revision=daa890af6b2ce30838ece83fd2d5e9a8daa93df5

should be the current patch on phab, reducing what we granted based on the profiler content and what you shared in comment 99, let us know :)

Sorry for the delay. I missed this earlier.

Yes that works as expected.

Perfect, I'd just need more context on that ifdef thing, why MOZ_ENABLE_V4L2 would be incorrect? It's defined on our aarch64 builds

can you give a try to https://treeherder.mozilla.org/jobs?repo=try&revision=4d3c516a087c972adffcce3e4e621dcacb2dcf60 ?

Flags: needinfo?(dofficialgman)

(In reply to :gerard-majax from comment #106)

Perfect, I'd just need more context on that ifdef thing, why MOZ_ENABLE_V4L2 would be incorrect? It's defined on our aarch64 builds

if MOZ_ENABLE_V4L2 were to be disabled, the issue would occur again (decoding would fail). the firefox hwdecode V4L2 codepaths (https://searchfox.org/mozilla-central/source/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp#590-596) aren't taken for these tegra nvv4l2 decoders but instead the generic ffmpeg codepaths (https://searchfox.org/mozilla-central/source/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp#602-605). So it would be incorrect to place it within that ifdef

Flags: needinfo?(dofficialgman)

(In reply to :gerard-majax from comment #107)

can you give a try to https://treeherder.mozilla.org/jobs?repo=try&revision=4d3c516a087c972adffcce3e4e621dcacb2dcf60 ?

this fails because (_ARM64_) appears to only be defined in Windows ARM64. you probably should be using (__aarch64__)

Attachment #9408827 - Attachment description: WIP: Bug 1903688 - NVIDIA Tegra hardware decoding via V4L2 → Bug 1903688 - NVIDIA Tegra hardware decoding r?gcp!

you're right, i was doing too many things at once; https://treeherder.mozilla.org/jobs?repo=try&revision=a624f289e7d269bf52a3fc03dd0a1f9f436f11c5 should be good

Flags: needinfo?(dofficialgman)
Pushed by alissy@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/6b204e1c4234 NVIDIA Tegra hardware decoding r=gcp

(In reply to :gerard-majax from comment #110)

you're right, i was doing too many things at once; https://treeherder.mozilla.org/jobs?repo=try&revision=a624f289e7d269bf52a3fc03dd0a1f9f436f11c5 should be good

yes that works. thank you

Flags: needinfo?(dofficialgman)

https://hg.mozilla.org/mozilla-central/rev/6b204e1c4234

Status: UNCONFIRMED → RESOLVED
Closed: 1 year ago
status-firefox130: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 130 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: