Open Bug 1911635 Opened 1 year ago Updated 1 year ago

[openbsd] delay rdd process sandbox initialization to enable vaapi

Categories

(Core :: Security: Process Sandboxing, enhancement, P3)

Product:
Core
Component:
Security: Process Sandboxing
Platform:
All
OpenBSD
Type:
enhancement

Tracking

()

Tracking Flags:
Tracking Status
firefox131 --- affected

People

(Reporter: gaston, Unassigned, NeedInfo)

Details

the RDD process is sandboxed on OpenBSD using pledge/unveil since bug 1713745, and that was fixed later on in 1856301.

at that time OpenBSD didnt have VAAPI support, but this has recently been imported, and is in a state where other media decoders (eg plain ffmpeg, vlc) can make use of VAAPI.

For firefox though, it enables lots of new codepaths, which fail pretty hard because of sandboxing. Some people have it (eg media hw decoding via vaapi) running fine once turning some knobs in about:config and opening wide the sandboxing config (eg https://marc.info/?l=openbsd-ports&m=172204141320769&w=2) , but that's not acceptable for our default packaging config.

There are various issues to fix:

i was initially toying with preopening early the drm fd before sandboxing and reusing it via a static var (eg https://cgit.rhaalovely.net/mozilla-firefox/commit/?h=vaapi&id=522e3de0d5f86176f740feeba8b92f5284f4262f) but that still fails, as the VAAPI initialization code does an fstat again on the drm device, and tries to open it in write mode, which isnt permitted by the sandbox.

for security reasons, we might not want to open wide the sandbox config, which only has so far the bare minimum. So im now considering putting the rdd process sandbox initialization right after initrddpdm in https://searchfox.org/mozilla-central/source/dom/media/platforms/PDMFactory.cpp#202 which seems to be where most of the initialization is called from.

so, all this talk to get feedback from ppl working in the sandboxing area wrt VAAPI, if you have insight about the proper way to move code around... that's welcome. After that, i'll put my wip patches on phabricator, once i've finally something that 'works'.

Severity: -- → S3
Flags: needinfo?(lissyx+mozillians)
Priority: -- → P3

my initial try with this:

Index: dom/media/platforms/PDMFactory.cpp
--- dom/media/platforms/PDMFactory.cpp.orig
+++ dom/media/platforms/PDMFactory.cpp
@@ -60,6 +60,11 @@
 #endif
 #include "FFVPXRuntimeLinker.h"
 
+#if defined(__OpenBSD__) && defined(MOZ_SANDBOX)
+#  include "mozilla/SandboxSettings.h"
+#  include "prlink.h"
+#endif
+
 #include <functional>
 
 using DecodeSupport = mozilla::media::DecodeSupport;
@@ -201,6 +206,9 @@ void PDMInitializer::InitPDMs() {
   } else if (XRE_IsRDDProcess()) {
     PDM_INIT_LOG("Init PDMs in RDD process");
     InitRddPDMs();
+#if defined(__OpenBSD__) && defined(MOZ_SANDBOX)
+  StartOpenBSDSandbox(GeckoProcessType_RDD);
+#endif
   } else if (XRE_IsUtilityProcess()) {
     PDM_INIT_LOG("Init PDMs in Utility process");
     InitUtilityPDMs();

fails, because it's still too early, per a MOZ_LOG=PlatformDecoderModule:5,OpenBSDSandbox:5,Dmabuf:5 output log for the rdd process:

[RDD 77005: Main Thread]: D/PlatformDecoderModule PDMInitializer, Init PDMs in RDD process
[RDD 77005: Main Thread]: D/PlatformDecoderModule FFMPEG: version: 0x20, macro: 58, micro: 100, isFFMpeg: yes
[RDD 77005: Main Thread]: D/PlatformDecoderModule FFMPEG: version: 0x80, macro: 61, micro: 101, isFFMpeg: yes
[RDD 77005: Main Thread]: D/PlatformDecoderModule FFVPX: Link result: Success
[RDD 77005: Main Thread]: D/OpenBSDSandbox /tmp: unveil(/tmp, rwc)
[RDD 77005: Main Thread]: D/OpenBSDSandbox /etc/firefox/pledge.rdd: pledge(stdio rpath tmppath recvfd sendfd unix)
....
[RDD 77005: MediaSupervisor #1]: D/PlatformDecoderModule FFmpeg decoder rejects requested type 'video/avc'
[RDD 77005: MediaSupervisor #1]: D/PlatformDecoderModule FFmpeg decoder supports requested type 'video/avc'
[RDD 77005: MediaSupervisor #1]: D/PlatformDecoderModule FFmpeg decoder supports requested type 'video/avc'
[RDD 77005: MediaSupervisor #1]: D/FFmpegVideo FFMPEG: FFmpegVideoDecoder::FFmpegVideoDecoder MIME video/avc Codec ID 27
[Child 94570: RemVidChild]: D/PlatformDecoderModule RemoteMediaDataDecoder[6acb3b90cb0] ::RemoteMediaDataDecoder: 6acb3b90cb0 is created
[RDD 77005: MediaPDecoder #1]: D/FFmpegVideo FFMPEG: Initialising VA-API FFmpeg decoder
[RDD 77005: MediaPDecoder #1]: D/FFmpegVideo FFMPEG:   codec h264 : H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
[RDD 77005: MediaPDecoder #1]: D/Dmabuf DMABufDevice::Configure()
[RDD 77005: MediaPDecoder #1]: D/Dmabuf Loading DMABuf system library libgbm.so ...
[RDD 77005: MediaPDecoder #1]: D/Dmabuf Failed to load libgbm.so, dmabuf isn't available.
[RDD 77005: MediaPDecoder #1]: D/Dmabuf GbmLib is not available!
[RDD 77005: MediaPDecoder #1]: D/Dmabuf Using DRM device /dev/dri/renderD128

can i directly put the StartOpenBSDSandbox() call in https://searchfox.org/mozilla-central/source/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp#298 so that i can be sure it's called after vaapi initialization ? or there's a better spot ? how can i ensure it will be only called once in the rdd process lifetime ? what about the other decoders ?

my next try will be putting it here at the end of FFmpegVideoDecoder<LIBAV_VER>::Init() but what about the other potential decoders ? I'm also a bit lost with the codepaths taken by the promises, all those async calls make it hard to figure out the callers/code flow.

a bit more progress by:

  • moving sandbox init right after InitVAAPIDecoder()
  • adding drm to pledge.rdd config, because vaapi is going to make ioctl() calls on the drm device one cant get away without it.

with that, vaapi initializes fine, but then i crash immediately in CreateEGLImage

#0  std::__1::shared_ptr<mozilla::gl::EglDisplay>::operator->[abi:v160006]() const (this=0xa68) at /usr/include/c++/v1/__memory/shared_ptr.h:857                                                                     
#1  DMABufSurfaceYUV::CreateEGLImage (this=0x11f9967e1c0, aGLContext=0x0, aPlane=<optimized out>) at /usr/obj/ports/firefox-130.0beta2/firefox-130.0/widget/gtk/DMABufSurface.cpp:1462                               
#2  0x0000011f7910cb5e in DMABufSurfaceYUV::VerifyTextureCreation (this=0x11f9967e1c0) at /usr/obj/ports/firefox-130.0beta2/firefox-130.0/widget/gtk/DMABufSurface.cpp:1573                                          
#3  0x0000011f783bc7ff in mozilla::VideoFramePool<58>::GetVideoFrameSurface (this=0x11f2ac89400, aVaDesc=..., aWidth=600, aHeight=600, aAVCodecContext=0x11f2ac76000, aAVFrame=0x11f99691800,                        
    aLib=0x11f7abb6480 <mozilla::sLibAV>) at /usr/obj/ports/firefox-130.0beta2/firefox-130.0/dom/media/platforms/ffmpeg/FFmpegVideoFramePool.cpp:239                                                                 
#4  mozilla::FFmpegVideoDecoder<58>::CreateImageVAAPI (this=0x11f2ac7f280, aOffset=2717, aPts=0, aDuration=30000, aResults=...)                                                                                      
    at /usr/obj/ports/firefox-130.0beta2/firefox-130.0/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp:1572                                                                                                        
#5  mozilla::FFmpegVideoDecoder<58>::DoDecode (this=0x11f2ac7f280, aSample=<optimized out>, aData=<optimized out>, aSize=<optimized out>, aGotFrame=0x1201f677917, aResults=...)                                     
    at /usr/obj/ports/firefox-130.0beta2/firefox-130.0/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp:1198                                                                                                        
#6  0x0000011f783b3609 in mozilla::FFmpegDataDecoder<58>::DoDecode (this=0x11f2ac7f280, aSample=0x11f9967e380, aGotFrame=0x1201f677918, aResults=...)                                                                
    at /usr/obj/ports/firefox-130.0beta2/firefox-130.0/dom/media/platforms/ffmpeg/FFmpegDataDecoder.cpp:213

GLcontext is null, as confirms the debug log:

[RDD 7962: MediaPDecoder #1]: D/FFmpegVideo FFMPEG: Frame decode takes 3.01 ms average decode time 3.01 ms frame duration 30.00 average frame duration 30.00 decoded 1 frames
[RDD 7962: MediaPDecoder #1]: D/FFmpegVideo FFMPEG: VA-API Got one frame output with pts=0 dts=60000 duration=30000
[RDD 7962: MediaPDecoder #1]: D/Dmabuf VideoFramePool::VideoFramePool() pool size 17
[RDD 7962: MediaPDecoder #1]: D/Dmabuf VideoFrameSurface: creating surface UID 1
[RDD 7962: MediaPDecoder #1]: D/Dmabuf Using VA-API DMABufSurface UID 1 FFMPEG ID 0x4000018
[RDD 7962: MediaPDecoder #1]: D/Dmabuf Surface pool size 1 used copied 0 used ffmpeg 0 (max 17) free ratio 1.000000
[RDD 7962: MediaPDecoder #1]: D/Dmabuf DMABufSurfaceYUV::UpdateYUVData() UID 1 copy 0
[RDD 7962: MediaPDecoder #1]: D/Dmabuf DMABufSurfaceYUV::ImportPRIMESurfaceDescriptor() UID 1
[RDD 7962: MediaPDecoder #1]: D/Dmabuf     plane 0 size 600 x 600 format 20203852
[RDD 7962: MediaPDecoder #1]: D/Dmabuf     plane 1 size 300 x 300 format 38385247
[RDD 7962: MediaPDecoder #1]: D/Dmabuf DMABufSurfaceYUV::VerifyTextureCreation() UID 1
[RDD 7962: MediaPDecoder #1]: D/Dmabuf ClaimSnapshotGLContext: Failed to create snapshot GLContext.
[RDD 7962: MediaPDecoder #1]: D/Dmabuf DMABufSurfaceYUV::CreateEGLImage() UID 1 plane 0

so https://searchfox.org/mozilla-central/source/gfx/gl/GLContextProviderLinux.cpp#36 probably fails. how can i know if i'm using EGL or GLX ?

about:support shows available in front of X11_EGL so i guess EGL is used.. and now i think about it, maybe the code tries to load libGL/libEGL super late, which might be prevented by sandboxing again.

preloading libGL/libEGL before sandboxing helps:

      rv = InitVAAPIDecoder();
      if (NS_SUCCEEDED(rv)) {
 +#if defined(__OpenBSD__) && defined(MOZ_SANDBOX)
+      PR_LoadLibrary("libGL.so");
+      PR_LoadLibrary("libEGL.so");
+      StartOpenBSDSandbox(GeckoProcessType_RDD);
 +#endif

but that still crashes later on, because the GL stack initialization reaches into way too much places:

  • (re)opening /dev/dri/* device nodes, read write (while the device nodes are already opened by firefox)
  • loading dri modules from /usr/X11R6/lib/modules/dri
  • does some sandbox-forbidden syscalls (tries an mmap with prot_exec)
  • and in the end something tries to do an mkdir deep down in the dri stack, which at that point is forbidden by the sandboxing:
firefox[31630]: pledge "cpath", syscall 136

#0  mkdir () at /tmp/-:2
#1  0xd9b6507c13ee8177 in ?? ()
#2  0x00000eb98417499a in ?? () from /usr/X11R6/lib/modules/dri/iris_dri.so
#3  0x00000eb984174801 in ?? () from /usr/X11R6/lib/modules/dri/iris_dri.so
#4  0x00000eb984172aab in ?? () from /usr/X11R6/lib/modules/dri/iris_dri.so
#5  0x00000eb9841729bf in ?? () from /usr/X11R6/lib/modules/dri/iris_dri.so
#6  0x00000eb984dbac9a in ?? () from /usr/X11R6/lib/modules/dri/iris_dri.so
#7  0x00000eb984da4137 in ?? () from /usr/X11R6/lib/modules/dri/iris_dri.so
#8  0x00000eb983cae05c in ?? () from /usr/X11R6/lib/modules/dri/iris_dri.so
#9  0x00000eb984489842 in ?? () from /usr/X11R6/lib/modules/dri/iris_dri.so
#10 0x00000eb983cb1c37 in ?? () from /usr/X11R6/lib/modules/dri/iris_dri.so
#11 0x00000eb983cb5234 in ?? () from /usr/X11R6/lib/modules/dri/iris_dri.so
#12 0x00000eb91bb48156 in dri2_create_screen (disp=0xeb9f30045c0) at /usr/xenocara/lib/mesa/mk/libEGL/../../src/egl/drivers/dri2/egl_dri2.c:1020
#13 0x00000eb91bb517bd in dri2_initialize_device (disp=0xeb9f30045c0) at /usr/xenocara/lib/mesa/mk/libEGL/../../src/egl/drivers/dri2/platform_device.c:367
#14 0x00000eb91bb4957f in dri2_initialize (disp=0xeb9f30045c0) at /usr/xenocara/lib/mesa/mk/libEGL/../../src/egl/drivers/dri2/egl_dri2.c:1130
#15 0x00000eb91bb55969 in eglInitialize (dpy=<optimized out>, major=0x0, minor=0x0) at /usr/xenocara/lib/mesa/mk/libEGL/../../src/egl/main/eglapi.c:701
#16 0x00000eb934a760ac in mozilla::gl::GLLibraryEGL::fInitialize (this=0xeb9f3002500, dpy=0xeb9f30045c0, major=0x0, minor=0x0) at /usr/obj/ports/firefox-130.0beta4/firefox-130.0/gfx/gl/GLLibraryEGL.h:347
#17 mozilla::gl::EglDisplay::Create (lib=..., display=0xeb9f30045c0, isWarp=true, aProofOfLock=...) at /usr/obj/ports/firefox-130.0beta4/firefox-130.0/gfx/gl/GLLibraryEGL.cpp:802
#18 0x00000eb934a775a8 in mozilla::gl::GetAndInitDeviceDisplay (egl=..., aProofOfLock=...) at /usr/obj/ports/firefox-130.0beta4/firefox-130.0/gfx/gl/GLLibraryEGL.cpp:210
#19 mozilla::gl::GLLibraryEGL::CreateDisplayLocked (this=0xeb9f3002500, forceAccel=<optimized out>, out_failureId=0xeb90e373738, aProofOfLock=...)
    at /usr/obj/ports/firefox-130.0beta4/firefox-130.0/gfx/gl/GLLibraryEGL.cpp:941
#20 0x00000eb934a76b82 in mozilla::gl::GLLibraryEGL::DefaultDisplay (this=0xeb9f3002500, out_failureId=0xeb90e373738) at /usr/obj/ports/firefox-130.0beta4/firefox-130.0/gfx/gl/GLLibraryEGL.cpp:871
#21 0x00000eb934a75c9a in mozilla::gl::DefaultEglDisplay (out_failureId=0xeb90e373738) at /usr/obj/ports/firefox-130.0beta4/firefox-130.0/gfx/gl/GLContextEGL.h:30
#22 mozilla::gl::GLContextProviderEGL::CreateHeadless (desc=..., out_failureId=0xeb90e373738) at /usr/obj/ports/firefox-130.0beta4/firefox-130.0/gfx/gl/GLContextProviderEGL.cpp:1245
#23 0x00000eb93739c270 in ClaimSnapshotGLContext () at /usr/obj/ports/firefox-130.0beta4/firefox-130.0/widget/gtk/DMABufSurface.cpp:89
#24 0x00000eb9373a4a63 in DMABufSurfaceYUV::VerifyTextureCreation (this=0xeb9f2ffe8c0) at /usr/obj/ports/firefox-130.0beta4/firefox-130.0/widget/gtk/DMABufSurface.cpp:1566
#25 0x00000eb9366546cf in mozilla::VideoFramePool<58>::GetVideoFrameSurface (this=0xeb9922cb880, aVaDesc=..., aWidth=80, aHeight=56, aAVCodecContext=0xeb9922b5800, aAVFrame=0xeb9922a4c00, 
    aLib=0xeb938e4f780 <mozilla::sLibAV>) at /usr/obj/ports/firefox-130.0beta4/firefox-130.0/dom/media/platforms/ffmpeg/FFmpegVideoFramePool.cpp:239
#26 mozilla::FFmpegVideoDecoder<58>::CreateImageVAAPI (this=0xeb9f8683000, aOffset=944, aPts=0, aDuration=333333, aResults=...)

so, trying to go down more into this rabbit hole seems .. a dead end.

would it be possible to do the GLContext initialization 'early' in the RDD process lifetime ? from my understanding at that point we're in the process mainloop (eg the top of the traceback is in do_decode or something like this) so i cant really push the sandbox init further, and in that case my 'solution' is to initialize as much as possible what needs 'special access' before sandboxing.

You need to log in before you can comment on or make changes to this bug.