1446654 - Switching to lewton for vorbis decoding

Status: NEW

(Core :: Audio/Video: Playback, enhancement, P3)

Description

[est31]

Recently, Firefox has been struck with CVE-2018-5146 (bug 1446062) which is an out of bounds write vulnerability according to [1]. Safe Rust prevents out of bounds writes or reads. Some time ago, I've written the 100% safe Rust lewton [2] library to decode vorbis audio. It is being used successfully in the popular rodio library [3], the ggez framework [4], as well as other places in the Rust ecosystem (including the Mozilla-funded rust-av project).

I'd love to start discussion on using lewton instead of libvorbis for vorbis stream decoding inside Firefox.

Some points:

* IIRC Firefox also performs encoding of vorbis streams, inside WebRTC. lewton only performs decoding. So libvorbis would need to be kept. The decoder part of the library would disappear from the attack surface though. I'm not sure if there are build time options to disable decoder compilation but if not, the linker would most likely prune it.
* In running comparative benchmarks between lewton and libvorbis, my results are inconclusive and hardware dependent. In some settings, lewton is faster than libvorbis but in most settings it is slower... I fear I'm somewhat comparing clang to gcc which is probably a bad idea :)... Would love to see comprehensive performance comparisons across a wide range of hardware.
* Beyond performance, I know of a few rough edges around / bugs in lewton (see its issue tracker). Maybe there are even more lurking undiscovered. If there is general interest in inclusion, it'd be great to know which ones of these issues are blockers.
* Lewton currently assumes hard floats. I'm not certain what tremor is being used for in Firefox (presumably there are soft float Firefox builds), but most likely it'd have to stay.

Thank you!

[1]: https://www.mozilla.org/en-US/security/advisories/mfsa2018-08/
[2]: https://github.com/RustAudio/lewton
[3]: https://github.com/tomaka/rodio
[4]: https://github.com/ggez/ggez

PS: I also have an Ogg container decoder and encoder, written in Rust as well. But discussing that one is probably better left to a separate issue.

Comment 1

[Nils Ohlmeier [:drno]]

Thomas, was this the Rust implementation you mentioned to replace libvorbis in Firefox?

Comment 2

[Thomas Daede [:TD-Linux]]

Yes, it was. I'm actually not sure we use the encoding ability anywhere - it's definitely not used in WebRTC, I think the only APIs that could potentially access it are MediaRecorder and the Web Audio API.

Firefox no longer runs on any soft float devices, so that should not be a blocker - at this point Tremor is a historical oddity. Likewise for performance - libvorbis is totally lacking in even low-hanging SIMD, which tells me it was never treated as a primary concern.

I will need to maintain libvorbis in perpetuity as there are lots of projects that depend on it (hopefully it improves due to renewed attempts at fuzzing) but I would love to see lewton become the go-to decoder implementation. Bonus points if lewton gets a libvorbis-compatible API.

Comment 3

[Jean-Yves Avenard [:jya]]

Wouldn't moving to out of process decoding alleviate the security potential for all decoders (and encoders), regardless of codec?

Seems like a better approach to me than replace decoders one at a time, with untested entity.
This isn't a judgement in the quality of Leeton of course.

Comment 4

[est31]

> I'm actually not sure we use the encoding ability anywhere

I've searched for some encoding related functions (vorbis_analysis_buffer, vorbis_bitrate_addblock) using searchfox and didn't find anything... If there is vorbis encoding in Firefox it doesn't come from libvorbis :).

> I would love to see lewton become the go-to decoder implementation

Awesome! That would be really great :).

> Bonus points if lewton gets a libvorbis-compatible API.

Providing a libvorbis API should be possible. libvorbisfile would be a different matter (sample-correct seeking is hard :p) but apparently Firefox doesn't use it.

> Wouldn't moving to out of process decoding alleviate the security potential for all decoders (and encoders), regardless of codec?

To my limited knowledge of Firefox's process model, decoding happens inside the content process, which already is sandboxed. On the most recent Pwn2Own contest however, above mentioned CVE was used in combination with an attack on a Windows component [1] to escape the sandbox. So sandboxes are good, but no final solution. Nor is fuzzing. Not even safe Rust is a silver bullet to fix all security bugs. But each of the strategies improve security.

> Seems like a better approach to me than replace decoders one at a time, with untested entity.

Honestly I myself am a bit sceptical whether lewton supports all of the edge cases that libvorbis supports. In fact, I know of one file that libvorbis can play back but lewton can't... Most files do work really well however. But there is no better way to find out how good lewton is than by thorough testing.

[1]: https://www.thezdi.com/blog/2018/3/15/pwn2own-2018-day-two-results-and-master-of-pwn

Comment 5

[Jean-Yves Avenard [:jya]]

> To my limited knowledge of Firefox's process model, decoding happens inside the content process, which already is sandboxed. On the most recent Pwn2Own contest however, above mentioned CVE was used in combination with an attack on a Windows component [1] to escape the sandbox. So sandboxes are good, but no final solution. Nor is fuzzing. Not even safe Rust is a silver bullet to fix all security bugs. But each of the strategies improve security.

Exactly, the decoding is done in the content process, it should be moved away from that. Just as we are doing with the audio playback

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

> > Wouldn't moving to out of process decoding alleviate the security potential for all decoders (and encoders), regardless of codec?
> 
> To my limited knowledge of Firefox's process model, decoding happens inside
> the content process, which already is sandboxed. On the most recent Pwn2Own
> contest however, above mentioned CVE was used in combination with an attack
> on a Windows component [1] to escape the sandbox. So sandboxes are good, but
> no final solution. Nor is fuzzing. Not even safe Rust is a silver bullet to
> fix all security bugs. But each of the strategies improve security.

As part of the Process Isolation investigations I've been doing over the past year, one option that's been considered is to move all codecs (or at least any we're not actively fuzzing) to run in a GMP sandbox (the same sandbox used for EME, which is extremely locked-down compared to a Content Process).  There's no need to support a JIT, so GMP processes can't mark pages as executable, for example.  Pretty much all they can do is allocate memory, create threads, and send and receive IPC messages.  There's a small downside in performance -- however, this was designed to be used for codecs, so we already know the performance is acceptable, and the code-paths to support pushing codec data through GMP processes exist.

That would not eliminate the buffer-overruns in vorbis - but it would make them dramatically harder to exploit.  And this doesn't preclude switching to lewton, is compliments it - and as you say, Rust is not a silver bullet against all security issues anyways.

> 
> > Seems like a better approach to me than replace decoders one at a time, with untested entity.
> 
> Honestly I myself am a bit sceptical whether lewton supports all of the edge
> cases that libvorbis supports. In fact, I know of one file that libvorbis
> can play back but lewton can't... Most files do work really well however.
> But there is no better way to find out how good lewton is than by thorough
> testing.

Right - and that testing takes time and resources, for a codec that is largely legacy at this point.  GMP sandboxing can be completed much more quickly, and validating and switching to lewton can still occur later.

Chris - what would be required to switch vorbis to GMP, and what would be required to switch all our other SW codecs?  (I'm less concerned with OS-provided HW/SW decoders, though you can make an argument for sandboxing even those.)

Comment 7

[Chris Pearce [:cpearce (Not reading bugmail)]]

It would be nice to be able to switch over to decoders written in safe languages, in general, but it comes at a cost that we need to trade off against everything else we're doing. The existing libraries we have been using for a number of years have a big head start in terms of compatibility.

Also, we'd need to be certain of the level of support we can expect from maintainers of any library we come to depend on. We have the people working on libvorbis on our payroll, which gives us certainty there.

The approach we took when we were replacing our C++ MP4 demuxer with the new Rust demuxer was to run both, but use the old C++ demuxer's result, and report via telemetry whether the Rust demuxer's output matched the C++ demuxer's output. When the difference small enough we switched. However it's probably more expensive to run two decoders vs running two demuxers concurrently, so we'd need to be careful there.

In terms of process isolation, rather than moving to putting everything in GMPs, I think it makes sense to move all media out of process to a single sandboxed media process that is shared by all content processes. This would reduce our per-process overhead, and help us scale. I believe Jean-Yves has plans in this area, so I'll defer to him.

Comment 8

[Tom Ritter [:tjr] (OOTO until April)]

(In reply to Chris Pearce (:cpearce) from comment #7)
> In terms of process isolation, rather than moving to putting everything in
> GMPs, I think it makes sense to move all media out of process to a single
> sandboxed media process that is shared by all content processes. This would
> reduce our per-process overhead, and help us scale. I believe Jean-Yves has
> plans in this area, so I'll defer to him.

Any time we share a utility process among multiple content processes, we create a vector for bypassing site isolation. It doesn't mean the design is out of the question; but it's something I think we should consider carefully before choosing it.

Comment 9

[Randell Jesup [:jesup] (needinfo me)]

Agreed - there certainly are tradeoffs.  And also externally-loaded code like CDMs and probably OpenH264 will need to remain in GMP for other reasons (the security assumption is that any process running that code is already compromised).

As Tom says, this bypasses a part of site isolation, and because of the huge attack surface of codecs this is especially vulnerable code.  It's amazing that so few exploits leveraging codecs have been public.  It helps that codecs are fairly fuzzable - but random fuzzing doesn't catch everything (or doesn't in any reasonable time/resource limit).  Moving them out of Content makes total sense.  The question is where they move to, and the overheads and security/privacy implications implied.

Also, the more other-code/other-threads/etc running in the sandbox with a codec with a vulnerability, the easier it is to turn that buffer overrun/UAF/etc into a working exploit.  In a single-purpose sandbox, it's much harder.  So I wouldn't discount the idea of using GMPs for codecs yet - and the answer might be nuanced, with some codecs being "more trusted" than others.  

Note also that once you move media handling into another process, WebAudio (and worklets) become a huge issue.  If you allow multiple origins to run JS in a shared Media Process, you've seriously compromised site isolation, and added a ton of vulnerability vectors.  Another option would be a Media Process per origin -- but that also has impacts, and even that adds considerable vulnerability if the codecs are allowed to run directly there along with site JS.

Comment 10

[est31]

During the past months, I've focused on decoder correctness to reduce mismatches from libvorbis. With the 0.9.2 release [1], lewton now decodes the same samples on the xiph dataset as libvorbis, including other datasets.
In order to ensure 100% correctness, I'm now planning to run lewton on the approx. 500 GB dataset of ogg/vorbis files from Wikipedia/Wikimedia Commons [2]. I've requested a Wikipedia VPS instance for this purpose [3].

[1]: https://github.com/RustAudio/lewton/blob/0.9.3/CHANGELOG.md#release-093---october-28-2018
[2]: https://github.com/RustAudio/lewton/issues/36
[3]: https://phabricator.wikimedia.org/T212063