1917475 - Memory Bug in grcov: AddressSanitizer SEGV on Unknown Address

Status: RESOLVED FIXED

(Testing :: Code Coverage, defect)

Description

[Dongyeon Yu]

Attachment: output.rs

Hi,
We have recently developed a fuzzing tool to identify memory bugs in Rust programs (it can be combined with Address Sanitizer and activated with -Zsanitizer=address). We have built/run grcov using our fuzzing tool and also found a memory bug in it.

Environment:

• OS: Ubuntu 22.04.4 LTS
• Rust Compiler: rustc 1.83.0-nightly (9c01301c5, 2024-09-05)
• Cargo: cargo 1.83.0-nightly (c1fa840a8, 2024-08-29)
• grcov: Latest version (commit cc77ce34164fc3ea80ac579d1c15f36c9734133c, 2023-10-29)

Steps to reproduce:
(1) Replace grcov/benches/output.rs with the attached output.rs file.
(2) Build the benchmarks using the ASan flag.

RUSTFLAGS="-Zsanitizer=address" cargo bench --bench output --no-run --target x86_64-unknown-linux-gnu -- --release

(3) Run the binary

cd target/x86_64-unknown-linux-gnu/release/deps/
./output-544d6628beac7b41

---

Details:
// grcov/src/covdir.rs,

    fn get_coverage(coverage: BTreeMap<u32, u64>) -> (usize, usize, Vec<i64>) {
        let mut covered = 0;
        let last_line = *coverage.keys().last().unwrap_or(&0) as usize;
        let total = coverage.len();
        let mut lines: Vec<i64> = vec![-1; last_line];
        for (line_num, line_count) in coverage.iter() {
            let line_count = *line_count;
            unsafe {
                *lines.get_unchecked_mut((*line_num - 1) as usize) = line_count as i64;
            }
            covered += (line_count > 0) as usize;
        }
        (total, covered, lines)
    }

In this case, get_coverage uses the unsafe keyword to call get_unchecked_mut, which accesses memory without performing bounds checking. This violates Rust’s memory safety rules, as it can lead to invalid memory access if line_num is zero (0), potentially causing a segmentation fault (SEGV) at an unknown address

Actual results:
==1620241==ERROR: AddressSanitizer: SEGV on unknown address 0x00017fff8000 (pc 0x5555556cb4ea bp 0x7ffff4cfddd0 sp 0x7ffff4cfdce0 T1)
==1620241==The signal is caused by a READ memory access.
#0 0x5555556cb4ea in grcov::covdir::$LT$impl$u20$grcov..defs..CDFileStats$GT$::new::h0dc96c847a00dbdc (/home/dy3199/Fuzzing-Test/grcov_latest/grcov/target/x86_64-unknown-linux-gnu/release/deps/output-544d6628beac7b41+0x1774ea) (BuildId: 6897f926b78dbc228ff43fc5e6642c45c8c507ed)
#1 0x5555556bd955 in grcov::output::output_covdir::h9dd23d0655895cf0 (/home/dy3199/Fuzzing-Test/grcov_latest/grcov/target/x86_64-unknown-linux-gnu/release/deps/output-544d6628beac7b41+0x169955) (BuildId: 6897f926b78dbc228ff43fc5e6642c45c8c507ed)
#2 0x5555556e5f96 in test::bench::ns_iter_inner::h3439c21bed0dbccc output.fafd8d79820aac-cgu.06
#3 0x5555556e6289 in test::bench::Bencher::iter::h109937523f0a2f10 output.fafd8d79820aac-cgu.06
#4 0x5555556e3185 in core::ops::function::FnOnce::call_once::hae742c2051958e63 output.fafd8d79820aac-cgu.05
#5 0x55555577ffe0 in test::__rust_begin_short_backtrace::h19cb61a5553a6659 test.d5700e731e315db8-cgu.0
#6 0x55555577f33d in test::run_test::$u7b$$u7b$closure$u7d$$u7d$::hd3797bb249c9617e test.d5700e731e315db8-cgu.0
#7 0x5555557411b9 in std::sys::backtrace::__rust_begin_short_backtrace::hb64fc62652aa8df9 test.d5700e731e315db8-cgu.0
#8 0x555555744f28 in core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h9d7e8c2f99530c64 test.d5700e731e315db8-cgu.0
#9 0x55555572e14a in std::sys::pal::unix::thread::Thread::new::thread_start::h7725d3f895339174 std.c093c6b05171ebdd-cgu.0
#10 0x55555566f276 in asan_thread_start(void*) asan_interceptors.cpp
#11 0x7ffff7c94ac2 in start_thread nptl/pthread_create.c:442:8
#12 0x7ffff7d2684f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

==1620241==Register values:
rax = 0x0000000100000000 rbx = 0x00007ffff4cfdce0 rcx = 0x0000000000000000 rdx = 0x0000000000000008

rdi = 0x0000000800000000 rsi = 0x0000000000000000 rbp = 0x00007ffff4cfddd0 rsp = 0x00007ffff4cfdce0

r8 = 0x00000ffffe860015 r9 = 0x0000000000000000 r10 = 0x00000ffffe860017 r11 = 0x0000000000000000

r12 = 0x0000000000000000 r13 = 0x00007ffff43000a0 r14 = 0x00007ffff4300110 r15 = 0x00000ffffe860000

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/home/dy3199/Fuzzing-Test/grcov_latest/grcov/target/x86_64-unknown-linux-gnu/release/deps/output-544d6628beac7b41+0x1774ea) (BuildId: 6897f926b78dbc228ff43fc5e6642c45c8c507ed) in grcov::covdir::_$LT$impl$u20$grcov..defs..CDFileStats$GT$::new::h0dc96c847a00dbdc
Thread T1 created by T0 here:
#0 0x555555657111 in pthread_create (/home/dy3199/Fuzzing-Test/grcov_latest/grcov/target/x86_64-unknown-linux-gnu/release/deps/output-544d6628beac7b41+0x103111) (BuildId: 6897f926b78dbc228ff43fc5e6642c45c8c507ed)
#1 0x55555572dfac in std::sys::pal::unix::thread::Thread::new::hda210cf6917ea0c9 (/home/dy3199/Fuzzing-Test/grcov_latest/grcov/target/x86_64-unknown-linux-gnu/release/deps/output-544d6628beac7b41+0x1d9fac) (BuildId: 6897f926b78dbc228ff43fc5e6642c45c8c507ed)
#2 0x55555577d19b in test::run_test::heab5abb72715d4e8 (/home/dy3199/Fuzzing-Test/grcov_latest/grcov/target/x86_64-unknown-linux-gnu/release/deps/output-544d6628beac7b41+0x22919b) (BuildId: 6897f926b78dbc228ff43fc5e6642c45c8c507ed)
#3 0x55555575cf03 in test::console::run_tests_console::hdff422c5dd6bcc62 (/home/dy3199/Fuzzing-Test/grcov_latest/grcov/target/x86_64-unknown-linux-gnu/release/deps/output-544d6628beac7b41+0x208f03) (BuildId: 6897f926b78dbc228ff43fc5e6642c45c8c507ed)
#4 0x55555577a05f in test::test_main::h7f90b11b27d94337 (/home/dy3199/Fuzzing-Test/grcov_latest/grcov/target/x86_64-unknown-linux-gnu/release/deps/output-544d6628beac7b41+0x22605f) (BuildId: 6897f926b78dbc228ff43fc5e6642c45c8c507ed)
#5 0x55555577b05a in test::test_main_static::h6e0b16bced6ba111 (/home/dy3199/Fuzzing-Test/grcov_latest/grcov/target/x86_64-unknown-linux-gnu/release/deps/output-544d6628beac7b41+0x22705a) (BuildId: 6897f926b78dbc228ff43fc5e6642c45c8c507ed)
#6 0x5555556e5ce2 in std::sys::backtrace::__rust_begin_short_backtrace::h8af5994837567ba7 output.fafd8d79820aac-cgu.06
#7 0x5555556e5c34 in main (/home/dy3199/Fuzzing-Test/grcov_latest/grcov/target/x86_64-unknown-linux-gnu/release/deps/output-544d6628beac7b41+0x191c34) (BuildId: 6897f926b78dbc228ff43fc5e6642c45c8c507ed)
#8 0x7ffff7c29d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

==1620241==ABORTING

Comment 1

[Daniel Veditz [:dveditz]]

Not sure what security rating to give this. This is running in our Firefox CI so exploitation would be really indirect. Could someone with "try" access check in code that would trigger this bug, or do the coverage data itself need to be corrupt (and presumably we don't know of a bug in the data-generation tools that create the corrupted output).

Comment 2

[Dongyeon Yu]

(In reply to Daniel Veditz [:dveditz] from comment #1)

Not sure what security rating to give this. This is running in our Firefox CI so exploitation would be really indirect. Could someone with "try" access check in code that would trigger this bug, or do the coverage data itself need to be corrupt (and presumably we don't know of a bug in the data-generation tools that create the corrupted output).

Thank you for your response! I'd like to provide additional information to clarify the issue and emphasize its severity:

1. Corruption in Coverage Data : The bug occurs when the coverage data is corrupted—specifically when the line_num value is set to 0, which is atypical in correctly generated coverage files. This case is particularly concerning because:

• grcov is implemented in Rust, a language renowned for its strong memory safety guarantees.
• Rust's safety model typically prevents such issues by enforcing strict ownership and borrowing rules at compile-time.
• However, this specific case of corrupted input appears to bypass those safeguards and trigger a memory bug.
• Memory safety issues in Rust code are relatively rare and often carry higher expectations for safety compared to other languages.

The fact that this bug manifests in Rust code is significant. It suggests a vulnerability in handling malformed inputs that has managed to circumvent Rust's robust safety mechanisms. This makes the bug more severe than it might be in a language without such strong safety guarantees, as it represents a failure of one of Rust's core value propositions.

2. Severity and Exploitation: While the bug occurs in Firefox CI, potentially limiting direct exploitation, memory safety issues can have severe consequences if exploited.
3. Reproducing the Bug: This memory bug can be triggered by manually creating a corrupted coverage file with line_num = 0 and processing it with grcov.
4. Cause of Corrupted Data: I am uncertain whether the tools generating the coverage reports could produce such malformed data (line_num = 0) under normal circumstances. I've only observed the issue when corrupted data is manually introduced. Further investigation might be needed to determine if any coverage tools in the pipeline could inadvertently generate this type of corrupted data.
5. Recommended Next Steps: Given the potential severity of memory safety issues, I would suggest: Implementing input validation in grcov to safely handle unexpected line_num values.

Here's a sample code snippet

for (line_num, line_count) in coverage.iter() {
    // Validate line_num before use
    if *line_num == 0 || *line_num > lines.len() as u32 {
        return Err(format!("Invalid line number: {}", line_num));
    }
    let line_count = *line_count;
    lines[(*line_num - 1) as usize] = line_count as i64;
    covered += (line_count > 0) as usize;
}

Comment 3

[Marco Castelluccio [:marco]]

Thanks for the report Dongyeon.

Daniel, I don't think an attacker could do anything with this. It would be quite hard to exploit, and if they had access to try server, they could already do whatever they want on the try server.

Note: it's true that grcov is implemented in Rust, but when using unsafe code of course Rust safety guarantees do not apply.

Comment 4

[Dongyeon Yu]

Hi Marco,

Thank you for your response and for considering the issue.

I understand that, in this specific context, exploitation of this memory bug may be difficult, especially given the access control on the try server. While the bug may not be immediately exploitable due to the limited access on the try server, it still represents a memory safety issue within the project.

Could this issue qualify for a CVE under certain circumstances, particularly given Rust’s emphasis on memory safety and the risk of undefined behavior in unsafe code?

Comment 5

[Marco Castelluccio [:marco]]

(In reply to Dongyeon Yu from comment #4)

I understand that, in this specific context, exploitation of this memory bug may be difficult, especially given the access control on the try server. While the bug may not be immediately exploitable due to the limited access on the try server, it still represents a memory safety issue within the project.

I'm not sure how one could exploit such a memory bug wihout access to the machine where the tool is running, and if the attacker has access to the machine then they don't need the exploit because they can just already run whatever they want.

Could this issue qualify for a CVE under certain circumstances,

I'm not sure exactly what qualifies for a CVE. Dan, can you help?

particularly given Rust’s emphasis on memory safety and the risk of undefined behavior in unsafe code?

Again: Rust has nothing to do with this. If you use "unsafe", you're losing all of the guarantees Rust normally offers. If anything, this is another example that writing code in safe Rust would have prevented a memory bug.

Comment 6

[Daniel Veditz [:dveditz]]

I'm not sure exactly what qualifies for a CVE. Dan, can you help?

We'd need more evidence this is actually a vulnerability and not an unexploitable access violation. In the browser we give hypothetical attackers a giant benefit of the doubt because the ability to run scripts inside the application gives an attacker the ability to "groom" memory and time the triggering of the vulnerability. On top of that, there are millions of potential victims for browser bugs, not tens of CI environments running this tool.

Comment 7

[Marco Castelluccio [:marco]]

Fixed in https://github.com/mozilla/grcov/commit/c8219563bc91615dd4a27884a5c63f09db8d03bb.

Comment 8

[Dongyeon Yu]

Thank you so much for the quick patch!

Now that this issue has been resolved, will this bug report be made public automatically? Or could you please change it to public directly? It would be greatly helpful to other users who might experience similar issues.

Thank you.

As this is fixed and resolved, I want this (In reply to Marco Castelluccio [:marco] from comment #7)

Fixed in https://github.com/mozilla/grcov/commit/c8219563bc91615dd4a27884a5c63f09db8d03bb.

Comment 9

[Dongyeon Yu]

(In reply to Marco Castelluccio [:marco] from comment #7)

Fixed in https://github.com/mozilla/grcov/commit/c8219563bc91615dd4a27884a5c63f09db8d03bb.

Thank you so much for the quick patch!

Now that this issue has been resolved, will this bug report be made public automatically? Or could you please change it to public directly? It would be greatly helpful to other users who might experience similar issues.

Thank you.

Comment 10

[Andrew McCreight [:mccr8]]

With normal Firefox security bugs, we wait until the fix has been shipped for a while, but since we've determined there's not a real vulnerability given the level of privilege you'd need to be able to trigger this problem, I'll go ahead and open it right now. Thank you for the report.