Closed Bug 1921924 Opened 1 year ago Closed 1 year ago

Permanent for some pushes: Windows ccov check> thread 'main' panicked at toolkit\crashreporter\client\app\src\test.rs:48:9:

Categories

(Toolkit :: Crash Reporting, defect)

defect

Tracking

()

RESOLVED DUPLICATE of bug 1937295
Tracking Status
firefox-esr115 --- unaffected
firefox-esr128 --- unaffected
firefox131 --- unaffected
firefox132 --- unaffected
firefox133 --- affected

People

(Reporter: intermittent-bug-filer, Unassigned)

References

(Regression)

Details

(Keywords: intermittent-failure, regression)

Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=476344537&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/eDfvI1DIQp-FBbrWglbufg/runs/2/artifacts/public/logs/live_backing.log


[task 2024-10-01T00:39:16.553Z] 00:39:16     INFO -  check> test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.76s
[task 2024-10-01T00:39:16.553Z] 00:39:16     INFO -  check>
[task 2024-10-01T00:39:16.554Z] 00:39:16     INFO -  check>      Running `set CARGO='\\?\D:\task_172773934938410\fetches\rustc\bin\cargo.exe'&& set CARGO_MANIFEST_DIR='D:\task_172773934938410\build\src\toolkit\crashreporter\client\app'&& set CARGO_PKG_AUTHORS=''&& set CARGO_PKG_DESCRIPTION=''&& set CARGO_PKG_HOMEPAGE=''&& set CARGO_PKG_LICENSE=''&& set CARGO_PKG_LICENSE_FILE=''&& set CARGO_PKG_NAME=crashreporter&& set CARGO_PKG_README=''&& set CARGO_PKG_REPOSITORY=''&& set CARGO_PKG_RUST_VERSION=''&& set CARGO_PKG_VERSION=1.0.0&& set CARGO_PKG_VERSION_MAJOR=1&& set CARGO_PKG_VERSION_MINOR=0&& set CARGO_PKG_VERSION_PATCH=0&& set CARGO_PKG_VERSION_PRE=''&& set GLEAN_METRICS_FILE='D:/task_172773934938410/workspace/obj-build\toolkit\crashreporter\client\app\glean_metrics.rs'&& set OUT_DIR='D:/task_172773934938410/workspace/obj-build\x86_64-pc-windows-msvc\release\build\crashreporter-45e89f464b7134af\out'&& set PATH='D:/task_172773934938410/workspace/obj-build\x86_64-pc-windows-msvc\release\build\lmdb-rkv-sys-efaad72280c08b68\out;D:/task_172773934938410/workspace/obj-build\x86_64-pc-windows-msvc\release\build\whatsys-9f37387a9f329d0f\out;D:/task_172773934938410/workspace/obj-build\x86_64-pc-windows-msvc\release\deps;D:/task_172773934938410/workspace/obj-build\x86_64-pc-windows-msvc\release;D:\task_172773934938410\fetches\rustc\lib\rustlib\x86_64-pc-windows-msvc\lib;C:/mozilla-build/msys2/usr/bin;C:\mozilla-build\bin;D:\task_172773934938410\.mozbuild\srcdirs\src-19ee8823e225\_virtualenvs\build\Scripts;C:\Users\task_172773934938410\.mozbuild\srcdirs\src-19ee8823e225\_virtualenvs\common\Scripts;D:/task_172773934938410/fetches/python;C:\mozilla-build\msys2\usr\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files\Git\cmd;C:\Program Files\Puppet Labs\Puppet\bin;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files (x86)\GNU\GnuPG\pub;C:\Program Files\Mercurial\;C:\Program Files\Mercurial;C:\mozilla-build\kdiff;C:\mozilla-build\moztools-x64\bin;C:\mozilla-build\mozmake;C:\mozilla-build\nsis-3.01;C:\mozilla-build\python;C:\mozilla-build\python\Scripts;C:\mozilla-build\python3;C:\mozilla-build\msys\bin;C:\mozilla-build\msys\local\bin;C:\mozilla-build\msys2;C:\Users\task_172773934938410\AppData\Local\Microsoft\WindowsApps;C:/Users/task_172773934938410/.cargo/bin;D:/task_172773934938410/fetches/rustc/bin'&& D:/task_172773934938410/workspace/obj-build\x86_64-pc-windows-msvc\release\deps\crashreporter-44f8311c78d2bf34.exe`
[task 2024-10-01T00:39:16.557Z] 00:39:16     INFO -  check>
[task 2024-10-01T00:39:16.557Z] 00:39:16     INFO -  check> running 35 tests
[task 2024-10-01T00:39:16.557Z] 00:39:16     INFO -  check> test data::test::event ... ok
[task 2024-10-01T00:39:16.557Z] 00:39:16     INFO -  check> test net::report::test::report_http ... ok
[task 2024-10-01T00:39:16.557Z] 00:39:16     INFO -  check> test net::http::test::curl_quote ... ok
[task 2024-10-01T00:39:16.557Z] 00:39:16     INFO -  check> test data::test::synchronized ... ok
[task 2024-10-01T00:39:16.557Z] 00:39:16     INFO -  check> test test::delete_dump ... ok
[task 2024-10-01T00:39:16.557Z] 00:39:16     INFO -  check> test net::report::test::report_response ... ok
[task 2024-10-01T00:39:16.558Z] 00:39:16     INFO -  check> test test::comment ... ok
[task 2024-10-01T00:39:16.558Z] 00:39:16     INFO -  check> test data::test::ondemand ... ok
[task 2024-10-01T00:39:16.558Z] 00:39:16     INFO -  check> test test::error_dialog ... ok
[task 2024-10-01T00:39:16.558Z] 00:39:16     INFO -  check> test test::no_submit ... ok
[task 2024-10-01T00:39:16.558Z] 00:39:16     INFO -  check> test test::include_url ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16     INFO -  check> test test::curl_library ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16     INFO -  check> test test::auto_submit ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16     INFO -  check> test test::eol_version ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16     INFO -  check> test test::data_dir_default ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16     INFO -  check> test test::minidump_analyzer_error ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16     INFO -  check> test test::no_extra_file ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16     INFO -  check> test test::ping_and_event_files ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16     INFO -  check> test test::quit ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16     INFO -  check> test test::no_dump_file ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16     INFO -  check> test test::response_indicates_discarded ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16     INFO -  check> test test::report_not_sent ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16     INFO -  check> test test::rename_failure_uses_copy ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16     INFO -  check> test test::pingsender_failure ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16     INFO -  check> test test::real_curl_library ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16     INFO -  check> test test::curl_binary ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16     INFO -  check> test test::no_restart_with_windows_error_reporting ... ok
[task 2024-10-01T00:39:16.561Z] 00:39:16     INFO -  check> test test::glean_ping_extra_stack_trace_fields ... ok
[task 2024-10-01T00:39:16.561Z] 00:39:16     INFO -  check> test test::report_response_failed ... ok
[task 2024-10-01T00:39:16.561Z] 00:39:16     INFO -  check> test test::response_stop_sending_reports ... ok
[task 2024-10-01T00:39:16.561Z] 00:39:16     INFO -  check> test test::response_view_url ... ok
[task 2024-10-01T00:39:16.561Z] 00:39:16     INFO -  check> test test::restart ... ok
[task 2024-10-01T00:39:16.562Z] 00:39:16     INFO -  check> test test::details_window ... ok
[task 2024-10-01T00:39:16.562Z] 00:39:16     INFO -  check> test test::real_curl_binary ... ok
[task 2024-10-01T00:39:16.562Z] 00:39:16     INFO -  check> test test::glean_ping ... FAILED
[task 2024-10-01T00:39:16.562Z] 00:39:16     INFO -  check>
[task 2024-10-01T00:39:16.562Z] 00:39:16     INFO -  check> failures:
[task 2024-10-01T00:39:16.562Z] 00:39:16     INFO -  check>
[task 2024-10-01T00:39:16.562Z] 00:39:16     INFO -  check> ---- test::glean_ping stdout ----
[task 2024-10-01T00:39:16.562Z] 00:39:16     INFO -  check> ---- test::glean_ping stderr ----
[task 2024-10-01T00:39:16.563Z] 00:39:16     INFO -  check> [ERROR crashreporter::net::ping] failed to send legacy ping: failed to open ping file ping_dir\00000000-0000-0000-0000-000000000000.json for writing: entity not found
[task 2024-10-01T00:39:16.563Z] 00:39:16     INFO -  check> [WARN  crashreporter::logic] failed to update events file: failed to open event file at events_dir\minidump: entity not found
[task 2024-10-01T00:39:16.563Z] 00:39:16     INFO -  check> [WARN  crashreporter::logic] failed to write submission event: failed to write event to events_dir\minidump-submission
[task 2024-10-01T00:39:16.563Z] 00:39:16    ERROR -  check> thread 'main' panicked at toolkit\crashreporter\client\app\src\test.rs:48:9:
[task 2024-10-01T00:39:16.563Z] 00:39:16     INFO -  check> assertion `left == right` failed
[task 2024-10-01T00:39:16.563Z] 00:39:16     INFO -  check>   left: 2
[task 2024-10-01T00:39:16.563Z] 00:39:16     INFO -  check>  right: 1
[task 2024-10-01T00:39:16.563Z] 00:39:16     INFO -  check> stack backtrace:
[task 2024-10-01T00:39:16.564Z] 00:39:16     INFO -  check>    0:     0x7ff7cf75e3ad - std::backtrace_rs::backtrace::dbghelp64::trace
[task 2024-10-01T00:39:16.564Z] 00:39:16     INFO -  check>                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\..\..\backtrace\src\backtrace\dbghelp64.rs:91
[task 2024-10-01T00:39:16.564Z] 00:39:16     INFO -  check>    1:     0x7ff7cf75e3ad - std::backtrace_rs::backtrace::trace_unsynchronized
[task 2024-10-01T00:39:16.565Z] 00:39:16     INFO -  check>                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\..\..\backtrace\src\backtrace\mod.rs:66
[task 2024-10-01T00:39:16.565Z] 00:39:16     INFO -  check>    2:     0x7ff7cf75e3ad - std::sys::backtrace::_print_fmt
[task 2024-10-01T00:39:16.565Z] 00:39:16     INFO -  check>                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\sys\backtrace.rs:65
[task 2024-10-01T00:39:16.565Z] 00:39:16     INFO -  check>    3:     0x7ff7cf75e3ad - std::sys::backtrace::impl$0::print::impl$0::fmt
[task 2024-10-01T00:39:16.565Z] 00:39:16     INFO -  check>                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\sys\backtrace.rs:40
[task 2024-10-01T00:39:16.565Z] 00:39:16     INFO -  check>    4:     0x7ff7cf77b2b9 - core::fmt::rt::Argument::fmt
[task 2024-10-01T00:39:16.566Z] 00:39:16     INFO -  check>                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\core\src\fmt\rt.rs:173
[task 2024-10-01T00:39:16.566Z] 00:39:16     INFO -  check>    5:     0x7ff7cf77b2b9 - core::fmt::write
[task 2024-10-01T00:39:16.566Z] 00:39:16     INFO -  check>                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\core\src\fmt\mod.rs:1182
[task 2024-10-01T00:39:16.567Z] 00:39:16     INFO -  check>    6:     0x7ff7cf7592d1 - std::io::Write::write_fmt<std::sys::pal::windows::stdio::Stderr>
[task 2024-10-01T00:39:16.567Z] 00:39:16     INFO -  check>                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\io\mod.rs:1827
[task 2024-10-01T00:39:16.567Z] 00:39:16     INFO -  check>    7:     0x7ff7cf7608c7 - std::panicking::default_hook::closure$1
[task 2024-10-01T00:39:16.567Z] 00:39:16     INFO -  check>                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\panicking.rs:269
[task 2024-10-01T00:39:16.567Z] 00:39:16     INFO -  check>    8:     0x7ff7cf7604b9 - std::panicking::default_hook
[task 2024-10-01T00:39:16.567Z] 00:39:16     INFO -  check>                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\panicking.rs:296
[task 2024-10-01T00:39:16.567Z] 00:39:16     INFO -  check>    9:     0x7ff7cf579d41 - alloc::boxed::impl$50::call
[task 2024-10-01T00:39:16.568Z] 00:39:16     INFO -  check>                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\alloc\src\boxed.rs:2084
[task 2024-10-01T00:39:16.568Z] 00:39:16     INFO -  check>   10:     0x7ff7cf579d41 - test::run_test_in_spawned_subprocess::closure$0
[task 2024-10-01T00:39:16.568Z] 00:39:16     INFO -  check>                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\test\src\lib.rs:741

Michael, Tyson, this is a permanent bustage of the fuzzying code coverage on Windows and started with the merge of the webrtc v129 changes (bug 1918268). Please investigate.

Flags: needinfo?(twsmith)
Flags: needinfo?(mfroman)
Keywords: regression
Regressed by: 1918268

This failure is likely a concurrency bug: the glean tests rely on global state (in glean) and so must be serialized. The fact that it fails only on windows with fuzzing might imply that there is some specific ordering going on there (it's surprising that it is reproducible at all). Does fuzzing also fuzz thread scheduling?

Also, I don't think this is a sec bug.

Group: firefox-core-security

Not sure why this would only happen on fuzzing ccov builds. My guess is that this is related to code coverage and not fuzzing related flags. AFAIK the fuzzing builds are not doing anything different, they only make more features available for use.

Flags: needinfo?(twsmith)

Does this glean issue fall into your realm?

Flags: needinfo?(chutten)

This would be my realm, as it is only related to glean in that glean is imposing a few limitations on the implementation. I wrote the tests and code being tested here.

Flags: needinfo?(chutten)

FWIW, I have no idea how bug 1918268 can possibly relate to this.

(In reply to Alex Franchuk [:afranchuk] from comment #7)

FWIW, I have no idea how bug 1918268 can possibly relate to this.

That was my thinking as well, but I'm slowly bisecting and pushing build-win64-fuzzing-ccov/opt jobs one at a time to try in order to narrow down what change causes the issue. Let me know if there is anything else I can do in the meantime.

This does not reproduce with either the last push to elm before it got merged nor with the next push to central.

If it's observed again, glandium will be consulted.

Flags: needinfo?(mfroman)
Summary: Perma Windows ccov check> thread 'main' panicked at toolkit\crashreporter\client\app\src\test.rs:48:9: → Permanent for some pushes: Windows ccov check> thread 'main' panicked at toolkit\crashreporter\client\app\src\test.rs:48:9:

Such is the nature of concurrency bugs :( I really thought I removed the possible issues from occurring (namely, two tests that use glean running concurrently, which is what causes this failure).

It's good to note that the assertion which fails is being a bit more rigorous than it needs to be: if for some strange reason glean sent more than one ping out (which is what is being checked), that wouldn't have an adverse effect on the user. Admittedly, that could be caused by a logic bug, but in these cases it's simply caused by multiple glean tests trying to use the same static glean state.

:mjf, since you are the author of the regressor, bug 1918268, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Flags: needinfo?(mfroman)
Severity: -- → S4
Flags: needinfo?(mfroman)
See Also: → 1913084
See Also: → 1937295
Status: NEW → RESOLVED
Closed: 1 year ago
Duplicate of bug: 1937295
Resolution: --- → DUPLICATE
See Also: 1937295
You need to log in before you can comment on or make changes to this bug.