Permanent for some pushes: Windows ccov check> thread 'main' panicked at toolkit\crashreporter\client\app\src\test.rs:48:9:
Categories
(Toolkit :: Crash Reporting, defect)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox-esr115 | --- | unaffected |
| firefox-esr128 | --- | unaffected |
| firefox131 | --- | unaffected |
| firefox132 | --- | unaffected |
| firefox133 | --- | affected |
People
(Reporter: intermittent-bug-filer, Unassigned)
References
(Regression)
Details
(Keywords: intermittent-failure, regression)
Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=476344537&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/eDfvI1DIQp-FBbrWglbufg/runs/2/artifacts/public/logs/live_backing.log
[task 2024-10-01T00:39:16.553Z] 00:39:16 INFO - check> test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.76s
[task 2024-10-01T00:39:16.553Z] 00:39:16 INFO - check>
[task 2024-10-01T00:39:16.554Z] 00:39:16 INFO - check> Running `set CARGO='\\?\D:\task_172773934938410\fetches\rustc\bin\cargo.exe'&& set CARGO_MANIFEST_DIR='D:\task_172773934938410\build\src\toolkit\crashreporter\client\app'&& set CARGO_PKG_AUTHORS=''&& set CARGO_PKG_DESCRIPTION=''&& set CARGO_PKG_HOMEPAGE=''&& set CARGO_PKG_LICENSE=''&& set CARGO_PKG_LICENSE_FILE=''&& set CARGO_PKG_NAME=crashreporter&& set CARGO_PKG_README=''&& set CARGO_PKG_REPOSITORY=''&& set CARGO_PKG_RUST_VERSION=''&& set CARGO_PKG_VERSION=1.0.0&& set CARGO_PKG_VERSION_MAJOR=1&& set CARGO_PKG_VERSION_MINOR=0&& set CARGO_PKG_VERSION_PATCH=0&& set CARGO_PKG_VERSION_PRE=''&& set GLEAN_METRICS_FILE='D:/task_172773934938410/workspace/obj-build\toolkit\crashreporter\client\app\glean_metrics.rs'&& set OUT_DIR='D:/task_172773934938410/workspace/obj-build\x86_64-pc-windows-msvc\release\build\crashreporter-45e89f464b7134af\out'&& set PATH='D:/task_172773934938410/workspace/obj-build\x86_64-pc-windows-msvc\release\build\lmdb-rkv-sys-efaad72280c08b68\out;D:/task_172773934938410/workspace/obj-build\x86_64-pc-windows-msvc\release\build\whatsys-9f37387a9f329d0f\out;D:/task_172773934938410/workspace/obj-build\x86_64-pc-windows-msvc\release\deps;D:/task_172773934938410/workspace/obj-build\x86_64-pc-windows-msvc\release;D:\task_172773934938410\fetches\rustc\lib\rustlib\x86_64-pc-windows-msvc\lib;C:/mozilla-build/msys2/usr/bin;C:\mozilla-build\bin;D:\task_172773934938410\.mozbuild\srcdirs\src-19ee8823e225\_virtualenvs\build\Scripts;C:\Users\task_172773934938410\.mozbuild\srcdirs\src-19ee8823e225\_virtualenvs\common\Scripts;D:/task_172773934938410/fetches/python;C:\mozilla-build\msys2\usr\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files\Git\cmd;C:\Program Files\Puppet Labs\Puppet\bin;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files (x86)\GNU\GnuPG\pub;C:\Program Files\Mercurial\;C:\Program Files\Mercurial;C:\mozilla-build\kdiff;C:\mozilla-build\moztools-x64\bin;C:\mozilla-build\mozmake;C:\mozilla-build\nsis-3.01;C:\mozilla-build\python;C:\mozilla-build\python\Scripts;C:\mozilla-build\python3;C:\mozilla-build\msys\bin;C:\mozilla-build\msys\local\bin;C:\mozilla-build\msys2;C:\Users\task_172773934938410\AppData\Local\Microsoft\WindowsApps;C:/Users/task_172773934938410/.cargo/bin;D:/task_172773934938410/fetches/rustc/bin'&& D:/task_172773934938410/workspace/obj-build\x86_64-pc-windows-msvc\release\deps\crashreporter-44f8311c78d2bf34.exe`
[task 2024-10-01T00:39:16.557Z] 00:39:16 INFO - check>
[task 2024-10-01T00:39:16.557Z] 00:39:16 INFO - check> running 35 tests
[task 2024-10-01T00:39:16.557Z] 00:39:16 INFO - check> test data::test::event ... ok
[task 2024-10-01T00:39:16.557Z] 00:39:16 INFO - check> test net::report::test::report_http ... ok
[task 2024-10-01T00:39:16.557Z] 00:39:16 INFO - check> test net::http::test::curl_quote ... ok
[task 2024-10-01T00:39:16.557Z] 00:39:16 INFO - check> test data::test::synchronized ... ok
[task 2024-10-01T00:39:16.557Z] 00:39:16 INFO - check> test test::delete_dump ... ok
[task 2024-10-01T00:39:16.557Z] 00:39:16 INFO - check> test net::report::test::report_response ... ok
[task 2024-10-01T00:39:16.558Z] 00:39:16 INFO - check> test test::comment ... ok
[task 2024-10-01T00:39:16.558Z] 00:39:16 INFO - check> test data::test::ondemand ... ok
[task 2024-10-01T00:39:16.558Z] 00:39:16 INFO - check> test test::error_dialog ... ok
[task 2024-10-01T00:39:16.558Z] 00:39:16 INFO - check> test test::no_submit ... ok
[task 2024-10-01T00:39:16.558Z] 00:39:16 INFO - check> test test::include_url ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16 INFO - check> test test::curl_library ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16 INFO - check> test test::auto_submit ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16 INFO - check> test test::eol_version ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16 INFO - check> test test::data_dir_default ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16 INFO - check> test test::minidump_analyzer_error ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16 INFO - check> test test::no_extra_file ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16 INFO - check> test test::ping_and_event_files ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16 INFO - check> test test::quit ... ok
[task 2024-10-01T00:39:16.559Z] 00:39:16 INFO - check> test test::no_dump_file ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16 INFO - check> test test::response_indicates_discarded ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16 INFO - check> test test::report_not_sent ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16 INFO - check> test test::rename_failure_uses_copy ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16 INFO - check> test test::pingsender_failure ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16 INFO - check> test test::real_curl_library ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16 INFO - check> test test::curl_binary ... ok
[task 2024-10-01T00:39:16.560Z] 00:39:16 INFO - check> test test::no_restart_with_windows_error_reporting ... ok
[task 2024-10-01T00:39:16.561Z] 00:39:16 INFO - check> test test::glean_ping_extra_stack_trace_fields ... ok
[task 2024-10-01T00:39:16.561Z] 00:39:16 INFO - check> test test::report_response_failed ... ok
[task 2024-10-01T00:39:16.561Z] 00:39:16 INFO - check> test test::response_stop_sending_reports ... ok
[task 2024-10-01T00:39:16.561Z] 00:39:16 INFO - check> test test::response_view_url ... ok
[task 2024-10-01T00:39:16.561Z] 00:39:16 INFO - check> test test::restart ... ok
[task 2024-10-01T00:39:16.562Z] 00:39:16 INFO - check> test test::details_window ... ok
[task 2024-10-01T00:39:16.562Z] 00:39:16 INFO - check> test test::real_curl_binary ... ok
[task 2024-10-01T00:39:16.562Z] 00:39:16 INFO - check> test test::glean_ping ... FAILED
[task 2024-10-01T00:39:16.562Z] 00:39:16 INFO - check>
[task 2024-10-01T00:39:16.562Z] 00:39:16 INFO - check> failures:
[task 2024-10-01T00:39:16.562Z] 00:39:16 INFO - check>
[task 2024-10-01T00:39:16.562Z] 00:39:16 INFO - check> ---- test::glean_ping stdout ----
[task 2024-10-01T00:39:16.562Z] 00:39:16 INFO - check> ---- test::glean_ping stderr ----
[task 2024-10-01T00:39:16.563Z] 00:39:16 INFO - check> [ERROR crashreporter::net::ping] failed to send legacy ping: failed to open ping file ping_dir\00000000-0000-0000-0000-000000000000.json for writing: entity not found
[task 2024-10-01T00:39:16.563Z] 00:39:16 INFO - check> [WARN crashreporter::logic] failed to update events file: failed to open event file at events_dir\minidump: entity not found
[task 2024-10-01T00:39:16.563Z] 00:39:16 INFO - check> [WARN crashreporter::logic] failed to write submission event: failed to write event to events_dir\minidump-submission
[task 2024-10-01T00:39:16.563Z] 00:39:16 ERROR - check> thread 'main' panicked at toolkit\crashreporter\client\app\src\test.rs:48:9:
[task 2024-10-01T00:39:16.563Z] 00:39:16 INFO - check> assertion `left == right` failed
[task 2024-10-01T00:39:16.563Z] 00:39:16 INFO - check> left: 2
[task 2024-10-01T00:39:16.563Z] 00:39:16 INFO - check> right: 1
[task 2024-10-01T00:39:16.563Z] 00:39:16 INFO - check> stack backtrace:
[task 2024-10-01T00:39:16.564Z] 00:39:16 INFO - check> 0: 0x7ff7cf75e3ad - std::backtrace_rs::backtrace::dbghelp64::trace
[task 2024-10-01T00:39:16.564Z] 00:39:16 INFO - check> at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\..\..\backtrace\src\backtrace\dbghelp64.rs:91
[task 2024-10-01T00:39:16.564Z] 00:39:16 INFO - check> 1: 0x7ff7cf75e3ad - std::backtrace_rs::backtrace::trace_unsynchronized
[task 2024-10-01T00:39:16.565Z] 00:39:16 INFO - check> at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\..\..\backtrace\src\backtrace\mod.rs:66
[task 2024-10-01T00:39:16.565Z] 00:39:16 INFO - check> 2: 0x7ff7cf75e3ad - std::sys::backtrace::_print_fmt
[task 2024-10-01T00:39:16.565Z] 00:39:16 INFO - check> at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\sys\backtrace.rs:65
[task 2024-10-01T00:39:16.565Z] 00:39:16 INFO - check> 3: 0x7ff7cf75e3ad - std::sys::backtrace::impl$0::print::impl$0::fmt
[task 2024-10-01T00:39:16.565Z] 00:39:16 INFO - check> at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\sys\backtrace.rs:40
[task 2024-10-01T00:39:16.565Z] 00:39:16 INFO - check> 4: 0x7ff7cf77b2b9 - core::fmt::rt::Argument::fmt
[task 2024-10-01T00:39:16.566Z] 00:39:16 INFO - check> at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\core\src\fmt\rt.rs:173
[task 2024-10-01T00:39:16.566Z] 00:39:16 INFO - check> 5: 0x7ff7cf77b2b9 - core::fmt::write
[task 2024-10-01T00:39:16.566Z] 00:39:16 INFO - check> at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\core\src\fmt\mod.rs:1182
[task 2024-10-01T00:39:16.567Z] 00:39:16 INFO - check> 6: 0x7ff7cf7592d1 - std::io::Write::write_fmt<std::sys::pal::windows::stdio::Stderr>
[task 2024-10-01T00:39:16.567Z] 00:39:16 INFO - check> at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\io\mod.rs:1827
[task 2024-10-01T00:39:16.567Z] 00:39:16 INFO - check> 7: 0x7ff7cf7608c7 - std::panicking::default_hook::closure$1
[task 2024-10-01T00:39:16.567Z] 00:39:16 INFO - check> at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\panicking.rs:269
[task 2024-10-01T00:39:16.567Z] 00:39:16 INFO - check> 8: 0x7ff7cf7604b9 - std::panicking::default_hook
[task 2024-10-01T00:39:16.567Z] 00:39:16 INFO - check> at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\std\src\panicking.rs:296
[task 2024-10-01T00:39:16.567Z] 00:39:16 INFO - check> 9: 0x7ff7cf579d41 - alloc::boxed::impl$50::call
[task 2024-10-01T00:39:16.568Z] 00:39:16 INFO - check> at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\alloc\src\boxed.rs:2084
[task 2024-10-01T00:39:16.568Z] 00:39:16 INFO - check> 10: 0x7ff7cf579d41 - test::run_test_in_spawned_subprocess::closure$0
[task 2024-10-01T00:39:16.568Z] 00:39:16 INFO - check> at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library\test\src\lib.rs:741
Comment 1•1 year ago
|
||
Michael, Tyson, this is a permanent bustage of the fuzzying code coverage on Windows and started with the merge of the webrtc v129 changes (bug 1918268). Please investigate.
Comment 2•1 year ago
|
||
This failure is likely a concurrency bug: the glean tests rely on global state (in glean) and so must be serialized. The fact that it fails only on windows with fuzzing might imply that there is some specific ordering going on there (it's surprising that it is reproducible at all). Does fuzzing also fuzz thread scheduling?
Comment 3•1 year ago
|
||
Also, I don't think this is a sec bug.
Updated•1 year ago
|
Comment 4•1 year ago
|
||
Not sure why this would only happen on fuzzing ccov builds. My guess is that this is related to code coverage and not fuzzing related flags. AFAIK the fuzzing builds are not doing anything different, they only make more features available for use.
Comment 6•1 year ago
•
|
||
This would be my realm, as it is only related to glean in that glean is imposing a few limitations on the implementation. I wrote the tests and code being tested here.
Comment 7•1 year ago
•
|
||
FWIW, I have no idea how bug 1918268 can possibly relate to this.
Comment 8•1 year ago
|
||
(In reply to Alex Franchuk [:afranchuk] from comment #7)
FWIW, I have no idea how bug 1918268 can possibly relate to this.
That was my thinking as well, but I'm slowly bisecting and pushing build-win64-fuzzing-ccov/opt jobs one at a time to try in order to narrow down what change causes the issue. Let me know if there is anything else I can do in the meantime.
Comment 9•1 year ago
|
||
This does not reproduce with either the last push to elm before it got merged nor with the next push to central.
If it's observed again, glandium will be consulted.
Comment 10•1 year ago
•
|
||
Such is the nature of concurrency bugs :( I really thought I removed the possible issues from occurring (namely, two tests that use glean running concurrently, which is what causes this failure).
It's good to note that the assertion which fails is being a bit more rigorous than it needs to be: if for some strange reason glean sent more than one ping out (which is what is being checked), that wouldn't have an adverse effect on the user. Admittedly, that could be caused by a logic bug, but in these cases it's simply caused by multiple glean tests trying to use the same static glean state.
Comment 11•1 year ago
|
||
:mjf, since you are the author of the regressor, bug 1918268, could you take a look? Also, could you set the severity field?
For more information, please visit BugBot documentation.
Updated•1 year ago
|
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
Updated•1 year ago
|
Description
•