2010162 - Perma Windows ccov Automation Error: mozharness timed out after 4800 seconds running <...> | 'mach build -v' did not run successfully. Please check log for errors.

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Treeherder Bug Filer]

Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=543331458&repo=mozilla-central&task=Yc9IwElWTxKKi0D9z_6o6A.1
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Yc9IwElWTxKKi0D9z_6o6A/runs/1/artifacts/public/logs/live_backing.log

---

[task 2026-01-13T13:07:46.938+00:00] 13:07:46     INFO -  check> TEST-PASS | check_js_msg_encoding.py | D:\task_176830572889440\build\src\js/xpconnect/src/jsshell.msg | ok
[task 2026-01-13T13:07:46.938+00:00] 13:07:46     INFO -  check> mozmake[4]: Leaving directory 'D:/task_176830572889440/workspace/obj-build/js/src'
[taskcluster 2026-01-13T13:08:36.134Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/Yc9IwElWTxKKi0D9z_6o6A/1/on/centralindia/vm-bx7p4omtu6fthojsmv2qetvmtdtsb64vqu7/until/1768310915.998
[task 2026-01-13T13:11:13.651+00:00] 13:11:13     INFO -  check> mozmake[4]: Entering directory 'D:/task_176830572889440/workspace/obj-build/ipc/ipdl/test/ipdl'
[task 2026-01-13T13:11:13.665+00:00] 13:11:13     INFO -  check> D:/task_176830572889440/.mozbuild/srcdirs/src-2b63e8e1c092/_virtualenvs/build/Scripts/python.exe 'D:/task_176830572889440/build/src/ipc/ipdl/test/ipdl/runtests.py' 'D:/task_176830572889440/build/src/ipc/ipdl/test/ipdl/ok' 'D:/task_176830572889440/build/src/ipc/ipdl/test/ipdl/error' D:/task_176830572889440/.mozbuild/srcdirs/src-2b63e8e1c092/_virtualenvs/build/Scripts/python.exe 'D:/task_176830572889440/build/src/ipc/ipdl/ipdl.py' --sync-msg-list 'D:/task_176830572889440/build/src/ipc/ipdl/test/ipdl/sync-messages.ini' --msg-metadata 'D:/task_176830572889440/build/src/ipc/ipdl/test/ipdl/message-metadata.ini' --ok-tests 'D:/task_176830572889440/build/src/ipc/ipdl/test/ipdl/ok/*.ipdl' 'D:/task_176830572889440/build/src/ipc/ipdl/test/ipdl/ok/*.ipdlh' --error-tests 'D:/task_176830572889440/build/src/ipc/ipdl/test/ipdl/error/*.ipdl' 'D:/task_176830572889440/build/src/ipc/ipdl/test/ipdl/error/*.ipdlh'
[task 2026-01-13T13:11:13.665+00:00] 13:11:13     INFO -  check> .....................................................................................................................................................................
[task 2026-01-13T13:11:13.666+00:00] 13:11:13     INFO -  check> ----------------------------------------------------------------------
[task 2026-01-13T13:11:13.666+00:00] 13:11:13     INFO -  check> Ran 165 tests in 208.272s
[task 2026-01-13T13:11:13.666+00:00] 13:11:13     INFO -  check>
[task 2026-01-13T13:11:13.666+00:00] 13:11:13     INFO -  check> OK
[task 2026-01-13T13:11:13.666+00:00] 13:11:13     INFO -  check> mozmake[4]: Leaving directory 'D:/task_176830572889440/workspace/obj-build/ipc/ipdl/test/ipdl'
[taskcluster 2026-01-13T13:25:36.461Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/Yc9IwElWTxKKi0D9z_6o6A/1/on/centralindia/vm-bx7p4omtu6fthojsmv2qetvmtdtsb64vqu7/until/1768311936.332
[taskcluster 2026-01-13T13:42:36.766Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/Yc9IwElWTxKKi0D9z_6o6A/1/on/centralindia/vm-bx7p4omtu6fthojsmv2qetvmtdtsb64vqu7/until/1768312956.633
[taskcluster 2026-01-13T13:59:37.026Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/Yc9IwElWTxKKi0D9z_6o6A/1/on/centralindia/vm-bx7p4omtu6fthojsmv2qetvmtdtsb64vqu7/until/1768313976.902
[taskcluster 2026-01-13T14:16:37.345Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/Yc9IwElWTxKKi0D9z_6o6A/1/on/centralindia/vm-bx7p4omtu6fthojsmv2qetvmtdtsb64vqu7/until/1768314997.202
[task 2026-01-13T14:31:13.749+00:00] 14:31:13     INFO - Automation Error: mozharness timed out after 4800 seconds running ['C:\\Users\\task_176830572889440\\.mozbuild\\srcdirs\\src-2b63e8e1c092\\_virtualenvs\\common\\Scripts\\python.exe', 'mach', '--log-no-times', 'build', '-v']
[task 2026-01-13T14:31:13.750+00:00] 14:31:13     INFO - Return code: 1
[task 2026-01-13T14:31:13.750+00:00] 14:31:13  WARNING - setting return code to 2
[task 2026-01-13T14:31:13.750+00:00] 14:31:13    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2026-01-13T14:31:13.750+00:00] 14:31:13    FATAL - Running post_fatal callback...
[task 2026-01-13T14:31:13.750+00:00] 14:31:13    FATAL - Exiting -1
[task 2026-01-13T14:31:13.750+00:00] 14:31:13     INFO - [mozharness: 2026-01-13 14:31:13.750547Z] Finished build step (failed)
[task 2026-01-13T14:31:13.751+00:00] 14:31:13     INFO - Running post-run listener: _parse_build_tests_ccov
[task 2026-01-13T14:31:13.751+00:00] 14:31:13     INFO - Running command: ['C:\\Users\\task_176830572889440\\.mozbuild\\srcdirs\\src-2b63e8e1c092\\_virtualenvs\\common\\Scripts\\python.exe', 'mach', 'python', 'testing\\parse_build_tests_ccov.py'] in D:/task_176830572889440/build/src
[task 2026-01-13T14:31:13.751+00:00] 14:31:13     INFO - Copy/paste: C:\Users\task_176830572889440\.mozbuild\srcdirs\src-2b63e8e1c092\_virtualenvs\common\Scripts\python.exe mach python testing\parse_build_tests_ccov.py
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO - Using env: {'ACCEPTED_MAR_CHANNEL_IDS': 'firefox-mozilla-central',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'ALLUSERSPROFILE': 'C:\\ProgramData',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'APPDATA': 'D:\\task_176830572889440\\AppData\\Roaming',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'COMMONPROGRAMFILES': 'C:\\Program Files\\Common Files',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'COMMONPROGRAMFILES(X86)': 'C:\\Program Files (x86)\\Common Files',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'COMMONPROGRAMW6432': 'C:\\Program Files\\Common Files',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'COMPUTERNAME': 'VM-BX7P4OMTU6FT',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'COMSPEC': 'C:\\Windows\\system32\\cmd.exe',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'DRIVERDATA': 'C:\\Windows\\System32\\Drivers\\DriverData',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'DXSDK_DIR': 'C:\\Program Files (x86)\\Microsoft DirectX SDK (June 2010)',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'EXTRA_MOZHARNESS_CONFIG': '{"mozconfig_variant":"fuzzing-ccov","objdir":"obj-build","update_channel":"nightly"}',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'GECKO_BASE_REPOSITORY': 'https://hg.mozilla.org/mozilla-unified',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'GECKO_HEAD_REF': 'caa037c3068b676922535eb88d2608015aed6653',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/mozilla-central',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'GECKO_HEAD_REV': 'caa037c3068b676922535eb88d2608015aed6653',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'GECKO_PATH': 'D:/task_176830572889440/build/src',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'GECKO_REPOSITORY_TYPE': 'hg',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'GRCOV_PATH': 'D:/task_176830572889440/fetches\\grcov\\grcov.exe',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'HG_CACHE': 'd:\\hg-cache',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'HG_SHARE_BASE_DIR': 'y:\\hg-shared',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'HG_STORE_PATH': 'y:/hg-shared',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'HOMEDRIVE': 'C:',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'HOMEPATH': '\\Users\\task_176830572889440',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'LOCALAPPDATA': 'C:\\Users\\task_176830572889440\\AppData\\Local',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'LOGONSERVER': '\\\\VM-BX7P4OMTU6FT',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE': 'system',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MACH_MAIN_PID': '4888',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MACH_SYSTEM_ASSERTED_COMPATIBLE_WITH_COMMON_SITE': '1',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MACH_SYSTEM_ASSERTED_COMPATIBLE_WITH_MACH_SITE': '1',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MAR_CHANNEL_ID': 'firefox-mozilla-central',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MH_BRANCH': 'mozilla-central',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZBUILD_STATE_PATH': 'D:\\task_176830572889440\\.mozbuild',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZILLABUILD': 'C:\\mozilla-build',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZPYTHONHOME': 'D:/task_176830572889440/fetches/python',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZ_ARTIFACT_PLATFORM': 'win64-fuzzing-ccov',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZ_AUTOMATION': '1',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZ_BUILD_DATE': '20260113093356',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZ_CRASHREPORTER_NO_REPORT': '1',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZ_FETCHES': '[{"artifact":"project/gecko/vs/vs.tar.zst","extract":true,"task":"AzJVk345QpOw7YvhMT6XiA"},{"artifact":"public/build/cbindgen.tar.zst","extract":true,"task":"XkHsO8NLQveB1WT3xrsUJw"},{"artifact":"public/build/clang.tar.zst","extract":true,"task":"PctUjlTYTD2yp212cEglrg"},{"artifact":"public/build/dump_syms.tar.zst","extract":true,"task":"I2qji4U9T7yMxeuHCT8-RA"},{"artifact":"public/build/dxc-x86_64-pc-windows-msvc.tar.zst","extract":true,"task":"Ed3lK6bwQlOBWG2xjiHb4w"},{"artifact":"public/build/grcov.tar.zst","extract":true,"task":"dSr8KjpCQ_G4r7OBMofryg"},{"artifact":"public/build/mozmake.tar.zst","extract":true,"task":"cq2mF_NmT4-4hG_CxfFS4Q"},{"artifact":"public/build/nasm.tar.zst","extract":true,"task":"E7l5vKkLSRSMM6BTOR8QDQ"},{"artifact":"public/build/node.tar.zst","extract":true,"task":"H60zb8X1Qq-lJfp-Gvq8YQ"},{"artifact":"public/build/nsis.tar.zst","extract":true,"task":"GdCe_FnhT2yPHiRJMaajaw"},{"artifact":"public/build/onnxruntime-x86_64-windows-msvc.tar.zst","extract":true,"task":"M_dqjDNPS52xgLks37QKkA"},{"artifact":"public/build/python.tar.zst","extract":true,"task":"Y42zVb9oSdy-viZq-NubVA"},{"artifact":"public/build/rust-size.tar.zst","extract":true,"task":"Lkn8uZdjQeqzENE6Fz8VpQ"},{"artifact":"public/build/rustc.tar.zst","extract":true,"task":"OMj3lt_MTR2R-_t6WdTAew"},{"artifact":"public/build/sccache.tar.zst","extract":true,"task":"FeUSv_hfSguBKJhVXkNpTQ"},{"artifact":"public/build/sysroot-wasm32-wasi.tar.zst","extract":true,"task":"VwHKQM1XTk244AMGa1iVeA"},{"artifact":"public/build/terser.tar.zst","extract":true,"task":"WxPQxKycQ3iL9xFI7mVYFQ"},{"artifact":"public/build/uv.tar.zst","extract":true,"task":"R-psVxwaQ2qnGpwJJpN7bQ"},{"artifact":"public/build/winappsdk-x86_64-pc-windows-msvc.tar.zst","extract":true,"task":"L0YsWqVvRaaAHiSd8XuIyQ"},{"artifact":"public/build/winchecksec.tar.zst","extract":true,"task":"d_of9ncYRp-yQTtS6YVKSg"},{"artifact":"public/windows-rs.tar.zst","extract":true,"task":"YWw5-VTzTCazKiQTrD6hzw"}]',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZ_FETCHES_DIR': 'D:/task_176830572889440/fetches',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZ_OBJDIR': 'D:\\task_176830572889440\\workspace\\obj-build',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZ_PYTHON_HOME': 'D:/task_176830572889440/fetches/python',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZ_SCM_LEVEL': '3',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZ_SIMPLE_PACKAGE_NAME': 'target',
[task 2026-01-13T14:31:13.752+00:00] 14:31:13     INFO -  'MOZ_SOURCE_CHANGESET': 'caa037c3068b676922535eb88d2608015aed6653',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'MOZ_SOURCE_REPO': 'https://hg.mozilla.org/mozilla-central',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'MOZ_UV_HOME': 'D:/task_176830572889440/fetches/uv',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'MSYSTEM': 'MINGW32',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'NUMBER_OF_PROCESSORS': '32',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'OS': 'Windows_NT',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PATH': 'C:\\Users\\task_176830572889440\\.mozbuild\\srcdirs\\src-2b63e8e1c092\\_virtualenvs\\common\\Scripts;D:/task_176830572889440/fetches/uv;D:/task_176830572889440/fetches/python;C:\\mozilla-build\\msys2\\usr\\bin;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\System32\\OpenSSH\\;C:\\Program '
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -          'Files\\dotnet\\;C:\\Program Files\\Git\\cmd;C:\\Program '
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -          'Files\\Puppet Labs\\Puppet\\bin;C:\\Program Files (x86)\\Windows '
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -          'Kits\\10\\Windows Performance Toolkit\\;C:\\Program '
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -          'Files\\Mercurial\\;C:\\mozilla-build\\bin;C:\\mozilla-build\\kdiff3;C:\\mozilla-build\\msys2;C:\\mozilla-build\\python3;C:\\mozilla-build\\mozmake;C:\\Users\\task_176830572889440\\AppData\\Local\\Microsoft\\WindowsApps',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PERFHERDER_EXTRA_OPTIONS': 'fuzzing-ccov',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PERFHERDER_FETCH_CONTENT_JSON_PATH': './perf/perfherder-data-fetch-content.json',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PIP_DOWNLOAD_CACHE': 'd:\\pip-cache',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PROCESSOR_ARCHITECTURE': 'AMD64',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PROCESSOR_IDENTIFIER': 'AMD64 Family 25 Model 17 Stepping 1, AuthenticAMD',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PROCESSOR_LEVEL': '25',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PROCESSOR_REVISION': '1101',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PROGRAMDATA': 'C:\\ProgramData',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PROGRAMFILES': 'C:\\Program Files',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PROGRAMFILES(X86)': 'C:\\Program Files (x86)',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PROGRAMW6432': 'C:\\Program Files',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PROMPT': '$P$G',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PSMODULEPATH': '%ProgramFiles%\\WindowsPowerShell\\Modules;C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\Modules',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PUBLIC': 'C:\\Users\\Public',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'PYTHONDONTWRITEBYTECODE': '1',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'REPOSITORIES': '{"gecko":"Mozilla Firefox"}',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'REQUIRE_GPU': '0',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'RUN_ID': '1',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'RUSTC_BOOTSTRAP': '1',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'SCCACHE_DISABLE': '1',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'SYSTEMDRIVE': 'C:',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'SYSTEMROOT': 'C:\\Windows',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'TASKCLUSTER_INSTANCE_TYPE': 'Standard_D32ads_v5',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'TASKCLUSTER_PROXY_URL': 'http://127.0.0.1:80',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'TASKCLUSTER_ROOT_URL': 'https://firefox-ci-tc.services.mozilla.com',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'TASKCLUSTER_WORKER_LOCATION': '{"cloud":"azure","region":"CentralIndia"}',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'TASK_GROUP_ID': 'b7XNJoVOQDuId4M8u15USQ',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'TASK_ID': 'Yc9IwElWTxKKi0D9z_6o6A',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'TASK_WORKDIR': 'D:/task_176830572889440',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'TEMP': 'C:\\Users\\task_176830572889440\\AppData\\Local\\Temp',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'TINDERBOX_OUTPUT': '1',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'TMP': 'C:\\Users\\task_176830572889440\\AppData\\Local\\Temp',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'TOOLTOOL_CACHE': 'c:/builds/tooltool_cache',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'TOOLTOOL_HOME': '/c/builds',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'UPLOAD_DIR': 'D:/task_176830572889440/public/build',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'USERDOMAIN': 'VM-BX7P4OMTU6FT',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'USERDOMAIN_ROAMINGPROFILE': 'VM-BX7P4OMTU6FT',
[task 2026-01-13T14:31:13.753+00:00] 14:31:13     INFO -  'USERNAME': 'task_176830572889440',
[task 2026-01-13T14:31:13.754+00:00] 14:31:13     INFO -  'USERPROFILE': 'C:\\Users\\task_176830572889440',
[task 2026-01-13T14:31:13.754+00:00] 14:31:13     INFO -  'VIRTUAL_ENV': 'C:\\Users\\task_176830572889440\\.mozbuild\\srcdirs\\src-2b63e8e1c092\\_virtualenvs\\common',
[task 2026-01-13T14:31:13.754+00:00] 14:31:13     INFO -  'WINDIR': 'C:\\Windows',
[task 2026-01-13T14:31:13.754+00:00] 14:31:13     INFO -  'WORKSPACE': 'D:\\task_176830572889440',
[task 2026-01-13T14:31:13.754+00:00] 14:31:13     INFO -  'XPCOM_DEBUG_BREAK': 'stack-and-abort'}
[task 2026-01-13T14:31:17.534+00:00] 14:31:17     INFO -  14:31:17 [ERROR] Error when running gcov: Failed to execute gcov process
[task 2026-01-13T14:31:17.534+00:00] 14:31:17     INFO -  14:31:17 [ERROR] Error when running gcov: Failed to execute gcov process
[task 2026-01-13T14:31:17.535+00:00] 14:31:17     INFO -  14:31:17 [ERROR] Error when running gcov: Failed to execute gcov process
<...>
[task 2026-01-13T14:31:19.578+00:00] 14:31:19     INFO - Return code: 0
[task 2026-01-13T14:31:19.578+00:00] 14:31:19     INFO - Running post-run listener: _relocate_artifacts
[task 2026-01-13T14:31:19.584+00:00] 14:31:19     INFO - The artifact D:/task_176830572889440/public/build\target.crashreporter-symbols-full.tar.zst is not present in this build. Skipping
[task 2026-01-13T14:31:19.584+00:00] 14:31:19     INFO - The artifact D:/task_176830572889440/public/build\sccache.log is not present in this build. Skipping
[task 2026-01-13T14:31:19.584+00:00] 14:31:19     INFO - The artifact D:/task_176830572889440/public/build\sccache-stats.json is not present in this build. Skipping
[task 2026-01-13T14:31:19.584+00:00] 14:31:19     INFO - Running post-run listener: _summarize
[task 2026-01-13T14:31:19.584+00:00] 14:31:19     INFO - [mozharness: 2026-01-13 14:31:19.584675Z] FxDesktopBuild summary:
[task 2026-01-13T14:31:19.704+00:00] Creating default state directory: C:\Users\task_176830572889440/.mozbuild
[task 2026-01-13T14:31:19.704+00:00] Running "pip check" to verify compatibility between the system Python and the "mach" site.
[task 2026-01-13T14:31:19.704+00:00] Creating local state directory: C:\Users\task_176830572889440\.mozbuild\srcdirs\src-2b63e8e1c092
[task 2026-01-13T14:31:19.704+00:00] Running "pip check" to verify compatibility between the system Python and the "common" site.
[task 2026-01-13T14:31:19.704+00:00] Creating the 'common' site at C:\Users\task_176830572889440\.mozbuild\srcdirs\src-2b63e8e1c092\_virtualenvs\common
[taskcluster 2026-01-13T14:31:19.771Z]                        Exit Code: 4294967295
[taskcluster 2026-01-13T14:31:19.771Z]                        User Time: 0s
[taskcluster 2026-01-13T14:31:19.771Z]                      Kernel Time: 46.875ms
[taskcluster 2026-01-13T14:31:19.771Z]                        Wall Time: 2h10m46.2113046s
[taskcluster 2026-01-13T14:31:19.771Z]  Average Available System Memory: 118.94 GiB
[taskcluster 2026-01-13T14:31:19.771Z]       Average System Memory Used: 9.06 GiB
[taskcluster 2026-01-13T14:31:19.771Z]          Peak System Memory Used: 38.91 GiB
[taskcluster 2026-01-13T14:31:19.771Z]              Total System Memory: 128.00 GiB
[taskcluster 2026-01-13T14:31:19.771Z]                           Result: FAILED
[taskcluster 2026-01-13T14:31:19.771Z] === Task Finished ===
[taskcluster 2026-01-13T14:31:19.771Z] Task Duration: 2h10m46.2134813s

Comment 1

[Serban Stanca [:SerbanS]]

It started with this autoland to central merge.

Hi Mike! Do you have any idea about this?

Thank you!

Comment 2

[Cristina Horotan [:chorotan]]

Since the backfills from autoland failed, I have tried on central, but it failed where it was green before.

Comment 3

[Intermittent Failures Robot]

44 failures were associated with this bug in the last 7 days.

This failure happened more than 30 times this week! Resolving this bug is a high priority.

Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled.

Repository breakdown:

• autoland: 1
• mozilla-central: 43

Table

	no_variant
linux-x86/ccov	39
linux-x86/debug	2
linux-x86/opt	2
macosx64-x86/debug	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2010162&startday=2026-01-12&endday=2026-01-18&tree=all

Comment 4

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

Thanks for the backfills, :chorotan! Per bug 2006698, the regression likely went back to Dec 15th. Could you backfill around that date to see if we can pinpoint the regression to an in-tree change?

Comment 5

[Cristina Horotan [:chorotan]]

Started backfills here

Comment 6

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

Thank you! It looks like it happened somewhere between d285bcf388090c90e4e285c18caec0525093104b and 70d8110fe32192bfb1880ebbebcc8dd993f296c7. Given it's intermittent, I retriggered 10 tasks per push in this range. Let's see how it goes.

Comment 7

[Cristina Horotan [:chorotan]]

I've added the job on autoland and started investigation up to here

Comment 8

[Intermittent Failures Robot]

41 failures were associated with this bug in the last 7 days.

This failure happened more than 30 times this week! Resolving this bug is a high priority.

Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled.

Repository breakdown:

• mozilla-central: 41

Table

	no_variant
linux-x86/ccov	38
linux-x86/opt	1
linux64-x86/ccov	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2010162&startday=2026-01-19&endday=2026-01-25&tree=all

Comment 9

[Sandor Molnar[:smolnar]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #6)

Thank you! It looks like it happened somewhere between d285bcf388090c90e4e285c18caec0525093104b and 70d8110fe32192bfb1880ebbebcc8dd993f296c7. Given it's intermittent, I retriggered 10 tasks per push in this range. Let's see how it goes.

@Johan, it seems it's coming from Bug 1989439 as seen from the backfills

Comment 10

[Alex Franchuk [:afranchuk]]

https://hg-edge.mozilla.org/mozilla-central/rev/9de2a5e939e2 should have disabled the test from running in ccov entirely. Is MOZ_CODE_COVERAGE not adequate to check this?

It seems to get stuck when running mach build. Is that running tests, too? Or is the new code somehow causing ccov compilation to get stuck?

Comment 11

[Mike Hommey [:glandium]]

(In reply to Alex Franchuk [:afranchuk] from comment #10)

https://hg-edge.mozilla.org/mozilla-central/rev/9de2a5e939e2 should have disabled the test from running in ccov entirely. Is MOZ_CODE_COVERAGE not adequate to check this?

It seems to get stuck when running mach build. Is that running tests, too?

mach build runs rust tests in ccov builds.

Comment 12

[Alex Franchuk [:afranchuk]]

All right, that's what I suspected must be occurring. Though it's still a mystery why this hangs when, I think, the test ought to be disabled.

Comment 13

[Intermittent Failures Robot]

38 failures were associated with this bug in the last 7 days.

This failure happened more than 30 times this week! Resolving this bug is a high priority.

Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled.

Repository breakdown:

• try: 1
• mozilla-central: 37

Table

	no_variant
linux-x86/ccov	37
linux64-x86/tsan	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2010162&startday=2026-01-26&endday=2026-02-01&tree=all

Comment 14

[Alex Franchuk [:afranchuk]]

I think there's some issue where the windows rust tests that are run with --enable-rust-tests don't have the correct mozconfig info, maybe?

I just tried an alternate approach to disabling the test (manually adding to RUST_TEST_FEATURES in toolkit/library/rust/moz.build), and a try run encountered the same timeout issue (implying the test still ran). FWIW, a similar timeout issue was happening on Linux, but in that case the timeout was the very short timeout built into the test itself, not that of mozharness. So it's possible this timeout is happening due to something a bit different (like a zombie child process, maybe).

I had to set up a new win11 VM so there was some delay, but now I can check this out in a windows-native environment.

Comment 15

[Alex Franchuk [:afranchuk]]

I've not been able to reproduce the behavior seen in CI (./mach build does not seem to run rusttests, or if it silently does, there is no hang). I also was not able to reproduce any hang when running ./mach rusttests with a ccov build.

glandium: do you know which flag(s) cause the ccov ./mach build in CI to also run rust tests?

Comment 16

[Mike Hommey [:glandium]]

ac_add_options --enable-rust-tests

Comment 17

[Intermittent Failures Robot]

57 failures were associated with this bug in the last 7 days.
This is the #32 most frequent failure this week.

This failure happened more than 30 times this week! Resolving this bug is a high priority.

Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled.

Repository breakdown:

• try: 4
• autoland: 3
• mozilla-central: 50

Table

	no_variant	s
android-x86/opt	2
android-x86_64/debug		1
linux-x86/ccov	50
linux-x86/debug	2
linux-x86/opt	1
linux64-x86/ccov	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2010162&startday=2026-02-02&endday=2026-02-08&tree=all

Comment 18

[Alex Franchuk [:afranchuk]]

Hmm, yeah that's what I have configured. Does it run silently? I don't see the usual rust test output.

Comment 19

[Alex Franchuk [:afranchuk]]

As a sanity check, I did a try push with the test force-ignored (rather than relying on a cfg value). The build still times out. So the test itself isn't the issue. It seems as if the presence of the code alone is the cause of the timeout. And my local builds don't time out, so this is something specific to the CI/mozharness environment.

Comment 20

[Alex Franchuk [:afranchuk]]

Attachment: Bug 2010162 - Disable the crashping tests in ccov builds r=gsvelto

Some tests cause timeouts on Windows and Linux (see bug 2006492) that
aren't locally reproducible, so it is more straightforward to simply
disable the crate's tests for ccov entirely.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/8f748c280ffa
https://hg.mozilla.org/integration/autoland/rev/fce37811e23f
Disable the crashping tests in ccov builds r=gsvelto

Comment 22

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/fce37811e23f

Comment 23

[Intermittent Failures Robot]

15 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 14
• try: 1

Table

	no_variant
linux-x86/ccov	14
linux-x86/debug	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2010162&startday=2026-02-09&endday=2026-02-15&tree=all