1806634 - Perma win ccov RuntimeError: dump_syms failed to produce the expected output

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=400208230&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Uj3yZh0kRWOBpkbmC-xI1A/runs/0/artifacts/public/logs/live_backing.log

---

[task 2022-12-20T10:50:07.782Z] 10:50:07     INFO -  Z:/task_167153007048083/build/src/xpcom/tests/gtest/TestLogging.cpp(49,13): note: On Windows executed functions: fopen, fopen_s, open, _open, _sopen, _sopen_s, OpenFile, CreateFileA should never be used due to lossy conversion from UTF8 to ANSI.
[task 2022-12-20T10:50:07.782Z] 10:50:07     INFO -  2 warnings generated.
[task 2022-12-20T10:50:07.782Z] 10:50:07     INFO -  mozmake[4]: Leaving directory 'z:/task_167153007048083/workspace/obj-build/xpcom/tests/gtest'
[task 2022-12-20T10:50:17.629Z] 10:50:17     INFO -  mozmake[4]: Entering directory 'z:/task_167153007048083/workspace/obj-build/build/win32'
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  z:/task_167153007048083/workspace/obj-build/_virtualenvs/build/Scripts/python.exe -m mozbuild.action.dumpsymbols z:/task_167153007048083/workspace/obj-build/dist/bin/crashinject.exe z:/task_167153007048083/workspace/obj-build/build/win32/crashinject.exe_syms.track
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  Running: z:\task_167153007048083\workspace\obj-build\_virtualenvs\build\Scripts\python.exe Z:/task_167153007048083/build/src\toolkit\crashreporter\tools\symbolstore.py -c --vcs-info -i --install-manifest=z:/task_167153007048083/workspace/obj-build\_build_manifests\install\dist_include,z:/task_167153007048083/workspace/obj-build\dist\include -s Z:/task_167153007048083/build/src Z:/task_167153007048083/fetches/dump_syms/dump_syms.exe z:/task_167153007048083/workspace/obj-build\dist\crashreporter-symbols z:\task_167153007048083\workspace\obj-build\dist\bin\crashinject.exe
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  Beginning work for file: z:\task_167153007048083\workspace\obj-build\dist\bin\crashinject.exe
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  Processing file: z:\task_167153007048083\workspace\obj-build\dist\bin\crashinject.exe
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  Z:\task_167153007048083\fetches\dump_syms\dump_syms.exe --inlines z:\task_167153007048083\workspace\obj-build\dist\bin\crashinject.exe
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  Unexpected error: dump_syms failed to produce the expected output
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  ----------
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  file: z:\task_167153007048083\workspace\obj-build\dist\bin\crashinject.exe
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  ----------
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  return code: 3221225781
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  ----------
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  first line of output:
[task 2022-12-20T10:50:17.643Z] 10:50:17    ERROR -  Traceback (most recent call last):
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -    File "Z:/task_167153007048083/build/src\toolkit\crashreporter\tools\symbolstore.py", line 1111, in <module>
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -      main()
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -    File "Z:/task_167153007048083/build/src\toolkit\crashreporter\tools\symbolstore.py", line 1106, in main
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -      dumper.Process(args[2], options.count_ctors)
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -    File "Z:/task_167153007048083/build/src\toolkit\crashreporter\tools\symbolstore.py", line 539, in Process
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -      self.ProcessFile(file_to_process, count_ctors=count_ctors)
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -    File "Z:/task_167153007048083/build/src\toolkit\crashreporter\tools\symbolstore.py", line 554, in ProcessFile
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -      file, arch_num, arch, vcs_root, dsymbundle, count_ctors=count_ctors
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -    File "Z:/task_167153007048083/build/src\toolkit\crashreporter\tools\symbolstore.py", line 700, in ProcessFileWork
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -      raise RuntimeError("\n----------\n".join(message))
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  RuntimeError: dump_syms failed to produce the expected output
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  ----------
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  file: z:\task_167153007048083\workspace\obj-build\dist\bin\crashinject.exe
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  ----------
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  return code: 3221225781
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  ----------
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -  first line of output:
[task 2022-12-20T10:50:17.643Z] 10:50:17    ERROR -  Traceback (most recent call last):
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -    File "c:\mozilla-build\python3\lib\runpy.py", line 193, in _run_module_as_main
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -      "__main__", mod_spec)
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -    File "c:\mozilla-build\python3\lib\runpy.py", line 85, in _run_code
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -      exec(code, run_globals)
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -    File "z:\task_167153007048083\build\src\python\mozbuild\mozbuild\action\dumpsymbols.py", line 110, in <module>
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -      sys.exit(main(sys.argv[1:]))
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -    File "z:\task_167153007048083\build\src\python\mozbuild\mozbuild\action\dumpsymbols.py", line 106, in main
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -      return dump_symbols(args.library_or_program, args.tracking_file, args.count_ctors)
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -    File "z:\task_167153007048083\build\src\python\mozbuild\mozbuild\action\dumpsymbols.py", line 86, in dump_symbols
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -      out_files = subprocess.check_output(args, universal_newlines=True)
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -    File "c:\mozilla-build\python3\lib\subprocess.py", line 336, in check_output
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -      **kwargs).stdout
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -    File "c:\mozilla-build\python3\lib\subprocess.py", line 418, in run
[task 2022-12-20T10:50:17.643Z] 10:50:17     INFO -      output=stdout, stderr=stderr)
[task 2022-12-20T10:50:17.657Z] 10:50:17     INFO -  subprocess.CalledProcessError: Command '['z:\\task_167153007048083\\workspace\\obj-build\\_virtualenvs\\build\\Scripts\\python.exe', 'Z:/task_167153007048083/build/src\\toolkit\\crashreporter\\tools\\symbolstore.py', '-c', '--vcs-info', '-i', '--install-manifest=z:/task_167153007048083/workspace/obj-build\\_build_manifests\\install\\dist_include,z:/task_167153007048083/workspace/obj-build\\dist\\include', '-s', 'Z:/task_167153007048083/build/src', 'Z:/task_167153007048083/fetches/dump_syms/dump_syms.exe', 'z:/task_167153007048083/workspace/obj-build\\dist\\crashreporter-symbols', 'z:\\task_167153007048083\\workspace\\obj-build\\dist\\bin\\crashinject.exe']' returned non-zero exit status 1.
[task 2022-12-20T10:50:17.657Z] 10:50:17    ERROR -  mozmake[4]: *** [Z:/task_167153007048083/build/src/config/rules.mk:655: crashinject.exe_syms.track] Error 1
[task 2022-12-20T10:50:17.657Z] 10:50:17     INFO -  mozmake[4]: Leaving directory 'z:/task_167153007048083/workspace/obj-build/build/win32'
[task 2022-12-20T10:50:17.657Z] 10:50:17     INFO -  mozmake[4]: Target 'syms' not remade because of errors.
[task 2022-12-20T10:50:17.657Z] 10:50:17    ERROR -  mozmake[3]: *** [Z:/task_167153007048083/build/src/config/recurse.mk:72: build/win32/syms] Error 2

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1797419

:glandium, since you are the author of the regressor, bug 1797419, could you take a look?

For more information, please visit auto_nag documentation.

Comment 2

[Mike Hommey [:glandium]]

This is actually another regression from bug 1806073. Rebuilding dump-syms on a changeset after its backout fixes the issue.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

I manually triggered dump_syms rebuilds today and Windows ccov builds are green again on m-c tip.

Comment 4

[Intermittent Failures Robot]

12 failures in 2999 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 11
• try: 1

Platform and build breakdown:

• windows2012-64: 12
  • ccov: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1806634&startday=2022-12-19&endday=2022-12-25&tree=all