1355453 - Windows Opt x86 and x64 build failure on 2017-04-11: subprocess.CalledProcessError: - mozmake.exe[5]: *** [crashinjectdll.dll_syms.track] Error 1

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect)

Description

[Jorg K (CEST = GMT+2)]

mozmake.exe[5]: Leaving directory 'c:/builds/moz2_slave/tb-c-cen-w32-ntly-000000000000/build/objdir-tb/build/win32/crashinjectdll'
c:/builds/moz2_slave/tb-c-cen-w32-ntly-000000000000/build/mozilla/config/recurse.mk:73: recipe for target 'build/win32/crashinjectdll/syms' failed
Traceback (most recent call last):
  File "c:\mozilla-build\python27\Lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\mozilla-build\python27\Lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "c:\builds\moz2_slave\tb-c-cen-w32-ntly-000000000000\build\mozilla\python\mozbuild\mozbuild\action\dumpsymbols.py", line 79, in <module>
    sys.exit(main(sys.argv[1:]))
  File "c:\builds\moz2_slave\tb-c-cen-w32-ntly-000000000000\build\mozilla\python\mozbuild\mozbuild\action\dumpsymbols.py", line 75, in main
    return dump_symbols(*argv)
  File "c:\builds\moz2_slave\tb-c-cen-w32-ntly-000000000000\build\mozilla\python\mozbuild\mozbuild\action\dumpsymbols.py", line 64, in dump_symbols
    out_files = subprocess.check_output(args)
  File "c:\mozilla-build\python27\Lib\subprocess.py", line 575, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['c:/builds/moz2_slave/tb-c-cen-w32-ntly-000000000000/build/objdir-tb/_virtualenv/Scripts/python.exe', u'c:/builds/moz2_slave/tb-c-cen-w32-ntly-000000000000/build/mozilla\\toolkit\\crashreporter\\tools\\symbolstore.py', '-c', '--vcs-info', '-i', u'--install-manifest=c:/builds/moz2_slave/tb-c-cen-w32-ntly-000000000000/build/objdir-tb\\_build_manifests\\install\\dist_include,c:/builds/moz2_slave/tb-c-cen-w32-ntly-000000000000/build/objdir-tb\\dist\\include', '-s', u'c:/builds/moz2_slave/tb-c-cen-w32-ntly-000000000000/build/mozilla', u'c:/builds/moz2_slave/tb-c-cen-w32-ntly-000000000000/build/objdir-tb\\dist\\host\\bin\\dump_syms.exe', u'c:/builds/moz2_slave/tb-c-cen-w32-ntly-000000000000/build/objdir-tb\\dist\\crashreporter-symbols', 'c:\\builds\\moz2_slave\\tb-c-cen-w32-ntly-000000000000\\build\\objdir-tb\\build\\win32\\crashinjectdll\\crashinjectdll.dll']' returned non-zero exit status 1
mozmake.exe[5]: *** [crashinjectdll.dll_syms.track] Error 1
mozmake.exe[4]: *** [build/win32/crashinjectdll/syms] Error 2
mozmake.exe[4]: *** Waiting for unfinished jobs....

Last successful build (on try): M-C: 731639fccc709a4dd95fed7e9dda88efb2
Failures with this M-C: 7031c811659d90275e7b8e155702779108

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=731639fccc709a4dd95fed7e9dda88efb2&tochange=7031c811659d90275e7b8e155702779108

Comment 1

[Jorg K (CEST = GMT+2)]

Edmund, any idea here?

Comment 2

[Frank-Rainer Grahl (:frg)]

My money would be on bug 1337986 - Move the work of buildsymbols into the compile tier

Comment 3

[Jorg K (CEST = GMT+2)]

Sounds plausible.

Chris, can you tell whether/how/why/where bug 1337986 is breaking our Windows Opt builds, debug builds are fine:
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=5a9983b82898ad03268862080953083e0b626be0&selectedJob=90401658

Comment 4

[Chris Manchester (limited bugmail, email directly)]

Yes, it looks like PDBSTR_PATH is set to a unix style path in the environment for debug builds. I had to convert this to a native windows path in bug 1337986 for various Firefox builds.

Comment 5

[Jorg K (CEST = GMT+2)]

Oh, I didn't paste the relevant error in comment #0:
PDBSTR_PATH=/c/Program Files (x86)/Windows Kits/8.0/Debuggers/x64/srcsrv/pdbstr.exe
Invalid path to pdbstr.exe - please set/check PDBSTR_PATH.

Where do we need to set this, I can't see this at all in C-C:
https://dxr.mozilla.org/comm-central/search?q=PDBSTR_PATH&redirect=false

The search only shows it in Makefile.in in mozilla/ which is basically M-C.

Could you please give me a few more hints where/how to fix this problem. Or does it need fixing in M-C?

Comment 6

[Richard Marti (:Paenglab)]

The path changes are done in this changeset: https://hg.mozilla.org/mozilla-central/rev/17fc25af0768 but does TB access them during tests? I doubt so.

The strange thing is, Windows opt has the wrong path in its log but the debug builds have nothing of this path in their logs.

Maybe the question is, why does opt (or tries to) run this test and debug not?

Comment 7

[Chris Manchester (limited bugmail, email directly)]

(In reply to Jorg K (GMT+2) from comment #5)
> Oh, I didn't paste the relevant error in comment #0:
> PDBSTR_PATH=/c/Program Files (x86)/Windows
> Kits/8.0/Debuggers/x64/srcsrv/pdbstr.exe
> Invalid path to pdbstr.exe - please set/check PDBSTR_PATH.
> 
> Where do we need to set this, I can't see this at all in C-C:
> https://dxr.mozilla.org/comm-central/search?q=PDBSTR_PATH&redirect=false
> 
> The search only shows it in Makefile.in in mozilla/ which is basically M-C.
> 
> Could you please give me a few more hints where/how to fix this problem. Or
> does it need fixing in M-C?

I'm sorry to say I don't know exactly where to set this, but the instances I see in that dxr search under mozilla/ are unix style paths. If those are being used in the Thunderbird builds that would be an issue, but I'd also look at https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/thunderbird_config.py as a wild guess.

Comment 8

[Richard Marti (:Paenglab)]

Attachment: Bug1355453.patch

Chris, thank you for the hint.

Rail, as a reference what I've done, see https://hg.mozilla.org/mozilla-central/rev/17fc25af0768. And when it's r+ could you land it then as I still not know how to do this?

Comment 9

[Jorg K (CEST = GMT+2)]

Rail, could you please push this to production as requested in comment #8. Thank you in advance.

Comment 10

[Edmund Wong (:ewong)]

I guess this has been solved.

Comment 11

[Jorg K (CEST = GMT+2)]

(In reply to Edmund Wong (:ewong) from comment #10)
> I guess this has been solved.
If someone could do the push.

Comment 12

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

https://hg.mozilla.org/build/buildbot-configs/rev/7ba6e65ecb152b810371f76fd483413b723075a3

Comment 13

[Jorg K (CEST = GMT+2)]

Rail, what's the process to get this merged into production? Is this automatic?

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Merged to prod with https://hg.mozilla.org/build/buildbot-configs/rev/11f5e306fe32b678656ef771cb4d70dc5d1ea742, reconfig in 20 minutes.