Closed Bug 1273812 Opened 10 years ago Closed 10 years ago

taskcluster win64 (debug) build failure: Command line error D8050 : cannot execute 'x:\Task_1463478272\build\src\vs2015u2\VC\bin\amd64\c1.dll': failed to get command line into debug records

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: grenade, Unassigned)

References

Details

build: https://treeherder.mozilla.org/#/jobs?repo=try&revision=22b59f95028b60784592ba0bd4fc8dd11537b40e&selectedJob=20964870 raw-log: https://public-artifacts.taskcluster.net/N8zTBkWFTAWWlyFT6oYhIA/0/public/logs/command_000007.log failure: 15:43:23 INFO - x:/Task_1463478272/build/src/vs2015u2/VC/bin/amd64/cl.EXE -Fox:/Task_1463478272/build/src/obj-firefox/security/nss/lib/softoken/pkcs11.obj -c -O1 -Zi -Fdx:/Task_1463478272/build/src/obj-firefox/security/nss/lib/softoken/ -MD -w44267 -w44244 -w44018 -w44312 -FS -W3 -nologo -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_WARNINGS -DXP_PC -DSHLIB_SUFFIX=\"dll\" -DSHLIB_PREFIX=\"\" -DSOFTOKEN_LIB_NAME=\"softokn3.dll\" -DSHLIB_VERSION=\"3\" -UDEBUG -DNDEBUG -DWIN64 -D_AMD64_ -D_WINDOWS -DWIN95 -DNSS_NO_INIT_SUPPORT -DUSE_UTIL_DIRECTLY -DNO_NSPR_10_SUPPORT -DSSL_DISABLE_DEPRECATED_CIPHER_SUITE_NAMES -Ix:/Task_1463478272/build/src/obj-firefox/dist/include -Ix:/Task_1463478272/build/src/obj-firefox/dist/include/nspr -Ix:/Task_1463478272/build/src/obj-firefox/dist/include/nspr -Ix:/Task_1463478272/build/src/obj-firefox/dist/include/nss -Ix:/Task_1463478272/build/src/obj-firefox/dist/private/nss "/x/Task_1463478272/build/src/security/nss/lib/softoken/pkcs11.c" 15:43:23 INFO - fipstest.c 15:43:23 INFO - cl : Command line error D8050 : cannot execute 'x:\Task_1463478272\build\src\vs2015u2\VC\bin\amd64\c1.dll': failed to get command line into debug records 15:43:23 INFO - ../../coreconf/rules.mk:388: recipe for target 'x:/Task_1463478272/build/src/obj-firefox/security/nss/lib/softoken/fipstest.obj' failed 15:43:23 INFO - mozmake.EXE[6]: *** [x:/Task_1463478272/build/src/obj-firefox/security/nss/lib/softoken/fipstest.obj] Error 2 15:43:23 INFO - mozmake.EXE[6]: *** Waiting for unfinished jobs.... 15:43:23 INFO - fipsaudt.c 15:43:23 INFO - fipstokn.c 15:43:23 INFO - lgglue.c 15:43:23 INFO - lowkey.c 15:43:23 INFO - lowpbe.c 15:43:23 INFO - pkcs11.c 15:43:23 INFO - padbuf.c 15:43:24 INFO - mozmake.EXE[6]: Leaving directory 'x:/Task_1463478272/build/src/security/nss/lib/softoken' 15:43:24 INFO - Makefile:455: recipe for target 'libs-nss/lib/softoken' failed 15:43:24 INFO - mozmake.EXE[5]: *** [libs-nss/lib/softoken] Error 2 15:43:24 INFO - mozmake.EXE[5]: Leaving directory 'x:/Task_1463478272/build/src/obj-firefox/config/external/nss' 15:43:24 INFO - x:/Task_1463478272/build/src/config/recurse.mk:71: recipe for target 'config/external/nss/target' failed 15:43:24 INFO - mozmake.EXE[4]: *** [config/external/nss/target] Error 2 15:43:24 INFO - mozmake.EXE[4]: Leaving directory 'x:/Task_1463478272/build/src/obj-firefox' 15:43:24 INFO - x:/Task_1463478272/build/src/config/recurse.mk:32: recipe for target 'compile' failed 15:43:24 INFO - mozmake.EXE[3]: *** [compile] Error 2 15:43:24 INFO - mozmake.EXE[3]: Leaving directory 'x:/Task_1463478272/build/src/obj-firefox' 15:43:24 INFO - x:/Task_1463478272/build/src/config/rules.mk:540: recipe for target 'default' failed 15:43:24 INFO - mozmake.EXE[2]: *** [default] Error 2 15:43:24 INFO - mozmake.EXE[2]: Leaving directory 'x:/Task_1463478272/build/src/obj-firefox' 15:43:24 INFO - x:/Task_1463478272/build/src/client.mk:410: recipe for target 'realbuild' failed 15:43:24 INFO - mozmake.EXE[1]: *** [realbuild] Error 2 15:43:24 INFO - mozmake.EXE[1]: Leaving directory 'x:/Task_1463478272/build/src' 15:43:24 INFO - client.mk:168: recipe for target 'build' failed 15:43:24 INFO - mozmake.EXE: *** [build] Error 2 15:43:24 INFO - 240 compiler warnings present. 15:43:26 INFO - 2 15:43:26 ERROR - Return code: 1 15:43:26 WARNING - setting return code to 2 15:43:26 FATAL - 'mach build' did not run successfully. Please check log for errors.
good find :glandium!
I also checked this after finding that SO answer, but as shown in the build log, TEMP and TMP were set. I did come across this posting on reddit https://www.reddit.com/r/VisualStudio2015/comments/4doeen/simultaneous_command_line_builds_getting_errors which suggests that it's fixable by setting a distinct TMP and TEMP path for parallel processes. Any idea where in the codebase such a setting could go? eg: for a try push to see if it has any impact.
Flags: needinfo?(mh+mozilla)
There is no easy place for that. OTOH, there is something interesting in your log: there is a msys path for the source file (/x/... instead of x:/), which shouldn't be there in the first place. Whether this is related or not is not granted, but there's that. Another question is why is this, and many of the recently filed bugs, are happening on those jobs while they definitely aren't happening on buildbot slaves or local builds, nor have I seen them on the AWS machines I do "local" windows builds on.
Flags: needinfo?(mh+mozilla)
So the main reason we get new errors in TC that we don't get elsewhere, is that we are documenting (in one place) every single builder dependency as we build our builders (https://github.com/MozRelOps/OpenCloudConfig/blob/master/userdata/Manifest/win2012.json). Literally every bit of magic sauce required to put a vanilla ec2 win 2012 ami into a state where it can build firefox under mozharness. This isn't true of buildbot slaves where a combination of imported gpo config, puppet and userdata powershell fiddling as well as a good dose of manual tinkering were combined to produce an ami that can build firefox under mozharness and buildbot. The json manifest above, is what is actually used to transform vanilla amis into builders. There's no manual step, no secret sauce, no intervention from someone with years of experience hacking a system environment variable, just what's in the manifest. This includes: - Environment Variables - Registry Settings - Path and symlink hacks (like copying python27.dll to C:\\mozilla-build\\python\\Scripts\\python27.dll, so that hg will start working) - Software Installations (either msis or exes) - Commands that alter the system (like `pip install --upgrade virtualenv`) If it's not in the manifest or the try push, it wasn't used in the build. No exceptions, no ambiguity. For mspdb140.dll errors, I have no idea why it happens when running mach without -j1 on win32 only. All I can say is that it's exceptionally reproducible, with absolute consistency. And that if we knew what magic environment variable or registry value or software install or path hack would make it go away, we'd document it in the manifest to make sure it doesn't happen again. There's no equivalent manifest for buildbot slaves or 'local builds' that we can reference to get that answer.
I forgot to say thanks for spotting the msys path. That's peculiar and suspect. Going to climb down that rabbit hole now...
FWIW, I can build win32 Firefox just fine on a fresh windows 2012 ec2 instance with MSVC and MozillaBuild freshly installed, using the "standard" start-shell-msvc-2015 script and nothing more.
Thanks Mike. If you can elaborate on what you do after running start-shell-msvc-2015, I'll try to replicate on our builders and report back. When I run start-shell-msvc-2015, I just get a bash shell where vcvars has been run. No Firefox build. For the purpose of removing any ambiguity, this is what I do to build Firefox under mozharness, manually on one of our builders (the first 3 var sets change regularly, the others stay pretty much constant): set platform=win32 set build_type=opt set GECKO_HEAD_REV=4f38fe78c1523e30a0d7c4a9066f09b3ae5ac057 set MOZHARNESS_CONFIG=builds\taskcluster_firefox_%platform%.py if "%build_type%" == "debug" set MH_CUSTOM_BUILD_VARIANT_CFG=debug if "%build_type%" == "opt" set MH_CUSTOM_BUILD_VARIANT_CFG= set WORKSPACE=x:\manual-%platform%-%build_type% set MH_BRANCH=try set MH_BUILD_POOL=taskcluster-windows set TOOLTOOL_CACHE=c:\builds\tooltool-cache set UPLOAD_HOST=localhost set UPLOAD_PATH=%WORKSPACE%\public\build set GECKO_BASE_REPOSITORY=c:\builds\hg-shared\mozilla-central set GECKO_HEAD_REPOSITORY=https://hg.mozilla.org/try set TOOLS_BASE_REPOSITORY=c:\builds\hg-shared\build\tools mkdir %WORKSPACE%\build\src hg share %GECKO_BASE_REPOSITORY% %WORKSPACE%\build\src hg pull -u -R %WORKSPACE%\build\src --rev %GECKO_HEAD_REV% %GECKO_HEAD_REPOSITORY% hg update -R %WORKSPACE%\build\src --rev %GECKO_HEAD_REV% mkdir %UPLOAD_PATH% if defined MH_CUSTOM_BUILD_VARIANT_CFG set MH_CUSTOM_BUILD_VARIANT_FLAGS=--platform windows --custom-build-variant-cfg %MH_CUSTOM_BUILD_VARIANT_CFG% x: && cd %WORKSPACE% python -u %WORKSPACE%\build\src\testing\mozharness\scripts\fx_desktop_build.py --config %MOZHARNESS_CONFIG% %MH_CUSTOM_BUILD_VARIANT_FLAGS% --branch=%MH_BRANCH% --build-pool=%MH_BUILD_POOL% --skip-buildbot-actions --work-dir=%WORKSPACE%\build --clone-tools --build
Flags: needinfo?(mh+mozilla)
(In reply to Rob Thijssen (:grenade - GMT) from comment #8) > Thanks Mike. If you can elaborate on what you do after running > start-shell-msvc-2015, I'll try to replicate on our builders and report > back. When I run start-shell-msvc-2015, I just get a bash shell where vcvars > has been run. No Firefox build. $ hg clone https://hg.mozilla.org/mozilla-central $ cd mozilla-central $ export NO_MERCURIAL_SETUP_CHECK=dammit $ ./mach build is all I do.
Flags: needinfo?(mh+mozilla)
Thanks for that Mike! I spent the day ripping mozharness out of my try push and established that I get no errors at all in any build (32/64, debug/opt) when mach isn't parented by a mozharness call. https://treeherder.mozilla.org/#/jobs?repo=try&revision=1a2a64024f329f8074532c975f950085323841b5 So it looks like I'll be spending some time getting to grips with what mh is up to...
this error has gone away. no idea what sorted it.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → INVALID
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.