Closed
Bug 1440917
Opened 6 years ago
Closed 6 years ago
Intermittent subprocess.CalledProcessError: Command '['/builds/worker/workspace/obj-haz-shell/dist/bin/js', '/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/computeGCFunctions.js', 'callgraph.txt', 'gcFunctions.tmp'
Categories
(Core :: JavaScript Engine, defect, P5)
Core
JavaScript Engine
Tracking
()
RESOLVED
INCOMPLETE
People
(Reporter: intermittent-bug-filer, Unassigned)
Details
(Keywords: intermittent-failure, Whiteboard: [stockwell needswork])
Filed by: ncsoregi [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=164106529&repo=mozilla-central https://queue.taskcluster.net/v1/task/ZoYyzIuwQGusDDlh-JGgtA/runs/0/artifacts/public/logs/live_backing.log [task 2018-02-24T10:59:07.306Z] PATH="/builds/worker/workspace/sixgill/usr/bin:${PATH}" SOURCE='/builds/worker/checkouts/gecko' ANALYZED_OBJDIR='/builds/worker/workspace/obj-analyzed' LD_LIBRARY_PATH='/builds/worker/workspace/obj-haz-shell/dist/bin' XDB='/builds/worker/workspace/sixgill/usr/bin/xdb.so' /builds/worker/workspace/obj-haz-shell/dist/bin/js /builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/computeCallgraph.js typeInfo.txt > callgraph.txt [task 2018-02-24T10:59:26.352Z] Finished loading data structures [task 2018-02-24T11:11:59.451Z] Running gcFunctions to generate ('gcFunctions.txt', 'gcFunctions.lst', 'gcEdges.txt', 'suppressedFunctions.lst') [task 2018-02-24T11:11:59.451Z] PATH="/builds/worker/workspace/sixgill/usr/bin:${PATH}" SOURCE='/builds/worker/checkouts/gecko' ANALYZED_OBJDIR='/builds/worker/workspace/obj-analyzed' LD_LIBRARY_PATH='/builds/worker/workspace/obj-haz-shell/dist/bin' XDB='/builds/worker/workspace/sixgill/usr/bin/xdb.so' /builds/worker/workspace/obj-haz-shell/dist/bin/js /builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/computeGCFunctions.js callgraph.txt gcFunctions.txt gcFunctions.lst gcEdges.txt suppressedFunctions.lst [task 2018-02-24T11:12:50.486Z] Traceback (most recent call last): [task 2018-02-24T11:12:50.486Z] File "/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyze.py", line 323, in <module> [task 2018-02-24T11:12:50.486Z] run_job(step, data) [task 2018-02-24T11:12:50.486Z] File "/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyze.py", line 199, in run_job [task 2018-02-24T11:12:50.486Z] subprocess.check_call(command, env=env(config)) [task 2018-02-24T11:12:50.486Z] File "/usr/lib/python2.7/subprocess.py", line 540, in check_call [task 2018-02-24T11:12:50.486Z] raise CalledProcessError(retcode, cmd) [task 2018-02-24T11:12:50.486Z] subprocess.CalledProcessError: Command '['/builds/worker/workspace/obj-haz-shell/dist/bin/js', '/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/computeGCFunctions.js', 'callgraph.txt', 'gcFunctions.tmp', 'gcFunctions_list.tmp', 'gcEdges.tmp', 'suppressedFunctions_list.tmp']' returned non-zero exit status -11 [task 2018-02-24T11:12:50.489Z] + onexit [task 2018-02-24T11:12:50.489Z] + grab_artifacts /builds/worker/workspace/analysis /builds/worker/artifacts [task 2018-02-24T11:12:50.489Z] + local analysis_dir [task 2018-02-24T11:12:50.489Z] + analysis_dir=/builds/worker/workspace/analysis [task 2018-02-24T11:12:50.489Z] + local artifacts [task 2018-02-24T11:12:50.489Z] + artifacts=/builds/worker/artifacts [task 2018-02-24T11:12:50.489Z] + cd /builds/worker/workspace/analysis [task 2018-02-24T11:12:50.490Z] + ls -lah [task 2018-02-24T11:12:50.491Z] total 3.7G
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 3•6 years ago
|
||
There are 31 failures in the past 7 days, all of them on Linux X64 debug. Recent failure log: https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=164373725&lineNumber=39277 Relevant part of the log: [task 2018-02-26T17:30:45.521Z] Traceback (most recent call last): [task 2018-02-26T17:30:45.521Z] File "/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyze.py", line 323, in <module> [task 2018-02-26T17:30:45.521Z] run_job(step, data) [task 2018-02-26T17:30:45.521Z] File "/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyze.py", line 199, in run_job [task 2018-02-26T17:30:45.521Z] subprocess.check_call(command, env=env(config)) [task 2018-02-26T17:30:45.521Z] File "/usr/lib/python2.7/subprocess.py", line 540, in check_call [task 2018-02-26T17:30:45.521Z] raise CalledProcessError(retcode, cmd) [task 2018-02-26T17:30:45.521Z] subprocess.CalledProcessError: Command '['/builds/worker/workspace/obj-haz-shell/dist/bin/js', '/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/computeGCFunctions.js', 'callgraph.txt', 'gcFunctions.tmp', 'gcFunctions_list.tmp', 'gcEdges.tmp', 'suppressedFunctions_list.tmp']' returned non-zero exit status -11 [task 2018-02-26T17:30:45.524Z] + onexit [task 2018-02-26T17:30:45.524Z] + grab_artifacts /builds/worker/workspace/analysis /builds/worker/artifacts [task 2018-02-26T17:30:45.524Z] + local analysis_dir [task 2018-02-26T17:30:45.524Z] + analysis_dir=/builds/worker/workspace/analysis [task 2018-02-26T17:30:45.524Z] + local artifacts [task 2018-02-26T17:30:45.524Z] + artifacts=/builds/worker/artifacts [task 2018-02-26T17:30:45.525Z] + cd /builds/worker/workspace/analysis [task 2018-02-26T17:30:45.525Z] + ls -lah [task 2018-02-26T17:30:45.526Z] total 3.7G This bug blocks 903519 which has some recent activity
Whiteboard: [stockwell needswork]
Comment 4•6 years ago
|
||
(In reply to Treeherder Bug Filer from comment #0) > [task 2018-02-24T11:12:50.486Z] subprocess.CalledProcessError: Command > '['/builds/worker/workspace/obj-haz-shell/dist/bin/js', > '/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/ > computeGCFunctions.js', 'callgraph.txt', 'gcFunctions.tmp', > 'gcFunctions_list.tmp', 'gcEdges.tmp', 'suppressedFunctions_list.tmp']' > returned non-zero exit status -11 This is a SIGSEGV. sfink: could these hazard builds use the same Breakpad integration bits you added for other JS builds?
Component: Build Config → JavaScript Engine
Flags: needinfo?(sphink)
Comment 5•6 years ago
|
||
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4) > (In reply to Treeherder Bug Filer from comment #0) > > [task 2018-02-24T11:12:50.486Z] subprocess.CalledProcessError: Command > > '['/builds/worker/workspace/obj-haz-shell/dist/bin/js', > > '/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/ > > computeGCFunctions.js', 'callgraph.txt', 'gcFunctions.tmp', > > 'gcFunctions_list.tmp', 'gcEdges.tmp', 'suppressedFunctions_list.tmp']' > > returned non-zero exit status -11 > > This is a SIGSEGV. sfink: could these hazard builds use the same Breakpad > integration bits you added for other JS builds? Yes. I started working on that last night. I should file the bug though... bug 1441946. I actually hadn't realized how frequent JS crashes during the analysis are; it looks like it has been generating quite a few intermittents for a while. The first thing I've run into is that the binary I built and am successfully using for the SM jobs is incompatible with the hazard environment -- minidump_stackwalk wants a newer libstdc++ than the hazard builds have. Which makes no sense to me, since they both get gcc from the toolchain task, and they're using libstdc++ from that gcc?! But this is motivating me to finally make a toolchain task for breakpad. I'm scrambling at the moment to see if there's some way I might be able to squeeze nursery strings in before the soft freeze, which is looking highly unlikely, but once that deadline whooshes by I'll get back to it.
Flags: needinfo?(sphink)
Comment 6•6 years ago
|
||
(In reply to Steve Fink [:sfink] [:s:] from comment #5) > The first thing I've run into is that the binary I built and am successfully > using for the SM jobs is incompatible with the hazard environment -- > minidump_stackwalk wants a newer libstdc++ than the hazard builds have. > Which makes no sense to me, since they both get gcc from the toolchain > task, and they're using libstdc++ from that gcc?! Oh, the minidump_stackwalk binaries in tooltool are not built with the toolchain gcc because I have never gotten around to fixing bug 1391408. That might be a lot easier nowadays because we have cross-compiled Mac and mingw builds in CI, so I could just reuse most of those toolchains for building minidump_stackwalk binaries for Mac/Windows. The minidump_stackwalk binaries were built with this script (which builds binaries for all of linux32, linux64, mac, and win32): https://dxr.mozilla.org/mozilla-central/source/taskcluster/scripts/misc/minidump_stackwalk.sh in this Docker image: https://hub.docker.com/r/luser/breakpad-builder/ built from the Dockerfile + scripts here: https://github.com/luser/breakpad-taskcluster Humorously, I did write a script to do the actual build itself in Taskcluster, but it predates toolchain tasks.
Comment hidden (Intermittent Failures Robot) |
Comment 8•6 years ago
|
||
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INCOMPLETE
You need to log in
before you can comment on or make changes to this bug.
Description
•