1609623 - Intermittent [hazard] subprocess.CalledProcessError: Command 'analyzeRoots.js' returned non-zero exit status 9

Status: RESOLVED INCOMPLETE

(Firefox Build System :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=285150900&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PxdHaunXSASnQHw18iYZSA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-01-16T06:42:26.072Z] 68:39.08 Notification center failed: Install notify-send (usually part of the libnotify package) to get a notification when the build finishes.
[task 2020-01-16T06:42:26.076Z] To view resource usage of the build, run |mach resource-usage|.
[task 2020-01-16T06:42:26.076Z] 68:39.08 We know it took a while, but your build finally finished successfully!
[task 2020-01-16T06:42:26.077Z] To take your build for a test drive, run: |mach run|
[task 2020-01-16T06:42:26.077Z] For more information on what to do now, see https://developer.mozilla.org/docs/Developer_Guide/So_You_Just_Built_Firefox
[task 2020-01-16T06:42:26.208Z] Received connection.
[task 2020-01-16T06:42:26.209Z] Final transaction received, finishing...
[task 2020-01-16T06:42:53.021Z] Waiting for manager to finish (build status 0)...
[task 2020-01-16T06:42:53.021Z] Exiting with status 0
[task 2020-01-16T06:42:53.022Z] build finished (status 0): Thu Jan 16 06:42:53 2020
[task 2020-01-16T06:42:53.022Z] Exiting run_complete with status 0
[task 2020-01-16T06:42:53.024Z] Running gcTypes to generate ('gcTypes.txt', 'typeInfo.txt')
[task 2020-01-16T06:42:53.024Z] ANALYZED_OBJDIR='/builds/worker/workspace/obj-analyzed' SOURCE='/builds/worker/checkouts/gecko' LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/builds/worker/workspace/obj-haz-shell/dist/bin" XDB='/builds/worker/fetches/sixgill/usr/bin/xdb.so' PATH="/builds/worker/fetches/sixgill/usr/bin:${PATH}" /builds/worker/workspace/obj-haz-shell/dist/bin/js /builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/computeGCTypes.js gcTypes.txt typeInfo.txt
[task 2020-01-16T06:43:23.546Z] Running callgraph to generate ('callgraph.txt',)
[task 2020-01-16T06:43:23.546Z] ANALYZED_OBJDIR='/builds/worker/workspace/obj-analyzed' SOURCE='/builds/worker/checkouts/gecko' LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/builds/worker/workspace/obj-haz-shell/dist/bin" XDB='/builds/worker/fetches/sixgill/usr/bin/xdb.so' PATH="/builds/worker/fetches/sixgill/usr/bin:${PATH}" /builds/worker/workspace/obj-haz-shell/dist/bin/js /builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/computeCallgraph.js typeInfo.txt callgraph.txt
[task 2020-01-16T06:43:54.859Z] Finished loading data structures
[task 2020-01-16T07:06:19.586Z] Running gcFunctions to generate ('gcFunctions.txt', 'gcFunctions.lst', 'gcEdges.txt', 'limitedFunctions.lst')
[task 2020-01-16T07:06:19.587Z] ANALYZED_OBJDIR='/builds/worker/workspace/obj-analyzed' SOURCE='/builds/worker/checkouts/gecko' LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/builds/worker/workspace/obj-haz-shell/dist/bin" XDB='/builds/worker/fetches/sixgill/usr/bin/xdb.so' PATH="/builds/worker/fetches/sixgill/usr/bin:${PATH}" /builds/worker/workspace/obj-haz-shell/dist/bin/js /builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/computeGCFunctions.js callgraph.txt gcFunctions.txt gcFunctions.lst gcEdges.txt limitedFunctions.lst
[task 2020-01-16T07:07:52.462Z] Writing gcFunctions.tmp
[task 2020-01-16T07:08:08.297Z] Writing gcFunctions_list.tmp
[task 2020-01-16T07:08:12.458Z] Writing gcEdges.tmp
[task 2020-01-16T07:08:12.458Z] Writing limitedFunctions_list.tmp
[task 2020-01-16T07:08:14.662Z] Running allFunctions to generate allFunctions.txt

[task 2020-01-16T07:10:45.239Z] ANALYZED_OBJDIR='/builds/worker/workspace/obj-analyzed' SOURCE='/builds/worker/checkouts/gecko' LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/builds/worker/workspace/obj-haz-shell/dist/bin" XDB='/builds/worker/fetches/sixgill/usr/bin/xdb.so' PATH="/builds/worker/fetches/sixgill/usr/bin:${PATH}" /builds/worker/workspace/obj-haz-shell/dist/bin/js /builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyzeRoots.js gcFunctions.lst gcEdges.txt limitedFunctions.lst gcTypes.txt typeInfo.txt 31 31 tmp.31 > rootingHazards.31
[task 2020-01-16T07:10:45.240Z] Traceback (most recent call last):
[task 2020-01-16T07:10:45.240Z] File "/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyze.py", line 339, in <module>
[task 2020-01-16T07:10:45.240Z] run_job(step, data)
[task 2020-01-16T07:10:45.240Z] File "/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyze.py", line 181, in run_job
[task 2020-01-16T07:10:45.240Z] cmdspec(config, outfiles)
[task 2020-01-16T07:10:45.240Z] File "/builds/worker/checkouts/gecko/js/src/devtools/rootAnalysis/analyze.py", line 110, in generate_hazards
[task 2020-01-16T07:10:45.240Z] raise subprocess.CalledProcessError(final_status, 'analyzeRoots.js')
[task 2020-01-16T07:10:45.240Z] subprocess.CalledProcessError: Command 'analyzeRoots.js' returned non-zero exit status 9
[task 2020-01-16T07:10:45.252Z] + onexit
[task 2020-01-16T07:10:45.252Z] + grab_artifacts /builds/worker/workspace/analysis /builds/worker/artifacts
[task 2020-01-16T07:10:45.252Z] + local analysis_dir
[task 2020-01-16T07:10:45.253Z] + analysis_dir=/builds/worker/workspace/analysis
[task 2020-01-16T07:10:45.253Z] + local artifacts
[task 2020-01-16T07:10:45.253Z] + artifacts=/builds/worker/artifacts
[task 2020-01-16T07:10:45.253Z] + cd /builds/worker/workspace/analysis
[task 2020-01-16T07:10:45.253Z] + ls -lah
[task 2020-01-16T07:10:45.268Z] total 6.1G
[task 2020-01-16T07:10:45.268Z] drwxr-xr-x 3 worker worker 4.0K Jan 16 07:08 .
[task 2020-01-16T07:10:45.268Z] drwxr-xr-x 6 worker worker 4.0K Jan 16 05:33 ..

Comment 1

[Daniel Varga [:dvarga]]

Andrew, can you please take a look at this fail?
It seems to start after bug 1609268 got landed.
Retrigger link: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=Linux%2Cx64%2Cdebug%2Chazard-linux64-haz%2Fdebug%2C%28H%29&tochange=15fbe779dd3015b318505c5f42cad4127bfac3ad&fromchange=39add523bc263ad91af942fa1ea3e43cabeee12b

Comment 2

[Intermittent Failures Robot]

24 failures in 3773 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 24

Platform breakdown:

• linux64: 24

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1609623&startday=2020-01-13&endday=2020-01-19&tree=all

Comment 3

[Andrew Osmond [:aosmond] (he/him)]

Reviewing orange factor, it looks like it follows bug 1597154 a lot more closely than it does mine. It started with it and stopped when it got backed out.

Comment 4

[Chris Peterson [:cpeterson]]

pbone said he will take a look.

Tentatively tracking for Fission Dogfooding (M5).

Comment 5

[Cosmin Sabou [:CosminS]]

The spike here has the same root cause as Bug 1614852. There are a lot of failures due to the switch of b-linux workers from AWS to GCP.
It is now permafailing and failing also when reruning green jobs so it is infra related. https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception&searchStr=linux%2Cx64%2Cdebug%2Chazard-linux64-haz%2Fdebug%2C%28h%29&tochange=be916959142fc9ed315e855e2dce4b57fcac8e05&fromchange=92dd7149719c1537792107db865bedbc5f87688c&selectedJob=288459724

Comment 6

[Andreea Pavel [:apavel]]

Root cause of this bug and Bug 1614852 has been reverted here: https://bugzilla.mozilla.org/show_bug.cgi?id=1614852#c12
https://hg.mozilla.org/integration/autoland/rev/7dcbf3debe6a3a527df82c422446ca42a3de78c4

Comment 7

[Chris Peterson [:cpeterson]]

(In reply to Andreea Pavel [:apavel] from comment #6)

Root cause of this bug and Bug 1614852 has been reverted here: https://bugzilla.mozilla.org/show_bug.cgi?id=1614852#c12
https://hg.mozilla.org/integration/autoland/rev/7dcbf3debe6a3a527df82c422446ca42a3de78c4

Since GCP bug 1614852 has reverted the root cause, can close this bug as fixed? Or is there additional work needed to verify hazard analysis is working with Fission on GCP?

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The plan to switch remains, getting Hazard ready before the next switch would be welcome.

Kendall, please let Chris know about the next date for the AWS to GCP switch once it is scheduled.

For testing it before, the worker type will need to be added to the task config like this: https://searchfox.org/mozilla-central/rev/a4be2fbe9bd4f405c91cc16e4e3a80400f5a9301/taskcluster/ci/build/android.yml#384

Comment 9

[Intermittent Failures Robot]

18 failures in 4690 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 18

Platform breakdown:

• linux64: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1609623&startday=2020-02-10&endday=2020-02-16&tree=all

Comment 10

[Paul Bone [:pbone]]

Looks like others found a root cause before I had time to look at this bug. Removing my NI.

Comment 11

[Chris Peterson [:cpeterson]]

Not a Fission bug

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

Clear a needinfo that is pending on an inactive user.

Inactive users most likely will not respond; if the missing information is essential and cannot be collected another way, the bug maybe should be closed as INCOMPLETE.

For more information, please visit auto_nag documentation.