Intermittent Linux static rooting hazard analysis "analysis failed, can't continue."

RESOLVED WORKSFORME

Status

defect
RESOLVED WORKSFORME
5 years ago
a year ago

People

(Reporter: cbook, Unassigned)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

(URL)

(Reporter)

Description

5 years ago
linux64-br-haz_b2g-inbound_dep on 2014-01-26 19:16:51 PST for push a20d51b01065

slave: bld-linux64-ec2-082

https://tbpl.mozilla.org/php/getParsedLog.php?id=33613541&tree=B2g-Inbound

20:24:15    FATAL - analysis failed, can't continue.

20:24:15    ERROR -  Traceback (most recent call last):
20:24:15     INFO -    File "/builds/slave/l64-br-haz_b2g-in_dep-00000000/build/source/js/src/devtools/rootAnalysis/analyze.py", line 279, in <module>
20:24:15     INFO -      run_job(step, data)
20:24:15     INFO -    File "/builds/slave/l64-br-haz_b2g-in_dep-00000000/build/source/js/src/devtools/rootAnalysis/analyze.py", line 175, in run_job
20:24:15     INFO -      subprocess.check_call(command, stdout=output, env=env(config))
20:24:15     INFO -    File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
20:24:15     INFO -      raise CalledProcessError(retcode, cmd)
20:24:15     INFO -  subprocess.CalledProcessError: Command '['/builds/slave/l64-br-haz_b2g-in_dep-00000000/build/obj-opt-js/dist/bin/js', '/builds/slave/l64-br-haz_b2g-in_dep-00000000/build/source/js/src/devtools/rootAnalysis/computeCallgraph.js']' returned non-zero exit status -6
20:24:15    ERROR - Return code: 1
20:24:15    FATAL - analysis failed, can't continue.
20:24:15    FATAL - Running post_fatal callback...
20:24:15    FATAL - Exiting 2
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Steve, any ideas with this? :-)
Flags: needinfo?(sphink)
Summary: Intermittent FATAL - analysis failed, can't continue. → Intermittent Linux static rooting hazard analysis "analysis failed, can't continue."
Comment hidden (Legacy TBPL/Treeherder Robot)
The relevant log line is:

20:24:13     INFO -  sh: line 1: 14890 Aborted                 (core dumped) /usr/bin/xmanager -terminate-on-assert > ./build_manager.log 2>&1

it'd be nice to see the build_manager.log log file. I would rerun this push on my slave, but the VPN is currently mangled and I can't reach it.
Flags: needinfo?(sphink)
Do not clear needinfo. Still needinfo.
Flags: needinfo?(sphink)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
I've been getting something like this over and over with a patch I'm trying to land, viz:
https://tbpl.mozilla.org/?tree=Try&rev=16911cb402de

I don't know if it's the same. (Mine appears to be a python exception.) I do know, however, that it is annoying me greatly :(
(In reply to Nicholas Nethercote [:njn] from comment #41)
> I've been getting something like this over and over with a patch I'm trying
> to land, viz:
> https://tbpl.mozilla.org/?tree=Try&rev=16911cb402de
> 
> I don't know if it's the same. (Mine appears to be a python exception.) I do
> know, however, that it is annoying me greatly :(

Not the same. Yours is segfaulting the JS shell when running the analysis. Could that be your patch?

I think the others are very intermittent, and are aborts not segvs. I have a patch I still need to push that'll at least get a little more error output into the logs. I don't think it'll help for yours, though.

I'm running your push on my slave right now. Hopefully it'll hit the same problem.
Flags: needinfo?(sphink)
> Yours is segfaulting the JS shell when running the analysis.
> Could that be your patch?

Quite possibly! How can you tell that? I can see anything in the log about it.
23:37:20     INFO -  subprocess.CalledProcessError: Command '['/builds/slave/l64-br-haz_try_dep-00000000000/build/obj-opt-js/dist/bin/js', '/builds/slave/l64-br-haz_try_dep-00000000000/build/source/js/src/devtools/rootAnalysis/computeCallgraph.js']' returned non-zero exit status -11

I'm guessing status -11 means seg fault.
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Steve, this intermittent failure masked real bustage - since the parsed log output that appears in TBPL looked virtually the same (compare comment 50 and comment 54 for example).

Please could we try to output something in the log to separate these two cases?
Flags: needinfo?(sphink)
(In reply to Ed Morley [:edmorley UTC+0] from comment #56)
> Steve, this intermittent failure masked real bustage - since the parsed log
> output that appears in TBPL looked virtually the same (compare comment 50
> and comment 54 for example).
> 
> Please could we try to output something in the log to separate these two
> cases?

Yes. Your comment came in while I was staring at that previous output and noticing that it didn't look like the usual failure here. (I was looking because I recently landed a patch that puts more info in the logs and I wanted to see if it caught anything here. It didn't.)

I think I mostly just need to change the format of the JS failure messages to be something the log parser will pick up on, so I can tell which phase of the analysis broke. Which means diving back in to figure out what our tangled web of regexes is scanning for.

Filed bug 1008308
Flags: needinfo?(sphink)
Comment hidden (Legacy TBPL/Treeherder Robot)
(In reply to Steve Fink [:sfink] from comment #57)
> Filed bug 1008308

Thank you :-)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Comment hidden (Legacy TBPL/Treeherder Robot)
Depends on: 1126544
Inactive; closing (see bug 1180138).
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → WORKSFORME

Updated

a year ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.