1117313 - Not getting stacks for crashes in mozharness-run jobs after bug 1113460 deploy, all trees closed

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Phil Ringnalda (:philor)]

What could possibly go wrong with deploying bug 1113460 on a Friday, a Friday when apparently nobody was particularly watching trees?

Now when we crash, we get https://treeherder.mozilla.org/logviewer.html#?job_id=5049376&repo=mozilla-inbound, where instead of a stack we get

17:17:16 INFO - Traceback (most recent call last):
17:17:16 INFO - File "/builds/slave/talos-slave/test/build/tests/mochitest/runtests.py", line 1909, in doTests
17:17:16 INFO - quiet=options.quiet
17:17:16 INFO - File "/builds/slave/talos-slave/test/build/tests/mochitest/runtests.py", line 1598, in runApp
17:17:16 INFO - test=self.lastTestSeen)
17:17:16 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 121, in log_crashes
17:17:16 INFO - stackwalk_binary=stackwalk_binary):
17:17:16 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 200, in __iter__
17:17:16 INFO - rv = self._process_dump_file(path, extra)
17:17:16 INFO - File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/mozcrash/mozcrash.py", line 235, in _process_dump_file
17:17:16 INFO - stderr=subprocess.PIPE)
17:17:16 INFO - File "/tools/python27/lib/python2.7/subprocess.py", line 679, in __init__
17:17:16 INFO - errread, errwrite)
17:17:16 INFO - File "/tools/python27/lib/python2.7/subprocess.py", line 1249, in _execute_child
17:17:16 INFO - raise child_exception
17:17:16 INFO - OSError: [Errno 13] Permission denied

which I suspect means the tooltool-provided minidump_stackwalk isn't executable.

The trees we care about are closed.

Comment 1

[Jordan Lund (:jlund)]

backed out on default:
remote:   https://hg.mozilla.org/build/mozharness/rev/84461b54c692
remote:   https://hg.mozilla.org/build/mozharness/rev/db42592f6c20

merged to production:
remote:   https://hg.mozilla.org/build/mozharness/rev/a1caf218cb3c

poking a slave confirms the permission bits for stackwalk binary are -x:
-rw-------  1 cltbld  staff  1316240  3 Jan 00:02 /builds/slave/talos-slave/test/build/macosx64-minidump_stackwalk

the above got its copy from tooltool_cache. looks like we need an extra step to chmod the binaries. will add that and update ToolTool docs.

apologies for the inconvenience. this should be resolved