980165 - directly link C stackwalker with Python Socorro Processor

Status: RESOLVED WONTFIX

(Socorro :: Backend, task)

Description

[K Lars Lohn [:lars] [:klohn]]

In the quest for more speed for eventual 100% processing:

the stackwalker code is not thread safe, that's why we've never used it as a callable C library.  With the ability now to choose between threaded and non threaded processors, we  should revisit that decision.  We could have a performance win with single threaded code by using the cores more effectively without child processes and the overhead of interprocess communication.  

We do not know why the stackwalker hangs.  If it is an interprocess communication deadlock, then having the library linked directly will eliminate that problem.  

However, if the stackwalker hang is inherent in the stackwalker itself, then this would likely hang a the single threaded processor.

Comment 1

[(not currently active) Ted Mielczarek]

We could experiment with this by splitting the stackwalker binary into a shared library + a stub binary that calls into the shared library, which would allow you to load the shared library with ctypes and call the same method via Python.

FWIW, the C++ classes that the stackwalker uses are not threadsafe, but I don't think anything relies on global shared state either, so it should be safe to do stackwalking in multiple threads in the same process.

Comment 2

[(not currently active) Ted Mielczarek]

I did the split work, it's in a branch here:
https://github.com/luser/socorro/tree/stackwalker-dylib

Here's a sample script of how to call it using ctypes:
https://github.com/luser/socorro/blob/stackwalker-dylib/minidump-stackwalk/test.py

Comment 3

[Lonnen :lonnen]

We are likely to go a different way with this now -- either use sandboxed subprocess or rewrite in rust.