907277 - [tracker] rewrite processor as thread-free service

Status: NEW

(Socorro :: Processor, task, P3)

Description

[Lonnen :lonnen]

Processor threads sometimes fail to shut down cleanly while they hang on calls to MDSW. We suspect this is because during long IO calls the threads cannot respond to signals.

Instead of multithreading, we could simplify the processor to run in a single thread and let upstart, circus, supervisord, etc. handle running multiple copies of processor on a single box.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Oy.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this a P2. This is something we want to do and I'm going to move a bunch of bugs to block on this one that involve rewriting bits of the processor.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Unassigning myself since I'm not going to get to this any time soon.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

It's been a long time since this bug was created, so it's prudent to update the mission here.

In our current infrastructure, each processor node runs a Docker container with a single processor process in it. Each processor process runs multiple threads. The processor spends the bulk of its time running minidump-stackwalk on crash reports and moving data over the network with S3 and Elasticsearch.

There are a few disadvantages to the current threaded model:

1. Threaded architectures are more complex because they have to deal with contention between threads and resources. If we rewrite to a non-threaded model, we can remove some of this contention-handling code and the corresponding tests.
2. Python threads doing network i/o block the entire process. One way to speed up applications that are network i/o heavy, is to change the model to multiprocess, or coroutines, or switch to asyncio and asyncio-aware libraries. The latter is hard especially since (at the time of this writing) boto3 isn't ascynio-aware, yet. Going multiprocess or coroutines seems helpful. We use multiprocess in some parts of Socorro already and we use coroutines in the collector.
3. The processor spends the bulk of its time waiting for minidump-stackwalk to run in a separate process. The LLT team is rewriting breakpad bits in Rust and that will eventually involve us switching to a different minidump-stackwalk. We may not want to run that in a separate process. It's a lot simpler to plan for that if we're not dealing with multiple threads.
4. The current processor architecture doesn't work in other contexts. It's not possible to write a cli that takes a crash id, processes the crash report, and then ends. Having something like that would be really helpful.

Thus, I want to rewrite the processor as a single-thread application and either switch to coroutines or move the concurrency up a level to either a process manager or to the infrastructure.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this block GCP migration because this will make instance sizing and scaling easier.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Removing this from the GCP migration. We can migrate the processor we've got as is.