1426375 - [ops infra socorro] processor nodes kicking up "cannot allocate memory" errors

Status: RESOLVED FIXED

(Socorro :: Infra, task, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Processor nodes in -stage-new are kicking up "Cannot allocate memory" errors:

https://sentry.prod.mozaws.net/operations/socorro-new-stage/issues/823071/

"""
OSError: [Errno 12] Cannot allocate memory
  File "socorro/lib/transform_rules.py", line 147, in action
    return self._action(*args, **kwargs)
  File "socorro/processor/breakpad_transform_rules.py", line 369, in _action
    processor_meta
  File "socorro/processor/breakpad_transform_rules.py", line 273, in _execute_external_process
    )._execute_external_process(command_line, processor_meta)
  File "socorro/processor/breakpad_transform_rules.py", line 130, in _execute_external_process
    stdout=subprocess.PIPE
  File "python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "python2.7/subprocess.py", line 917, in _execute_child
    self.pid = os.fork()
"""

This bug covers looking into why we get this in -stage-new and not -stage/-prod and then fixing it.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this a P1 because it's pretty critical. We've seen a number of these over the last day.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

We're running a lot of threads on the processor. We can probably reduce those. I'll find the configuration for that.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Miles increased the instance size and we haven't seen the issue since, so marking it as FIXED.