Closed Bug 1549201 Opened 7 years ago Closed 7 years ago

[traceback] [Errno 28] No space left on device: '/tmp/...'

Categories

(Socorro :: Processor, defect, P2)

Product:
Socorro
Component:
Processor
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: willkg, Assigned: brian)

Details

This just happened in prod today to one of the processor nodes:

https://sentry.prod.mozaws.net/operations/socorro-new-prod/issues/5678177/?query=is:unresolved

OSError: [Errno 28] No space left on device: '/tmp/04a52fc0-db4d-488c-8465-141fe0190505.Thread-7.TEMPORARY.json'
  File "socorro/processor/processor_2015.py", line 268, in process_crash
    rule.act(raw_crash, raw_dumps, processed_crash, processor_meta_data)
  File "socorro/processor/rules/base.py", line 66, in act
    self.action(raw_crash, raw_dumps, processed_crash, processor_meta_data)
  File "socorro/processor/breakpad_transform_rules.py", line 323, in action
    with self._temp_raw_crash_json_file(raw_crash, raw_crash.uuid) as raw_crash_pathname:
  File "python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "socorro/processor/breakpad_transform_rules.py", line 248, in _temp_raw_crash_json_file
    with open(file_pathname, "w") as f:

This bug covers capturing details, fixing the immediate issue, and then looking at what code is using /tmp and why it got into this situation.

Type: task → defect
Priority: -- → P2

Miles looked at the disk usage of the docker host, and the docker file system is taking up all the space. /tmp is in the docker container.

I have vague memories that we put symbols cache in the shared drive so all the processors can use it, but we put the minidump files for minidump-stackwalker in /tmp in the container. I think that was intentional.

Miles spun up some more processors and I think took the bad processor out.

Miles pointed out that we've got a bunch of core dumps. That's almost certainly from minidump-stackwalk. We should:

  1. clean up from minidump-stackwalk failures better
  2. have a better process for surfacing issues and debugging minidump-stackwalk

Tomorrow, I'll look at the logs and see whether we can determine which crash reports minidump stackwalk died on and if I can reproduce locally.

Also, relatedly, I literally just updated minidump-stackwalk but that update hasn't gotten to prod, yet. It's possible that update fixes some/most/all/none of the issues.

Grabbing this to work on on Monday.

Assignee: nobody → willkg
Status: NEW → ASSIGNED

It looks like the rule that runs minidump-stackwalk just keeps on rolling in the case of an out-of-space error. I'm pretty sure there are a bunch of crashes with "EMPTY: no crashing thread" signatures because of this.

I spot-check reprocessed a handful and every one of them came up with a valid signature after reprocessing.

We should reprocess all of those over the last XX hours. Maybe anything that came in on Saturday and Sunday? I'll do that tomorrow.

I still haven't found a crash report that triggers a segfault. I think it's buried in that group.

Will, if you can provide me with on of the minidumps that are causing minidump-stackwalk to segfault I can have a look.

On Thursday, I updated the breakpad client mdsw uses to 4d550cceca107f36c4bc1ea1126b7d32cc50f424. That change is in stage, but not prod. I was thinking once I find a bad crash report, I can test it with mdsw built with the new breakpad client.

If that works, I'll just do a deploy and the problems can go away. If that doesn't work, I'll send it to you.

Thank you for the offer!

I found a crash report that reproduces the issue and it dumps core in the local dev environment docker container. I uploaded it via google drive with the crash id.

That gives you the core dump we're getting and the mini dumps.

minidump-stackwalk is crashing with both the new breakpad client (4d550cceca107f36c4bc1ea1126b7d32cc50f424) as well as the previous one we had (44384d80b32a5bb361c2ec3bee667f7ccee566d7).

Gabriele: Any help would be appreciated!

Priority: P2 → P1
Priority: P1 → P2

Brian did an infra fix so that nodes create at most one core file and keep stomping on it with future segfaults. That makes the symptoms of running out of space go away.

I'm going to rescope this bug slightly to figure out why minidump-stackwalk is segfaulting and fixing that.

Turns out I had code tracking minidump-stackwalk exit codes. Seems like Socorro gets a -6 whenever it segfaults. [1]

In the data we've got, we see the first -6 on February 4th.

We upgraded the breakpad client in pull 4797: https://github.com/mozilla-services/socorro/pull/4797

That got pushed to prod on January 31st in 361: https://bugzilla.mozilla.org/show_bug.cgi?id=1524019

That seems like fishy timing, but could be a coincidence.

I'm working on a fix to do better logging so we can facet on what's causing segfaults: products, versions, buildids, etc. That might be illuminating.

[1] No clue why it's -6 or whether that could mean other things, too. I thought exit codes ranged from 0-255.

Ugh. I did this change on February 1st:

https://github.com/mozilla-services/socorro/pull/4800

I screwed up the handling. .wait() returns a negative number if it exited because of a signal. -6 indicates signal 6 which is SIGABRT which before this change did an exit code of 134 because we were using shell=True.

Given that, we've been seeing segfaults for as long as I've been collecting exit code data--since November 2018. So comment #7 is a red herring and I have another bug to fix.

We had another processor node run out of space and then go bad last night. Looking into that.

My idea to rescope this bug was a bad one. I spun off the "minidump-stackwalk is segfaulting" problem into bug #1549717. This bug will cover the problem where processor nodes are running out of space.

It wasn't a new processor node that ran out of space--it was a processor node that had run out of space before but was never shut down.

I'm going to wait another 10 minutes to make sure we're not still seeing the problem and then reprocess the affected crash reports. I'll go back a week again because the bad node was still processing so who knows how many are bad.

No longer blocks: 1549717

There haven't been any new instances of this issue in the last 24 hours, so I'm going to close this out. The minidump-stackwalk assertion failure issue is bug #1549717.

Assignee: willkg → bpitts
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.