1740775 - OSError: [Errno 28] No space left on device

Status: RESOLVED FIXED

(Tecken :: General, defect, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Sentry: https://sentry.prod.mozaws.net/operations/symbols-prod/issues/14867213/

Traceback:

OSError: [Errno 28] No space left on device
  File "django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "markus/main.py", line 515, in _timer_decorator
    return fun(*args, **kwargs)
  File "tecken/base/decorators.py", line 128, in inner
    return func(request, *args, **kwargs)
  File "django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "tecken/base/decorators.py", line 30, in inner
    return view_func(request, *args, **kwargs)
  File "django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "tecken/base/decorators.py", line 201, in inner
    return func(*args, **kwargs)
  File "tecken/upload/views.py", line 193, in upload_archive
    for name in request.FILES:
  File "django/core/handlers/wsgi.py", line 124, in FILES
    self._load_post_and_files()
  File "django/http/request.py", line 315, in _load_post_and_files
    self._post, self._files = self.parse_file_upload(self.META, data)
  File "django/http/request.py", line 275, in parse_file_upload
    return parser.parse()
  File "django/http/multipartparser.py", line 254, in parse
    chunk = handler.receive_data_chunk(chunk, counters[i])
  File "django/core/files/uploadhandler.py", line 143, in receive_data_chunk
    self.file.write(raw_data)
  File "python3.9/tempfile.py", line 474, in func_wrapper
    return func(*args, **kwargs)

Saw two of these today where a node goes unhealthy while handling symbols uploads requests.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

One of the nodes ran out of space. Jason did a du -hs and saw this:

2.9G raw-uploads7lrkh3eb
2.8G raw-uploadsln4y_tyr
2.9G raw-uploadsnzpwrw14
2.7G raw-uploads_rdu4r_3
1.1G tmphevf6df3.upload.zip
1.1G tmphf3o21fj.upload.zip
1.1G tmpw_p3mzf6.upload.zip
1.1G tmpx0aaxcve.upload.zip

Nodes have 2 virtual processors, so there should be 5 gunicorn workers (2 * 2 + 1) running.

Also, the stack shows Django is saving temp files as it parses the multipart form in the HTTP request, so it's possible that there's even more space being used up.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

This deployed to prod in bug #1744050 just now.

Marking as FIXED.