consider changing UPLOAD_TEMPDIR_ORPHANS_CUTOFF to 15 minutes
Categories
(Tecken :: General, task, P2)
Tracking
(Not tracked)
People
(Reporter: willkg, Assigned: willkg)
Details
Attachments
(1 file)
There's a UPLOAD_TEMPDIR_ORPHANS_CUTOFF setting which is set to 60 minutes currently which denotes the point at which we consider a file in the upload tempdir to be orphaned--namely that there is no active upload handling tied to that file.
We chose 60 minutes initially because we were being really conservative. Any file in that directory older than this value is absolutely an orphaned file.
However, once a tecken instance has hit a state where it's got orphaned files, it tends to accumulate them and it could accumulate them quickly over the course of an hour when they would be culled by the command that removes them.
We're seeing this situation now:
One or more uploads was attempted multiple times incurring timeouts where the related files were orphaned. This happened on multiple tecken instances. Now we're in a situation where they're periodically kicking up out-of-space errors. Because it takes 60 minutes for them to start culling those files, it takes a while for them to recover.
I think we should drop the UPLOAD_TEMPDIR_ORPHANS_CUTOFF value to 10 minutes. That still exceeds the idle timeouts scattered through the system (6 minutes) which will terminate upload API handling. It's much lower than 60 minutes. It would mean an instance would be more likely to recover.
|
Assignee | |
Updated•1 year ago
|
|
|
Assignee | |
Comment 1•1 year ago
|
||
|
|
Assignee | |
Comment 2•1 year ago
|
||
|
|
Assignee | |
Comment 3•1 year ago
|
||
I pushed this to prod just now in bug #1867844. Marking as FIXED.
Description
•