Probe-scraper failing with OOM
Categories
(Data Platform and Tools :: General, defect, P1)
Tracking
(Not tracked)
People
(Reporter: frank, Assigned: mdroettboom)
Details
Attachments
(3 files)
We are seeing the probe-scraper exceed its memory limit with >20GB in use. Clearly something is going wrong. The immediate fix is to up the memory limit, but we need to profile the probe-scraper and figure out why its memory use is so high.
Comment 1•4 years ago
|
||
Reporter | ||
Comment 2•4 years ago
|
||
The PR I posted will not fix the issue, since our highmem pool only has 26GB of memory and we will exceed that. We need to profile this code to see where the memory leak is.
Assignee | ||
Comment 3•4 years ago
|
||
I've had reasonable luck with memory-profiler in the past.
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Updated•4 years ago
|
Reporter | ||
Comment 4•4 years ago
|
||
Jason was able to copy the probe-scraper cache over to gs://fbertsch-test-bucket/cache/probe-scraper/
. I gave :mdroettboom access to that bucket; if anyone else needs it for their own tests we can add them as well.
Comment 5•4 years ago
|
||
One thing to note here is that a schemas deploy did succeed yesterday, which implies probe-scraper did as well (on the second attempt). Looking at the usage graphs it does seem to still have high memory usage, but schemas are currently up-to-date.
Comment 6•4 years ago
|
||
Comment 7•4 years ago
|
||
Assignee | ||
Updated•4 years ago
|
Updated•2 years ago
|
Description
•