Closed Bug 1689167 Opened 4 years ago Closed 4 years ago

Probe-scraper failing with OOM

Categories

(Data Platform and Tools :: General, defect, P1)

defect
Points:
3

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: frank, Assigned: mdroettboom)

Details

Attachments

(3 files)

We are seeing the probe-scraper exceed its memory limit with >20GB in use. Clearly something is going wrong. The immediate fix is to up the memory limit, but we need to profile the probe-scraper and figure out why its memory use is so high.

The PR I posted will not fix the issue, since our highmem pool only has 26GB of memory and we will exceed that. We need to profile this code to see where the memory leak is.

I've had reasonable luck with memory-profiler in the past.

Assignee: nobody → mdroettboom
Priority: -- → P1

Jason was able to copy the probe-scraper cache over to gs://fbertsch-test-bucket/cache/probe-scraper/. I gave :mdroettboom access to that bucket; if anyone else needs it for their own tests we can add them as well.

One thing to note here is that a schemas deploy did succeed yesterday, which implies probe-scraper did as well (on the second attempt). Looking at the usage graphs it does seem to still have high memory usage, but schemas are currently up-to-date.

Commit merged into master by Frank Bertsch (fbertsch) Authored by Michael Droettboom (mdboom)
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Component: Pipeline Ingestion → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: