1569980 - Web dynos exhausting memory

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Armen [:armenzg]]

Attachment: image.png

For a reason I fail to understand, after midnight the web dynos started bloating.

I increased the web dynos from 5 to 7 without an improvement.
I set the WEB_CONCURRENCY to 1 (instead of the default of 2) and that brought us down within reasonable limits.

We changed the dynos from Performance-M to Standard 2x about 24 hours ago, however, everything looked smooth until last night at midnight.

NewRelic did not show anything that will catch my eye.

We're out of the woods, however, I wanted to keep you on the loop and monitor this a bit more.

Comment 1

[Armen [:armenzg]]

Attachment: treeherder-prod - Heroku web dynos metrics

We went from too much memory being used to too many responses timing out.
I'm reverting this change back to Performance-M workers.

Comment 2

[Armen [:armenzg]]

I've also removed the WEB_CONCURRENCY env value.

Comment 3

[Armen [:armenzg]]

It all seems to be back to normal.