997702 - git1 memory

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Kendall Libby [:fubar] (he/him)]

git1.dmz.scl3 is running into OOM issues and burning trees nightly. It's a BL460C G7 blade, and docs say it'll take up to 48GB memory; it's currently got 36. Do we have spare memory around to upgrade it? (Enough to upgrade it and git2, it's warm standby?)

Comment 1

[Van Le [:van]]

:fubar, we have enough spare memory on site to bump each of these hosts up an additional 8gb, will that help?

Comment 2

[Kendall Libby [:fubar] (he/him)]

Since this will cause an outage, and other changes have made the immediate pain go away, we're going to defer this. Once we get n+1 servers up, we may well put in an order for more ram. Thanks!

Comment 3

[Van Le [:van]]

just a heads up, we were able to allocate some more dimms into the spare pool after the spring clean up so we can bump these up to 48gb if needed.

Comment 4

[Ed Morley [:emorley]]

Wontfix seems like the correct resolution given comment 2 and comment 3?

Comment 5

[Hal Wine [:hwine] use NI!]

We need to reconsider this, as the prior bandaids are starting to leak. :/

Comment 6

[Van Le [:van]]

worked with :fubar to upgrade these nodes to 64gb.