1401559 - Increase memory on dxr-proocessor[1-4]

Status: RESOLVED FIXED

(Infrastructure & Operations :: Virtualization, task)

Description

[Kendall Libby [:fubar] (he/him)]

I'd like to get a memory increase on the 4 DXR indexing nodes, dxr-processor[1-4].dmz.scl3. They're currently at 16gb but the firefox trees, particularly mozilla-central and -beta, have been failing a lot recently due to oom_killer whacking processes. I've reduced the number of workers on the jobs from 4 to 3 to try and stave it off, but it's continued to get worse. Decreasing the number of workers has also had the negative side effect of increasing indexing times from the previous ~4 hours to 6-8 hours, which means developers are always looking at effectively old data.

I don't have a target amount of memory offhand, so I'm open to suggestions.

Comment 1

[Chris Knowles [:cknowles]]

OK, looking at things from the VMware side, I can see the memory contentions several times through the day.  

However, 16GB is pretty large.  And there's a diminishing return issue, in that more memory doesn't necessarily lead to more happiness.  We'd *prefer* to increase the count of VMs if possible.  (Say, another 2 workers? at the same 16G specs)  However, that's dependent on the load being flexible and able to be spread out that way.

If that's not possible... then given the amount of RAM I was about to hand out in the above bit - how about +8G on all 4 nodes?

If we do expand the memory, we'll need ~15 minutes outage per node to increase the RAM.

Comment 2

[Kendall Libby [:fubar] (he/him)]

(In reply to Chris Knowles [:cknowles] from comment #1)
 
> We'd *prefer*
> to increase the count of VMs if possible.  (Say, another 2 workers? at the
> same 16G specs)  However, that's dependent on the load being flexible and
> able to be spread out that way.

Nope, we can't spread individual indexing jobs horizontally. By workers, I meant DXR's internal worker processes, not indexing hosts. :-(

> If that's not possible... then given the amount of RAM I was about to hand
> out in the above bit - how about +8G on all 4 nodes?

That'd be great. I don't expect to need more any time soon, between DC migration and this current issue is due to the addition of Stylo. (famous last words)

> If we do expand the memory, we'll need ~15 minutes outage per node to
> increase the RAM.

You are free to do this at your convenience. Some of the builds in progress are already burnt due to oom_killer, the rest are easy to restart.

Comment 3

[Chris Knowles [:cknowles]]

Bumped all 4 to 24G RAM. Updated our tracking sheets.

:fubar reports things are already picking up jobs.

Closing out.  Let me know if I can do anything else for you.