799826 - Investigate Jenkin's resourcing troubles

Status: RESOLVED FIXED

(Infrastructure & Operations :: IT-Managed Tools, task, P4)

Description

[Lonnen :lonnen]

Jenkin's is complaining that it is under resourced, and we've had some hanging builds lately.

from https://ci.mozilla.org/manage:
"Threads are not keeping up with the demands. Check if your polling is hanging, and/or increase the number of threads if necessary."

from https://ci.mozilla.org/descriptor/hudson.triggers.SCMTrigger/:
"There are more SCM polling activities scheduled than handled, so the threads are not keeping up with the demands. Check if your polling is hanging, and/or increase the number of threads if necessary."

If I understand things right, most of our jobs poll SCM. If we can't increase the Jenkin's resources we might convert more of our jobs to listening for push notifications from GH.

Comment 1

[Jake Maul [:jakem]]

I think this is likely a bit better now, due to improvements in the socorro build processes which result in far less disk I/O on jenkins1.dmz.phx1.

jenkins1.dmz.phx1 has 37GB RAM, but is currently 14GB into swap... there's enough free mem + cache to empty out swap, so we could do that and tweak swapiness... might help some.

A better solution is probably to set up some jenkins slaves and move some of the workload over to them.

Comment 2

[Lonnen :lonnen]

@jakem the bug you caught in my build scripts was introduced after this bug was filed. I'm sure it wasn't helping but I don't think I/O is the problem.

Comment 3

[Brandon Burton [:solarce]]

Job clean up, disk clean, and jenkins tuning has improved this enough not to be an issue.