483595 - Linux mozilla-1.9.1 unit test % fails to link libxul, timing out on multiple slaves

Status: RESOLVED DUPLICATE of bug 482463

(Release Engineering :: General, defect, P2)

Description

[Axel Hecht [:Pike]]

http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox3.1&maxdate=1237207079&hours=6&legend=0&norules=1 shows three timeouts in linking libxul, on slaves moz2-linux-slave11, 13, and 14.

Comment 1

[Axel Hecht [:Pike]]

... plus a mochitest timeout and a nightly build failure for WINNT 5.2.

Intel cluster badness?

Comment 2

[[e-mail] (:bhearsum)]

Our VM hosts seem fine:
HOST        | CPU % | MEM % |
bm-vmware01 |  57%  |  77%  |
bm-vmware02 |  53%  |  71%  |
bm-vmware03 |  64%  |  73%  |
bm-vmware04 |  32%  |  64%  |
bm-vmware05 |  52%  |  81%  |
bm-vmware06 |  77%  |  83%  |
bm-vmware07 |  58%  |  68%  |
bm-vmware08 |  54%  |  79%  |
bm-vmware09 |  72%  |  82%  |
bm-vmware10 |  71%  |  78%  |
bm-vmware11 |  62%  |  79%  |
bm-vmware12 |  74%  |  86%  |
bm-vmware13 |  46%  |  82%  |

Right now *all* of our Linux and Win32 VMs are building, too, so I suspect it doesn't spike much higher than this.


(Also, this might be related to bug 482463)

Comment 3

[[e-mail] (:bhearsum)]

Our Linux machines only have 768MB of RAM. I'm going to watch for swapping.

Comment 4

[Axel Hecht [:Pike]]

Are there cron jobs on either the VMs or the actual hardware around that time? Possibly they take additional toll on the memory and trigger swapping or something.

Comment 5

[[e-mail] (:bhearsum)]

Looks like we may be swapping:
from one machine:
Mem:    775228k total,   766732k used,     8496k free,    33380k buffers
Swap:   522104k total,    91932k used,   430172k free,   342560k cached
from another:
Mem:    775228k total,   766796k used,     8432k free,    24224k buffers
Swap:   522104k total,    78036k used,   444068k free,   402268k cached

Comment 6

[[e-mail] (:bhearsum)]

On a daily basis we run updatedb and logrotate - at 4am pacific. updatedb probably does Bad Bad things considering all the file changes over the course of the day. I'll look into disable this.