Closed
Bug 469904
Opened 16 years ago
Closed 15 years ago
tinderbox.bugzilla.lan is so slow that our Bugzilla QA tests can't pass
Categories
(mozilla.org Graveyard :: Server Operations: Projects, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: mkanat, Assigned: phong)
References
Details
tinderbox.bugzilla.lan (192.168.99.10 on cg-vmware01) is the tinderbox client for the Bugzilla tests. I recently set up our Selenium tests to run on that server, and with everything else that's going on, things are now so slow that the QA tests actually time out when they try to run, meaning that we can't get accurate test results. The problem both the disk and the CPU. Adding another CPU to the system (and possibly allocating it more RAM) would probably help. If that doesn't help, it'll just need faster disk access, if that's in any way possible.
Comment 1•16 years ago
|
||
Phong is our vmware guy, so passing this to him.
Assignee: server-ops → phong
Comment 2•16 years ago
|
||
Are the vmware guest tools running?
Reporter | ||
Comment 3•16 years ago
|
||
I think so: ps -Af | grep vm root 1735 1 0 Nov26 ? 00:09:13 [vmmemctl] root 1770 1 0 Nov26 ? 00:55:36 /usr/sbin/vmware-guestd --background /var/run/vmware-guestd.pid
Assignee | ||
Comment 4•16 years ago
|
||
This ESX host only has 4GB of RAM total. We will need to add more to this host. This will require downtime to add RAM.
Assignee | ||
Updated•16 years ago
|
Flags: needs-downtime+
Flags: colo-trip+
Reporter | ||
Comment 5•16 years ago
|
||
Really? I could have sworn it had 8GB. Is it maybe only addressing 4GB because the host OS is 32-bit?
Comment 6•16 years ago
|
||
The ESX host only has 4GB of physical memory and more than 70% of it is in use. The tinderbox.bugzilla.lan VM is only configured for 1GB RAM.
Assignee | ||
Comment 7•16 years ago
|
||
Max: Can I take this cluster down tomorrow afternoon to add more RAM?
Reporter | ||
Comment 8•16 years ago
|
||
(In reply to comment #7) > Max: Can I take this cluster down tomorrow afternoon to add more RAM? Sure, that would be fine. If I'm on IRC (mkanat) just let me know before it goes down.
Comment 9•16 years ago
|
||
(In reply to comment #7) > Max: Can I take this cluster down tomorrow afternoon to add more RAM? Which cluster is that? All of cg-vmware01? That affects way more than just Bugzilla, so please make sure you get all necessary parties involved.
Comment 10•16 years ago
|
||
> Which cluster is that? All of cg-vmware01? That affects way more than just
> Bugzilla, so please make sure you get all necessary parties involved.
I wouldn't say "way more" but Reed's right - I forget who ownes cg-ecmascript01 and cg-centos01 but I suspect you could easily get a window from those owners or pause the VMs before taking the host down.
displayName = "cg-ecmascript01"
displayName = "landfill"
displayName = "tinderbox.bugzilla"
displayName = "cg-centos01"
displayName = "windows.bugzilla"
displayName = "oracle.bugzilla"
Comment 11•16 years ago
|
||
I own cg-centos01, fyi. ;)
Assignee | ||
Comment 12•16 years ago
|
||
Who are the owners of the remaining VM's?
Comment 13•16 years ago
|
||
(In reply to comment #12) > Who are the owners of the remaining VM's? "cg-ecmascript01" -- Dave Herman (dherman@ccs.neu.edu) [however, I think this VM isn't used anymore... should ask] "landfill" -- Bugzilla Project "tinderbox.bugzilla" -- Bugzilla Project "cg-centos01" -- reed "windows.bugzilla" -- Bugzilla Project "oracle.bugzilla" -- Bugzilla Project
Comment 14•16 years ago
|
||
Re: cg-ecmascript01, IIRC we didn't end up using this, but I'd double check with Brendan before deleting. Thx
Comment 15•16 years ago
|
||
And I have to check with graydon, my memory is failing me. Probably that means we never used this. /be
Comment 16•16 years ago
|
||
nope, not used.
Assignee | ||
Comment 17•16 years ago
|
||
are we ready for me to take these down? I'll also delete cg-ecmascript01.
Comment 18•16 years ago
|
||
(In reply to comment #17) > are we ready for me to take these down? I'll also delete cg-ecmascript01. ok from me for cg-centos01
Comment 19•16 years ago
|
||
(In reply to comment #8) > Sure, that would be fine. If I'm on IRC (mkanat) just let me know before it > goes down. mkanat is not on IRC, so let's go.
Assignee | ||
Comment 20•16 years ago
|
||
I was able to added 1 GB of RAM to the ESX host. I also bumped tinderbox.bugzilla RAM up to 1536.
Reporter | ||
Comment 21•16 years ago
|
||
Great! So, we're not hitting the swap anymore, but the machine is still too slow for the QA tests to pass. Any chance of allocating it another CPU? If that doesn't do it, then it will need to be moved to its own machine (which is funny, since it's one of the whole reasons we got cg-vmware01).
Assignee | ||
Comment 22•16 years ago
|
||
I can add a second CPU for this VM, but it will require a quick shutdown to make the change.
Reporter | ||
Comment 23•16 years ago
|
||
(In reply to comment #22) > I can add a second CPU for this VM, but it will require a quick shutdown to > make the change. That's fine, you can shut it down whenever.
Assignee | ||
Comment 24•16 years ago
|
||
second CPU added.
Assignee | ||
Comment 25•16 years ago
|
||
Please reopen if you run into more issues.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
Reporter | ||
Comment 26•15 years ago
|
||
The machine is still too slow for our tests to all be running and pass. (Also, unfortunately, I don't have a good way to run them in-order instead of in parallel.) The machine is currently swapping actively--it's using about 1GB of swap. I suspect the main limiter is the disk, though. Ideally we'd have an additional machine for certain disk-heavy tests.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 27•15 years ago
|
||
This ESX host has 4GB RAM and is using nearly 70% of it right now. It doesn't at all look CPU constrained. Disk I/O doesn't look strained either. My gut feeling is that the ESX host is memory constrained. I'd recommend a query to Community Giving about upgrading either the entire ESX host or getting additional memory.
Component: Server Operations → Server Operations: Projects
Comment 28•15 years ago
|
||
The server upgrade is being tracked elsewhere, can I close this?
Flags: needs-downtime-
Flags: needs-downtime+
Flags: colo-trip-
Flags: colo-trip+
Reporter | ||
Comment 29•15 years ago
|
||
Sure.
Updated•15 years ago
|
Status: REOPENED → RESOLVED
Closed: 16 years ago → 15 years ago
Resolution: --- → FIXED
Updated•9 years ago
|
Product: mozilla.org → mozilla.org Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•