Closed
Bug 1115930
Opened 9 years ago
Closed 9 years ago
Swap on gitmirror.dmz.phx1.mozilla.com is WARNING: SWAP WARNING - 50% free (3063 MB out of 6143 MB)
Categories
(Developer Services :: General, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: dgarvey, Unassigned)
References
()
Details
(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/4250] [id=nagios1.private.phx1.mozilla.com:408453])
Guys this git front server is now running Critical in swap. The problem appears to be git itself. Can it be restarted to free some memory? Hostname: gitmirror.dmz.phx1.mozilla.com Service: Swap State: WARNING Output: SWAP WARNING - 50% free (3063 MB out of 6143 MB) Runbook: http://m.allizom.org/Swap
Comment 1•9 years ago
|
||
gitmirror is currently a "best effort" service, so no checkin with developer services is needed before restarts. Filed bug 1115958 to get the docs updated. :)
Comment 2•9 years ago
|
||
I do not believe that this is managed by webops any longer. Feel free to hand it back over if I am mistaken.
Assignee: server-ops-webops → nobody
Component: WebOps: IT-Managed Tools → Git
Product: Infrastructure & Operations → Developer Services
QA Contact: nmaul
Updated•9 years ago
|
Whiteboard: [id=nagios1.private.phx1.mozilla.com:408453] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/4250] [id=nagios1.private.phx1.mozilla.com:408453]
Updated•9 years ago
|
Component: Git → General
Comment 3•9 years ago
|
||
Thu 00:19:02 PST [1735] gitmirror.dmz.phx1.mozilla.com:http - gitmirror.m.o is CRITICAL: CRITICAL - Socket timeout after 10 seconds Thu 00:18:52 PST [1734] gitmirror.dmz.phx1.mozilla.com:Load is WARNING: WARNING - load average: 21.29, 21.09, 17.91 Box is responsive in accordance to the service documentation. Running several processes of the following; apache 30910 5.5 0.7 1213648 60808 ? S Feb04 16:25 git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://github.com/mozilla-b2g/gaia.git/ Perhaps due to a recent commit in the last few minutes.
Comment 4•9 years ago
|
||
`git fetch-pack` is the process that runs on the Git server when a client is doing a clone or a fetch. Here, the culprit seems to be the gaia.git repo, which I suspect is related to TaskCluster load testing. The server side of a `git fetch` will potentially spin up several processes on the server (1 per available core IIRC). If you have multiple clients coming in at the same time, there is a potential for server-side process explosion and memory requirements that come along with it. This is because the Git server will need to dynamically generate a packfile meeting the client's requirements. The bigger the difference between the client's local state and the server's, the more work the server will do.
Reporter | ||
Comment 5•9 years ago
|
||
Well I will continue to monitor frequency of this event. Perhaps a memory upgrade might be in order?
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•