Guys this git front server is now running Critical in swap. The problem appears to be git itself. Can it be restarted to free some memory? Hostname: gitmirror.dmz.phx1.mozilla.com Service: Swap State: WARNING Output: SWAP WARNING - 50% free (3063 MB out of 6143 MB) Runbook: http://m.allizom.org/Swap
gitmirror is currently a "best effort" service, so no checkin with developer services is needed before restarts. Filed bug 1115958 to get the docs updated. :)
I do not believe that this is managed by webops any longer. Feel free to hand it back over if I am mistaken.
Thu 00:19:02 PST  gitmirror.dmz.phx1.mozilla.com:http - gitmirror.m.o is CRITICAL: CRITICAL - Socket timeout after 10 seconds Thu 00:18:52 PST  gitmirror.dmz.phx1.mozilla.com:Load is WARNING: WARNING - load average: 21.29, 21.09, 17.91 Box is responsive in accordance to the service documentation. Running several processes of the following; apache 30910 5.5 0.7 1213648 60808 ? S Feb04 16:25 git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://github.com/mozilla-b2g/gaia.git/ Perhaps due to a recent commit in the last few minutes.
`git fetch-pack` is the process that runs on the Git server when a client is doing a clone or a fetch. Here, the culprit seems to be the gaia.git repo, which I suspect is related to TaskCluster load testing. The server side of a `git fetch` will potentially spin up several processes on the server (1 per available core IIRC). If you have multiple clients coming in at the same time, there is a potential for server-side process explosion and memory requirements that come along with it. This is because the Git server will need to dynamically generate a packfile meeting the client's requirements. The bigger the difference between the client's local state and the server's, the more work the server will do.
Well I will continue to monitor frequency of this event. Perhaps a memory upgrade might be in order?