1115930 - Swap on gitmirror.dmz.phx1.mozilla.com is WARNING: SWAP WARNING - 50% free (3063 MB out of 6143 MB)

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[david garvey:dgarvey]

Guys this git front server is now running Critical in swap. The problem appears to be git itself. Can it be restarted to free some memory?



Hostname: gitmirror.dmz.phx1.mozilla.com
Service:  Swap
State:    WARNING
Output:   SWAP WARNING - 50% free (3063 MB out of 6143 MB)

Runbook:  http://m.allizom.org/Swap

Comment 1

[Hal Wine [:hwine] use NI!]

gitmirror is currently a "best effort" service, so no checkin with developer services is needed before restarts. Filed bug 1115958 to get the docs updated. :)

Comment 2

[Jason Crowe [:jd]]

I do not believe that this is managed by webops any longer. Feel free to hand it back over if I am mistaken.

Comment 3

[Ryan C [:ryanc] (UTC-4)]

Thu 00:19:02 PST [1735] gitmirror.dmz.phx1.mozilla.com:http - gitmirror.m.o is CRITICAL: CRITICAL - Socket timeout after 10 seconds

Thu 00:18:52 PST [1734] gitmirror.dmz.phx1.mozilla.com:Load is WARNING: WARNING - load average: 21.29, 21.09, 17.91

Box is responsive in accordance to the service documentation. Running several processes of the following;

apache   30910  5.5  0.7 1213648 60808 ?       S    Feb04  16:25 git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://github.com/mozilla-b2g/gaia.git/

Perhaps due to a recent commit in the last few minutes.

Comment 4

[Gregory Szorc [:gps]]

`git fetch-pack` is the process that runs on the Git server when a client is doing a clone or a fetch. Here, the culprit seems to be the gaia.git repo, which I suspect is related to TaskCluster load testing.

The server side of a `git fetch` will potentially spin up several processes on the server (1 per available core IIRC). If you have multiple clients coming in at the same time, there is a potential for server-side process explosion and memory requirements that come along with it. This is because the Git server will need to dynamically generate a packfile meeting the client's requirements. The bigger the difference between the client's local state and the server's, the more work the server will do.

Comment 5

[david garvey:dgarvey]

Well I will continue to monitor frequency of this event. Perhaps a memory upgrade might be in order?