If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Swap on gitmirror.dmz.phx1.mozilla.com is WARNING: SWAP WARNING - 50% free (3063 MB out of 6143 MB)

RESOLVED FIXED

Status

Developer Services
General
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: dgarvey, Unassigned)

Tracking

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/4250] [id=nagios1.private.phx1.mozilla.com:408453], URL)

(Reporter)

Description

3 years ago
Guys this git front server is now running Critical in swap. The problem appears to be git itself. Can it be restarted to free some memory?



Hostname: gitmirror.dmz.phx1.mozilla.com
Service:  Swap
State:    WARNING
Output:   SWAP WARNING - 50% free (3063 MB out of 6143 MB)

Runbook:  http://m.allizom.org/Swap
See Also: → bug 1115958
gitmirror is currently a "best effort" service, so no checkin with developer services is needed before restarts. Filed bug 1115958 to get the docs updated. :)

Comment 2

3 years ago
I do not believe that this is managed by webops any longer. Feel free to hand it back over if I am mistaken.
Assignee: server-ops-webops → nobody
Component: WebOps: IT-Managed Tools → Git
Product: Infrastructure & Operations → Developer Services
QA Contact: nmaul

Updated

3 years ago
Whiteboard: [id=nagios1.private.phx1.mozilla.com:408453] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/4250] [id=nagios1.private.phx1.mozilla.com:408453]
Component: Git → General
Thu 00:19:02 PST [1735] gitmirror.dmz.phx1.mozilla.com:http - gitmirror.m.o is CRITICAL: CRITICAL - Socket timeout after 10 seconds

Thu 00:18:52 PST [1734] gitmirror.dmz.phx1.mozilla.com:Load is WARNING: WARNING - load average: 21.29, 21.09, 17.91

Box is responsive in accordance to the service documentation. Running several processes of the following;

apache   30910  5.5  0.7 1213648 60808 ?       S    Feb04  16:25 git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://github.com/mozilla-b2g/gaia.git/

Perhaps due to a recent commit in the last few minutes.

Comment 4

3 years ago
`git fetch-pack` is the process that runs on the Git server when a client is doing a clone or a fetch. Here, the culprit seems to be the gaia.git repo, which I suspect is related to TaskCluster load testing.

The server side of a `git fetch` will potentially spin up several processes on the server (1 per available core IIRC). If you have multiple clients coming in at the same time, there is a potential for server-side process explosion and memory requirements that come along with it. This is because the Git server will need to dynamically generate a packfile meeting the client's requirements. The bigger the difference between the client's local state and the server's, the more work the server will do.
(Reporter)

Comment 5

3 years ago
Well I will continue to monitor frequency of this event. Perhaps a memory upgrade might be in order?
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.