Closed
Bug 1037260
(mac-v2-signing4)
Opened 10 years ago
Closed 10 years ago
mac-v2-signing4 problem tracking
Categories
(Infrastructure & Operations Graveyard :: CIDuty, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: nthomas, Unassigned)
References
Details
At 16:30 nagios began reporting timeouts: Thu 16:30:22 PDT [4102] mac-signing4.srv.releng.scl3.mozilla.com:load is CRITICAL: CHECK_NRPE: Socket timeout after 15 seconds. Thu 16:32:11 PDT [4103] mac-signing4.srv.releng.scl3.mozilla.com:signing-server is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. Thu 16:34:11 PDT [4104] mac-signing4.srv.releng.scl3.mozilla.com:ntp time is CRITICAL: CHECK_NRPE: Socket timeout after 15 seconds. Thu 16:35:01 PDT [4105] mac-signing4.srv.releng.scl3.mozilla.com:disk - / is CRITICAL: CHECK_NRPE: Socket timeout after 60 seconds. Thu 16:35:21 PDT [4106] mac-signing4.srv.releng.scl3.mozilla.com:load is CRITICAL: CHECK_NRPE: Socket timeout after 15 seconds. Puppet was having problems running processes like /builds/signing/dep-key-signing-server/bin/python /tools/misc-python/pip-check.py pip==1.5.5 and signing jobs were timing out. Attempting to launch that python directly lead to a hanging process with no output. Nothing to see in /var/log/system.log
Reporter | ||
Comment 1•10 years ago
|
||
I ended up rebooting it and then python would launch OK. Restarted the signing instances. bhearsum, is this a new or known failure case ?
Flags: needinfo?(bhearsum)
Comment 2•10 years ago
|
||
This is nothing I've heard of before, but it sounds vaguely familiar to some issues we see on slaves sometimes. Might be worthwhile running hardware diagnostics.
Flags: needinfo?(bhearsum)
Reporter | ||
Comment 3•10 years ago
|
||
Lets let this ride once, and blame ... *spins bofh-excuses * ... cosmic rays from static rulers.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Reporter | ||
Comment 4•10 years ago
|
||
Bug 1008052 might be a symptom of flaky hardware here.
See Also: → 1008052
Comment 5•10 years ago
|
||
This machine got upgraded as part of bug 1082685.
Summary: mac-signing4 problem tracking → mac-v4-signing4 problem tracking
Updated•10 years ago
|
Alias: mac-signing4 → mac-v2-signing4
Summary: mac-v4-signing4 problem tracking → mac-v2-signing4 problem tracking
Reporter | ||
Comment 6•9 years ago
|
||
The dep signing instance hasn't been reloading the config since 2014-10-17. Looks like signing.pid got lost and then http://hg.mozilla.org/build/puppet/file/0bd2c271b526/modules/signingserver/manifests/instance.pp#l160. I recreated the .pid and ran the --reload command.
Updated•6 years ago
|
Product: Release Engineering → Infrastructure & Operations
Updated•4 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•