mac-v2-signing2 has some weird issues with the rel-key-signing-server directory
Categories
(Infrastructure & Operations :: RelOps: Hardware, task)
Tracking
(Not tracked)
People
(Reporter: mozilla, Assigned: dhouse)
References
Details
|
Reporter | |
Comment 1•7 years ago
|
||
|
|
Reporter | |
Comment 2•7 years ago
|
||
Also /builds/signing/dep-key-signing-server/lib/python2.7/site-packages/.broken .
I've run diskutil verifyvolume and get an error (exit code 8, exec format error). So I'll research this and plan to run a repair and see if that fixes it.
fsck requires running in recovery or single-user mode. So we can do that if diskutil cannot repair the disk.
[dhouse@mac-v2-signing2.srv.releng.mdc1.mozilla.com ~]$ df -k
Filesystem 1024-blocks Used Available Capacity iused ifree %iused Mounted on
/dev/disk2 731241344 22662456 708322888 4% 5729612 177080722 3% /
devfs 180 180 0 100% 626 0 100% /dev
/dev/disk1s2 732238672 616200 731622472 1% 154048 182905618 0% /Volumes/Untitled 1
map -hosts 0 0 0 100% 0 0 100% /net
map auto_home 0 0 0 100% 0 0 100% /home
[dhouse@mac-v2-signing2.srv.releng.mdc1.mozilla.com ~]$ diskutil list
/dev/disk0
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *750.2 GB disk0
1: EFI EFI 209.7 MB disk0s1
2: Apple_CoreStorage 749.2 GB disk0s2
3: Apple_Boot Recovery HD 650.0 MB disk0s3
/dev/disk1
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *750.2 GB disk1
1: EFI EFI 209.7 MB disk1s1
2: Apple_HFS Untitled 1 749.8 GB disk1s2
/dev/disk2
#: TYPE NAME SIZE IDENTIFIER
0: Apple_HFS Macintosh HD *748.8 GB disk2
Logical Volume on disk0s2
9DA46A49-0CFF-4D33-9DCE-4F5644843B81
Unencrypted
[dhouse@mac-v2-signing2.srv.releng.mdc1.mozilla.com ~]$ sudo diskutil verifyvolume /
Started file system verification on disk2 Macintosh HD
Verifying storage system
Checking volume
disk0s2: Scan for Volume Headers
disk0s2: Scan for Disk Labels
Logical Volume Group 21197F5C-296B-40DF-982A-F76B7B81C888 on 1 device
disk0s2: Scan for Metadata Volume
Logical Volume Group has a 24 MB Metadata Volume with double redundancy
Start scanning metadata for a valid checkpoint
Load and verify Segment Headers
Load and verify Checkpoint Payload
Load and verify Transaction Segment
Incorporate 0 newer non-checkpoint transactions
Load and verify Virtual Address Table
Load and verify Segment Usage Table
Load and verify Metadata Superblock
Load and verify Logical Volumes B-Trees
Logical Volume Group contains 1 Logical Volume
Load and verify C1889711-0B89-4E5C-9565-D1186909DB02
Load and verify 9DA46A49-0CFF-4D33-9DCE-4F5644843B81
Load and verify Freespace Summary
Load and verify Block Accounting
Load and verify Live Virtual Addresses
Newest transaction commit checkpoint is valid
Load and verify Segment Cleaning
The volume 21197F5C-296B-40DF-982A-F76B7B81C888 appears to be OK
Storage system check exit code is 0
Verifying file system
Using live mode
Performing live verification
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
The volume Macintosh HD could not be verified completely
File system check exit code is 8
Error: -69845: File system verify or repair failed
Underlying error: 8: POSIX reports: Exec format error
[dhouse@mac-v2-signing2.srv.releng.mdc1.mozilla.com ~]$
Aki, would it be okay with you if I take mac-v2-signing2 offline today or Monday to run fsck and try to repair the disk? I'll ask QTS to reboot into recovery mode and run fsck (until it shows no errors, or about 5 times if it doesn't fix the problems).
|
|
Reporter | |
Comment 6•7 years ago
|
||
I've removed the nightly and release secrets, so we'll need to copy those back in before restarting.
I've opened case REQ0267519 with QTS to run fsck (as many times as needed or phone me) on the machine from recovery mode.
QTS rebooted this machine but was not able to get it to come back up.
It powers-off 1/4 into boot (recovery/diagnostics/etc attempted).
We tried resetting smc (unplugged 30+s, plugged power back in. 5s wait. power on).
And we tried resetting nvram (command+option+p+r).
And we tried to enter hardware diagnostics.
Jake will be in MDC1 later this month and suggested he could look at it or pull the disk to ensure it is wiped/zeroed securely to decommission it.
This machine is off warranty: Mac mini Server (Mid 2011). Serial Number: C07HT05CDKDJ
https://checkcoverage.apple.com/us/en/?sn=C07HT05CDKDJ
mac-v2-signing2.srv.releng.mdc1.mozilla.com
Rack: MDC1::IT41
Shelf: 14.1
Asset# 11719
pdu1.rit41.ops.releng.mdc1.mozilla.com:ac6
|
||
Comment 10•7 years ago
|
||
(In reply to Dave House [:dhouse] from comment #9)
Jake will be in MDC1 later this month and suggested he could look at it or pull the disk to ensure it is wiped/zeroed securely to decommission it.
(on site @ mdc1) I was able to see the power off issue as described by QTS but was able to start the recovery console. I've initiated a single pass wipe of the disk. eta 1:40h
|
|
||
Comment 11•7 years ago
|
||
The disk wipe worked fine but I got curious as to if it would take an image. And it did, just fine. I'm wondering if we should put it back in production again. If not, I'll file a bug to have IT decomm it next time they are in mdc1.
aki, thoughts?
|
|
Reporter | |
Comment 12•7 years ago
|
||
Nice! I'll copy the secrets over and we can see how it does.
Thanks Jake!
|
|
Assignee | |
Comment 13•7 years ago
|
||
Aki, I'm going to close this bug. Nagios hasn't seen a problem with the host for 18 days: https://nagios1.private.releng.mdc1.mozilla.com/releng-mdc1/cgi-bin/status.cgi?host=mac-v2-signing2.srv.releng.mdc1.mozilla.com
If you have any concerns, or if you saw problems with it, we can reopen this.
Thanks!
Description
•