1339124 - investigate sns notifications from install.test.releng.scl3.mozilla.com

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Amy Rich [:arr] [:arich]]

Received an irc ping regarding SNS alerts in #buildduty:

[sns alert] Feb 13 03:32:31 install.test.releng.scl3.mozilla.com FileStatsAgent: Failed to enumerate: (null), error: Error Domain=NSCocoaErrorDomain Code=257 "The file “Downloads” couldn’t be opened because you don’t have permission to view it." UserInfo=0x7fc3a8c2f800 {NSURL=file:///Users/dsadmin/Downloads, NSFilePath=/Users/dsadmin/Downloads, NSUnderlyingError=0x7fc3a8c2f760 "The operation couldn’t be completed.
13:35:10 Permission denied"}
13:35:11 [sns alert] Feb 13 03:32:31 install.test.releng.scl3.mozilla.com FileStatsAgent: Failed to enumerate: (null), error: Error Domain=NSCocoaErrorDomain Code=257 "The file “Backup” couldn’t be opened because you don’t have permission to view it." UserInfo=0x7fc3a8d07da0 {NSURL=file:///Users/dsadmin/Library/Application%20Support/MobileSync/Backup/, NSFilePath=/Users/dsadmin/Library/Application Support/MobileSync/Backup,
13:35:12 NSUnderlyingError=0x7fc3a8d07d40 "The operation couldn’t be completed. Permission denied"}
13:35:13 [sns alert] Feb 13 03:32:31 install.test.releng.scl3.mozilla.com FileStatsAgent: Metadata.framework [Error]: couldn't get the client port
13:35:14 [sns alert] Feb 13 03:32:31 install.test.releng.scl3.mozilla.com ReportCrash: File creation failed /Users/dsadmin/Library/Logs/DiagnosticReports/cloudd_2017-02-13-033231_install-test-releng-scl3-mozilla-com.crash - uid: 543647 gid: 20, euid: 543647 egid: 20 - error 13: Permission denied
13:35:15 [sns alert] Feb 13 03:32:31 install.test.releng.scl3.mozilla.com ReportCrash: *** Terminating app due to uncaught exception 'NSGenericException', reason: 'Error opening db at /Users/USER/Library/Caches/*/CloudKitMetadata: <CKError 0x7ff4f14a6eb0: "Internal Error" (1)>'
13:35:16 [sns alert] Feb 13 03:32:31 install.test.releng.scl3.mozilla.com ReportCrash: 3   CloudKit                            0x00007fff8fbd4d5e +[CKSQLiteError raise:code:extended:] + 1005
13:35:17 [sns alert] Feb 13 03:32:31 install.test.releng.scl3.mozilla.com ReportCrash: Error Code:      0x02000148

Comment 1

[Amy Rich [:arr] [:arich]]

I looked through all of the alerts in papertrail and am unable to determine which one would have sent this. The DS servers have no targeted alerts, so the only thing applicable would have been one of the generic alerts. None of them match this output pattern.

I did reboot install.test since it had been up for > 100 days and there were a lot of ReportCrash lines in the logs. It came back up, but then froze after I logged in. I power cycled it from the PDU outlet and it seemed to come back up okay the second time.

I verified both disks (both reported no errors), did a manual timemachine backup (which succeeded), and verified and fixed disk permissions for the boot drive.

If SNS alerts from install.test appear again, please reopen this bug.