859148 - investigate "humpty dumpty" errors from machines that cant connect to puppet

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

For over a week, I've been deleting these alerts every time I refresh my email inbox, without having a chance to file a bug. Now, on a sunny Sunday, I finally have time! 

Not sure how many days?weeks? these machines have been out-of-commission, but the following machines need manual intervention:

talos-r4-lion-085
talos-r4-lion-086
talos-r4-lion-087
talos-r4-lion-088
talos-r4-lion-089
talos-r4-lion-090
cltblds-Mac-mini-186
cltblds-Mac-mini-187
cltblds-Mac-mini-230
cltblds-Mac-mini-249
cltblds-Mac-mini-254
cltblds-Mac-mini-277
talos-r4-snow-029
Client-Builders-Mac-mini-8
Client-Builders-Mac-mini-76

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Re-arranged slightly, most of the mail is coming from the first two blocks.

(In reply to John O'Duinn [:joduinn] from comment #0)
> talos-r4-lion-085
> talos-r4-lion-086
> talos-r4-lion-087
> talos-r4-lion-088
> talos-r4-lion-089
> talos-r4-lion-090

These are from bug 848992, foopies reimaged as testers. I'll file a bug to get them set up on our side.

> cltblds-Mac-mini-186
> cltblds-Mac-mini-187
> cltblds-Mac-mini-249
> cltblds-Mac-mini-254
> cltblds-Mac-mini-277

Hosts from above without proper hostnames.

> cltblds-Mac-mini-230

Actually talos-r4-lion-039, suspected bad RAM. DCOps running diagnostics.

> talos-r4-snow-029

Got reimaged in bug 856626, still with DCOps for diagnostics.

> Client-Builders-Mac-mini-8

talos-r4-snow-001 from reverse DNS on IP. In production, intermittent fail.

> Client-Builders-Mac-mini-76

talos-r4-snow-002 from reverse DNS on IP. In production, intermittent fail.