709591 - most linux64 builders not running

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

These appear to be stuck in puppet.  Puppet is stuck due to

Dec 11 08:29:24 scl-production-puppet-new puppetmasterd[3663]: Could not parse for environment production: Node talos-r3-leopard-061 is already defined at /etc/puppet/manifests/scl-production.pp:1127; cannot redefine at /etc/puppet/manifests/scl-production.pp:1131

which was landed in bug 683734.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m709591.patch

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Landed as a break-fix patch.  scl-production-puppet is getting hammered, but hosts seem to be coming back, if slowly.

Over to releng to monitor and close when this is confirmed fixed.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

It wasn't coming back - it was just thrashing.

I used iptables to firewall off all but 10.12.48.0/24, and get those puppetized, and I'll keep adding /24's as each one gets finished.  It seems to be recovering this way.

Comment 4

[Chris Cooper [:coop] (he/him)]

catlee landed a fix this morning (bug 711604). I'll make sure any offline slaves sync-up with puppet and make it back into production.

Comment 5

[Chris Cooper [:coop] (he/him)]

Oops. I did an initial pass back in December, and another one now. Only known-bad linux64 slaves are currently offline.