604272 - talos-r3-fed-013, talos-r3-fed-036, talos-r3-fed-039 need setting up

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Looks like it was reimaged, probably as part of a reboot bug because I can't find any note of it in bugmail. Has the wrong host name, no buildbot.tac, will need puppet key fixing etc etc

Comment 1

[Chris Cooper [:coop] (he/him)]

These are all reconnected to puppet:

013 -> scl
036 & 039 -> mv 

All are now also reporting to buildbot masters:

013 -> buildbot-master1:8011
036 & 039 -> test-master02:8012

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

talos-r3-fed-013 is hitting errors in puppet:
Nov 10 19:41:22 talos-r3-fed-013 puppetd[1953]: Starting catalog run
Nov 10 19:41:23 talos-r3-fed-013 puppetd[1953]: (//Node[talos-r3-fed-013.build.mozilla.org]/talosslave/talos_fedora/File[/home/cltbld/.bash_profile]) Failed to retrieve current state of resource: Cannot access mount[production] Could not describe /production/fedora12-i686/test/home/cltbld/.bash_profile: Cannot access mount[production] at /etc/puppet/manifests/os/talos_fedora.pp:46
Nov 10 19:41:23 talos-r3-fed-013 puppetd[1953]: (//Node[talos-r3-fed-013.build.mozilla.org]/talosslave/talos_fedora/File[/home/cltbld/.ssh/authorized_keys]) Failed to retrieve current state of resource: Cannot access mount[production] Could not describe /production/fedora12-i686/test/home/cltbld/.ssh/authorized_keys: Cannot access mount[production] at /etc/puppet/manifests/os/talos_fedora.pp:46
Nov 10 19:41:23 talos-r3-fed-013 puppetd[1953]: (//Node[talos-r3-fed-013.build.mozilla.org]/talosslave/talos_fedora/File[/home/cltbld/run-puppet-and-buildbot.sh]) Failed to retrieve current state of resource: Cannot access mount[production] Could not describe /production/fedora12-i686/test/home/cltbld/run-puppet-and-buildbot.sh: Cannot access mount[production] at /etc/puppet/manifests/os/talos_fedora.pp:46
Nov 10 19:41:23 talos-r3-fed-013 puppetd[1953]: (//Node[talos-r3-fed-013.build.mozilla.org]/talosslave/talos_fedora/File[/home/cltbld/.fonts.conf]) Failed to retrieve current state of resource: Cannot access mount[production] Could not describe /production/fedora12-i686/test/etc/fonts.conf: Cannot access mount[production] at /etc/puppet/manifests/os/talos_fedora.pp:46
Nov 10 19:41:23 talos-r3-fed-013 puppetd[1953]: (//Node[test]/puppet-config/File[/home/cltbld/.config/autostart/gnome-terminal.desktop]) Failed to retrieve current state of resource: Cannot access mount[production] Could not describe /production/fedora12-i686/test/local/home/cltbld/.config/autostart/gnome-terminal.desktop: Cannot access mount[production] at /etc/puppet/manifests/packages/puppet-config.pp:30
Nov 10 19:41:23 talos-r3-fed-013 puppetd[1953]: (//Node[test]/puppet-config/Exec[reset-ssl]) Dependency file[/home/cltbld/.config/autostart/gnome-terminal.desktop] has 1 failures
Nov 10 19:41:23 talos-r3-fed-013 puppetd[1953]: (//Node[test]/puppet-config/Exec[reset-ssl]) Skipping because of failed dependencies
Nov 10 19:41:23 talos-r3-fed-013 puppetd[1953]: (//Node[test]/puppet-config/Exec[restart]) Dependency file[/home/cltbld/.config/autostart/gnome-terminal.desktop] has 1 failures
Nov 10 19:41:23 talos-r3-fed-013 puppetd[1953]: (//Node[test]/puppet-config/Exec[restart]) Skipping because of failed dependencies
Nov 10 19:41:25 talos-r3-fed-013 ntpdate[2101]: step time server 72.29.161.5 offset -0.001805 sec
Nov 10 19:41:25 talos-r3-fed-013 puppetd[1953]: (//Node[talos-r3-fed-013.build.mozilla.org]/talosslave/talos_fedora/Service[ntpdate]/ensure) ensure changed 'stopped' to 'running'
Nov 10 19:41:26 talos-r3-fed-013 puppetd[1953]: Finished catalog run in 4.01 seconds

My attempt to kill puppet off, remove the contents of /var/lib/puppet, puppetca --clean it and reboot didn't help.

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to comment #2)
> My attempt to kill puppet off, remove the contents of /var/lib/puppet, puppetca
> --clean it and reboot didn't help.

I can't even get to it now. I've marked it for reboot, and will progress to re-imaging if that doesn't help.

Comment 4

[Chris Cooper [:coop] (he/him)]

talos-r3-fed-013 came back from reboot, is now successfully syncing with puppet and is running jobs on buildbot-master1:8011