750044 - (talos-r3-fed64-030) talos-r3-fed64-030 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Needs a reboot.

Comment 1

[Jake Watkins [:dividehex]]

grey screen | rebooted

Comment 2

[Jake Watkins [:dividehex]]

grey screen | rebooted

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Please reboot.

Comment 4

[Jake Watkins [:dividehex]]

grey screen | rebooted

Comment 5

[Rail Aliiev [:rail]]

Up and running.

Comment 6

[Armen [:armenzg]]

[56] talos-r3-fed64-030.build.scl1:PING is CRITICAL: PING CRITICAL - Packet loss = 100%

Comment 7

[[e-mail] (:bhearsum)]

Back in production.

Comment 9

[Kim Moir [:kmoir] ET]

Machine seems to be unavailable on the network today.

Comment 10

[Matt Brubeck (:mbrubeck)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14238884&tree=Mozilla-Aurora

Looks like maybe the system clock is set to the wrong date.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Disabled in slavealloc while I take a quick look.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

I fixed the date up with ntpdate (it thought it was 2001), but there's may be a puppet issue to follow up on.

Comment 13

[Chris AtLee [:catlee]]

rebooted it and it re-puppetized correctly.

Comment 14

[Armen [:armenzg]]

Reboot needed.

Comment 15

[Justin Wood (:Callek)]

[18:12:13]	nagios-releng	Wed 15:12:36 PST [437] talos-r3-fed64-030.build.scl1.mozilla.com is DOWN :PING CRITICAL - Packet loss = 100%

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Needs a reboot.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Back in action.

Comment 18

[Phil Ringnalda (:philor)]

Just like comment 10, the clock is 13 years behind.

Comment 19

[Justin Wood (:Callek)]

disabled in slavealloc

Comment 20

[Armen [:armenzg]]

It seems that we still have clock issues and puppet issues.
I recreated the certs and cleaned around but I get this error.

[cltbld@talos-r3-fed64-030 ~]$ date
Wed Jan  3 18:14:25 PST 2001
[cltbld@talos-r3-fed64-030 ~]$ su -
Password: 
[root@talos-r3-fed64-030 ~]# puppetd --test --server scl-production-puppet.build.scl1.mozilla.com
notice: Lock file /var/lib/puppet/state/puppetdlock exists; skipping catalog run
[root@talos-r3-fed64-030 ~]# ls -l /var/lib/puppet/state/puppetdlock 
-rw-r--r-- 1 root root 0 2012-12-17 10:27 /var/lib/puppet/state/puppetdlock
[root@talos-r3-fed64-030 ~]# vi /var/lib/puppet/state/puppetdlock 
[root@talos-r3-fed64-030 ~]# rm /var/lib/puppet/state/puppetdlock 
rm: remove regular empty file `/var/lib/puppet/state/puppetdlock'? y
[root@talos-r3-fed64-030 ~]# puppetd --test --server scl-production-puppet.build.scl1.mozilla.com
warning: Certificate validation failed; consider using the certname configuration option
err: Could not retrieve catalog: Certificates were not trusted: certificate verify failed
warning: Not using cache on failed catalog
[root@talos-r3-fed64-030 ~]#  puppetd --test --server scl-production-puppet.build.scl1.mozilla.com
warning: Certificate validation failed; consider using the certname configuration option
err: Could not retrieve catalog: Certificates were not trusted: certificate verify failed
warning: Not using cache on failed catalog
You have new mail in /var/spool/mail/root
[root@talos-r3-fed64-030 ~]#  rm -rf /var/lib/puppet/ssl/certs/*
[root@talos-r3-fed64-030 ~]# puppetd --test --server scl-production-puppet.build.scl1.mozilla.com
warning: peer certificate won't be verified in this SSL session
notice: Did not receive certificate
notice: Set to run 'one time'; exiting with no certificate
[root@talos-r3-fed64-030 ~]# puppetd --test --server scl-production-puppet.build.scl1.mozilla.com
warning: peer certificate won't be verified in this SSL session
notice: Got signed certificate
warning: Certificate validation failed; consider using the certname configuration option
err: Could not retrieve catalog: Certificates were not trusted: certificate verify failed
warning: Not using cache on failed catalog

Comment 21

[[e-mail] (:bhearsum)]

Back in production.

Comment 22

[[e-mail] (:bhearsum)]

And I should note that I found no clock issues on this machine even after multiple reboots.

Comment 23

[[e-mail] (:bhearsum)]

needs a reboot

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Back in production.

Comment 25

[Carsten Book [:Tomcat]]

01:11 < nagios-releng> Tue 01:11:28 PST [494] talos-r3-fed64-030.build.scl1.mozilla.com is DOWN :PING CRITICAL - Packet loss = 100%