1175409 - Web2.releng.webapp stopped reporting to papertrail

Status: RESOLVED FIXED

(Infrastructure & Operations :: Infrastructure: Other, task)

Description

[Justin Wood (:Callek)]

According to papertrail (https://papertrailapp.com/groups/1421834) stopped reporting 8 days ago

And there is:

[[e-mail]3 ~]# uptime
 23:10:08 up 8 days,  7:10,  1 user,  load average: 0.13, 0.11, 0.09

While web1 has:

[[e-mail]3 ~]# uptime
 23:10:25 up 59 days, 12:01,  1 user,  load average: 1.19, 0.44, 0.20

Comment 1

[Amy Rich [:arr] [:arich]]

they all say "last seem moments ago"

Comment 2

[Justin Wood (:Callek)]

(In reply to Amy Rich [:arr] [:arich] from comment #1)
> they all say "last seem moments ago"

Only because web2 is no longer even on the list:

https://papertrailapp.com/groups/1421834/

[[e-mail]3 ~]# facter fqdn
web2.releng.webapp.scl3.mozilla.com

Comment 3

[Amy Rich [:arr] [:arich]]

digi, it looks like the syslog modifications you made to look at hiera have mysteriously stopped working on these two hosts (relengwebadm is fine). I just added a couple more machines today, and they're working fine, too. On web1 and web2, the config file doesn't exist, and running puppet doesn't restore it.

Comment 4

[Amy Rich [:arr] [:arich]]

I take that back, it's JUST web2.

Comment 5

[:Atoll]

Committing to svn+ssh://svn.mozilla.org/sysadmins/puppet/trunk ...
	R	hiera/secrets/nodes/web2.releng.webapp.scl3.mozilla.com => hiera/secrets/nodes/web2.releng.webapp.scl3.mozilla.com.yaml
Committed r105559

This should fix it within an hour.

Comment 6

[Amy Rich [:arr] [:arich]]

Huh, okay, even though my local copy of svn had the hiera file ending in .yaml, the remote version apparently did not. Thanks to atoll for catching that.
I forced the puppet run so it's fixed now.