887740 - Syslog data from puppet masters should be sent to data center syslog servers

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Michal Purzynski [:michal`] (use NEEDINFO)]

Hello,

The syslog from puppetmasters does not currently send us any data. It should be forwarding to:

syslog1.scl3.mozilla.com
syslog1.phx1.mozilla.com

depending on the datacenter.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Michal, are those still the right hostnames?  They don't seem to exist.  I see syslog1.private.scl3.mozilla.com, but that doesn't allow me access on 6514/tcp:

dmitchell@releng-puppet2 ~ $ nc -vz syslog1.private.scl3.mozilla.com 6514
nc: connect to syslog1.private.scl3.mozilla.com port 6514 (tcp) failed: Connection timed out

same for the proxy:

dmitchell@releng-puppet2 ~ $ nc -vz syslog-proxy1.dmz.scl3.mozilla.com 6514
nc: connect to syslog-proxy1.dmz.scl3.mozilla.com port 6514 (tcp) failed: Connection timed out

Comment 2

[Michal Purzynski [:michal`] (use NEEDINFO)]

It does exist and has a listening syslog instance. What are the source IP you are trying to connect from?

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

It = syslog1.private.scl3, then :)

I'm connecting from releng-puppet{1,2}.{build.{scl1,mtv1},srv.releng.{use1,usw2,scl3}}.mozilla.com.

Comment 4

[Michal Purzynski [:michal`] (use NEEDINFO)]

Are you having any troubles connecting?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

yes, see comment 1

Comment 6

[Michal Purzynski [:michal`] (use NEEDINFO)]

SCL3 and PHX1

I can see the policy allowing connections to syslog1.private.{scl3,phx1}.mozilla.com on port 514. Can you try that? If the data is coming to us inside the IPSEC tunnel we can just ship cleartext on the application layer.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug887740.patch

Messages appeared.  UDP works, and anyway that's what we want so rsyslog doesn't queue up messages if syslog1 fails.

We don't have anything in phx1, so there's no need to have a conditional to switch between the syslog servers.

Comment 8

[Justin Wood (:Callek)]

Comment on attachment 8348858 [details] [diff] [review]
bug887740.patch

I don't see ::rsyslog and I suspect we'd want the .conf to be absent if no syslog server specified.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

What does "don't see ::rsyslog" mean?  /\<::rsyslog\>/ appears exactly once in the patch, and /::rsyslog/ three times.

The rsyslog module manages the conf directory, so if no server is specified, it will be removed.

Comment 10

[Justin Wood (:Callek)]

bah! totally missed that rsyslog was already in repo, I meant I hadn't seen it included in patch.

Consider my concerns revoked

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug887740-p1.patch

Callek pointed out I was wrong - rsyslog.d wasn't purged.  Now it is.

Comment 12

[Jake Watkins [:dividehex]]

Comment on attachment 8348877 [details] [diff] [review]
bug887740-p1.patch

Sorry, somehow I missed this review req.  lgtm r+

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Seems to be working!