Syslog data from puppet masters should be sent to data center syslog servers

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps: Puppet
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: michal, Assigned: dustin)

Tracking

Details

Attachments

(1 attachment, 1 obsolete attachment)

Hello,

The syslog from puppetmasters does not currently send us any data. It should be forwarding to:

syslog1.scl3.mozilla.com
syslog1.phx1.mozilla.com

depending on the datacenter.
Assignee: server-ops-releng → dustin
Group: infra
Component: Server Operations: RelEng → RelOps: Puppet
Product: mozilla.org → Infrastructure & Operations
QA Contact: arich → dustin
Severity: normal → enhancement
Assignee: dustin → relops
Severity: enhancement → normal
Assignee: relops → dustin
Michal, are those still the right hostnames?  They don't seem to exist.  I see syslog1.private.scl3.mozilla.com, but that doesn't allow me access on 6514/tcp:

dmitchell@releng-puppet2 ~ $ nc -vz syslog1.private.scl3.mozilla.com 6514
nc: connect to syslog1.private.scl3.mozilla.com port 6514 (tcp) failed: Connection timed out

same for the proxy:

dmitchell@releng-puppet2 ~ $ nc -vz syslog-proxy1.dmz.scl3.mozilla.com 6514
nc: connect to syslog-proxy1.dmz.scl3.mozilla.com port 6514 (tcp) failed: Connection timed out
Flags: needinfo?(mpurzynski)
It does exist and has a listening syslog instance. What are the source IP you are trying to connect from?
Flags: needinfo?(mpurzynski)
It = syslog1.private.scl3, then :)

I'm connecting from releng-puppet{1,2}.{build.{scl1,mtv1},srv.releng.{use1,usw2,scl3}}.mozilla.com.
Flags: needinfo?(mpurzynski)
Are you having any troubles connecting?
Flags: needinfo?(mpurzynski)
yes, see comment 1
Flags: needinfo?(mpurzynski)
SCL3 and PHX1

I can see the policy allowing connections to syslog1.private.{scl3,phx1}.mozilla.com on port 514. Can you try that? If the data is coming to us inside the IPSEC tunnel we can just ship cleartext on the application layer.
Flags: needinfo?(mpurzynski)
Created attachment 8348858 [details] [diff] [review]
bug887740.patch

Messages appeared.  UDP works, and anyway that's what we want so rsyslog doesn't queue up messages if syslog1 fails.

We don't have anything in phx1, so there's no need to have a conditional to switch between the syslog servers.
Attachment #8348858 - Flags: review?(jwatkins)
Comment on attachment 8348858 [details] [diff] [review]
bug887740.patch

I don't see ::rsyslog and I suspect we'd want the .conf to be absent if no syslog server specified.
Attachment #8348858 - Flags: feedback-
What does "don't see ::rsyslog" mean?  /\<::rsyslog\>/ appears exactly once in the patch, and /::rsyslog/ three times.

The rsyslog module manages the conf directory, so if no server is specified, it will be removed.
bah! totally missed that rsyslog was already in repo, I meant I hadn't seen it included in patch.

Consider my concerns revoked
Created attachment 8348877 [details] [diff] [review]
bug887740-p1.patch

Callek pointed out I was wrong - rsyslog.d wasn't purged.  Now it is.
Attachment #8348858 - Attachment is obsolete: true
Attachment #8348858 - Flags: review?(jwatkins)
Attachment #8348877 - Flags: review?(jwatkins)
Comment on attachment 8348877 [details] [diff] [review]
bug887740-p1.patch

Sorry, somehow I missed this review req.  lgtm r+
Attachment #8348877 - Flags: review?(jwatkins) → review+
Seems to be working!
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.