Closed Bug 911193 Opened 11 years ago Closed 11 years ago

all socorro apps logging settings are not correct

Categories

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task)

x86_64
Linux
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: lars, Assigned: bburton)

Details

I noticed that the collectors stopped logging in production in April.  On investigation, I see that the "syslog_host" in the file /etc/socorro/common_logging.ini is not correct.  It looks like it was never set properly as it currently has "whitewater.local" listed (that's my old laptop's name).  This error is propagated all over the system on every collector, admin, and processors boxes.  

Socorro logging has two destinations: stderr and syslog.  We generally look at the redirected stderr output in /var/log/socorro.  However, the collector can't write to stderr because it runs under Apache.  To see collector logging, we need syslog set properly.

we need to verify and get the proper values for common_logging.ini into puppet:

syslog_host
syslog_facility_string
syslog_port

my guess for these values is as follows, but only IT knows for sure.

syslog_host=localhost
syslog_facility_string=local2
syslog_port=514
Assignee: nobody → server-ops-webops
Severity: normal → major
Component: Infra → WebOps: Socorro
Product: Socorro → Infrastructure & Operations
QA Contact: nmaul
Version: unspecified → other
Assignee: server-ops-webops → bburton
Severity: major → normal
Here are the current common_logging.ini settings from staging

# name: stderr_error_logging_level
# doc: logging level for the logging to stderr (10 - DEBUG, 20 - INFO, 30 - WARNING, 40 - ERROR, 50 - CRITICAL)
# converter: int
# OPS - may change.
stderr_error_logging_level='10'

# name: syslog_error_logging_level
# doc: logging level for the log file (10 - DEBUG, 20 - INFO, 30 - WARNING, 40 - ERROR, 50 - CRITICAL)
# converter: int
# OPS - may change
syslog_error_logging_level='40'

# name: stderr_line_format_string
# doc: python logging system format for logging to stderr
# converter: str
# OPS set once
stderr_line_format_string='{asctime} {levelname} - {threadName} - {message}'

# name: syslog_facility_string
# doc: syslog facility string ("user", "local0", etc)
# converter: str
# OPS set once
syslog_facility_string='local2'

# name: syslog_host
# doc: syslog hostname
# converter: str
# OPS set once
syslog_host='localhost'

# name: syslog_line_format_string
# doc: python logging system format for syslog entries
# converter: str
# OPS set once
syslog_line_format_string='crontabber (pid {process}): {asctime} {levelname} - {threadName} - {message}'

# name: syslog_port
# doc: syslog port
# converter: int
# OPS set once
syslog_port='514'
Status: NEW → ASSIGNED
I reviewed the SVN history of the production file and the current file was committed from https://bugzilla.mozilla.org/attachment.cgi?id=736387&action=edit in bug 860835 ( https://bugzilla.mozilla.org/show_bug.cgi?id=860835#c7 )
bburton@althalus [08:28:13] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn diff
Index: modules/socorro/files/prod/etc-socorro/common_logging.ini
===================================================================
--- modules/socorro/files/prod/etc-socorro/common_logging.ini	(revision 74200)
+++ modules/socorro/files/prod/etc-socorro/common_logging.ini	(working copy)
@@ -26,7 +26,7 @@
 # doc: syslog hostname
 # converter: str
 # OPS set once
-syslog_host='whitewater.local'
+syslog_host='localhost'

 # name: syslog_line_format_string
 # doc: python logging system format for syslog entries
bburton@althalus [08:28:15] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn ci -m "fixing socorro prod syslog to use localhost, bug 911193"
Sending        trunk/modules/socorro/files/prod/etc-socorro/common_logging.ini
Transmitting file data .
Committed revision 74201.
With that change, prod looks like

# name: stderr_error_logging_level
# doc: logging level for the logging to stderr (10 - DEBUG, 20 - INFO, 30 - WARNING, 40 - ERROR, 50 - CRITICAL)
# converter: int
# OPS - may change
stderr_error_logging_level='10'

# name: syslog_error_logging_level
# doc: logging level for the log file (10 - DEBUG, 20 - INFO, 30 - WARNING, 40 - ERROR, 50 - CRITICAL)
# converter: int
# OPS - may change
syslog_error_logging_level='40'

# name: stderr_line_format_string
# doc: python logging system format for logging to stderr
# converter: str
# OPS set once
stderr_line_format_string='{asctime} {levelname} - {threadName} - {message}'

# name: syslog_facility_string
# doc: syslog facility string ("user", "local0", etc)
# converter: str
# OPS set once
syslog_facility_string='local2'

# name: syslog_host
# doc: syslog hostname
# converter: str
# OPS set once
syslog_host='localhost'

# name: syslog_line_format_string
# doc: python logging system format for syslog entries
# converter: str
# OPS set once
syslog_line_format_string='{app_name} (pid {process}): {asctime} {levelname} - {threadName} - {message}'

# name: syslog_port
# doc: syslog port
# converter: int
# OPS set once
syslog_port='514'
This has been deployed everywhere

[root@sp-admin01.phx1 ~]# issue-multi-command all grep 'syslog_host=' /etc/socorro/common_logging.ini
[sp-processor01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor02.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor03.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor04.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor05.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor06.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor07.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor08.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor09.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor10.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-collector01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-collector02.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-collector03.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-collector04.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-collector05.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-collector06.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-mware01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-mware02.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-mware03.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-web01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-web02.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-web03.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-web04.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-web05.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-admin01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor02.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.407s)
[sp-processor02.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.416s)
[sp-processor01.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor06.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.585s)
[sp-processor06.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor08.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.590s)
[sp-processor08.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor04.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.601s)
[sp-processor04.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor05.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.609s)
[sp-processor05.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor03.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.612s)
[sp-processor03.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor09.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.610s)
[sp-processor09.phx1.mozilla.com] out: syslog_host='localhost'
[sp-collector03.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.673s)
[sp-collector03.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor10.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.150s)
[sp-processor10.phx1.mozilla.com] out: syslog_host='localhost'
[sp-mware03.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.148s)
[sp-mware03.phx1.mozilla.com] out: syslog_host='localhost'
[sp-admin01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.133s)
[sp-admin01.phx1.mozilla.com] out: syslog_host='localhost'
[sp-web02.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.151s)
[sp-web02.phx1.mozilla.com] out: syslog_host='localhost'
[sp-web01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.158s)
[sp-web01.phx1.mozilla.com] out: syslog_host='localhost'
[sp-collector02.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.208s)
[sp-collector02.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor07.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.224s)
[sp-processor07.phx1.mozilla.com] out: syslog_host='localhost'
[sp-web05.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.194s)
[sp-web05.phx1.mozilla.com] out: syslog_host='localhost'
[sp-collector04.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.220s)
[sp-collector04.phx1.mozilla.com] out: syslog_host='localhost'
[sp-collector05.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.236s)
[sp-collector05.phx1.mozilla.com] out: syslog_host='localhost'
[sp-collector06.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.237s)
[sp-collector06.phx1.mozilla.com] out: syslog_host='localhost'
[sp-collector01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.263s)
[sp-collector01.phx1.mozilla.com] out: syslog_host='localhost'
[sp-web03.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.673s)
[sp-web03.phx1.mozilla.com] out: syslog_host='localhost'
[sp-web04.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.673s)
[sp-web04.phx1.mozilla.com] out: syslog_host='localhost'
[sp-mware01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (2.210s)
[sp-mware01.phx1.mozilla.com] out: syslog_host='localhost'
[sp-mware02.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (2.210s)
[sp-mware02.phx1.mozilla.com] out: syslog_host='localhost'
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.