all socorro apps logging settings are not correct

RESOLVED FIXED

Status

Infrastructure & Operations Graveyard
WebOps: Socorro
RESOLVED FIXED
5 years ago
2 years ago

People

(Reporter: lars, Assigned: solarce)

Tracking

Details

(Reporter)

Description

5 years ago
I noticed that the collectors stopped logging in production in April.  On investigation, I see that the "syslog_host" in the file /etc/socorro/common_logging.ini is not correct.  It looks like it was never set properly as it currently has "whitewater.local" listed (that's my old laptop's name).  This error is propagated all over the system on every collector, admin, and processors boxes.  

Socorro logging has two destinations: stderr and syslog.  We generally look at the redirected stderr output in /var/log/socorro.  However, the collector can't write to stderr because it runs under Apache.  To see collector logging, we need syslog set properly.

we need to verify and get the proper values for common_logging.ini into puppet:

syslog_host
syslog_facility_string
syslog_port

my guess for these values is as follows, but only IT knows for sure.

syslog_host=localhost
syslog_facility_string=local2
syslog_port=514

Updated

5 years ago
Assignee: nobody → server-ops-webops
Severity: normal → major
Component: Infra → WebOps: Socorro
Product: Socorro → Infrastructure & Operations
QA Contact: nmaul
Version: unspecified → other
(Assignee)

Updated

5 years ago
Assignee: server-ops-webops → bburton
Severity: major → normal
(Assignee)

Comment 1

5 years ago
Here are the current common_logging.ini settings from staging

# name: stderr_error_logging_level
# doc: logging level for the logging to stderr (10 - DEBUG, 20 - INFO, 30 - WARNING, 40 - ERROR, 50 - CRITICAL)
# converter: int
# OPS - may change.
stderr_error_logging_level='10'

# name: syslog_error_logging_level
# doc: logging level for the log file (10 - DEBUG, 20 - INFO, 30 - WARNING, 40 - ERROR, 50 - CRITICAL)
# converter: int
# OPS - may change
syslog_error_logging_level='40'

# name: stderr_line_format_string
# doc: python logging system format for logging to stderr
# converter: str
# OPS set once
stderr_line_format_string='{asctime} {levelname} - {threadName} - {message}'

# name: syslog_facility_string
# doc: syslog facility string ("user", "local0", etc)
# converter: str
# OPS set once
syslog_facility_string='local2'

# name: syslog_host
# doc: syslog hostname
# converter: str
# OPS set once
syslog_host='localhost'

# name: syslog_line_format_string
# doc: python logging system format for syslog entries
# converter: str
# OPS set once
syslog_line_format_string='crontabber (pid {process}): {asctime} {levelname} - {threadName} - {message}'

# name: syslog_port
# doc: syslog port
# converter: int
# OPS set once
syslog_port='514'
Status: NEW → ASSIGNED
(Assignee)

Comment 2

5 years ago
I reviewed the SVN history of the production file and the current file was committed from https://bugzilla.mozilla.org/attachment.cgi?id=736387&action=edit in bug 860835 ( https://bugzilla.mozilla.org/show_bug.cgi?id=860835#c7 )
(Assignee)

Comment 3

5 years ago
bburton@althalus [08:28:13] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn diff
Index: modules/socorro/files/prod/etc-socorro/common_logging.ini
===================================================================
--- modules/socorro/files/prod/etc-socorro/common_logging.ini	(revision 74200)
+++ modules/socorro/files/prod/etc-socorro/common_logging.ini	(working copy)
@@ -26,7 +26,7 @@
 # doc: syslog hostname
 # converter: str
 # OPS set once
-syslog_host='whitewater.local'
+syslog_host='localhost'

 # name: syslog_line_format_string
 # doc: python logging system format for syslog entries
bburton@althalus [08:28:15] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn ci -m "fixing socorro prod syslog to use localhost, bug 911193"
Sending        trunk/modules/socorro/files/prod/etc-socorro/common_logging.ini
Transmitting file data .
Committed revision 74201.
(Assignee)

Comment 4

5 years ago
With that change, prod looks like

# name: stderr_error_logging_level
# doc: logging level for the logging to stderr (10 - DEBUG, 20 - INFO, 30 - WARNING, 40 - ERROR, 50 - CRITICAL)
# converter: int
# OPS - may change
stderr_error_logging_level='10'

# name: syslog_error_logging_level
# doc: logging level for the log file (10 - DEBUG, 20 - INFO, 30 - WARNING, 40 - ERROR, 50 - CRITICAL)
# converter: int
# OPS - may change
syslog_error_logging_level='40'

# name: stderr_line_format_string
# doc: python logging system format for logging to stderr
# converter: str
# OPS set once
stderr_line_format_string='{asctime} {levelname} - {threadName} - {message}'

# name: syslog_facility_string
# doc: syslog facility string ("user", "local0", etc)
# converter: str
# OPS set once
syslog_facility_string='local2'

# name: syslog_host
# doc: syslog hostname
# converter: str
# OPS set once
syslog_host='localhost'

# name: syslog_line_format_string
# doc: python logging system format for syslog entries
# converter: str
# OPS set once
syslog_line_format_string='{app_name} (pid {process}): {asctime} {levelname} - {threadName} - {message}'

# name: syslog_port
# doc: syslog port
# converter: int
# OPS set once
syslog_port='514'
(Assignee)

Comment 5

5 years ago
This has been deployed everywhere

[root@sp-admin01.phx1 ~]# issue-multi-command all grep 'syslog_host=' /etc/socorro/common_logging.ini
[sp-processor01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor02.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor03.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor04.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor05.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor06.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor07.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor08.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor09.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor10.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-collector01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-collector02.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-collector03.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-collector04.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-collector05.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-collector06.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-mware01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-mware02.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-mware03.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-web01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-web02.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-web03.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-web04.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-web05.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-admin01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini
[sp-processor02.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.407s)
[sp-processor02.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.416s)
[sp-processor01.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor06.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.585s)
[sp-processor06.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor08.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.590s)
[sp-processor08.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor04.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.601s)
[sp-processor04.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor05.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.609s)
[sp-processor05.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor03.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.612s)
[sp-processor03.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor09.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.610s)
[sp-processor09.phx1.mozilla.com] out: syslog_host='localhost'
[sp-collector03.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.673s)
[sp-collector03.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor10.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.150s)
[sp-processor10.phx1.mozilla.com] out: syslog_host='localhost'
[sp-mware03.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.148s)
[sp-mware03.phx1.mozilla.com] out: syslog_host='localhost'
[sp-admin01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.133s)
[sp-admin01.phx1.mozilla.com] out: syslog_host='localhost'
[sp-web02.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.151s)
[sp-web02.phx1.mozilla.com] out: syslog_host='localhost'
[sp-web01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.158s)
[sp-web01.phx1.mozilla.com] out: syslog_host='localhost'
[sp-collector02.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.208s)
[sp-collector02.phx1.mozilla.com] out: syslog_host='localhost'
[sp-processor07.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.224s)
[sp-processor07.phx1.mozilla.com] out: syslog_host='localhost'
[sp-web05.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.194s)
[sp-web05.phx1.mozilla.com] out: syslog_host='localhost'
[sp-collector04.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.220s)
[sp-collector04.phx1.mozilla.com] out: syslog_host='localhost'
[sp-collector05.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.236s)
[sp-collector05.phx1.mozilla.com] out: syslog_host='localhost'
[sp-collector06.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.237s)
[sp-collector06.phx1.mozilla.com] out: syslog_host='localhost'
[sp-collector01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.263s)
[sp-collector01.phx1.mozilla.com] out: syslog_host='localhost'
[sp-web03.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.673s)
[sp-web03.phx1.mozilla.com] out: syslog_host='localhost'
[sp-web04.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.673s)
[sp-web04.phx1.mozilla.com] out: syslog_host='localhost'
[sp-mware01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (2.210s)
[sp-mware01.phx1.mozilla.com] out: syslog_host='localhost'
[sp-mware02.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (2.210s)
[sp-mware02.phx1.mozilla.com] out: syslog_host='localhost'
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.