Closed
Bug 911193
Opened 11 years ago
Closed 11 years ago
all socorro apps logging settings are not correct
Categories
(Infrastructure & Operations Graveyard :: WebOps: Socorro, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: lars, Assigned: bburton)
Details
I noticed that the collectors stopped logging in production in April. On investigation, I see that the "syslog_host" in the file /etc/socorro/common_logging.ini is not correct. It looks like it was never set properly as it currently has "whitewater.local" listed (that's my old laptop's name). This error is propagated all over the system on every collector, admin, and processors boxes. Socorro logging has two destinations: stderr and syslog. We generally look at the redirected stderr output in /var/log/socorro. However, the collector can't write to stderr because it runs under Apache. To see collector logging, we need syslog set properly. we need to verify and get the proper values for common_logging.ini into puppet: syslog_host syslog_facility_string syslog_port my guess for these values is as follows, but only IT knows for sure. syslog_host=localhost syslog_facility_string=local2 syslog_port=514
Updated•11 years ago
|
Assignee: nobody → server-ops-webops
Severity: normal → major
Component: Infra → WebOps: Socorro
Product: Socorro → Infrastructure & Operations
QA Contact: nmaul
Version: unspecified → other
Assignee | ||
Updated•11 years ago
|
Assignee: server-ops-webops → bburton
Severity: major → normal
Assignee | ||
Comment 1•11 years ago
|
||
Here are the current common_logging.ini settings from staging # name: stderr_error_logging_level # doc: logging level for the logging to stderr (10 - DEBUG, 20 - INFO, 30 - WARNING, 40 - ERROR, 50 - CRITICAL) # converter: int # OPS - may change. stderr_error_logging_level='10' # name: syslog_error_logging_level # doc: logging level for the log file (10 - DEBUG, 20 - INFO, 30 - WARNING, 40 - ERROR, 50 - CRITICAL) # converter: int # OPS - may change syslog_error_logging_level='40' # name: stderr_line_format_string # doc: python logging system format for logging to stderr # converter: str # OPS set once stderr_line_format_string='{asctime} {levelname} - {threadName} - {message}' # name: syslog_facility_string # doc: syslog facility string ("user", "local0", etc) # converter: str # OPS set once syslog_facility_string='local2' # name: syslog_host # doc: syslog hostname # converter: str # OPS set once syslog_host='localhost' # name: syslog_line_format_string # doc: python logging system format for syslog entries # converter: str # OPS set once syslog_line_format_string='crontabber (pid {process}): {asctime} {levelname} - {threadName} - {message}' # name: syslog_port # doc: syslog port # converter: int # OPS set once syslog_port='514'
Status: NEW → ASSIGNED
Assignee | ||
Comment 2•11 years ago
|
||
I reviewed the SVN history of the production file and the current file was committed from https://bugzilla.mozilla.org/attachment.cgi?id=736387&action=edit in bug 860835 ( https://bugzilla.mozilla.org/show_bug.cgi?id=860835#c7 )
Assignee | ||
Comment 3•11 years ago
|
||
bburton@althalus [08:28:13] [~/code/mozilla/sysadmins/puppet/trunk] -> % svn diff Index: modules/socorro/files/prod/etc-socorro/common_logging.ini =================================================================== --- modules/socorro/files/prod/etc-socorro/common_logging.ini (revision 74200) +++ modules/socorro/files/prod/etc-socorro/common_logging.ini (working copy) @@ -26,7 +26,7 @@ # doc: syslog hostname # converter: str # OPS set once -syslog_host='whitewater.local' +syslog_host='localhost' # name: syslog_line_format_string # doc: python logging system format for syslog entries bburton@althalus [08:28:15] [~/code/mozilla/sysadmins/puppet/trunk] -> % svn ci -m "fixing socorro prod syslog to use localhost, bug 911193" Sending trunk/modules/socorro/files/prod/etc-socorro/common_logging.ini Transmitting file data . Committed revision 74201.
Assignee | ||
Comment 4•11 years ago
|
||
With that change, prod looks like # name: stderr_error_logging_level # doc: logging level for the logging to stderr (10 - DEBUG, 20 - INFO, 30 - WARNING, 40 - ERROR, 50 - CRITICAL) # converter: int # OPS - may change stderr_error_logging_level='10' # name: syslog_error_logging_level # doc: logging level for the log file (10 - DEBUG, 20 - INFO, 30 - WARNING, 40 - ERROR, 50 - CRITICAL) # converter: int # OPS - may change syslog_error_logging_level='40' # name: stderr_line_format_string # doc: python logging system format for logging to stderr # converter: str # OPS set once stderr_line_format_string='{asctime} {levelname} - {threadName} - {message}' # name: syslog_facility_string # doc: syslog facility string ("user", "local0", etc) # converter: str # OPS set once syslog_facility_string='local2' # name: syslog_host # doc: syslog hostname # converter: str # OPS set once syslog_host='localhost' # name: syslog_line_format_string # doc: python logging system format for syslog entries # converter: str # OPS set once syslog_line_format_string='{app_name} (pid {process}): {asctime} {levelname} - {threadName} - {message}' # name: syslog_port # doc: syslog port # converter: int # OPS set once syslog_port='514'
Assignee | ||
Comment 5•11 years ago
|
||
This has been deployed everywhere [root@sp-admin01.phx1 ~]# issue-multi-command all grep 'syslog_host=' /etc/socorro/common_logging.ini [sp-processor01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-processor02.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-processor03.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-processor04.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-processor05.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-processor06.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-processor07.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-processor08.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-processor09.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-processor10.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-collector01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-collector02.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-collector03.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-collector04.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-collector05.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-collector06.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-mware01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-mware02.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-mware03.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-web01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-web02.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-web03.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-web04.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-web05.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-admin01.phx1.mozilla.com] running: grep syslog_host= /etc/socorro/common_logging.ini [sp-processor02.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.407s) [sp-processor02.phx1.mozilla.com] out: syslog_host='localhost' [sp-processor01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.416s) [sp-processor01.phx1.mozilla.com] out: syslog_host='localhost' [sp-processor06.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.585s) [sp-processor06.phx1.mozilla.com] out: syslog_host='localhost' [sp-processor08.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.590s) [sp-processor08.phx1.mozilla.com] out: syslog_host='localhost' [sp-processor04.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.601s) [sp-processor04.phx1.mozilla.com] out: syslog_host='localhost' [sp-processor05.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.609s) [sp-processor05.phx1.mozilla.com] out: syslog_host='localhost' [sp-processor03.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.612s) [sp-processor03.phx1.mozilla.com] out: syslog_host='localhost' [sp-processor09.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.610s) [sp-processor09.phx1.mozilla.com] out: syslog_host='localhost' [sp-collector03.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (0.673s) [sp-collector03.phx1.mozilla.com] out: syslog_host='localhost' [sp-processor10.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.150s) [sp-processor10.phx1.mozilla.com] out: syslog_host='localhost' [sp-mware03.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.148s) [sp-mware03.phx1.mozilla.com] out: syslog_host='localhost' [sp-admin01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.133s) [sp-admin01.phx1.mozilla.com] out: syslog_host='localhost' [sp-web02.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.151s) [sp-web02.phx1.mozilla.com] out: syslog_host='localhost' [sp-web01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.158s) [sp-web01.phx1.mozilla.com] out: syslog_host='localhost' [sp-collector02.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.208s) [sp-collector02.phx1.mozilla.com] out: syslog_host='localhost' [sp-processor07.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.224s) [sp-processor07.phx1.mozilla.com] out: syslog_host='localhost' [sp-web05.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.194s) [sp-web05.phx1.mozilla.com] out: syslog_host='localhost' [sp-collector04.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.220s) [sp-collector04.phx1.mozilla.com] out: syslog_host='localhost' [sp-collector05.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.236s) [sp-collector05.phx1.mozilla.com] out: syslog_host='localhost' [sp-collector06.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.237s) [sp-collector06.phx1.mozilla.com] out: syslog_host='localhost' [sp-collector01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.263s) [sp-collector01.phx1.mozilla.com] out: syslog_host='localhost' [sp-web03.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.673s) [sp-web03.phx1.mozilla.com] out: syslog_host='localhost' [sp-web04.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (1.673s) [sp-web04.phx1.mozilla.com] out: syslog_host='localhost' [sp-mware01.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (2.210s) [sp-mware01.phx1.mozilla.com] out: syslog_host='localhost' [sp-mware02.phx1.mozilla.com] finished: grep syslog_host= /etc/socorro/common_logging.ini (2.210s) [sp-mware02.phx1.mozilla.com] out: syslog_host='localhost'
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Updated•8 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•