822360 - Make the logging directory the same for PostgreSQL on prod/stage/dev for Socorro

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Selena Deckelmann :selenamarie :selena]

Please put all the logs in /var/log/postgresql/ instead of /pgdata/9.2/data/pg_log

For all prod/stage/dev.

Thanks!

Comment 1

[Matt Pressman [:mpressman]]

stage and dev are already using pg_log relative to their data directory - I've modified the templates in puppet for master01 and master02 to use pg_log relative to their data directory

Comment 2

[Sheeri Cabral [:sheeri]]

double-plus good - make it managed by puppet.

Comment 3

[Sheeri Cabral [:sheeri]]

/me reads comment 1. Nevermind!!!

Comment 4

[Selena Deckelmann :selenamarie :selena]

We really should put logs in /var/log/postgresql/$VERSION for dev, stage and prod

The reasons for this are: 

* We need logs to be preserved even when replica instances are refreshed - the procedure for this currently removes everything including the logs during refresh.
* Most other supported services log to /var/log, making things easier on anyone troubleshooting

Is there an ops reason I'm missing to have them in /pgdata?

Comment 5

[Laura Thomson :laura]

+1 to comment #4.  We really need them to be out of the pgdata directory.

syslog would be nice too since it's the standard for Socorro apps.

Comment 6

[Sheeri Cabral [:sheeri]]

Let's standardize on /var/log/postgresql/CLUSTER/VERSION

Comment 7

[Matt Pressman [:mpressman]]

removed pg_log and added new location for the instances on dev and for stage.

Comment 8

[Selena Deckelmann :selenamarie :selena]

(In reply to Matt Pressman [:mpressman] from comment #7)
> removed pg_log and added new location for the instances on dev and for stage.

Thanks, Matt!

Could you attach the changes that were committed to the puppet repo?

Comment 9

[Selena Deckelmann :selenamarie :selena]

(In reply to Selena Deckelmann :selenamarie :selena from comment #8)
> (In reply to Matt Pressman [:mpressman] from comment #7)
> > removed pg_log and added new location for the instances on dev and for stage.
> 
> Thanks, Matt!
> 
> Could you attach the changes that were committed to the puppet repo?

Ping about this!

also: 

Current Postgres logging has the following problems: 

* We only log 1 week's worth of data
* Postgres log rotation truncates previous log and overwrites it
* We don't use logrotate
* Filenames do not reflect the actual date that logs were produced

Would like for all of this to change.

Should be changed in dev, stage and production.

Comment 11

[Selena Deckelmann :selenamarie :selena]

Just ran into this again this morning while trying to diagnose a deadlock problem. Things were being logged into /var/log/postgresql/9.0 and logs were only available for the last week. :(

Taking this to update puppet postgres2 module configs.

Comment 12

[Selena Deckelmann :selenamarie :selena]

Attachment: Diff of change for logfiles

Comment 13

[Sheeri Cabral [:sheeri]]

selena - I think you have commit access now, right? (not sure if this got into the puppet module yet and the diff is for reference, or if you want someone to update the puppet module).

Comment 14

[Matt Pressman [:mpressman]]

The changes have been made or verified to puppet in both the postgres2 and the new postgres module. On all the modules in puppet it is listed as:

log_directory = /var/log/postgres/<%=postgres_version%>
log_truncate_on_rotation = off
log_filename = 'activitylog-%Y%m%d.log'

The only difference is on dev where the instance name is listed in the log_directory naming:

log_directory = '/var/log/postgresql/<%=name%>/<%=postgres_version%>'

socorro1.dev.db.phx1:crash-stats-dev has all log configs set up and running
socorro1.dev.db.phx1:devdb has all log configs set up and running
socorro1.dev.db.phx1:pgslave has all log configs set up and running
socorro1.stage.db.phx1 has all log configs set up and running

tp-socorro01-master0[1-2] does not have log_truncate_on_rotation set to off.

Comment 15

[Matt Pressman [:mpressman]]

tp-socorro01-master01 and tp-socorro01-master02 now have log_truncate_on_rotation=off

prod/stage/dev for socorro logging params are the same in the configs

Additionally, the nagios check check_postgres_settings_checksum has been changed in puppet to account for the config changes.