983625 - Disk space on socorro3.db.phx1.mozilla.com

Status: RESOLVED FIXED

(Socorro :: Database, task)

Description

[Ryan Watson [:w0ts0n]]

Fri 05:26:14 PDT [1129] socorro3.db.phx1.mozilla.com:Disk - All is CRITICAL: DISK CRITICAL - free space: / 0 MB

Trying to get hold of a DBA to look.

Comment 1

[Matt Brandt [:mbrandt]]

Bumping priority and adding :sheeri to bug to help with diagnosis

Comment 2

[Selena Deckelmann :selenamarie :selena]

I think this alert cleared: 

[postgres@socorro3 wal_archive]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             273G   18G  241G   7% /
tmpfs                  40G     0   40G   0% /dev/shm
/dev/sda1             985M   65M  870M   7% /boot
/dev/sdb1             2.5T  1.8T  759G  70% /pgdata

Comment 3

[Sheeri Cabral [:sheeri]]

The alert did clear, we refreshed from production.

Comment 4

[Adrian J Fernandez [:Aj]]

Reoccurred;
Sun 17:51:56 PDT [1220] socorro3.db.phx1.mozilla.com:Disk - All is WARNING: DISK WARNING - free space: / 28801 MB (10% inode=99%)

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             273G  231G   29G  90% /

Comment 5

[Sheeri Cabral [:sheeri]]

I have a wrong path in the recovery.conf file:

archive_cleanup_command = '/usr/pgsql-9.2/bin/pg_archivecleanup /wal/9.2 %r'

but the datadir on socorr3 isn't in /wal - though it should be...

So I did this manually:
 /usr/pgsql-9.2/bin/pg_archivecleanup  /var/lib/pgsql/wal_archive/ 0000001D00001991000000E2

But I should either get puppet to put the wal files in /wal or change the line to:

archive_cleanup_command = '/usr/pgsql-9.2/bin/pg_archivecleanup var/lib/pgsql/wal_archive %r'

Comment 6

[Sheeri Cabral [:sheeri]]

Giving this to mpressman so he sees it - not sure which of the solutions in comment 5 we should use.

Comment 7

[Lonnen :lonnen]

mpressman - recommendation?

Comment 8

[Matt Pressman [:mpressman]]

Immediately we should point to /var/lib/pgsql/wal_archive for the conf file as that is where it currently resides. 

We also should do what we do on socorro2 - https://bugzilla.mozilla.org/show_bug.cgi?id=993104 - which is to make the /wal partition on socorro3 as was done on socorro2. Then we will have wal shipped to /wal/9.2.

Comment 9

[Sheeri Cabral [:sheeri]]

awesome! changing the title to be less heart-attack-inducing.

Comment 10

[Matt Pressman [:mpressman]]

socorro3.db.phx1 now has it's own partition on /wal

Comment 11

[Matt Pressman [:mpressman]]

restarted postgres service on socorro3 to verify and now socorro3.db.phx1 WAL is being written to a separate partition at /wal