Closed Bug 1030349 Opened 10 years ago Closed 10 years ago

socorro[1,2,3].db.phx1 disk are at 90% again

Categories

(Data & BI Services Team :: DB: MySQL, task)

Product:
Data & BI Services Team
Component:
DB: MySQL
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mpressman, Assigned: mpressman)

Details

[14:47:00] <nagios-phx1> Wed 13:47:00 PDT [1492] socorro2.db.phx1.mozilla.com:Disk 10 5 /pgdata is WARNING: DISK WARNING - free space: /pgdata 282610 MB (10% inode=99%): (http://m.mozilla.org/Disk+10+5+/pgdata)

socorro1.db.phx1 and socorro3.db.phx1 are at 89%

data needs to be truncated
A cron job is in the works to automatically truncate processed json, but until that is ready to go it is required to run it manually.

https://bugzilla.mozilla.org/show_bug.cgi?id=1008452#c27 show what needs to be truncated

breakpad=# select relname, pg_size_pretty(pg_table_size(relname::text)) from pg_class where relname ~ 'processed_crashes_201404' and relkind = 'r';
          relname           | pg_size_pretty
----------------------------+----------------
 processed_crashes_20140428 | 101 GB
 processed_crashes_20140421 | 98 GB
 processed_crashes_20140407 | 99 GB
 processed_crashes_20140414 | 96 GB
(4 rows)

breakpad=# select relname, pg_size_pretty(pg_table_size(relname::text)) from pg_class where relname ~ 'processed_crashes_201405' and relkind = 'r';
          relname           | pg_size_pretty
----------------------------+----------------
 processed_crashes_20140512 | 117 GB
 processed_crashes_20140526 | 120 GB
 processed_crashes_20140505 | 110 GB
 processed_crashes_20140519 | 117 GB
(4 rows)

This will leave processed_crashes_201406 still available
breakpad=# select relname, pg_size_pretty(pg_table_size(relname::text)) from pg_class where relname ~ 'processed_crashes_201406' and relkind = 'r';
          relname           | pg_size_pretty
----------------------------+----------------
 processed_crashes_20140609 | 123 GB
 processed_crashes_20140623 | 58 GB
 processed_crashes_20140602 | 123 GB
 processed_crashes_20140616 | 130 GB
 processed_crashes_20140630 | 8192 bytes
(5 rows)
Assignee: server-ops-database → mpressman
breakpad=# select 'TRUNCATE ' || relname || ';' from pg_class where relname ~ 'processed_crashes_201404' and relkind = 'r';
               ?column?
--------------------------------------
 TRUNCATE processed_crashes_20140428;
 TRUNCATE processed_crashes_20140421;
 TRUNCATE processed_crashes_20140407;
 TRUNCATE processed_crashes_20140414;
(4 rows)

breakpad=# TRUNCATE processed_crashes_20140428;
TRUNCATE TABLE
breakpad=# TRUNCATE processed_crashes_20140421;
TRUNCATE TABLE
breakpad=# TRUNCATE processed_crashes_20140407;
TRUNCATE TABLE
breakpad=# TRUNCATE processed_crashes_20140414;
TRUNCATE TABLE

[mpressman@socorro1.db.phx1 ~]$ df -hT /pgdata
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sdb1      xfs   2.5T  1.4T  1.2T  54% /pgdata
[mpressman@admin1a.private.phx1 ~]$ ssh -A mpressman@socorro2.db.phx1.mozilla.com "df -hT /pgdata"
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sdb1      xfs   2.5T  1.4T  1.2T  55% /pgdata

[mpressman@admin1a.private.phx1 ~]$ ssh -A mpressman@socorro3.db.phx1.mozilla.com "df -hT /pgdata"
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sdb1      xfs   2.5T  1.4T  1.2T  54% /pgdata
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.