849529 - socorro-collector[1-6].webapp.phx1 low disk

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Michael Burns [:mburns]]

Sat 09:05:44 PST [163] socorro-collector4.webapp.phx1.mozilla.com:Disk - All Early is WARNING: DISK WARNING - free space: / 67708 MB (15% inode=82%): /dev/shm 5947 MB (100% inode=99%): /boot 870 MB (93% inode=99%)

Sat 08:53:35 PST [158] socorro-collector3.webapp.phx1.mozilla.com:Disk - All Early is WARNING: DISK WARNING - free space: / 66572 MB (15% inode=82%): /dev/shm 5947 MB (100% inode=99%): /boot 870 MB (93% inode=99%)

Comment 1

[Laura Thomson :laura]

Crashes backed up on disk again.

e.g.
[[e-mail]1 primaryCrashStore]# find . -type f -name *.dump | wc -l
3335144

I'm not sure how to get crashmover to go back and try to resubmit those crashes; lars is on the road.  mburns is paging jakem for assistance since he was in the trenches on this last time.

Comment 2

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/c6833b40e079c56c8118e2e23535d02dc399e2ea
Merge pull request #1128 from twobraids/delete_all_dumps

Fixes Bug 849529

Comment 3

[Laura Thomson :laura]

Lars: are these actually deleted?  This bug should remain open until the piled up dumps have been removed.

Comment 4

[Adrian J Fernandez [:Aj]]

Related alert;

Sat 19:54:14 PST [144] sp-admin01.phx1.mozilla.com:Socorro Admin - cron_submitter-crash-reports.allizom.org.log is CRITICAL: FILE_AGE CRITICAL: /var/log/socorro/cron_submitter-crash-reports.allizom.org.log is 4295 seconds old and 85385563 bytes

Comment 5

[Adrian J Fernandez [:Aj]]

Another (possibly) related alert;
Sun 04:27:01 PDT [143] sp-admin01.phx1.mozilla.com:Socorro Admin - cron_create_partitions.log is CRITICAL: FILE_AGE CRITICAL: /var/log/socorro/cron_create_partitions.log is 605152 seconds old and 0 bytes

Comment 6

[Selena Deckelmann :selenamarie :selena]

(In reply to Adrian Fernandez [:Aj] from comment #5)
> Another (possibly) related alert;
> Sun 04:27:01 PDT [143] sp-admin01.phx1.mozilla.com:Socorro Admin -
> cron_create_partitions.log is CRITICAL: FILE_AGE CRITICAL:
> /var/log/socorro/cron_create_partitions.log is 605152 seconds old and 0 bytes

This is not related. I made a mistake in the ownership of a new table. I will investigate why this is still a problem -- was able to run the partitioner function today without an error: 

breakpad=# SELECT * FROM weekly_report_partitions();
weekly_report_partitions|t
Time: 5747.353 ms

Comment 7

[Peter Radcliffe [:pir]]

socorro-collector4.webapp.phx1 out of the zeus pools to be cleaned up by lars.

Comment 8

[Ashish Vijayaram [:ashish]]

socorro-collector1 undrained in Zeus, socorro-collector2 is drained while :lars works on it.

Comment 9

[Eric Ziegenhorn :ericz]

socorro-collector2 undrained in Zeus, socorro-collector3 is drained now.

Comment 10

[Peter Radcliffe [:pir]]

socorro-collector3 undrained and socorro-collector5 drained, as per lars's request on irc.

Comment 11

[Peter Radcliffe [:pir]]

collector 5 undrained and collector 6 drained as per lars's request on irc.