912039 - No space left on socorroadm.stage root partition

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task)

Description

[Selena Deckelmann :selenamarie :selena]

the stage system was alerting because there's no space left on the root partition: 


Traceback (most recent call last):
  File "/data/socorro/application/socorro/cron/crontabber.py", line 858, in <module>
    sys.exit(main(CronTabber))
  File "/data/socorro/application/socorro/app/generic_app.py", line 167, in main
    config_manager_cls
  File "/data/socorro/application/socorro/app/generic_app.py", line 264, in _do_main
    return fix_exit_code(app(config))
  File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__
    self.gen.throw(type, value, traceback)
  File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/config_manager.py", line 258, in context
    yield config
  File "/data/socorro/application/socorro/app/generic_app.py", line 258, in _do_main
    return fix_exit_code(instance.main())
  File "/data/socorro/application/socorro/cron/crontabber.py", line 521, in main
    self.run_all()
  File "/data/socorro/application/socorro/cron/crontabber.py", line 665, in run_all
    self._run_one(job_class, class_config)
  File "/data/socorro/application/socorro/cron/crontabber.py", line 720, in _run_one
    exc_type, exc_value, exc_tb)
  File "/data/socorro/application/socorro/cron/crontabber.py", line 820, in _log_run
    self.database.save(self.config.crontabber.database_file)
  File "/data/socorro/application/socorro/cron/crontabber.py", line 152, in save
    super(JSONAndPostgresJobDatabase, self).save(file_path)
  File "/data/socorro/application/socorro/cron/crontabber.py", line 113, in save
    f, indent=2)
  File "/usr/lib64/python2.6/json/__init__.py", line 181, in dump
    fp.write(chunk)
IOError: [Errno 28] No space left on device
2013-09-03 02:05:01,791 INFO - MainThread - app_name: crontabber
2013-09-03 02:05:01,791 INFO - MainThread - app_version: 1.1
2013-09-03 02:05:01,791 INFO - MainThread - current configuration:
2013-09-03 02:05:01,792 
[[e-mail]1 socorro]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      39091656  39084836         0 100% /
tmpfs                  1962520      2992   1959528   1% /dev/shm
/dev/sda1                99150     53440     40590  57% /boot
10.8.75.14:/symbols  4194304000 3737347296 456956704  90% /mnt/socorro/symbols

Comment 1

[Selena Deckelmann :selenamarie :selena]

I just realized I can clean this up... Taking a whack at /data first.

Comment 2

[Selena Deckelmann :selenamarie :selena]

Removing stuff from August: 

[[e-mail]1 data]# ls -1 | egrep '*-08-*' | xargs -iXX rm -rf XX

And things are looking better already: 

[[e-mail]1 persistent]$ df -hl
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       38G   33G  3.0G  92% /
tmpfs                 1.9G  3.0M  1.9G   1% /dev/shm
/dev/sda1              97M   53M   40M  57% /boot

There's a ticket open somewhere to start removing data after autodeploys so I'm going to just close this for now.