If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

No space left on socorroadm.stage root partition

RESOLVED FIXED

Status

Infrastructure & Operations Graveyard
WebOps: Socorro
--
blocker
RESOLVED FIXED
4 years ago
a year ago

People

(Reporter: selenamarie, Assigned: selenamarie)

Tracking

Details

the stage system was alerting because there's no space left on the root partition: 


Traceback (most recent call last):
  File "/data/socorro/application/socorro/cron/crontabber.py", line 858, in <module>
    sys.exit(main(CronTabber))
  File "/data/socorro/application/socorro/app/generic_app.py", line 167, in main
    config_manager_cls
  File "/data/socorro/application/socorro/app/generic_app.py", line 264, in _do_main
    return fix_exit_code(app(config))
  File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__
    self.gen.throw(type, value, traceback)
  File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman/config_manager.py", line 258, in context
    yield config
  File "/data/socorro/application/socorro/app/generic_app.py", line 258, in _do_main
    return fix_exit_code(instance.main())
  File "/data/socorro/application/socorro/cron/crontabber.py", line 521, in main
    self.run_all()
  File "/data/socorro/application/socorro/cron/crontabber.py", line 665, in run_all
    self._run_one(job_class, class_config)
  File "/data/socorro/application/socorro/cron/crontabber.py", line 720, in _run_one
    exc_type, exc_value, exc_tb)
  File "/data/socorro/application/socorro/cron/crontabber.py", line 820, in _log_run
    self.database.save(self.config.crontabber.database_file)
  File "/data/socorro/application/socorro/cron/crontabber.py", line 152, in save
    super(JSONAndPostgresJobDatabase, self).save(file_path)
  File "/data/socorro/application/socorro/cron/crontabber.py", line 113, in save
    f, indent=2)
  File "/usr/lib64/python2.6/json/__init__.py", line 181, in dump
    fp.write(chunk)
IOError: [Errno 28] No space left on device
2013-09-03 02:05:01,791 INFO - MainThread - app_name: crontabber
2013-09-03 02:05:01,791 INFO - MainThread - app_version: 1.1
2013-09-03 02:05:01,791 INFO - MainThread - current configuration:
2013-09-03 02:05:01,792 
[root@socorroadm.stage.private.phx1 socorro]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      39091656  39084836         0 100% /
tmpfs                  1962520      2992   1959528   1% /dev/shm
/dev/sda1                99150     53440     40590  57% /boot
10.8.75.14:/symbols  4194304000 3737347296 456956704  90% /mnt/socorro/symbols
Assignee: server-ops-webops → sdeckelmann
I just realized I can clean this up... Taking a whack at /data first.
Removing stuff from August: 

[root@socorroadm.stage.private.phx1 data]# ls -1 | egrep '*-08-*' | xargs -iXX rm -rf XX

And things are looking better already: 

[socorro@socorroadm.stage.private.phx1 persistent]$ df -hl
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       38G   33G  3.0G  92% /
tmpfs                 1.9G  3.0M  1.9G   1% /dev/shm
/dev/sda1              97M   53M   40M  57% /boot

There's a ticket open somewhere to start removing data after autodeploys so I'm going to just close this for now.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
See Also: → bug 912647
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.