1231910 - treeherder-etl1.private.scl3.mozilla.com:Disk - All is WARNING

Status: RESOLVED DUPLICATE of bug 1229020

(Tree Management :: Treeherder: Infrastructure, defect)

Description

[Vinh Hua [:vinh]]

nagios-scl3> Fri 03:55:43 PST [5048] treeherder-etl1.private.scl3.mozilla.com:Disk - All is WARNING: DISK WARNING - free space: / 3808 MB (10% inode=95%): (http://m.mozilla.org/Disk+-+All)

Comment 1

[Vinh Hua [:vinh]]

[[e-mail]3 celery]# ll -lSrh
total 24G
-rw-r--r-- 1 treeherder treeherder 1.1M Nov 22 03:41 celery_worker_buildapi.log-20151122.gz
-rw-r--r-- 1 treeherder treeherder 1.1M Nov  8 03:31 celery_worker_buildapi.log-20151108.gz
-rw-r--r-- 1 treeherder treeherder 4.1M Dec 11 12:30 celery_worker_buildapi.log
-rw-r--r-- 1 treeherder treeherder 8.1M Dec  6 03:30 celery_worker_buildapi.log-20151206
-rw-r--r-- 1 treeherder treeherder 9.1M Nov 22 03:42 celery_worker_pushlog.log-20151122.gz
-rw-r--r-- 1 treeherder treeherder  11M Nov  8 03:31 celery_worker_pushlog.log-20151108.gz
-rw-r--r-- 1 treeherder treeherder  12M Nov 15 03:41 celery_worker_pushlog.log-20151115.gz
-rw-r--r-- 1 treeherder treeherder  13M Nov 29 03:17 celery_worker_buildapi.log-20151129
-rw-r--r-- 1 treeherder treeherder  24M Nov 15 03:41 celery_worker_buildapi.log-20151115.gz
-rw-r--r-- 1 treeherder treeherder 754M Nov 29 03:17 celery_worker_pushlog.log-20151129.gz
-rw-r--r-- 1 treeherder treeherder  23G Dec 11 12:31 celery_worker_pushlog.log
[[e-mail]3 celery]# pwd
/var/log/celery

Comment 2

[Ryan C [:ryanc] (UTC-4)]

Lot of this in, celery_worker_pushlog.log

[2015-12-12 11:16:07,587: INFO/Worker-426] Starting new HTTPS connection (1): treeherder.mozilla.org
[2015-12-12 11:16:07,637: ERROR/Worker-426] HTTPError 429 submitting to https://treeherder.mozilla.org/api/project/try/resultset/: {"detail":"Request was throttled. Expected available in 1.0 second."}
[2015-12-12 11:16:07,644: INFO/Worker-426] Starting new HTTPS connection (1): treeherder.mozilla.org
[2015-12-12 11:16:07,792: INFO/Worker-426] Starting new HTTPS connection (1): treeherder.mozilla.org
[2015-12-12 11:16:07,846: ERROR/Worker-426] HTTPError 429 submitting to https://treeherder.mozilla.org/api/project/try/resultset/: {"detail":"Request was throttled. Expected available in 1.0 second."}
[2015-12-12 11:16:07,850: INFO/Worker-426] Starting new HTTPS connection (1): treeherder.mozilla.org
[2015-12-12 11:16:07,905: ERROR/Worker-426] HTTPError 429 submitting to https://treeherder.mozilla.org/api/project/try/resultset/: {"detail":"Request was throttled. Expected available in 1.0 second."}
[2015-12-12 11:16:07,911: INFO/Worker-426] Starting new HTTPS connection (1): treeherder.mozilla.org
[2015-12-12 11:16:08,346: INFO/Worker-426] Starting new HTTPS connection (1): treeherder.mozilla.org