864069 - Out of space on update.boot2gecko.org leading to infinite retries of b2g18 nightly

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

According to the 15 times that we built a b2g18 unagi nightly before I got sufficiently tired of seeing the infinite loop and killed the last one,

17:16:21     INFO - ##### Running upload-updates step.
17:16:21     INFO - #####
17:16:21     INFO - Uploading the contents of /builds/slave/b2g_m-b18_unagi_ntly-000000000/build/upload-updates to update.boot2gecko.org:/data/update-channels/nightly
17:16:21     INFO - Running command: ['ssh', '-oIdentityFile=/home/cltbld/.ssh/b2gbld_dsa', '[e-mail]', 'mkdir', '-p', '/data/update-channels/nightly'] in /builds/slave/b2g_m-b18_unagi_ntly-000000000/build
17:16:21     INFO - Copy/paste: ssh -oIdentityFile=/home/cltbld/.ssh/b2gbld_dsa [e-mail] mkdir -p /data/update-channels/nightly
17:16:22     INFO - Return code: 0
17:16:22     INFO - Running command: ['rsync', '-e', 'ssh -oIdentityFile=/home/cltbld/.ssh/b2gbld_dsa', '-azv', '--exclude=update.xml', '.', '[e-mail]:/data/update-channels/nightly/'] in /builds/slave/b2g_m-b18_unagi_ntly-000000000/build/upload-updates
17:16:22     INFO - Copy/paste: rsync -e "ssh -oIdentityFile=/home/cltbld/.ssh/b2gbld_dsa" -azv --exclude=update.xml . [e-mail]:/data/update-channels/nightly/
17:16:24     INFO -  sending incremental file list
17:16:24     INFO -  ./
17:16:24     INFO -  application_20130420070203.ini
17:16:24     INFO -  b2g_update_20130420070203.mar
17:16:26     INFO -  rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
17:16:26     INFO -  rsync: write failed on "/data/update-channels/nightly/b2g_update_20130420070203.mar": No space left on device (28)
17:16:26    ERROR -  rsync error: error in file IO (code 11) at receiver.c(301) [receiver=3.0.6]
17:16:26    ERROR -  rsync: connection unexpectedly closed (56 bytes received so far) [sender]
17:16:26    ERROR -  rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
17:16:26    ERROR - Return code: 12
17:16:26    ERROR - Unable to rsync /builds/slave/b2g_m-b18_unagi_ntly-000000000/build/upload-updates to update.boot2gecko.org:/data/update-channels/nightly!
17:16:26    ERROR - failed to upload

Comment 1

[Rail Aliiev [:rail]]

I copied /data/update-channels/betatest (which isn't used anymore) to bm36:betatest-bug864069 and them removed it. I'll removed it if noone screams in a couple of days.

Comment 2

[Rail Aliiev [:rail]]

(In reply to Rail Aliiev [:rail] from comment #1)
> I copied /data/update-channels/betatest (which isn't used anymore) to
> bm36:betatest-bug864069 and them removed it. I'll removed it if noone
> screams in a couple of days.

I'm going to delete the files this Friday. Any objections?

Comment 3

[Rail Aliiev [:rail]]

(In reply to Rail Aliiev [:rail] from comment #2)
> (In reply to Rail Aliiev [:rail] from comment #1)
> > I copied /data/update-channels/betatest (which isn't used anymore) to
> > bm36:betatest-bug864069 and them removed it. I'll removed it if noone
> > screams in a couple of days.
> 
> I'm going to delete the files this Friday. Any objections?

rm -rf betatest-bug864069

Comment 4

[Phil Ringnalda (:philor)]

Full again, currently on our third try of one nightly and our eighth try of another, on each of b2g18 and b2g18v1_0_1.

Comment 5

[Phil Ringnalda (:philor)]

And apparently in need of a nagios check to check up on the results of the automated cleanup it's also apparently in need of.

Comment 6

[Hal Wine [:hwine] use NI!]

(In reply to Phil Ringnalda (:philor) from comment #4)
> Full again, currently on our third try of one nightly and our eighth try of
> another, on each of b2g18 and b2g18v1_0_1.

In progress -- data has to be backed up first, no ETR at this point.

(In reply to Phil Ringnalda (:philor) from comment #5)
> And apparently in need of a nagios check to check up on the results of the
> automated cleanup it's also apparently in need of.

Machine is outside of nagios reach, but we'll set up an old fashioned cron job or something after this cleanup is done.

Comment 7

[Chris AtLee [:catlee]]

backed up /data/update-channels/nightly to /home/cltbld/b2g-nightly-updates on bm53, and then deleted *2012* in /data/update-channels/nightly. we're back down to 76% capacity.

Comment 8

[Hal Wine [:hwine] use NI!]

(In reply to Hal Wine [:hwine] from comment #6)
> Machine is outside of nagios reach, but we'll set up an old fashioned cron
> job or something after this cleanup is done.

daily email alert set up.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Hal Wine [:hwine] from comment #8)
> daily email alert set up.

Did that go off the last few days ?

Comment 10

[Hal Wine [:hwine] use NI!]

(In reply to Nick Thomas [:nthomas] from comment #9)
> (In reply to Hal Wine [:hwine] from comment #8)
> > daily email alert set up.
> 
> Did that go off the last few days ?

No, it hasn't. :( 

PATH problem in cronjob - fixed now.