Closed Bug 1079256 Opened 11 years ago Closed 11 years ago

B2G device image nightlies (non-eng only) constantly failing/retrying due to failure to upload to update.boot2gecko.org

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: pmoore, Unassigned)

References

Details

Failing with e.g.: rsync error: error in file IO (code 11) at receiver.c(301) [receiver=3.0.6] rsync: connection unexpectedly closed (56 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6] Return code: 12 Unable to rsync /builds/slave/b2g_m-aurora_nexus-4_ntly-0000/build/upload-updates to update.boot2gecko.org:/data/update-channels/nexus-4/2.1.0/nightly! failed to upload Probably a disk space issue on update.boot2gecko.org - checking now... May need to disable branches - only needed for v1.3t.
/data appears full... Will see what I can clean up... [ec2-user@ip-10-39-70-110 ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 7.9G 6.1G 1.8G 78% / tmpfs 3.4G 0 3.4G 0% /dev/shm /dev/xvdf1 32G 30G 25M 100% /data
[ec2-user@ip-10-39-70-110 update-channels]$ du -sb /data/update-channels/* | sort -n | column -t 87257 /data/update-channels/nightly 103635 /data/update-channels/leo 48110585 /data/update-channels/unagi 1396359749 /data/update-channels/tarako 3367777912 /data/update-channels/flame-kk 3568752571 /data/update-channels/dolphin 5584646824 /data/update-channels/hamachi 5734702890 /data/update-channels/flame 5889667221 /data/update-channels/nexus-4 6252036518 /data/update-channels/helix
Hey Ben, These are the versions we are publishing - what can we delete here? Thanks! Pete [ec2-user@ip-10-39-70-110 update-channels]$ find . -mindepth 2 -maxdepth 2 -type d | sort ./dolphin/1.4.0 ./dolphin/2.1.0 ./dolphin/2.2.0 ./flame/1.4.0 ./flame/2.0.0 ./flame/2.1.0 ./flame/2.2.0 ./flame/default ./flame-kk/2.0.0 ./flame-kk/2.1.0 ./flame-kk/2.2.0 ./hamachi/1.0.1 ./hamachi/1.1.0 ./hamachi/1.2.0 ./hamachi/1.3.0 ./hamachi/1.4.0 ./hamachi/2.0.0 ./hamachi/2.1.0 ./hamachi/2.2.0 ./hamachi/default ./helix/1.1.1 ./helix/1.2.0 ./helix/1.3.0 ./helix/1.4.0 ./helix/2.0.0 ./helix/2.1.0 ./helix/2.2.0 ./helix/default ./leo/1.1.0 ./leo/1.2.0 ./leo/1.3.0 ./leo/1.4.0 ./leo/2.0.0 ./leo/default ./nexus-4/1.4.0 ./nexus-4/2.0.0 ./nexus-4/2.1.0 ./nexus-4/2.2.0 ./nexus-4/default ./tarako/1.3.0t ./unagi/1.1.0
Flags: needinfo?(bhearsum)
(In reply to Pete Moore [:pete][:pmoore] from comment #3) > Hey Ben, > > These are the versions we are publishing - what can we delete here? > > Thanks! > Pete > > [ec2-user@ip-10-39-70-110 update-channels]$ find . -mindepth 2 -maxdepth 2 > -type d | sort > ./dolphin/1.4.0 > ./dolphin/2.1.0 > ./dolphin/2.2.0 > ./flame/1.4.0 > ./flame/2.0.0 > ./flame/2.1.0 > ./flame/2.2.0 > ./flame/default > ./flame-kk/2.0.0 > ./flame-kk/2.1.0 > ./flame-kk/2.2.0 > ./hamachi/1.0.1 > ./hamachi/1.1.0 > ./hamachi/1.2.0 > ./hamachi/1.3.0 > ./hamachi/1.4.0 > ./hamachi/2.0.0 > ./hamachi/2.1.0 > ./hamachi/2.2.0 > ./hamachi/default > ./helix/1.1.1 > ./helix/1.2.0 > ./helix/1.3.0 > ./helix/1.4.0 > ./helix/2.0.0 > ./helix/2.1.0 > ./helix/2.2.0 > ./helix/default > ./leo/1.1.0 > ./leo/1.2.0 > ./leo/1.3.0 > ./leo/1.4.0 > ./leo/2.0.0 > ./leo/default > ./nexus-4/1.4.0 > ./nexus-4/2.0.0 > ./nexus-4/2.1.0 > ./nexus-4/2.2.0 > ./nexus-4/default > ./tarako/1.3.0t > ./unagi/1.1.0 Only 1.3t matters - anything else can go.
Flags: needinfo?(bhearsum)
[ec2-user@ip-10-39-70-110 ~]$ df -k; find /data/update-channels/ -mindepth 2 -maxdepth 2 | grep -v 1.3.0t | while read dir; do find "${dir}" -name '*.mar'; done | while read file; do echo "deleting '${file}'"; rm "${file}"; done; df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvda1 8256952 6355880 1817200 78% / tmpfs 3559600 0 3559600 0% /dev/shm /dev/xvdf1 33025240 31322652 25004 100% /data deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140531040201.mar' deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140530160203.mar' deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140601160201.mar' deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140531160201.mar' deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140601040203.mar' deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140602040201.mar' deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140602072051.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140928160203.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140927000202.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141005000203.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141004000203.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141005160203.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141002160200.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140930000204.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140929000203.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140929160205.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141001000205.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140928000205.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141004160203.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140927160206.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141003000204.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141002000208.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140930160202.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141001160205.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141006000202.mar' deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141003160205.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141003160209.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141004000205.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141005000204.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141004160205.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140928000204.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140927000203.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140927160203.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141005160206.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141002160203.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140930160205.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140929000203.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141006000205.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141001000202.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141001160202.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140929160206.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140928160204.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140930000203.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141003000203.mar' deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141002000202.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141001040205.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140927160205.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140928040208.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141006040204.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140928160202.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140930160207.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141003040207.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141003160204.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140929040202.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141002160204.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140930040206.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141005160202.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141001160203.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141002093155.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140926185533.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140927040203.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141004160202.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140929160203.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141004040202.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141002040202.mar' deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141005040205.mar' deleting '/data/update-channels/nexus-4/1.4.0/nexus-4/1.4.0/nightly/b2g_update_20140602000203.mar' deleting '/data/update-channels/nexus-4/1.4.0/nexus-4/1.4.0/nightly/b2g_update_20140601000202.mar' deleting '/data/update-channels/nexus-4/1.4.0/nexus-4/1.4.0/nightly/b2g_update_20140531000203.mar' deleting '/data/update-channels/nexus-4/1.4.0/nexus-4/1.4.0/nightly/b2g_update_20140530160206.mar' deleting '/data/update-channels/nexus-4/1.4.0/nexus-4/1.4.0/nightly/b2g_update_20140531160202.mar' deleting '/data/update-channels/nexus-4/1.4.0/nexus-4/1.4.0/nightly/b2g_update_20140601160203.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141002160206.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140928160203.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141004000201.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141003000205.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141005000201.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140929000204.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141002000206.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140929160207.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140927000205.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141006000201.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141003160202.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140930000202.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141005160201.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141001000204.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141001160204.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140927160202.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140930160200.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141004160201.mar' deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140928000202.mar' deleting '/data/update-channels/helix/1.2.0/helix/1.2.0/nightly/b2g_update_20140531034002.mar' deleting '/data/update-channels/helix/1.2.0/helix/1.2.0/nightly/b2g_update_20140601034002.mar' deleting '/data/update-channels/helix/1.2.0/helix/1.2.0/nightly/b2g_update_20140602034005.mar' deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140531040201.mar' deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140530160203.mar' deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140601160201.mar' deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140531160201.mar' deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140601040203.mar' deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140602040201.mar' deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140602072051.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140928160203.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140927000202.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141005000203.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141004000203.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141005160203.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141002160200.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140930000204.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140929000203.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140929160205.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141001000205.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140928000205.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141004160203.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140927160206.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141003000204.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141002000208.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140930160202.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141001160205.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141006000202.mar' deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141003160205.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141003160209.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141004000205.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141005000204.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141004160205.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140928000204.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140927000203.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140927160203.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141005160206.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141002160203.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140930160205.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140929000203.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141006000205.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141001000202.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141001160202.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140929160206.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140928160204.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140930000203.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141003000203.mar' deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141002000202.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141001040205.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140927160205.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140928040208.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141006040204.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140928160202.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140930160207.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141003040207.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141003160204.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140929040202.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141002160204.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140930040206.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141005160202.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141001160203.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141002093155.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140926185533.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140927040203.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141004160202.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140929160203.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141004040202.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141002040202.mar' deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141005040205.mar' deleting '/data/update-channels/helix/1.4.0/helix/1.4.0/nightly/b2g_update_20140602000203.mar' deleting '/data/update-channels/helix/1.4.0/helix/1.4.0/nightly/b2g_update_20140601000202.mar' deleting '/data/update-channels/helix/1.4.0/helix/1.4.0/nightly/b2g_update_20140531000203.mar' deleting '/data/update-channels/helix/1.4.0/helix/1.4.0/nightly/b2g_update_20140530160206.mar' deleting '/data/update-channels/helix/1.4.0/helix/1.4.0/nightly/b2g_update_20140531160202.mar' deleting '/data/update-channels/helix/1.4.0/helix/1.4.0/nightly/b2g_update_20140601160203.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141002160206.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140928160203.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141004000201.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141003000205.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141005000201.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140929000204.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141002000206.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140929160207.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140927000205.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141006000201.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141003160202.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140930000202.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141005160201.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141001000204.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141001160204.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140927160202.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140930160200.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141004160201.mar' deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140928000202.mar' deleting '/data/update-channels/helix/1.3.0/helix/1.3.0/nightly/b2g_update_20140531024004.mar' deleting '/data/update-channels/helix/1.3.0/helix/1.3.0/nightly/b2g_update_20140601024001.mar' deleting '/data/update-channels/helix/1.3.0/helix/1.3.0/nightly/b2g_update_20140602024006.mar' deleting '/data/update-channels/hamachi/1.2.0/hamachi/1.2.0/nightly/b2g_update_20140601034002.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140928160203.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140927000202.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141005000203.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141004000203.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141005160203.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141002160200.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140930000204.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140929000203.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140929160205.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141001000205.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140928000205.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141004160203.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140927160206.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141003000204.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141002000208.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140930160202.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141001160205.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141006000202.mar' deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141003160205.mar' deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140531040201.mar' deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140530160203.mar' deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140601160201.mar' deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140531160201.mar' deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140601040203.mar' deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140602040201.mar' deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140602072051.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141003160209.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141004000205.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141005000204.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141004160205.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140928000204.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140927000203.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140927160203.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141005160206.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141002160203.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140930160205.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140929000203.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141006000205.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141001000202.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141001160202.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140929160206.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140928160204.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140930000203.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141003000203.mar' deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141002000202.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141001040205.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140927160205.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140928040208.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141006040204.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140928160202.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140930160207.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141003040207.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141003160204.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140929040202.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141002160204.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140930040206.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141005160202.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141001160203.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141002093155.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140926185533.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140927040203.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141004160202.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140929160203.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141004040202.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141002040202.mar' deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141005040205.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141002160206.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140928160203.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141004000201.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141003000205.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141005000201.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140929000204.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141002000206.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140929160207.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140927000205.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141006000201.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141003160202.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140930000202.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141005160201.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141001000204.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141001160204.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140927160202.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140930160200.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141004160201.mar' deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140928000202.mar' deleting '/data/update-channels/hamachi/1.4.0/hamachi/1.4.0/nightly/b2g_update_20140602000203.mar' deleting '/data/update-channels/hamachi/1.4.0/hamachi/1.4.0/nightly/b2g_update_20140601000202.mar' deleting '/data/update-channels/hamachi/1.4.0/hamachi/1.4.0/nightly/b2g_update_20140531000203.mar' deleting '/data/update-channels/hamachi/1.4.0/hamachi/1.4.0/nightly/b2g_update_20140530160206.mar' deleting '/data/update-channels/hamachi/1.4.0/hamachi/1.4.0/nightly/b2g_update_20140531160202.mar' deleting '/data/update-channels/hamachi/1.4.0/hamachi/1.4.0/nightly/b2g_update_20140601160203.mar' deleting '/data/update-channels/hamachi/1.3.0/hamachi/1.3.0/nightly/b2g_update_20140602024006.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20140928160203.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141005000203.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141004000203.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141005160203.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141002160200.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20140930000204.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20140929000203.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20140929160205.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141001000205.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141004160203.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141003000204.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141002000208.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20140930160202.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141001160205.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141006000202.mar' deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141003160205.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141003160209.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141004000205.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141005000204.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141004160205.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141005160206.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141002160203.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20140930160205.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20140929000203.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141006000205.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141001000202.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141001160202.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20140929160206.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20140928160204.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20140930000203.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141003000203.mar' deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141002000202.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141001040205.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20140928160202.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20140930160207.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141003040207.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141003160204.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20140929040202.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141002160204.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20140930040206.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141005160202.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141001160203.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141002093155.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141004160202.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20140929160203.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141004040202.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141002040202.mar' deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141005040205.mar' deleting '/data/update-channels/unagi/1.1.0/beta/b2g_beta_update_20130821041204.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141003160209.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141004000205.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141005000204.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141004160205.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140928000204.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140927000203.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140927160203.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141005160206.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141002160203.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140930160205.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140929000203.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141006000205.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141001000202.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141001160202.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140929160206.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140928160204.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140930000203.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141003000203.mar' deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141002000202.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141001040205.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140927160205.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140928040208.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140928160202.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140930160207.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141003040207.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141003160204.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140929040202.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141002160204.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140930040206.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141005160202.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141001160203.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141002093155.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140926185533.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140927040203.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141004160202.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140929160203.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141004040202.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141002040202.mar' deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141005040205.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141002160206.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140928160203.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141004000201.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141003000205.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141005000201.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140929000204.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141002000206.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140929160207.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140927000205.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141006000201.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141003160202.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140930000202.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141005160201.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141001000204.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141001160204.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140927160202.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140930160200.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141004160201.mar' deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140928000202.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140928160203.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140927000202.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141005000203.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141004000203.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141005160203.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141002160200.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140930000204.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140929000203.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140929160205.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141001000205.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140928000205.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141004160203.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140927160206.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141003000204.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141002000208.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140930160202.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141001160205.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141006000202.mar' deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141003160205.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141003160209.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141004000205.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141005000204.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141004160205.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140928000204.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140927000203.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140927160203.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141005160206.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141002160203.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140930160205.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140929000203.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141006000205.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141001000202.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141001160202.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140929160206.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140928160204.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140930000203.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141003000203.mar' deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141002000202.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141001040205.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140927160205.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140928040208.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140928160202.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140930160207.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141003040207.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141003160204.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140929040202.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141002160204.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140930040206.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141005160202.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141001160203.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141002093155.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140926185533.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140927040203.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141004160202.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140929160203.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141004040202.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141002040202.mar' deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141005040205.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141002160206.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140928160203.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141004000201.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141003000205.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141005000201.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140929000204.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141002000206.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140929160207.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140927000205.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141006000201.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141003160202.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140930000202.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141005160201.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141001000204.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141001160204.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140927160202.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140930160200.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141004160201.mar' deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140928000202.mar' deleting '/data/update-channels/flame/1.4.0/flame/1.4.0/nightly/b2g_update_20140602000203.mar' deleting '/data/update-channels/flame/1.4.0/flame/1.4.0/nightly/b2g_update_20140601000202.mar' deleting '/data/update-channels/flame/1.4.0/flame/1.4.0/nightly/b2g_update_20140531000203.mar' deleting '/data/update-channels/flame/1.4.0/flame/1.4.0/nightly/b2g_update_20140530160206.mar' deleting '/data/update-channels/flame/1.4.0/flame/1.4.0/nightly/b2g_update_20140531160202.mar' deleting '/data/update-channels/flame/1.4.0/flame/1.4.0/nightly/b2g_update_20140601160203.mar' Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvda1 8256952 6355880 1817200 78% / tmpfs 3559600 0 3559600 0% /dev/shm /dev/xvdf1 33025240 1565992 29781664 5% /data [ec2-user@ip-10-39-70-110 ~]$
See Also: → 1000217
Making bug 1000217 a blocker: we'll be pushing a rock up a hill here until we stop uploading unneeded mars.
Blocks: 1000217
(In reply to Chris Cooper [:coop] from comment #6) > Making bug 1000217 a blocker: we'll be pushing a rock up a hill here until > we stop uploading unneeded mars. And by blocker, I of course meant dependency.
No longer blocks: 1000217
Depends on: 1000217
Perhaps the ec2-user cron is also busted. I don't see any email from the /home/ec2-user/check_disk job which runs every hour, and should have warned us that the disk was filling up. And files like > deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140927000205.mar' should've been deleted by @daily find /data/update-channels/{hamachi,helix,leo,nexus-4,unagi,flame,flame-kk,dolphin}/*/nightly -type f -mtime +1 \( -name '*.xml' -o -name '*.ini' -o -name '*.mar' \) -delete
Might want to fix this at some point. It's full again, and if I didn't lose count along the way by the time I killed the pointless retries just now, we had built 210 pointless nightlies (or maybe 200, if you call the 1.3t ones non-pointless).
Cleaned up again by running the cron jobs manually: $ df -h /data/ Filesystem Size Used Avail Use% Mounted on /dev/xvdf1 32G 3.6G 27G 12% /data Cron is definitely busted: Sep 27 19:00:01 ip-10-39-70-110 crond[23532]: (ec2-user) FAILED to authorize user with PAM (Module is unknown) Sep 27 19:01:01 ip-10-39-70-110 crond[23533]: (root) FAILED to authorize user with PAM (Module is unknown) Restarted the cron service (per google results) and now it's working ok.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
BTW, probably got broken when we upgraded packages for one of the recent security issues.
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.