1079256 - B2G device image nightlies (non-eng only) constantly failing/retrying due to failure to upload to update.boot2gecko.org

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Pete Moore [:pmoore][:pete]]

Failing with e.g.:

rsync error: error in file IO (code 11) at receiver.c(301) [receiver=3.0.6]

rsync: connection unexpectedly closed (56 bytes received so far) [sender]

rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]

Return code: 12

Unable to rsync /builds/slave/b2g_m-aurora_nexus-4_ntly-0000/build/upload-updates to update.boot2gecko.org:/data/update-channels/nexus-4/2.1.0/nightly!

failed to upload

Probably a disk space issue on update.boot2gecko.org - checking now...

May need to disable branches - only needed for v1.3t.

Comment 1

[Pete Moore [:pmoore][:pete]]

/data appears full...

Will see what I can clean up...

[ec2-user@ip-10-39-70-110 ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            7.9G  6.1G  1.8G  78% /
tmpfs                 3.4G     0  3.4G   0% /dev/shm
/dev/xvdf1             32G   30G   25M 100% /data

Comment 2

[Pete Moore [:pmoore][:pete]]

[ec2-user@ip-10-39-70-110 update-channels]$ du -sb /data/update-channels/* | sort -n | column -t
87257       /data/update-channels/nightly
103635      /data/update-channels/leo
48110585    /data/update-channels/unagi
1396359749  /data/update-channels/tarako
3367777912  /data/update-channels/flame-kk
3568752571  /data/update-channels/dolphin
5584646824  /data/update-channels/hamachi
5734702890  /data/update-channels/flame
5889667221  /data/update-channels/nexus-4
6252036518  /data/update-channels/helix

Comment 3

[Pete Moore [:pmoore][:pete]]

Hey Ben,

These are the versions we are publishing - what can we delete here?

Thanks!
Pete

[ec2-user@ip-10-39-70-110 update-channels]$ find . -mindepth 2 -maxdepth 2 -type d | sort
./dolphin/1.4.0
./dolphin/2.1.0
./dolphin/2.2.0
./flame/1.4.0
./flame/2.0.0
./flame/2.1.0
./flame/2.2.0
./flame/default
./flame-kk/2.0.0
./flame-kk/2.1.0
./flame-kk/2.2.0
./hamachi/1.0.1
./hamachi/1.1.0
./hamachi/1.2.0
./hamachi/1.3.0
./hamachi/1.4.0
./hamachi/2.0.0
./hamachi/2.1.0
./hamachi/2.2.0
./hamachi/default
./helix/1.1.1
./helix/1.2.0
./helix/1.3.0
./helix/1.4.0
./helix/2.0.0
./helix/2.1.0
./helix/2.2.0
./helix/default
./leo/1.1.0
./leo/1.2.0
./leo/1.3.0
./leo/1.4.0
./leo/2.0.0
./leo/default
./nexus-4/1.4.0
./nexus-4/2.0.0
./nexus-4/2.1.0
./nexus-4/2.2.0
./nexus-4/default
./tarako/1.3.0t
./unagi/1.1.0

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Pete Moore [:pete][:pmoore] from comment #3)
> Hey Ben,
> 
> These are the versions we are publishing - what can we delete here?
> 
> Thanks!
> Pete
> 
> [ec2-user@ip-10-39-70-110 update-channels]$ find . -mindepth 2 -maxdepth 2
> -type d | sort
> ./dolphin/1.4.0
> ./dolphin/2.1.0
> ./dolphin/2.2.0
> ./flame/1.4.0
> ./flame/2.0.0
> ./flame/2.1.0
> ./flame/2.2.0
> ./flame/default
> ./flame-kk/2.0.0
> ./flame-kk/2.1.0
> ./flame-kk/2.2.0
> ./hamachi/1.0.1
> ./hamachi/1.1.0
> ./hamachi/1.2.0
> ./hamachi/1.3.0
> ./hamachi/1.4.0
> ./hamachi/2.0.0
> ./hamachi/2.1.0
> ./hamachi/2.2.0
> ./hamachi/default
> ./helix/1.1.1
> ./helix/1.2.0
> ./helix/1.3.0
> ./helix/1.4.0
> ./helix/2.0.0
> ./helix/2.1.0
> ./helix/2.2.0
> ./helix/default
> ./leo/1.1.0
> ./leo/1.2.0
> ./leo/1.3.0
> ./leo/1.4.0
> ./leo/2.0.0
> ./leo/default
> ./nexus-4/1.4.0
> ./nexus-4/2.0.0
> ./nexus-4/2.1.0
> ./nexus-4/2.2.0
> ./nexus-4/default
> ./tarako/1.3.0t
> ./unagi/1.1.0

Only 1.3t matters - anything else can go.

Comment 5

[Pete Moore [:pmoore][:pete]]

[ec2-user@ip-10-39-70-110 ~]$ df -k; find /data/update-channels/ -mindepth 2 -maxdepth 2 | grep -v 1.3.0t | while read dir; do find "${dir}" -name '*.mar'; done | while read file; do echo "deleting '${file}'"; rm "${file}"; done; df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda1             8256952   6355880   1817200  78% /
tmpfs                  3559600         0   3559600   0% /dev/shm
/dev/xvdf1            33025240  31322652     25004 100% /data
deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140531040201.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140530160203.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140601160201.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140531160201.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140601040203.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140602040201.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nexus-4/2.0.0/nightly/b2g_update_20140602072051.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140928160203.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140927000202.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141005000203.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141004000203.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141005160203.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141002160200.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140930000204.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140929000203.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140929160205.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141001000205.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140928000205.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141004160203.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140927160206.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141003000204.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141002000208.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20140930160202.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141001160205.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141006000202.mar'
deleting '/data/update-channels/nexus-4/2.0.0/nightly/b2g_update_20141003160205.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141003160209.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141004000205.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141005000204.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141004160205.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140928000204.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140927000203.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140927160203.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141005160206.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141002160203.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140930160205.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140929000203.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141006000205.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141001000202.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141001160202.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140929160206.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140928160204.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20140930000203.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141003000203.mar'
deleting '/data/update-channels/nexus-4/2.1.0/nightly/b2g_update_20141002000202.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141001040205.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140927160205.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140928040208.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141006040204.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140928160202.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140930160207.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141003040207.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141003160204.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140929040202.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141002160204.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140930040206.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141005160202.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141001160203.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141002093155.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140926185533.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140927040203.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141004160202.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20140929160203.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141004040202.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141002040202.mar'
deleting '/data/update-channels/nexus-4/2.2.0/nightly/b2g_update_20141005040205.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nexus-4/1.4.0/nightly/b2g_update_20140602000203.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nexus-4/1.4.0/nightly/b2g_update_20140601000202.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nexus-4/1.4.0/nightly/b2g_update_20140531000203.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nexus-4/1.4.0/nightly/b2g_update_20140530160206.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nexus-4/1.4.0/nightly/b2g_update_20140531160202.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nexus-4/1.4.0/nightly/b2g_update_20140601160203.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141002160206.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140928160203.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141004000201.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141003000205.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141005000201.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140929000204.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141002000206.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140929160207.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140927000205.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141006000201.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141003160202.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140930000202.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141005160201.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141001000204.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141001160204.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140927160202.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140930160200.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20141004160201.mar'
deleting '/data/update-channels/nexus-4/1.4.0/nightly/b2g_update_20140928000202.mar'
deleting '/data/update-channels/helix/1.2.0/helix/1.2.0/nightly/b2g_update_20140531034002.mar'
deleting '/data/update-channels/helix/1.2.0/helix/1.2.0/nightly/b2g_update_20140601034002.mar'
deleting '/data/update-channels/helix/1.2.0/helix/1.2.0/nightly/b2g_update_20140602034005.mar'
deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140531040201.mar'
deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140530160203.mar'
deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140601160201.mar'
deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140531160201.mar'
deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140601040203.mar'
deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140602040201.mar'
deleting '/data/update-channels/helix/2.0.0/helix/2.0.0/nightly/b2g_update_20140602072051.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140928160203.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140927000202.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141005000203.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141004000203.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141005160203.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141002160200.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140930000204.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140929000203.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140929160205.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141001000205.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140928000205.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141004160203.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140927160206.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141003000204.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141002000208.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20140930160202.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141001160205.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141006000202.mar'
deleting '/data/update-channels/helix/2.0.0/nightly/b2g_update_20141003160205.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141003160209.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141004000205.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141005000204.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141004160205.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140928000204.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140927000203.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140927160203.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141005160206.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141002160203.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140930160205.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140929000203.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141006000205.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141001000202.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141001160202.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140929160206.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140928160204.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20140930000203.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141003000203.mar'
deleting '/data/update-channels/helix/2.1.0/nightly/b2g_update_20141002000202.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141001040205.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140927160205.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140928040208.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141006040204.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140928160202.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140930160207.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141003040207.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141003160204.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140929040202.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141002160204.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140930040206.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141005160202.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141001160203.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141002093155.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140926185533.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140927040203.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141004160202.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20140929160203.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141004040202.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141002040202.mar'
deleting '/data/update-channels/helix/2.2.0/nightly/b2g_update_20141005040205.mar'
deleting '/data/update-channels/helix/1.4.0/helix/1.4.0/nightly/b2g_update_20140602000203.mar'
deleting '/data/update-channels/helix/1.4.0/helix/1.4.0/nightly/b2g_update_20140601000202.mar'
deleting '/data/update-channels/helix/1.4.0/helix/1.4.0/nightly/b2g_update_20140531000203.mar'
deleting '/data/update-channels/helix/1.4.0/helix/1.4.0/nightly/b2g_update_20140530160206.mar'
deleting '/data/update-channels/helix/1.4.0/helix/1.4.0/nightly/b2g_update_20140531160202.mar'
deleting '/data/update-channels/helix/1.4.0/helix/1.4.0/nightly/b2g_update_20140601160203.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141002160206.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140928160203.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141004000201.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141003000205.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141005000201.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140929000204.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141002000206.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140929160207.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140927000205.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141006000201.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141003160202.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140930000202.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141005160201.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141001000204.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141001160204.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140927160202.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140930160200.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20141004160201.mar'
deleting '/data/update-channels/helix/1.4.0/nightly/b2g_update_20140928000202.mar'
deleting '/data/update-channels/helix/1.3.0/helix/1.3.0/nightly/b2g_update_20140531024004.mar'
deleting '/data/update-channels/helix/1.3.0/helix/1.3.0/nightly/b2g_update_20140601024001.mar'
deleting '/data/update-channels/helix/1.3.0/helix/1.3.0/nightly/b2g_update_20140602024006.mar'
deleting '/data/update-channels/hamachi/1.2.0/hamachi/1.2.0/nightly/b2g_update_20140601034002.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140928160203.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140927000202.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141005000203.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141004000203.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141005160203.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141002160200.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140930000204.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140929000203.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140929160205.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141001000205.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140928000205.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141004160203.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140927160206.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141003000204.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141002000208.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20140930160202.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141001160205.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141006000202.mar'
deleting '/data/update-channels/hamachi/2.0.0/nightly/b2g_update_20141003160205.mar'
deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140531040201.mar'
deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140530160203.mar'
deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140601160201.mar'
deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140531160201.mar'
deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140601040203.mar'
deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140602040201.mar'
deleting '/data/update-channels/hamachi/2.0.0/hamachi/2.0.0/nightly/b2g_update_20140602072051.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141003160209.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141004000205.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141005000204.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141004160205.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140928000204.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140927000203.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140927160203.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141005160206.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141002160203.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140930160205.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140929000203.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141006000205.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141001000202.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141001160202.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140929160206.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140928160204.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20140930000203.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141003000203.mar'
deleting '/data/update-channels/hamachi/2.1.0/nightly/b2g_update_20141002000202.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141001040205.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140927160205.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140928040208.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141006040204.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140928160202.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140930160207.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141003040207.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141003160204.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140929040202.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141002160204.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140930040206.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141005160202.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141001160203.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141002093155.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140926185533.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140927040203.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141004160202.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20140929160203.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141004040202.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141002040202.mar'
deleting '/data/update-channels/hamachi/2.2.0/nightly/b2g_update_20141005040205.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141002160206.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140928160203.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141004000201.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141003000205.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141005000201.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140929000204.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141002000206.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140929160207.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140927000205.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141006000201.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141003160202.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140930000202.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141005160201.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141001000204.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141001160204.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140927160202.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140930160200.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20141004160201.mar'
deleting '/data/update-channels/hamachi/1.4.0/nightly/b2g_update_20140928000202.mar'
deleting '/data/update-channels/hamachi/1.4.0/hamachi/1.4.0/nightly/b2g_update_20140602000203.mar'
deleting '/data/update-channels/hamachi/1.4.0/hamachi/1.4.0/nightly/b2g_update_20140601000202.mar'
deleting '/data/update-channels/hamachi/1.4.0/hamachi/1.4.0/nightly/b2g_update_20140531000203.mar'
deleting '/data/update-channels/hamachi/1.4.0/hamachi/1.4.0/nightly/b2g_update_20140530160206.mar'
deleting '/data/update-channels/hamachi/1.4.0/hamachi/1.4.0/nightly/b2g_update_20140531160202.mar'
deleting '/data/update-channels/hamachi/1.4.0/hamachi/1.4.0/nightly/b2g_update_20140601160203.mar'
deleting '/data/update-channels/hamachi/1.3.0/hamachi/1.3.0/nightly/b2g_update_20140602024006.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20140928160203.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141005000203.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141004000203.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141005160203.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141002160200.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20140930000204.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20140929000203.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20140929160205.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141001000205.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141004160203.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141003000204.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141002000208.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20140930160202.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141001160205.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141006000202.mar'
deleting '/data/update-channels/flame-kk/2.0.0/nightly/b2g_update_20141003160205.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141003160209.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141004000205.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141005000204.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141004160205.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141005160206.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141002160203.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20140930160205.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20140929000203.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141006000205.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141001000202.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141001160202.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20140929160206.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20140928160204.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20140930000203.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141003000203.mar'
deleting '/data/update-channels/flame-kk/2.1.0/nightly/b2g_update_20141002000202.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141001040205.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20140928160202.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20140930160207.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141003040207.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141003160204.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20140929040202.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141002160204.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20140930040206.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141005160202.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141001160203.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141002093155.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141004160202.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20140929160203.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141004040202.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141002040202.mar'
deleting '/data/update-channels/flame-kk/2.2.0/nightly/b2g_update_20141005040205.mar'
deleting '/data/update-channels/unagi/1.1.0/beta/b2g_beta_update_20130821041204.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141003160209.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141004000205.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141005000204.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141004160205.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140928000204.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140927000203.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140927160203.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141005160206.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141002160203.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140930160205.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140929000203.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141006000205.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141001000202.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141001160202.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140929160206.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140928160204.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20140930000203.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141003000203.mar'
deleting '/data/update-channels/dolphin/2.1.0/nightly/b2g_update_20141002000202.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141001040205.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140927160205.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140928040208.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140928160202.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140930160207.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141003040207.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141003160204.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140929040202.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141002160204.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140930040206.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141005160202.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141001160203.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141002093155.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140926185533.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140927040203.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141004160202.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20140929160203.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141004040202.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141002040202.mar'
deleting '/data/update-channels/dolphin/2.2.0/nightly/b2g_update_20141005040205.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141002160206.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140928160203.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141004000201.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141003000205.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141005000201.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140929000204.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141002000206.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140929160207.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140927000205.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141006000201.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141003160202.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140930000202.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141005160201.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141001000204.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141001160204.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140927160202.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140930160200.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20141004160201.mar'
deleting '/data/update-channels/dolphin/1.4.0/nightly/b2g_update_20140928000202.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140928160203.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140927000202.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141005000203.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141004000203.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141005160203.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141002160200.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140930000204.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140929000203.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140929160205.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141001000205.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140928000205.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141004160203.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140927160206.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141003000204.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141002000208.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20140930160202.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141001160205.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141006000202.mar'
deleting '/data/update-channels/flame/2.0.0/nightly/b2g_update_20141003160205.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141003160209.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141004000205.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141005000204.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141004160205.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140928000204.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140927000203.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140927160203.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141005160206.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141002160203.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140930160205.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140929000203.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141006000205.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141001000202.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141001160202.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140929160206.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140928160204.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20140930000203.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141003000203.mar'
deleting '/data/update-channels/flame/2.1.0/nightly/b2g_update_20141002000202.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141001040205.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140927160205.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140928040208.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140928160202.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140930160207.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141003040207.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141003160204.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140929040202.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141002160204.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140930040206.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141005160202.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141001160203.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141002093155.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140926185533.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140927040203.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141004160202.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20140929160203.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141004040202.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141002040202.mar'
deleting '/data/update-channels/flame/2.2.0/nightly/b2g_update_20141005040205.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141002160206.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140928160203.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141004000201.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141003000205.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141005000201.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140929000204.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141002000206.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140929160207.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140927000205.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141006000201.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141003160202.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140930000202.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141005160201.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141001000204.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141001160204.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140927160202.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140930160200.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20141004160201.mar'
deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140928000202.mar'
deleting '/data/update-channels/flame/1.4.0/flame/1.4.0/nightly/b2g_update_20140602000203.mar'
deleting '/data/update-channels/flame/1.4.0/flame/1.4.0/nightly/b2g_update_20140601000202.mar'
deleting '/data/update-channels/flame/1.4.0/flame/1.4.0/nightly/b2g_update_20140531000203.mar'
deleting '/data/update-channels/flame/1.4.0/flame/1.4.0/nightly/b2g_update_20140530160206.mar'
deleting '/data/update-channels/flame/1.4.0/flame/1.4.0/nightly/b2g_update_20140531160202.mar'
deleting '/data/update-channels/flame/1.4.0/flame/1.4.0/nightly/b2g_update_20140601160203.mar'
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda1             8256952   6355880   1817200  78% /
tmpfs                  3559600         0   3559600   0% /dev/shm
/dev/xvdf1            33025240   1565992  29781664   5% /data
[ec2-user@ip-10-39-70-110 ~]$

Comment 6

[Chris Cooper [:coop] (he/him)]

Making bug 1000217 a blocker: we'll be pushing a rock up a hill here until we stop uploading unneeded mars.

Comment 7

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #6)
> Making bug 1000217 a blocker: we'll be pushing a rock up a hill here until
> we stop uploading unneeded mars.

And by blocker, I of course meant dependency.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Perhaps the ec2-user cron is also busted. I don't see any email from the /home/ec2-user/check_disk job which runs every hour, and should have warned us that the disk was filling up. And files like 
> deleting '/data/update-channels/flame/1.4.0/nightly/b2g_update_20140927000205.mar'
should've been deleted by 
@daily find /data/update-channels/{hamachi,helix,leo,nexus-4,unagi,flame,flame-kk,dolphin}/*/nightly -type f -mtime +1 \( -name '*.xml' -o -name '*.ini' -o -name '*.mar' \) -delete

Comment 9

[Phil Ringnalda (:philor)]

Might want to fix this at some point. It's full again, and if I didn't lose count along the way by the time I killed the pointless retries just now, we had built 210 pointless nightlies (or maybe 200, if you call the 1.3t ones non-pointless).

Comment 10

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-10-18T11:08:21
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-central&job_id=503993
repository: mozilla-central
who: philringnalda[at]gmail[dot]com
machine: unknown
buildname: b2g_mozilla-central_dolphin_nightly
revision: 33c0181c4a25

Comment 11

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-10-18T11:08:21
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-central&job_id=503999
repository: mozilla-central
who: philringnalda[at]gmail[dot]com
machine: unknown
buildname: b2g_mozilla-central_hamachi_nightly
revision: 33c0181c4a25

Comment 12

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-10-18T11:08:21
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-central&job_id=503997
repository: mozilla-central
who: philringnalda[at]gmail[dot]com
machine: unknown
buildname: b2g_mozilla-central_flame-kk_nightly
revision: 33c0181c4a25

Comment 13

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-10-18T11:08:21
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-central&job_id=503998
repository: mozilla-central
who: philringnalda[at]gmail[dot]com
machine: unknown
buildname: b2g_mozilla-central_nexus-4_nightly
revision: 33c0181c4a25

Comment 14

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-10-18T11:08:21
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-central&job_id=503995
repository: mozilla-central
who: philringnalda[at]gmail[dot]com
machine: unknown
buildname: b2g_mozilla-central_helix_nightly
revision: 33c0181c4a25

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Cleaned up again by running the cron jobs manually:

$ df -h /data/
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvdf1             32G  3.6G   27G  12% /data


Cron is definitely busted:
Sep 27 19:00:01 ip-10-39-70-110 crond[23532]: (ec2-user) FAILED to authorize user with PAM (Module is unknown)
Sep 27 19:01:01 ip-10-39-70-110 crond[23533]: (root) FAILED to authorize user with PAM (Module is unknown)

Restarted the cron service (per google results) and now it's working ok.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

BTW, probably got broken when we upgraded packages for one of the recent security issues.