1077869 - docker-worker: Formatting instance storage fails intermittently

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[James Lal [:lightsofapollo]]

At least once now (see node i-75a12c7a in papertrail) I have seen a worker node start without actually booting the docker-worker and pulling tasks off the queue. We should investigate why this is happening and actually iron out what we would like the aws-provisioner to do to nodes which become "zombies".

Comment 1

[James Lal [:lightsofapollo]]

also i-330e833c, i-cf0b86c0, i-0e0d8001 (this one did not even register a hostname on papertrail!), i-a2179aad, i-aa2ba6a5, i-e93db0e6, i-3856da37.

Clearly something is preventing upstart from even _starting_ the docker-worker/docker (it starts diamond at least in some of these cases).

Comment 2

[Greg Arndt [:garndt]]

Appears that the logical volume in the upstart job is getting created, but then the next step to format it is not happening.  Not sure why yet.  I didn't find anything in the logs indicating a larger failure.

Zombie instance:

root@ip-172-31-4-29:/home/ubuntu# lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL
NAME                          FSTYPE      SIZE MOUNTPOINT LABEL
xvda                                        8G            
└─xvda1                       ext4          8G /          cloudimg-rootfs
xvdb                          LVM2_member  75G            
└─instance_storage-all (dm-0)              75G       


Good instance:

NAME                          FSTYPE      SIZE MOUNTPOINT LABEL
xvda                                        8G            
└─xvda1                       ext4          8G /          cloudimg-rootfs
xvdb                          LVM2_member  75G            
└─instance_storage-all (dm-0) ext4         75G /mnt

Comment 3

[James Lal [:lightsofapollo]]

Hrm I wonder if this is something we can retry somehow... It looks like the process was at least partially completed.

Comment 4

[Greg Arndt [:garndt]]

We should be able to capture the status code and retry a few times.  I can add some logic around it and perhaps some useful logging to try to narrow down the conditions a failure happens and to at least let us know that's exactly what failed and was retried.

Comment 5

[James Lal [:lightsofapollo]]

++ I think we need some kind of retry logic for this particular case we may or may not have to destroy the lvm "disk" and start over... I also wonder if this is actually failing in ext4 formatting if there maybe is some verbose logging we can try.

Comment 6

[Jonas Finnemann Jensen (:jonasfj)]

It seems to me we should figure out why it fails sometimes.. This isn't a network thing it's not suppose to onyl work "sometimes".

If we can detect failure to format disk right we can just force shutdown. Thought this can create a spawn/shutdown loop... I hate those :)

Comment 7

[Greg Arndt [:garndt]]

We absolutely should figure out why it's failing.  Magical failing things are too mysterious for my liking :)

Comment 8

[Greg Arndt [:garndt]]

Attachment: GH PR 54

Over the course of a couple of days have spawns over a thousand nodes using a graph of 170 tasks and haven't had a zombie yet.  Perhaps it could be tested out with the tasks that were running on the gaia worker type.  I created this ami with the changes: ami-25185715

Comment 9

[Greg Arndt [:garndt]]

More info about the error..  It appears that the upstart job that created the logical volume was being respawned the first time because mkfs could not start because the system was reporting the volume was currently in use.  Every respawn after that failed because the commands to create the logical volume would fail because the LV was already present triggering subsequent respawns and never attempting mkfs again.

Comment 10

[James Lal [:lightsofapollo]]

Comment on attachment 8506989 [details] [review]
GH PR 54

Few comments here ... can you flag me again for r? after reviewing my comments

Comment 11

[James Lal [:lightsofapollo]]

r+ 

https://github.com/taskcluster/docker-worker/commit/d2d14ac0a5aae59f249e207bb1d862526c4815ad