1445445 - DEV-FAILED for deploys of balrog-admin on code changes

Status: RESOLVED FIXED

(Cloud Services :: Operations: Product Delivery, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

(not sure this is the right component)

The last few landings of Balrog code in the Github have resulted in #balrog irc messages like this:

10:38 <cloudops-ansible> balrog-web #71: master-bdf053f836e53d366aeb0b37dce7c7317bdb2dcd DEV STARTED
10:43 <cloudops-ansible> balrog-web #71: master-bdf053f836e53d366aeb0b37dce7c7317bdb2dcd DEV COMPLETE
10:43 <cloudops-ansible> balrog-admin #75: master-bdf053f836e53d366aeb0b37dce7c7317bdb2dcd DEV FAILED
10:48 <cloudops-ansible> balrog-admin #76: master-c847a51aee23f5ea9537a0339816777f5b887f55 DEV STARTED
10:51 <cloudops-ansible> balrog-admin #76: master-c847a51aee23f5ea9537a0339816777f5b887f55 DEV FAILED
10:51 <cloudops-ansible> balrog-web #72: master-c847a51aee23f5ea9537a0339816777f5b887f55 DEV COMPLETE

In particular the two DEV-FAILED for balrog-admin. 

bhearsum notes that the dev instances do report the current tip of master at
 https://balrog-admin.dev.mozaws.net/__version__
 https://aus5.dev.mozaws.net/__version__

Could you please investigate the CI logs to see what's up.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

oremj, do you have some time to look into this ?

Comment 2

[Jeremy Orem [:oremj]]

Fixed, it was just failing to remove old docker images.

Comment 3

[[e-mail] (:bhearsum)]

It looks like there's still some issues here:
14:47 -!- cloudops-ansible [[e-mail]] has joined #balrog
14:47 < cloudops-ansible> balrog-web #89: master-4e695f6f765d5b82c3332e765a2cd618cef63e78 DEV FAILED
14:47 -!- cloudops-ansible [[e-mail]] has left #balrog []
14:48 -!- cloudops-ansible [[e-mail]] has joined #balrog
14:48 < cloudops-ansible> balrog-admin #93: master-4e695f6f765d5b82c3332e765a2cd618cef63e78 DEV FAILED


admin still ends up with the new version, but web doesn't

Comment 4

[[e-mail] (:bhearsum)]

Bump?

Comment 5

[[e-mail] (:bhearsum)]

22:51 < jbuck> migration failed
22:52 < jbuck>  /usr/bin/docker-current: failed to register layer: devmapper: Thin Pool has 4316 free data blocks which is less than minimum required 4514 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change 
               behavior.
22:52 < jbuck> not enough space again maybe?
22:57 <~bhearsum> seems plausible
23:04 < jbuck> bhearsum: should be fixed for admin
23:04 < jbuck> I couldn't run the docker-gc because there wasn't enough space, heh
23:05 < jbuck> hmm
23:05 < jbuck> web has enough room I think though
23:07 <~bhearsum> hah
23:07 <~bhearsum> should we run docker-gc in a cronjob or something?
23:07 < jbuck> yeah
23:11 < jbuck> error: cannot open Packages index using db5 -  (-30973)\r\nerror: cannot open Packages database in /var/lib/rpm
23:11 < jbuck> that's what's wrong with web

Comment 6

[Jeremy Orem [:oremj]]

Looks like this is fixed again. Let me know when this happens again and I'll see if I can figure out why the clean up task is not running.