701179 - moz2-linux-slave10 is down

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Armen [:armenzg]]

We should able to lift this machine up with vcenter.

Comment 1

[Chris Cooper [:coop] (he/him)]

I was already connected to admin.build.m.o, so I reset this VM.

Comment 2

[Chris Cooper [:coop] (he/him)]

Double-checked this slave this morning when I noticed it still wasn't connected and saw it was having problems running puppet. Looks like it got into an indeterminate state WRT scratchbox (die, scratchbox, die). Swapping in the old scratchbox.deleteme/ dir briefly allowed it to complete it's puppet run and get back into service.

Comment 3

[Armen [:armenzg]]

Both moz2-linux-slave10 and moz2-linux-slave17 seem to be down.

Comment 4

[Armen [:armenzg]]

Shall we file a different bug? or reuse this one?

Comment 5

[Armen [:armenzg]]

It is also unable to free up enough space:
"Error: unable to free 6.00 GB of space. Free space only 2.86 GB"

Comment 6

[Aki Sasaki (not active)]

We should probably allow releases to be clobbered on staging slaves after X amount of time.

Comment 7

[Chris Cooper [:coop] (he/him)]

(In reply to Aki Sasaki [:aki] from comment #6)
> We should probably allow releases to be clobbered on staging slaves after X
> amount of time.

Let's file a new bug for that then.

As I mentioned to Armen in IRC, the extreme lack of space is likely why the initial puppet setup was failing for scratchbox on moz2-linux-slave10. That's a pretty large package IIRC.

Comment 8

[Mike Taylor [:bear]]

I had to manually clear space on this slave yesterday to get it back into the cycle.