855661 - (bld-lion-r5-049) bld-lion-r5-049 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Ryan VanderMeulen [:RyanVM]]

Builds are dying for various reasons.

https://tbpl.mozilla.org/php/getParsedLog.php?id=21199035&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=21200072&tree=Mozilla-Inbound

Comment 1

[Ed Morley [:emorley]]

Disabled in slavealloc.

Comment 2

[[e-mail] (:bhearsum)]

I don't think this slave is having problems. The first log shows it hitting bug 753223. The second one shows a missing file. That build was the second attempt to build the same rev on that machine, the first failing with bug 753223 again (which probably screwed up the objdir and broke the second attempt). I clobbered all of its build directories put this slave back into production.

Comment 3

[Phil Ringnalda (:philor)]

Well, now it has a problem, though I can't remember whether this is a problem fixed by a reboot, or a reimage; burned its last four attempts to build with

configure:4778: checking for minimum required perl version >= 5.006
perl: no user 28
configure: error: Perl 5.006 or higher is required.
*** Fix above errors and then restart with               "make -f client.mk build"

Disabled in slavealloc.

Comment 4

[Armen [:armenzg]]

Re-image and reboot triggered.

Next action: re-enable in slavealloc.

Comment 5

[Pete Moore [:pmoore][:pete]]

Added ssh keys for cltbld user.

Comment 6

[Armen [:armenzg]]

Boo me:
https://wiki.mozilla.org/ReleaseEngineering/How_To/Set_Up_a_Freshly_Imaged_Slave#Linux_.26_MacOS_X_.28puppet_again.29
"Build machines need to have their ssh keys checked"

We really need pre-flight tasks and remove humans from not following instructions!

Comment 7

[Armen [:armenzg]]

Added back after checking the keys.

Comment 8

[[e-mail] (:bhearsum)]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #6)
> Boo me:
> https://wiki.mozilla.org/ReleaseEngineering/How_To/
> Set_Up_a_Freshly_Imaged_Slave#Linux_.26_MacOS_X_.28puppet_again.29
> "Build machines need to have their ssh keys checked"
> 
> We really need pre-flight tasks and remove humans from not following
> instructions!

bug 792836 is fixing slave secret management