569790 - Maemo Try slaves don't get updates from puppet

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Exception on 0.8.0rc3

Several issues here
* the slaves (try-linux-slave02 and 03) have different version of buildbot, because they're not rebooting and hence not running puppet
* as a result try-linux-slave03 still has buildbot 0.8.0rc3, and 02 still has 0.7.10p1
* on 03 we are failing to scp the files up to build.m.o because the path is a unicode string. The exception in the slave log is the attached with this comment.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Potential fix

Sprinkle some str() around so that buildbot 0.8.0 doesn't barf. This is almost deprecated code, only maemo is using it still.

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 448957 [details] [diff] [review]
Potential fix

str() should fix it. I had the similar problems passing int to setCommand without wrapping it with str().

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 448957 [details] [diff] [review]
Potential fix

Letting puppet update the two slaves gets green builds without this patch. Thanks for the r+ anyway.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Morphing this now ...

The remaining issues are
 * these two slaves don't reboot after a build, so they never run puppet to pick up fixes
 * anything distributed via http://production-puppet doesn't work because the firewall doesn't allow that traffic out of the sandbox network

The latter means we don't install the yasm rpm.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Rebooting after each build

This should get us polling puppet.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Bug 569806 to open the firewall up.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to comment #6)
> Bug 569806 to open the firewall up.

Thanks for catching this, Nick.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 448967 [details] [diff] [review]
Rebooting after each build

http://hg.mozilla.org/build/buildbot-configs/rev/7a767539bd27

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

sm-try-master reconfig'd but there was an exception:
2010-06-03 14:30:39-0700 [Broker,16,10.2.76.36] BuildSlave.sendBuilderList (<BuildSlave 'try-mac-slave04', current builders: >) failed
2010-06-03 14:30:39-0700 [Broker,16,10.2.76.36] Unhandled Error
	Traceback (most recent call last):
	Failure: twisted.spread.pb.PBConnectionLost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.

which is odd given there are no mac builders anymore, and this master had a restart after they got removed. Also no 'configuration update finished', but it looks like the maemo factory got updated. Letting it run before more drastic action.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Factory was updated. There's a ~7 minute hang in puppet before it gives up on pulling the yasm rpm, which will go away when the firewall is modified.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

These two slaves are up to date and will stay that way.