537748 - update leopard talos ref image

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[alice nodelman [:alice] [:anode]]

With the new hardware we should take the opportunity to:

1) update the latest dot release for leopard/tiger
2) change user from mozqa to cltbld

Comment 1

[alice nodelman [:alice] [:anode]]

Leopard upgraded to 10.5.8 also:

- changed to cltbld user
- clean up twistd.logs on reboot
- clean up error_log apache log on reboot

Looks like tiger ref image won't need to be updated immediately as tiger will not run on the new hardware.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: [wip] buildbot-tac init script updates for new talos machines

Comment 3

[[e-mail] (:bhearsum)]

* Installed Puppet
* Modified buildbot launcher NOT to run on boot
* Installed Puppet launcher that runs once, waits for exit, then runs Buildbot
* Installed buildbot-tac generator, with init script patch from attachment
* Modified buildbot plist file for new username

After spending 3 hours finding and fixing little bugs I got all of the above done. This machine is running in staging over the weekend now.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to comment #3)
> After spending 3 hours finding and fixing little bugs I got all of the above
> done. This machine is running in staging over the weekend now.

This comment was posted too soon. I thought I fixed the last issues, but I still haven't been able to get Buildbot launching after Puppet runs. I'm continuing to look for a solution to this this morning.

Comment 5

[[e-mail] (:bhearsum)]

Finally got it working, after learning about WatchPaths for launchd plist files. This machine is running in staging now.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: updated buildbot plist file

This is the updated version of the buildbot plist file that will prevent buildbot from launching until Puppet has finished. WatchPaths will attempt to launch buildbot whenever that file is modified. Note that we don't have a StartInterval anymore - I doubt launching this way will be flakey like launching at startup was. Incoming is the Puppet launcher that touches this file after Puppet runs (regardless of success or failure).

Comment 7

[[e-mail] (:bhearsum)]

Attachment: Puppet launcher

Here's the script that launches Puppet. When Puppet runs for the first time on a new slave it will return 1 because the certificate will not be signed yet. Therefore, we run in a loop until the certificate gets signed and Puppet runs. By doing this we ensure that the slave will be up to date before it accepts any jobs.

Comment 8

[[e-mail] (:bhearsum)]

Attachment: Puppet plist launcher

This is pretty similar to the one we use for builds. We use the StartInterval/LaunchOnlyOnce combo to make sure it gets run if the first attempt doesn't get through. Note that we'll need a second copy of this when we have some staging slaves.

Comment 9

[[e-mail] (:bhearsum)]

Attachment: buildbot-tac updates for talos

Password stuff has been moved up, because we need the same logic to handle the tac file.
Might need to update the IGNORE_HOSTS - I don't know yet what we're doing for a Leopard v3 image, we may just continue using qm-leopard-ref.

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 421042 [details] [diff] [review]
updated buildbot plist file

Details in comment #6

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 421043 [details] [diff] [review]
Puppet launcher

Details in comment #7

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 421044 [details]
Puppet plist launcher

Details in comment #8

Comment 13

[Chris AtLee [:catlee]]

Comment on attachment 421044 [details]
Puppet plist launcher

tiny nit: I wouldn't call the script 'run-puppet-and-buildbot.sh', since it's not actually running buildbot.

Comment 14

[[e-mail] (:bhearsum)]

Just hit a problem with FileDownload on the dirty profiles on this machine. I don't quite understand why though - we're running the same Buildbot as the other slaves. Looking into it.

Comment 15

[[e-mail] (:bhearsum)]

Got the issue sorted out. Turns out that OS X 10.5.8 comes with Twisted 2.5.0 (10.5.2 comes with Twisted 2.4.0). After installing Twisted 2.4.0 and adjusting PYTHONPATH I managed to get the dirty profile tests working again. It's now running in staging. Initial numbers are looking good. There's a bit more deviation between runs, but nothing horrible, AFAICT:
leopard mozilla-1.9.1	dhtml	856.1, 852.6
svg_opacity	12174.5, 12190.5



More in the spreadsheet:
http://spreadsheets.google.com/ccc?key=0AhwotqoBm6lEdHJLN2w0U2dTa1o2bVpOLXNWZF9ZM0E&hl=en

Comment 16

[[e-mail] (:bhearsum)]

Just had a look at more numbers for talos-r3-leopard01 and the deviation shown is within the normal range (a couple ms for most tests, 100s of ms for nochrome gfx and svg_opacity).

However, there's a period of time overnight where the slave went through a ton of rapid disconnects/reconnects (from roughly 11:40pm PST yesterday to 5am PST today). I'm hoping this happening because of some machine futzing last night, but we'll see...

Comment 17

[[e-mail] (:bhearsum)]

Looks like somebody rebooted it:
Jan 12 23:48:19 talos-r3-leopard01 sudo[6666]:   cltbld : TTY=ttys001 ; PWD=/Users/cltbld/talos-slave/talos-data ; USER=root ; COMMAND=/sbin/reboot
Jan 12 23:48:20 talos-r3-leopard01 reboot[6666]: rebooted by cltbld
Jan 12 23:48:20 talos-r3-leopard01 reboot[6666]: SHUTDOWN_TIME: 1263368900 9105

Dunno why this would cause failures for 6 hours, though.

Comment 18

[[e-mail] (:bhearsum)]

Attachment: buildbot plist - set PYTHONPATH

Comment 19

[[e-mail] (:bhearsum)]

Comment on attachment 421043 [details] [diff] [review]
Puppet launcher

Landed with a better name:

Checking in run-puppet.sh;
/mofo/puppet-files/talos/mac/run-puppet.sh,v  <--  run-puppet.sh
initial revision: 1.1
done

Comment 20

[[e-mail] (:bhearsum)]

Comment on attachment 421044 [details]
Puppet plist launcher

Landed, with the name adjusted:
Checking in com.reductivelabs.puppet.plist;
/mofo/puppet-files/talos/mac/com.reductivelabs.puppet.plist,v  <--  com.reductivelabs.puppet.plist
initial revision: 1.1
done

Comment 21

[[e-mail] (:bhearsum)]

Comment on attachment 421054 [details] [diff] [review]
buildbot-tac updates for talos

buildbot-tac updates for talos

Comment 22

[[e-mail] (:bhearsum)]

(In reply to comment #16)
> Just had a look at more numbers for talos-r3-leopard01 and the deviation shown
> is within the normal range (a couple ms for most tests, 100s of ms for nochrome
> gfx and svg_opacity).
> 
> However, there's a period of time overnight where the slave went through a ton
> of rapid disconnects/reconnects (from roughly 11:40pm PST yesterday to 5am PST
> today). I'm hoping this happening because of some machine futzing last night,
> but we'll see...

Turns out that Phong had taken the machine down and cloned it to another one due to a miscommunication - so we had two machines trying to connect with the same name. So - this is a non-issue.

Comment 23

[[e-mail] (:bhearsum)]

Attachment: update puppet configs for new leopard slaves

Nothing big here - just need to make Puppet aware of the new and incoming machines.

Comment 24

[[e-mail] (:bhearsum)]

Comment on attachment 421503 [details] [diff] [review]
update puppet configs for new leopard slaves

changeset:   86:ba716aca3f32

Comment 25

[[e-mail] (:bhearsum)]

Phong, talos-r3-leopard-ref is ready to have its image taken. Feel free to shut it down whenever you're ready to take it.

A couple notes about imaging:
Please use 3 digit numbering - talos-r3-leopardNNN
Please be sure to set the full hostname with:
scutil --set HostNAme talos-r3-leopardNNN.mozilla.org

Our Puppet configs are dependent on that getting set properly.

Comment 26

[[e-mail] (:bhearsum)]

Comment on attachment 421446 [details]
buildbot plist - set PYTHONPATH

Checking in buildbot.start.talos.slave.plist;
/mofo/puppet-files/talos/mac/buildbot.start.talos.slave.plist,v  <--  buildbot.start.talos.slave.plist
initial revision: 1.1
done

Comment 27

[[e-mail] (:bhearsum)]

I'm pretty sure we're all done here.

Comment 28

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Per irc with catlee, not sure if this is caused by refimage or something else. Getting a lot of errors like this, for various different talos suites:
=====
-------- Original Message --------
Subject: buildbot failure in Talos on MacOSX Leopard 10.5.8 mozilla-1.9.0 talos
Date: Tue, 26 Jan 2010 19:13:04 -0800


The Buildbot has detected a failed build of MacOSX Leopard 10.5.8 mozilla-1.9.0 talos on Talos.
Full details are available at:
 http://talos-master.mozilla.org:8012builders/MacOSX%20Leopard%2010.5.8%20mozilla-1.9.0%20talos/builds/268

Buildbot URL: http://talos-master.mozilla.org:8012

Buildslave for this Build: talos-r3-leopard-011

Build Reason: 
Build Source Stamp: unavailable
Blamelist: http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/bm-xserve08-mozilla1.9.0/1264560300/

BUILD FAILED: failed cleanup slave lost

sincerely,
 -The Buildbot