537836 - update linux64 talos ref image

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[alice nodelman [:alice] [:anode]]

To be fixed:

a) apache logs out of control
b) twistd.log files out of control
c) switch to cltbld user

Comment 1

[Phong Tran [:phong]]

talos-rev3-x64-ref.mv.mozilla.com  10.250.48.227

Comment 2

[alice nodelman [:alice] [:anode]]

I believe that I killed the machine by attempting a reboot.  Which is something that we've hit with the other linux image - we'll have to figure out a way to reliably reboot or switch to another distro.

Comment 3

[Phong Tran [:phong]]

you most certainly did. it should be up and running again.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

taking bug while i get a base fedora image up

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

https://wiki.mozilla.org/User:John64/Talos-Fedora12 has the latest copy of the ref image document.  

the linux32 ref image will be following the same steps. Some things to think about:
-There are debuginfo packages that might be useful for stacks, they aren't installed
-Because we can't overwrite the system python installation we put a python2.5 installation in /home/cltbld/ .  the 'python' command is symlinked only for cltbld.  python --version will return 2.5.4 for cltbld and 2.6.X for root.  The versioned binaries (python2.5 and python2.6) will still work.

Comment 6

[[e-mail] (:bhearsum)]

We should get Puppet on this machine - what's the hostname?

Comment 7

[[e-mail] (:bhearsum)]

Attachment: update buildbot-tac.py for fedora talos machines

Comment 8

[[e-mail] (:bhearsum)]

Attachment: update puppet configs for new linux/mac talos slaves - with the right hostnames

I'm putting this here because it's as good of a place as any - it contains the updates to the production Puppet configs needed for all the new Linux and Mac Talos slaves.

Comment 9

[[e-mail] (:bhearsum)]

For the record, the IP of this machine is 10.250.2.200

Comment 10

[[e-mail] (:bhearsum)]

New ip: 10.250.48.241

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 422394 [details] [diff] [review]
update puppet configs for new linux/mac talos slaves - with the right hostnames

changeset:   89:adde60ad69fb

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 422385 [details] [diff] [review]
update buildbot-tac.py for fedora talos machines

changeset:   481:077fb60c8e35

Comment 13

[[e-mail] (:bhearsum)]

Here's what I just did:
* Installed Puppet ('yum install puppet')
* Dropped on the run-puppet-and-buildbot.sh script (to be posted)
* Added the following lines to /etc/sudoers, with 'visudo':
cltbld ALL=NOPASSWD: /sbin/reboot
cltbld ALL=NOPASSWD: /bin/bash /home/cltbld/run-puppet-and-buildbot.sh
production-puppet.build.mozilla.org
* Changed the 'talos-slave' startup job to run run-puppet-and-buildbot.sh
instead of buildbot.
* Put the latest version of the build/tools repo in /tools/build-tools
* Installed the buildbot-tac init script

Comment 14

[[e-mail] (:bhearsum)]

I'm done with this VM now, too.

Comment 15

[John Ford [:jhford] CET/CEST Berlin Time]

putting back in the pool as I am no longer doing anything on this.

Comment 16

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Not sure if this is caused by refimage or something else. Getting a lot of errors like this, for various different talos suites:

=====
Subject: buildbot failure in Talos on Fedora 12x64 mozilla-1.9.2 talos nochrome
Date: Tue, 26 Jan 2010 19:08:41 -0800

The Buildbot has detected a failed build of Fedora 12x64 mozilla-1.9.2 talos nochrome on Talos.
Full details are available at:
 http://talos-master.mozilla.org:8012builders/Fedora%2012x64%20mozilla-1.9.2%20talos%20nochrome/builds/6

Buildbot URL: http://talos-master.mozilla.org:8012

Buildslave for this Build: talos-r3-fed64-012

Build Reason: 
Build Source Stamp: unavailable
Blamelist: sendchange

BUILD FAILED: failed cleanup slave lost

sincerely,
 -The Buildbot

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

/bin/sh -c nohup rm -vrf *; cat nohup.out
 in dir /home/cltbld/talos-slave/mozilla-1.9.2-fedora64-nochrome/../talos-data 
...
 closing stdin
 using PTY: True
nohup: ignoring input and appending output to `nohup.out'
cat: nohup.out: No such file or directory
program finished with exit code 1

and then it continues to do the rest of the steps in the job.

Comment 18

[Aki Sasaki (not active)]

/bin/sh -c nohup rm -vrf *; if [ -f nohup.out ] ; then cat nohup.out; fi

?

(i am not a hack)

also, does the nohup rm -vrf nuke nohup.out if it exists but create it if it doesn't?

Comment 19

[alice nodelman [:alice] [:anode]]

re-claiming this bug.

Comment 20

[alice nodelman [:alice] [:anode]]

This is all done.  We are using the fedora image right now on rev3 machines.