664144 - set-up dev-stage01

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Please open a relops bug assigned to me to delete the old staging-stage system, when that's ready to happen.  Let me know if you need a backup.

Please file another relops bug to redirect monitoring from the old to the new system, too.

Comment 3

[[e-mail] (:bhearsum)]

To install Puppet, did (from https://wiki.mozilla.org/ReferencePlatforms/Linux-CentOS-5.0_64-bit#Puppet_Installation):
scp root@moz2-linux64-slave03:/var/cache/yum/epel/packages/*.rpm .
scp root@moz2-linux64-slave03:/var/cache/yum/updates/packages/*.rpm .
rpm -i ruby-libs-1.8.5-5.el5_4.8.x86_64.rpm
rpm -i ruby-1.8.5-5.el5_4.8.x86_64.rpm 
rpm -i augeas-libs-0.7.0-1.el5.x86_64.rpm
rpm -i facter-1.5.7-1.el5.noarch.rpm
rpm -i ruby-augeas-0.3.0-1.el5.x86_64.rpm 
rpm -i ruby-shadow-1.4.1-7.el5.x86_64.rpm
rpm -i puppet-0.24.8-4.el5.noarch.rpm

Then:
puppetd --test --server staging-puppet.build.mozilla.org
(accept key on the master)
puppetd --test --server staging-puppet.build.mozilla.org

I found that the /usr/local/bin/tools.sh doesn't quite work, so I modified it slightly in a way that works for this machine and preproduction-stage.

I had to run it twice, because the stage manifests don't properly require the user accounts. In any case, the second run was clean.

Now, I'm running tests in staging against it.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: support 64-bit stage machines, turn off iptables

These are the changes I needed to make to get dev-stage01 up and running.

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 542171 [details] [diff] [review]
support 64-bit stage machines, turn off iptables

Hmmm, this patch causes us to use preproduction ~/.ssh/authorized_keys files instead of staging ones. Need to figure out something here.

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 542171 [details] [diff] [review]
support 64-bit stage machines, turn off iptables

Oops, I was confusing dev-stage01 with dev-master01. This should still be valid.

Comment 7

[[e-mail] (:bhearsum)]

Here's what I've done so far
- Get dev-stage01 set-up through Puppet
- Start running builds & tests through dev-stage01
- Update known_hosts files in Puppet

Still to do:
- Land update puppet manifests
- Update known_hosts files in OPSI
- Look over build/test results
- Change staging configs to point at dev-stage01

Comment 8

[[e-mail] (:bhearsum)]

Attachment: add dev-stage01 to ssh-config OPSI package

Comment 9

[[e-mail] (:bhearsum)]

Attachment: update post_upload.py

Comment 10

[[e-mail] (:bhearsum)]

Attachment: s/staging-stage.build.mozilla.org/dev-stage01.build.sjc1.mozilla.com/g in buildbot-configs

Comment 11

[[e-mail] (:bhearsum)]

Attachment: fix tools.sh to work with different python versions

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

I created some directories so that cron would stop spamming:

find: /data/ftp/pub/firefox/nightly/20??/: No such file or directory
find: /data/ftp/pub/xulrunner/nightly/20??/: No such file or directory
find: /data/ftp/pub/mobile/nightly/20??/: No such file or directory

find: /data/ftp/pub/firefox/nightly/latest-*: No such file or directory
find: /data/ftp/pub/xulrunner/nightly/latest-*: No such file or directory
find: /data/ftp/pub/mobile/nightly/latest-*: No such file or directory

First and third with ffxbld ownership, second with xrbld.

Comment 13

[Rail Aliiev [:rail]]

Comment on attachment 542171 [details] [diff] [review]
support 64-bit stage machines, turn off iptables

Review of attachment 542171 [details] [diff] [review]:
-----------------------------------------------------------------

::: stage/stage-rpms.pp
@@ +8,5 @@
> +                    source => "${platform_httproot}/RPMs/cvs-1.11.22-7.el5.x86_64.rpm";
> +                "python25":
> +                    source => "${platform_httproot}/RPMs/python25-2.5.1-0moz1.x86_64.rpm";
> +                "mercurial":
> +                    source => "${platform_httproot}/RPMs/mercurial-1.1.2-0moz1.x86_64.rpm",

mercurial-1.1.2 looks a little bit outdated. Mind to use a fresher version?

Comment 14

[Rail Aliiev [:rail]]

Comment on attachment 542239 [details] [diff] [review]
s/staging-stage.build.mozilla.org/dev-stage01.build.sjc1.mozilla.com/g in buildbot-configs

Hmm, it doesn't look just like s/staging-stage.build.mozilla.org/dev-stage01.build.sjc1.mozilla.com/g, there are some new files added. Seems like you have broken symlinks. Are those files supposed to be standalone files instead of symlinks?

Comment 15

[[e-mail] (:bhearsum)]

(In reply to comment #12)
> I created some directories so that cron would stop spamming:
> 
> find: /data/ftp/pub/firefox/nightly/20??/: No such file or directory
> find: /data/ftp/pub/xulrunner/nightly/20??/: No such file or directory
> find: /data/ftp/pub/mobile/nightly/20??/: No such file or directory
> 
> find: /data/ftp/pub/firefox/nightly/latest-*: No such file or directory
> find: /data/ftp/pub/xulrunner/nightly/latest-*: No such file or directory
> find: /data/ftp/pub/mobile/nightly/latest-*: No such file or directory
> 
> First and third with ffxbld ownership, second with xrbld.

Thanks Nick, sorry about that.

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 542228 [details] [diff] [review]
add dev-stage01 to ssh-config OPSI package

Landed this, and set the new version of the package to roll out to all Windows build machines (staging and production).

Comment 17

[[e-mail] (:bhearsum)]

Comment on attachment 542239 [details] [diff] [review]
s/staging-stage.build.mozilla.org/dev-stage01.build.sjc1.mozilla.com/g in buildbot-configs

I'm not sure what happened here...maybe I forgot to rebase my branch. I'll repost.

Comment 18

[[e-mail] (:bhearsum)]

Attachment: configs patch, without the cruft

Turns out that using sed -i on symlinks is a bad thing.

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 542470 [details] [diff] [review]
configs patch, without the cruft

looks good!

Comment 20

[[e-mail] (:bhearsum)]

Attachment: puppet manifests, using Mercurial 1.6.3

Comment 21

[[e-mail] (:bhearsum)]

Comment on attachment 542474 [details] [diff] [review]
puppet manifests, using Mercurial 1.6.3

Landed this, and dev-stage01 is now successfully syncing against mpt-p-p.

Comment 22

[[e-mail] (:bhearsum)]

Rail, I'm poking at the existing crontabs and wondering what this line is for:
11 * * * *      find /data/ftp/pub/{firefox,xulrunner,mobile}/nightly/ -mindepth 2 -maxdepth 2 -name build? -type d -mtime +1 -exec rm -rf {} \;


There's other lines that clean up dated and latest dirs, so this would seem to only affect candidates directories. Do we really want to do be doing that, even on preproduction? If so, we'll have to cope with dev-stage01 and preproduction-stage having different requirements.

Comment 23

[[e-mail] (:bhearsum)]

(In reply to comment #22)
> Rail, I'm poking at the existing crontabs and wondering what this line is
> for:
> 11 * * * *      find /data/ftp/pub/{firefox,xulrunner,mobile}/nightly/
> -mindepth 2 -maxdepth 2 -name build? -type d -mtime +1 -exec rm -rf {} \;
> 
> 
> There's other lines that clean up dated and latest dirs, so this would seem
> to only affect candidates directories. Do we really want to do be doing
> that, even on preproduction? If so, we'll have to cope with dev-stage01 and
> preproduction-stage having different requirements.

I'm also curious about:
1 * * * *       find /data/ftp/pub/{firefox,xulrunner,mobile}/tinderbox-builds/ -mindepth 2 -maxdepth 2 -type d -mmin -43200 -name 1????????? -exec rm -rf {} \;

and other jobs that use negative numbers. It seems the intention here is to clean up after 60 hours, but this seems to match _all_ files that meet the other requirements. Eg:
[ffxbld@dev-stage01 tinderbox-builds]$ pwd
/home/ftp/pub/firefox/tinderbox-builds
[ffxbld@dev-stage01 tinderbox-builds]$ date
Tue Jun 28 11:30:34 PDT 2011
[ffxbld@dev-stage01 tinderbox-builds]$ ls -l mozilla-central-macosx64-debug/
total 4
drwxr-xr-x 2 ffxbld firefox 4096 Jun 28 11:01 1309280210
[ffxbld@dev-stage01 tinderbox-builds]$ find /data/ftp/pub/{firefox,xulrunner,mobile}/tinderbox-builds/ -mindepth 2 -maxdepth 2 -type d -mmin -43200 -name 1?????????
/data/ftp/pub/firefox/tinderbox-builds/mozilla-central-macosx64-debug/1309280210


It looks like these should be positive values based on some manual tests:
[ffxbld@dev-stage01 tinderbox-builds]$ find /data/ftp/pub/{firefox,xulrunner,mobile}/tinderbox-builds/ -mindepth 2 -maxdepth 2 -type d -mmin +43200 -name 1?????????
/data/ftp/pub/firefox/tinderbox-builds/mozilla-central-macosx64-debug/1309280210
[ffxbld@dev-stage01 tinderbox-builds]$ find /data/ftp/pub/{firefox,xulrunner,mobile}/tinderbox-builds/ -mindepth 2 -maxdepth 2 -type d -mmin +20 -name 1?????????
/data/ftp/pub/firefox/tinderbox-builds/mozilla-central-macosx64-debug/1309280210
[ffxbld@dev-stage01 tinderbox-builds]$ find /data/ftp/pub/{firefox,xulrunner,mobile}/tinderbox-builds/ -mindepth 2 -maxdepth 2 -type d -mmin +30 -name 1?????????


...but I want to confirm with you before I change anything.

Comment 24

[Rail Aliiev [:rail]]

(In reply to comment #22)
> Rail, I'm poking at the existing crontabs and wondering what this line is
> for:
> 11 * * * *      find /data/ftp/pub/{firefox,xulrunner,mobile}/nightly/
> -mindepth 2 -maxdepth 2 -name build? -type d -mtime +1 -exec rm -rf {} \;
> 
> 
> There's other lines that clean up dated and latest dirs, so this would seem
> to only affect candidates directories. Do we really want to do be doing
> that, even on preproduction? If so, we'll have to cope with dev-stage01 and
> preproduction-stage having different requirements.

This is for candidates directories, yes. Unfortunately preproduction-stage has not enough disk space to keep old builds. :/(In reply to comment #23)


> (In reply to comment #22)
> It looks like these should be positive values based on some manual tests:

Yeah, looks like they should be positive numbers. My fault. Thanks for catching this.

Comment 25

[[e-mail] (:bhearsum)]

Attachment: templatize root's crontab

This should cope with the difference between preprod and dev.

Comment 26

[[e-mail] (:bhearsum)]

All done here. Bug 669157 tracks shutting off staging-stage.