446421 - migrate buildbot master instances to staging-master and production-master

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[[e-mail] (:bhearsum)]

Historically, we've ended up with our various Buildbot masters on different machines. Now that we've unified them on the same version of Buildbot we need to start consolidating them onto two machines: one for staging, one for production.

When we're done, things should look like this:
staging-master:
* Staging Release Automation (Firefox 1.8)
* Staging Release Automation (Firefox 1.9)
* Staging Moz2/3.next (builds+leaktest)
* Staging Talos
* Unittest Staging (1.9)
* Unittest Staging (m-c)

For release automation we'll need to add another hard drive to make room for the CVS mirrors. The bootstrap configs for those will also need adjusting, since they'll both be hosted on the same machine.

Similarly, production-master will look like this:
* Release Automation (Firefox 1.8)
* Release Automation (Thunderbird 1.8)
* Release Automation (Firefox 1.9)
* Moz2/3.next
* Talos
* Unittest (1.9)
* Unittest (m-c)


I'm going to start consolidating the staging stuff pretty soon. Alice: I won't be touching any Talos stuff without co-ordinating with you first. Lukas: same for you re: unittests

Comment 1

[Armen [:armenzg]]

I am currently working on staging-1.9

Comment 2

[[e-mail] (:bhearsum)]

For our release automation environments we depend heavily on on the stage.m.o directory structure set-up on the master. It's going to be tougher to do this on staging-master, because it's a shared setup. I'd be worried about hard disk load, too. I'm going to re-purpose staging 1.8 as a real "staging stage.mozilla.org" server. I think this will be the best place to put our CVS mirrors, and having all of our staging machines push to.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #2)
> For our release automation environments we depend heavily on on the stage.m.o
> directory structure set-up on the master. 

Could you elaborate here? I thought we moved this job off to the linux slave for staging, and use stage.m.o for production. (bug 415970)

Comment 4

[[e-mail] (:bhearsum)]

(In reply to comment #3)
> (In reply to comment #2)
> > For our release automation environments we depend heavily on on the stage.m.o
> > directory structure set-up on the master. 
> 
> Could you elaborate here? I thought we moved this job off to the linux slave
> for staging, and use stage.m.o for production. (bug 415970)
> 

Ah, that's true. I always get confused when I see the old directory structure on the masters. Well, that simplifies things!

Comment 5

[[e-mail] (:bhearsum)]

We're still going to need a place to put the CVS mirrors, so I'm going to continue re-purposing staging-1.8-master for this.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: changes necessary for moving staging 1.8 automation master

Lots of little changes here.
* Get rid of SlaveLocks and move to max_builds, like the rest of the masters.
* Remove {staging,production}-{1.8,1.9}-master from the slave list since we won't need slave there anymore
* Add 'staging-stage' (i'm open to a better name) to the slave list so we can do the appropriate prestage cleanup (which might have to be reworked a little bit in the Makefile, i'm still looking at that)
* Remove 'prestage' builder altogether, have slavePrestage and cvsmirror run at the same time
* Bump to 2.0.0.17
* Switch ports to be more consistent on staging-master (moz2-master is 8010/9010).

Comment 7

[[e-mail] (:bhearsum)]

Attachment: changes necessary for moving staging 1.8 automation master

sorry, attached the patch before re-diffing

Comment 8

[[e-mail] (:bhearsum)]

Attachment: accompanying bootstrap updates for staging-1.8

Mostly trivial changes. Things of note are the movement of the aus server and symbol server to staging-stage.

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 330762 [details] [diff] [review]
accompanying bootstrap updates for staging-1.8

this has changed a bit, i'll post updated patches tomorrow.

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 330760 [details] [diff] [review]
changes necessary for moving staging 1.8 automation master

this has changed a bit, i'll post updated patches tomorrow.

Comment 11

[[e-mail] (:bhearsum)]

Attachment: [checked in] boostrap updates for updated cvsmirror/staging-1.8-master rename

Pre-landed these because they are staging and blocking finishing up this move.

Checking in fx-moz18-nightly-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-nightly-staging-bootstrap.cfg,v  <--  fx-moz18-nightly-staging-bootstrap.cfg
new revision: 1.6; previous revision: 1.5
done
Checking in fx-moz18-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-staging-bootstrap.cfg,v  <--  fx-moz18-staging-bootstrap.cfg
new revision: 1.28; previous revision: 1.27
done

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 331297 [details] [diff] [review]
[checked in] boostrap updates for updated cvsmirror/staging-1.8-master rename

>Index: fx-moz18-nightly-staging-bootstrap.cfg
>===================================================================
>+ausServer       = staging-stgae.build.mozilla.org

Looks good, just this typo.

Comment 13

[[e-mail] (:bhearsum)]

Attachment: staging 1.8 master.cfg updates

Here's the config I did a full staging run with (along with the already-checked-in-boostrap.cfg).

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 331783 [details] [diff] [review]
staging 1.8 master.cfg updates

This looks OK, except ...

>Index: master.cfg
>===================================================================
>-prestageFactory.addStep(ShellCommand, 
>- description='create staging area',
>- command=['make', 'stage'], 
>- haltOnFailure=1,

... I'm not sure why the removal of the prestageFactory is OK. Don't we still need to call "make stage" on the linux slave to get a copy of the files from the previous release ?

> cvsmirrorFactory = BootstrapFactory(
>  cvsroot=cvsroot, 
>  cvsmodule=cvsmodule, 
>@@ -320,26 +262,29 @@
> cvsmirrorFactory.addStep(ShellCommand, 
>  description='create cvsmirror cvsroot',
>  command=['rsync', '-av', '--delete-after',
>-  '/builds/cvsmirror.clean/cvsroot/', '/builds/cvsmirror/cvsroot/'], 
>+  '/builds/cvs/staging-1.8/cvsmirror.clean/cvsroot/',
>+  '/builds/cvs/staging-1.8/cvsmirror/cvsroot/'], 

This bit is stale now ? Should be /builds/cvsmirrors/stag.... ?

Comment 15

[[e-mail] (:bhearsum)]

(In reply to comment #14)
> (From update of attachment 331783 [details] [diff] [review])
> This looks OK, except ...
> 
> >Index: master.cfg
> >===================================================================
> >-prestageFactory.addStep(ShellCommand, 
> >- description='create staging area',
> >- command=['make', 'stage'], 
> >- haltOnFailure=1,
> 
> ... I'm not sure why the removal of the prestageFactory is OK. Don't we still
> need to call "make stage" on the linux slave to get a copy of the files from
> the previous release ?
> 

Yeah, I noticed this after the fact. Will post a fix.

> > cvsmirrorFactory = BootstrapFactory(
> >  cvsroot=cvsroot, 
> >  cvsmodule=cvsmodule, 
> >@@ -320,26 +262,29 @@
> > cvsmirrorFactory.addStep(ShellCommand, 
> >  description='create cvsmirror cvsroot',
> >  command=['rsync', '-av', '--delete-after',
> >-  '/builds/cvsmirror.clean/cvsroot/', '/builds/cvsmirror/cvsroot/'], 
> >+  '/builds/cvs/staging-1.8/cvsmirror.clean/cvsroot/',
> >+  '/builds/cvs/staging-1.8/cvsmirror/cvsroot/'], 
> 
> This bit is stale now ? Should be /builds/cvsmirrors/stag.... ?
> 

Same with this. I think I changed it locally on the master.cfg to workaround.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 331783 [details] [diff] [review]
staging 1.8 master.cfg updates

Ok, please set r? on the final patch.

Comment 17

[[e-mail] (:bhearsum)]

We've done as much of this as we're going to do, and experience has shown that putting everything on the same master hurts.

Comment 18

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.