427166 - merge 1.8/1.9 staging configs to production

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Robert Helmer [:rhelmer]]

Several changes have happened on staging that have not been merged to production yet, a couple of them fairly large. I've set them as dependencies on this bug, once they are closed out we should merge all of this to production.

Comment 1

[Robert Helmer [:rhelmer]]

For bug 415970, we need:

* bootstrap.cfg changes
* master.cfg changes

For bug 417779, we need:

* bootstrap.cfg changes
* master.cfg changes
* remove bootstrap-configs checkout on master
* checkout buildbotcustom on master

For bug 415180, we need:

* create Xr-Release tinderbox dir on each slave
OR
* land bug 397554

Comment 2

[Robert Helmer [:rhelmer]]

Attachment: merge 1.8 changes to production

This should be rolled out per comment 1

Comment 3

[Robert Helmer [:rhelmer]]

Comment on attachment 313746 [details] [diff] [review]
merge 1.8 changes to production

Actually, we should probably change all of the bootstrap configs in one go; I'll do a separate patch for those.

Comment 4

[Robert Helmer [:rhelmer]]

Attachment: merge bootstrap changes from staging to production

Comment 5

[Robert Helmer [:rhelmer]]

Attachment: merge 1.9 master.cfg changes from staging to production

Comment 6

[Robert Helmer [:rhelmer]]

Attachment: [checked in] merge 1.8 master.cfg changes from staging to production

Comment 7

[Robert Helmer [:rhelmer]]

Because of 2.0.0.14 release we can't check in 1.8 master.cfg changes :/

Maybe the master.cfg should be pulled from the RELEASE_AUTOMATION tag instead? Then this could be safely checked in on trunk.

Comment 8

[Robert Helmer [:rhelmer]]

Sorry, clobbered the Linux64 nightlies (it's not loaded on staging :/).

Maybe we put entries for it on staging, just to make future merges easier if nothing else.

Comment 9

[Robert Helmer [:rhelmer]]

Attachment: merge 1.9 master.cfg changes from staging to production take 2

spread the r? load a bit :)

This patch is the same as before but re-adds Linux64 builder.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 313747 [details] [diff] [review]
merge bootstrap changes from staging to production

>Index: release/configs/fx-moz18-bootstrap.cfg
>===================================================================
>+sourceDir       = /builds/source
>+sourceDir       = /builds/source

Nit: Remove double paste.

>Index: release/configs/fx-moz18-nightly-staging-bootstrap.cfg
>===================================================================
> # where release updates/builds go
>-bouncerServer   = staging-1.8-master.build..mozilla.org
>+bouncerServer   = staging-prometheus-vm.build..mozilla.org

Nit: Single dot before m.o

>Index: release/configs/fx-moz18-staging-bootstrap.cfg
>===================================================================
>+sourceDir       = /builds/source

Already present after the logDir definitions, use the standard location you added here instead ?

>Index: release/configs/fx-moz180-bootstrap.cfg
>===================================================================

Wouldn't bother with this one, there's plenty of other changes it never got.

>Index: release/configs/fx-moz19-staging-bootstrap.cfg
>===================================================================

Same as fx-moz18-staging-bootstrap.cfg

>Index: release/configs/tb-moz18-bootstrap.cfg
>===================================================================
> stageHome            = /data/cltbld
>+sourceDir       = /builds/source
> updateDir            = /builds/updates

Urgh, indent differences across files.

>Index: release/configs/tb-moz180-bootstrap.cfg
>===================================================================

Leave this out too.

r+ with suggested changes and adding sourceDir to release/bootstrap.cfg.example

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 314176 [details] [diff] [review]
merge 1.9 master.cfg changes from staging to production take 2

>Index: buildbot-configs/automation/production-1.9/master.cfg
>===================================================================
>-# For nightly builds
>-nightly_scheduler = Nightly(
>- name='nightly_scheduler',
>+# For buildbot-driven nightly builders
>+shark_scheduler = Nightly(
>+ name='nightly_shark',
>  branch='HEAD',
>  hour=[01],
>  builderNames=[

This Scheduler kicks off linux64 too - can we leave the name the same?


>-c['schedulers'].append(nightly_scheduler)
>+c['schedulers'].append(shark_scheduler)

Here too.


-prestage_scheduler = Scheduler(name="prestage", branch=None, treeStableTimer=0,
-                               builderNames=["prestage"])
+tag_depscheduler = Scheduler (

nit: no space between the class name and the parenthesis. name should be 'tag_scheduler' to match below

+ name="tag_scheduler",
+ treeStableTimer=0,
+ builderNames=["tag"],
+)
 
Need branch defined here too.


>-import buildbotcustom.steps.misc
>-import buildbotcustom.steps.transfer
>-reload(buildbotcustom.steps.misc)
>-reload(buildbotcustom.steps.transfer)
>-from buildbotcustom.steps.misc import GetBuildID
>-from buildbotcustom.steps.transfer import MozillaStageUpload

Linux64 needs these.


> linux64BuildFactory = factory.BuildFactory()

Since stgbld_cvsroot was removed you'll need to s/stgbld_cvsroot/cvsroot/ in this factory.


>+xrBuildFactory = BootstrapFactory(
>+ cvsroot=cvsroot, 
>+ cvsmodule=cvsmodule, 
>+ automation_tag=automation_tag,
>+ logdir='/builds/logs', 
>+ bootstrap_config='configs/xr-moz19-bootstrap.cfg',
>+)
>+xrBuildFactory.addStep(ShellCommand,
>+ description='clean tinder-config area',
>+ workdir='build',
>+ command=['rm', '-rfv', '/builds/config'],
>+)
>+xrBuildFactory.addStep(ShellCommand,
>+ description='TinderConfig',
>+ workdir='build',
>+ command=['perl', './release', '-o', 'TinderConfig'], 
>+ timeout=36000,
>+ haltOnFailure=1,
>+ env={'CVS_RSH': 'ssh'},
>+)
>+xrBuildFactory.addStep(ShellCommand,
>+ description='Build',
>+ workdir='build',
>+ command=['perl', './release', '-o', 'Build'], 
>+ timeout=36000,
>+ haltOnFailure=1,
>+ env={'CVS_RSH': 'ssh'},
>+)
>+
>+c['builders'].append(
>+ { 
>+  'name': 'linux_xr_build',
>+  'slavename': 'fx-linux-1.9-slave2',
>+  'builddir': 'linux_xr_build',
>+  'locks': [linux_lock],
>+  'category': 'release',
>+  'factory': xrBuildFactory,
>+ },
>+)
>+c['builders'].append(
>+ { 
>+  'name': 'win32_xr_build',
>+  'slavename': 'fx-win32-1.9-slave2',
>+  'builddir': 'win32_xr_build',
>+  'locks': [win32_lock],
>+  'category': 'release',
>+  'factory': xrBuildFactory,
>+ },
>+)
>+c['builders'].append(
>+ { 
>+ 'name': 'macosx_xr_build',
>+ 'slavename': 'fx-mac-1.9-slave2',
>+ 'builddir': 'macosx_xr_build',
>+ 'locks': [macosx_lock],
>+ 'category': 'release',
>+ 'factory': xrBuildFactory,
>+ },
>+)

This isn't currently in the production config; I just want to confirm that it was intended to be in this patch.


>+ fromaddr="[e-mail]",
>+ tree="MozillaTest",

Should be MozillaRelease here.


r- because it doesn't pass 'buildbot checkconfig' and the scheduler issues. Looks good overall though.

Comment 12

[Robert Helmer [:rhelmer]]

(In reply to comment #11)
> (From update of attachment 314176 [details] [diff] [review])

> > linux64BuildFactory = factory.BuildFactory()
> 
> Since stgbld_cvsroot was removed you'll need to s/stgbld_cvsroot/cvsroot/ in
> this factory.
> 

Ok. Why was stgbld used here anyway?

> 
> >+xrBuildFactory = BootstrapFactory(

> This isn't currently in the production config; I just want to confirm that it
> was intended to be in this patch.
> 


Yes, it's one of the bugs blocking this one (bug 415180).

> 
> >+ fromaddr="[e-mail]",
> >+ tree="MozillaTest",
> 
> Should be MozillaRelease here.
> 
> 
> r- because it doesn't pass 'buildbot checkconfig' and the scheduler issues.
> Looks good overall though.
> 

Hmm.. the 'buildbot checkconfig' error, is that:
          File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/
site-packages/buildbot/status/mail.py", line 139, in __init__
            assert mode in ('all', 'failing', 'problem')

If so that's 0.7.6 only, this seems to work ok under 0.7.5 for me. I'll post the latest patch momentarily.

Comment 13

[Robert Helmer [:rhelmer]]

Attachment: [checked in] merge 1.9 master.cfg changes from staging to production take 3

Comment 14

[[e-mail] (:bhearsum)]

(In reply to comment #12)
> (In reply to comment #11)
> > (From update of attachment 314176 [details] [diff] [review] [details])
> 
> > > linux64BuildFactory = factory.BuildFactory()
> > 
> > Since stgbld_cvsroot was removed you'll need to s/stgbld_cvsroot/cvsroot/ in
> > this factory.
> > 
> 
> Ok. Why was stgbld used here anyway?
> 

I added that when I added linux64. It should've been using :ext:stgbld@..., but apparently :ext:cltbld@ worked too!

> > 
> > >+xrBuildFactory = BootstrapFactory(
> 
> > This isn't currently in the production config; I just want to confirm that it
> > was intended to be in this patch.
> > 
> 
> 
> Yes, it's one of the bugs blocking this one (bug 415180).
> 

K.

> > 
> > >+ fromaddr="[e-mail]",
> > >+ tree="MozillaTest",
> > 
> > Should be MozillaRelease here.
> > 
> > 
> > r- because it doesn't pass 'buildbot checkconfig' and the scheduler issues.
> > Looks good overall though.
> > 
> 
> Hmm.. the 'buildbot checkconfig' error, is that:
>           File
> "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/
> site-packages/buildbot/status/mail.py", line 139, in __init__
>             assert mode in ('all', 'failing', 'problem')
> 
> If so that's 0.7.6 only, this seems to work ok under 0.7.5 for me. I'll post
> the latest patch momentarily.
> 

Yeah, we landed a patch to support 'passing mode' that isn't in upstream. I forgot about that failure ;). By not passing checkconfig I was talking about the scheduler typos.

Comment 15

[[e-mail] (:bhearsum)]

Comment on attachment 314357 [details] [diff] [review]
[checked in] merge 1.9 master.cfg changes from staging to production take 3

>Index: buildbot-configs/automation/production-1.9/master.cfg
>===================================================================
>+tag_scheduler = Scheduler (
>+ name="tag_scheduler",
>+ treeStableTimer=0,
>+ builderNames=["tag"],
>+ branch='HEAD',
>+)
>

To make this work the same as it currently does, the branch needs to be None -- otherwise you have to specify --branch= w/ 'buildbot sendchange'.
 
r=bhearsum with that change.

Comment 16

[Robert Helmer [:rhelmer]]

Attachment: [checked in] merge bootstrap changes from staging to production [as landed]

Checking in release/bootstrap.cfg.example;
/cvsroot/mozilla/tools/release/bootstrap.cfg.example,v  <--  bootstrap.cfg.example
new revision: 1.8; previous revision: 1.7
done
Checking in release/configs/fx-moz18-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-bootstrap.cfg,v  <--  fx-moz18-bootstrap.cfg
new revision: 1.42; previous revision: 1.41
done
Checking in release/configs/fx-moz18-nightly-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-nightly-bootstrap.cfg,v  <--  fx-moz18-nightly-bootstrap.cfg
new revision: 1.13; previous revision: 1.12
done
Checking in release/configs/fx-moz18-nightly-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-nightly-staging-bootstrap.cfg,v  <--  fx-moz18-nightly-staging-bootstrap.cfg
new revision: 1.4; previous revision: 1.3
done
Checking in release/configs/fx-moz18-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-staging-bootstrap.cfg,v  <--  fx-moz18-staging-bootstrap.cfg
new revision: 1.24; previous revision: 1.23
done
Checking in release/configs/fx-moz19-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz19-bootstrap.cfg,v  <--  fx-moz19-bootstrap.cfg
new revision: 1.24; previous revision: 1.23
done
Checking in release/configs/fx-moz19-nightly-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz19-nightly-bootstrap.cfg,v  <--  fx-moz19-nightly-bootstrap.cfg
new revision: 1.12; previous revision: 1.11
done
Checking in release/configs/fx-moz19-nightly-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz19-nightly-staging-bootstrap.cfg,v  <--  fx-moz19-nightly-staging-bootstrap.cfg
new revision: 1.3; previous revision: 1.2
done
Checking in release/configs/fx-moz19-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz19-staging-bootstrap.cfg,v  <--  fx-moz19-staging-bootstrap.cfg
new revision: 1.20; previous revision: 1.19
done
Checking in release/configs/tb-moz18-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/tb-moz18-bootstrap.cfg,v  <--  tb-moz18-bootstrap.cfg
new revision: 1.19; previous revision: 1.18
done
Checking in release/configs/tb-moz180-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/tb-moz180-bootstrap.cfg,v  <--  tb-moz180-bootstrap.cfg
new revision: 1.11; previous revision: 1.10
done
Checking in release/configs/xr-moz19-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/xr-moz19-bootstrap.cfg,v  <--  xr-moz19-bootstrap.cfg
new revision: 1.3; previous revision: 1.2
done
Checking in release/configs/xr-moz19-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/xr-moz19-staging-bootstrap.cfg,v  <--  xr-moz19-staging-bootstrap.cfg
new revision: 1.5; previous revision: 1.4
done

Comment 17

[Robert Helmer [:rhelmer]]

Comment on attachment 314357 [details] [diff] [review]
[checked in] merge 1.9 master.cfg changes from staging to production take 3

Landed with "branch" change to None on tagScheduler:

Checking in buildbot-configs/automation/production-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v  <--  master.cfg
new revision: 1.23; previous revision: 1.22
done

Comment 18

[Robert Helmer [:rhelmer]]

(In reply to comment #17)
> (From update of attachment 314357 [details] [diff] [review])
> Landed with "branch" change to None on tagScheduler:
> 


Also, I tagged the previous version as RELEASE_AUTOMATION_M8_1 so it can easily be rolled back if needed. We should tag the current config RELEASE_AUTOMATION_M9 before rolling it out.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 313749 [details] [diff] [review]
[checked in] merge 1.8 master.cfg changes from staging to production

r+. Should've diffed this against staging/master.cfg earlier, so much less scary :-)

Note to release monkeys: we'll need to revisit the required disk space for the linux slave since a bunch of jobs moved there from the master.

Comment 20

[Robert Helmer [:rhelmer]]

Comment on attachment 313749 [details] [diff] [review]
[checked in] merge 1.8 master.cfg changes from staging to production

Tagged rev 1.24 as RELEASE_AUTOMATION_M8_1 before landing:

Checking in buildbot-configs/automation/production/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production/master.cfg,v  <--  master.cfg
new revision: 1.25; previous revision: 1.24
done

Comment 21

[Robert Helmer [:rhelmer]]

Ok, this just needs to be rolled out on the production servers. Specifically:

* tag the master.cfg files

On the master:

* remove bootstrap-configs/ checkout
* checkout mozilla/tools/buildbotcustom to buildbotcustom/ in master's dir
* update buildbot-configs/ checkout to new tag

The BuildNotes should be updated as well, to reflect that this is tagged now. 

Comment 22

[Robert Helmer [:rhelmer]]

I pushed these changes to production-1.9-master like so:

(as buildbot user):
cd /builds/buildbot/trunk-automation
mv bootstrap-configs bootstrap-configs.old
cvs -d [e-mail]:/cvsroot co -d buildbotcustom mozilla/tools/buildbotcustom
cd buildbot-configs
cvs up -A
# merge the password changes in.. will probably conflict because of the whitespace changes
cd ../
buildbot stop .
buildbot start `pwd`

Note that I did not update the master.cfg to a tag because no tag has been applied :) I will leave that to be done as part of the setup steps for the release.. feel free to go ahead and do this now if you like. It should work in any case.

If you need to roll this back, then update master.cfg to RELEASE_AUTOMATION_M8_1, move the bootstrap-configs.old back to bootstrap-configs, and restart the master. All should be the same as it was before I touched it today.

Comment 23

[Robert Helmer [:rhelmer]]

Also, I noticed that the linux64 builder doesn't work unless the cvsroot uses the "stgbld" key; I put in a workaround (overrode that cvsroot for that builder), but this should probably be fixed.

Comment 24

[Robert Helmer [:rhelmer]]

This broke the nightly stuff (still in test mode for 1.9); I landed bustage fixes in the tinder-config.pl files to do s/externalStaging/ftpServer/

Also, I made a local patch to master.cfg so automation_tag="HEAD", to verify bug 415180 using a previous release.

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

I'm happy to herd this through rollout.

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Looks like we handled this in assorted other bugs.