merge 1.8/1.9 staging configs to production

RESOLVED FIXED

Status

Release Engineering
General
P3
normal
RESOLVED FIXED
10 years ago
5 years ago

People

(Reporter: rhelmer, Assigned: nthomas)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments, 4 obsolete attachments)

(Reporter)

Description

10 years ago
Several changes have happened on staging that have not been merged to production yet, a couple of them fairly large. I've set them as dependencies on this bug, once they are closed out we should merge all of this to production.
(Reporter)

Comment 1

10 years ago
For bug 415970, we need:

* bootstrap.cfg changes
* master.cfg changes

For bug 417779, we need:

* bootstrap.cfg changes
* master.cfg changes
* remove bootstrap-configs checkout on master
* checkout buildbotcustom on master

For bug 415180, we need:

* create Xr-Release tinderbox dir on each slave
OR
* land bug 397554
(Reporter)

Comment 2

10 years ago
Created attachment 313746 [details] [diff] [review]
merge 1.8 changes to production

This should be rolled out per comment 1
Attachment #313746 - Flags: review?(nrthomas)
(Reporter)

Comment 3

10 years ago
Comment on attachment 313746 [details] [diff] [review]
merge 1.8 changes to production

Actually, we should probably change all of the bootstrap configs in one go; I'll do a separate patch for those.
Attachment #313746 - Attachment is obsolete: true
Attachment #313746 - Flags: review?(nrthomas)
(Reporter)

Comment 4

10 years ago
Created attachment 313747 [details] [diff] [review]
merge bootstrap changes from staging to production
Attachment #313747 - Flags: review?(nrthomas)
(Reporter)

Comment 5

10 years ago
Created attachment 313748 [details] [diff] [review]
merge 1.9 master.cfg changes from staging to production
Attachment #313748 - Flags: review?(nrthomas)
(Reporter)

Comment 6

10 years ago
Created attachment 313749 [details] [diff] [review]
[checked in] merge 1.8 master.cfg changes from staging to production
Attachment #313749 - Flags: review?(nrthomas)
(Reporter)

Updated

10 years ago
Assignee: nobody → robert
(Reporter)

Updated

10 years ago
Whiteboard: waiting for review
(Reporter)

Updated

10 years ago
Priority: -- → P2
(Reporter)

Comment 7

10 years ago
Because of 2.0.0.14 release we can't check in 1.8 master.cfg changes :/

Maybe the master.cfg should be pulled from the RELEASE_AUTOMATION tag instead? Then this could be safely checked in on trunk.
Status: NEW → ASSIGNED
(Reporter)

Updated

10 years ago
Attachment #313748 - Flags: review?(nrthomas)
(Reporter)

Comment 8

10 years ago
Sorry, clobbered the Linux64 nightlies (it's not loaded on staging :/).

Maybe we put entries for it on staging, just to make future merges easier if nothing else.
(Reporter)

Comment 9

10 years ago
Created attachment 314176 [details] [diff] [review]
merge 1.9 master.cfg changes from staging to production take 2

spread the r? load a bit :)

This patch is the same as before but re-adds Linux64 builder.
Attachment #314176 - Flags: review?(bhearsum)
(Reporter)

Updated

10 years ago
Attachment #314176 - Attachment is patch: true
Attachment #314176 - Attachment mime type: application/octet-stream → text/plain
(Assignee)

Comment 10

10 years ago
Comment on attachment 313747 [details] [diff] [review]
merge bootstrap changes from staging to production

>Index: release/configs/fx-moz18-bootstrap.cfg
>===================================================================
>+sourceDir       = /builds/source
>+sourceDir       = /builds/source

Nit: Remove double paste.

>Index: release/configs/fx-moz18-nightly-staging-bootstrap.cfg
>===================================================================
> # where release updates/builds go
>-bouncerServer   = staging-1.8-master.build..mozilla.org
>+bouncerServer   = staging-prometheus-vm.build..mozilla.org

Nit: Single dot before m.o

>Index: release/configs/fx-moz18-staging-bootstrap.cfg
>===================================================================
>+sourceDir       = /builds/source

Already present after the logDir definitions, use the standard location you added here instead ?

>Index: release/configs/fx-moz180-bootstrap.cfg
>===================================================================

Wouldn't bother with this one, there's plenty of other changes it never got.

>Index: release/configs/fx-moz19-staging-bootstrap.cfg
>===================================================================

Same as fx-moz18-staging-bootstrap.cfg

>Index: release/configs/tb-moz18-bootstrap.cfg
>===================================================================
> stageHome            = /data/cltbld
>+sourceDir       = /builds/source
> updateDir            = /builds/updates

Urgh, indent differences across files.

>Index: release/configs/tb-moz180-bootstrap.cfg
>===================================================================

Leave this out too.

r+ with suggested changes and adding sourceDir to release/bootstrap.cfg.example
Attachment #313747 - Flags: review?(nrthomas) → review+

Updated

10 years ago
Attachment #313748 - Attachment is obsolete: true
Comment on attachment 314176 [details] [diff] [review]
merge 1.9 master.cfg changes from staging to production take 2

>Index: buildbot-configs/automation/production-1.9/master.cfg
>===================================================================
>-# For nightly builds
>-nightly_scheduler = Nightly(
>- name='nightly_scheduler',
>+# For buildbot-driven nightly builders
>+shark_scheduler = Nightly(
>+ name='nightly_shark',
>  branch='HEAD',
>  hour=[01],
>  builderNames=[

This Scheduler kicks off linux64 too - can we leave the name the same?


>-c['schedulers'].append(nightly_scheduler)
>+c['schedulers'].append(shark_scheduler)

Here too.


-prestage_scheduler = Scheduler(name="prestage", branch=None, treeStableTimer=0,
-                               builderNames=["prestage"])
+tag_depscheduler = Scheduler (

nit: no space between the class name and the parenthesis. name should be 'tag_scheduler' to match below

+ name="tag_scheduler",
+ treeStableTimer=0,
+ builderNames=["tag"],
+)
 
Need branch defined here too.


>-import buildbotcustom.steps.misc
>-import buildbotcustom.steps.transfer
>-reload(buildbotcustom.steps.misc)
>-reload(buildbotcustom.steps.transfer)
>-from buildbotcustom.steps.misc import GetBuildID
>-from buildbotcustom.steps.transfer import MozillaStageUpload

Linux64 needs these.


> linux64BuildFactory = factory.BuildFactory()

Since stgbld_cvsroot was removed you'll need to s/stgbld_cvsroot/cvsroot/ in this factory.


>+xrBuildFactory = BootstrapFactory(
>+ cvsroot=cvsroot, 
>+ cvsmodule=cvsmodule, 
>+ automation_tag=automation_tag,
>+ logdir='/builds/logs', 
>+ bootstrap_config='configs/xr-moz19-bootstrap.cfg',
>+)
>+xrBuildFactory.addStep(ShellCommand,
>+ description='clean tinder-config area',
>+ workdir='build',
>+ command=['rm', '-rfv', '/builds/config'],
>+)
>+xrBuildFactory.addStep(ShellCommand,
>+ description='TinderConfig',
>+ workdir='build',
>+ command=['perl', './release', '-o', 'TinderConfig'], 
>+ timeout=36000,
>+ haltOnFailure=1,
>+ env={'CVS_RSH': 'ssh'},
>+)
>+xrBuildFactory.addStep(ShellCommand,
>+ description='Build',
>+ workdir='build',
>+ command=['perl', './release', '-o', 'Build'], 
>+ timeout=36000,
>+ haltOnFailure=1,
>+ env={'CVS_RSH': 'ssh'},
>+)
>+
>+c['builders'].append(
>+ { 
>+  'name': 'linux_xr_build',
>+  'slavename': 'fx-linux-1.9-slave2',
>+  'builddir': 'linux_xr_build',
>+  'locks': [linux_lock],
>+  'category': 'release',
>+  'factory': xrBuildFactory,
>+ },
>+)
>+c['builders'].append(
>+ { 
>+  'name': 'win32_xr_build',
>+  'slavename': 'fx-win32-1.9-slave2',
>+  'builddir': 'win32_xr_build',
>+  'locks': [win32_lock],
>+  'category': 'release',
>+  'factory': xrBuildFactory,
>+ },
>+)
>+c['builders'].append(
>+ { 
>+ 'name': 'macosx_xr_build',
>+ 'slavename': 'fx-mac-1.9-slave2',
>+ 'builddir': 'macosx_xr_build',
>+ 'locks': [macosx_lock],
>+ 'category': 'release',
>+ 'factory': xrBuildFactory,
>+ },
>+)

This isn't currently in the production config; I just want to confirm that it was intended to be in this patch.


>+ fromaddr="bootstrap@mozilla.com",
>+ tree="MozillaTest",

Should be MozillaRelease here.


r- because it doesn't pass 'buildbot checkconfig' and the scheduler issues. Looks good overall though.
Attachment #314176 - Flags: review?(bhearsum) → review-
(Reporter)

Comment 12

10 years ago
(In reply to comment #11)
> (From update of attachment 314176 [details] [diff] [review])

> > linux64BuildFactory = factory.BuildFactory()
> 
> Since stgbld_cvsroot was removed you'll need to s/stgbld_cvsroot/cvsroot/ in
> this factory.
> 

Ok. Why was stgbld used here anyway?

> 
> >+xrBuildFactory = BootstrapFactory(

> This isn't currently in the production config; I just want to confirm that it
> was intended to be in this patch.
> 


Yes, it's one of the bugs blocking this one (bug 415180).

> 
> >+ fromaddr="bootstrap@mozilla.com",
> >+ tree="MozillaTest",
> 
> Should be MozillaRelease here.
> 
> 
> r- because it doesn't pass 'buildbot checkconfig' and the scheduler issues.
> Looks good overall though.
> 

Hmm.. the 'buildbot checkconfig' error, is that:
          File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/
site-packages/buildbot/status/mail.py", line 139, in __init__
            assert mode in ('all', 'failing', 'problem')

If so that's 0.7.6 only, this seems to work ok under 0.7.5 for me. I'll post the latest patch momentarily.
(Reporter)

Comment 13

10 years ago
Created attachment 314357 [details] [diff] [review]
[checked in] merge 1.9 master.cfg changes from staging to production take 3
Attachment #314176 - Attachment is obsolete: true
Attachment #314357 - Flags: review?(bhearsum)
(In reply to comment #12)
> (In reply to comment #11)
> > (From update of attachment 314176 [details] [diff] [review] [details])
> 
> > > linux64BuildFactory = factory.BuildFactory()
> > 
> > Since stgbld_cvsroot was removed you'll need to s/stgbld_cvsroot/cvsroot/ in
> > this factory.
> > 
> 
> Ok. Why was stgbld used here anyway?
> 

I added that when I added linux64. It should've been using :ext:stgbld@..., but apparently :ext:cltbld@ worked too!

> > 
> > >+xrBuildFactory = BootstrapFactory(
> 
> > This isn't currently in the production config; I just want to confirm that it
> > was intended to be in this patch.
> > 
> 
> 
> Yes, it's one of the bugs blocking this one (bug 415180).
> 

K.

> > 
> > >+ fromaddr="bootstrap@mozilla.com",
> > >+ tree="MozillaTest",
> > 
> > Should be MozillaRelease here.
> > 
> > 
> > r- because it doesn't pass 'buildbot checkconfig' and the scheduler issues.
> > Looks good overall though.
> > 
> 
> Hmm.. the 'buildbot checkconfig' error, is that:
>           File
> "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/
> site-packages/buildbot/status/mail.py", line 139, in __init__
>             assert mode in ('all', 'failing', 'problem')
> 
> If so that's 0.7.6 only, this seems to work ok under 0.7.5 for me. I'll post
> the latest patch momentarily.
> 

Yeah, we landed a patch to support 'passing mode' that isn't in upstream. I forgot about that failure ;). By not passing checkconfig I was talking about the scheduler typos.
Comment on attachment 314357 [details] [diff] [review]
[checked in] merge 1.9 master.cfg changes from staging to production take 3

>Index: buildbot-configs/automation/production-1.9/master.cfg
>===================================================================
>+tag_scheduler = Scheduler (
>+ name="tag_scheduler",
>+ treeStableTimer=0,
>+ builderNames=["tag"],
>+ branch='HEAD',
>+)
>

To make this work the same as it currently does, the branch needs to be None -- otherwise you have to specify --branch= w/ 'buildbot sendchange'.
 
r=bhearsum with that change.
Attachment #314357 - Flags: review?(bhearsum) → review+
(Reporter)

Comment 16

10 years ago
Created attachment 314358 [details] [diff] [review]
[checked in] merge bootstrap changes from staging to production [as landed]

Checking in release/bootstrap.cfg.example;
/cvsroot/mozilla/tools/release/bootstrap.cfg.example,v  <--  bootstrap.cfg.example
new revision: 1.8; previous revision: 1.7
done
Checking in release/configs/fx-moz18-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-bootstrap.cfg,v  <--  fx-moz18-bootstrap.cfg
new revision: 1.42; previous revision: 1.41
done
Checking in release/configs/fx-moz18-nightly-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-nightly-bootstrap.cfg,v  <--  fx-moz18-nightly-bootstrap.cfg
new revision: 1.13; previous revision: 1.12
done
Checking in release/configs/fx-moz18-nightly-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-nightly-staging-bootstrap.cfg,v  <--  fx-moz18-nightly-staging-bootstrap.cfg
new revision: 1.4; previous revision: 1.3
done
Checking in release/configs/fx-moz18-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-staging-bootstrap.cfg,v  <--  fx-moz18-staging-bootstrap.cfg
new revision: 1.24; previous revision: 1.23
done
Checking in release/configs/fx-moz19-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz19-bootstrap.cfg,v  <--  fx-moz19-bootstrap.cfg
new revision: 1.24; previous revision: 1.23
done
Checking in release/configs/fx-moz19-nightly-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz19-nightly-bootstrap.cfg,v  <--  fx-moz19-nightly-bootstrap.cfg
new revision: 1.12; previous revision: 1.11
done
Checking in release/configs/fx-moz19-nightly-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz19-nightly-staging-bootstrap.cfg,v  <--  fx-moz19-nightly-staging-bootstrap.cfg
new revision: 1.3; previous revision: 1.2
done
Checking in release/configs/fx-moz19-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz19-staging-bootstrap.cfg,v  <--  fx-moz19-staging-bootstrap.cfg
new revision: 1.20; previous revision: 1.19
done
Checking in release/configs/tb-moz18-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/tb-moz18-bootstrap.cfg,v  <--  tb-moz18-bootstrap.cfg
new revision: 1.19; previous revision: 1.18
done
Checking in release/configs/tb-moz180-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/tb-moz180-bootstrap.cfg,v  <--  tb-moz180-bootstrap.cfg
new revision: 1.11; previous revision: 1.10
done
Checking in release/configs/xr-moz19-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/xr-moz19-bootstrap.cfg,v  <--  xr-moz19-bootstrap.cfg
new revision: 1.3; previous revision: 1.2
done
Checking in release/configs/xr-moz19-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/xr-moz19-staging-bootstrap.cfg,v  <--  xr-moz19-staging-bootstrap.cfg
new revision: 1.5; previous revision: 1.4
done
Attachment #313747 - Attachment is obsolete: true
(Reporter)

Comment 17

10 years ago
Comment on attachment 314357 [details] [diff] [review]
[checked in] merge 1.9 master.cfg changes from staging to production take 3

Landed with "branch" change to None on tagScheduler:

Checking in buildbot-configs/automation/production-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v  <--  master.cfg
new revision: 1.23; previous revision: 1.22
done
Attachment #314357 - Attachment description: merge 1.9 master.cfg changes from staging to production take 3 → [checked in] merge 1.9 master.cfg changes from staging to production take 3
(Reporter)

Comment 18

10 years ago
(In reply to comment #17)
> (From update of attachment 314357 [details] [diff] [review])
> Landed with "branch" change to None on tagScheduler:
> 


Also, I tagged the previous version as RELEASE_AUTOMATION_M8_1 so it can easily be rolled back if needed. We should tag the current config RELEASE_AUTOMATION_M9 before rolling it out.
(Reporter)

Updated

10 years ago
Depends on: 427690
(Assignee)

Comment 19

10 years ago
Comment on attachment 313749 [details] [diff] [review]
[checked in] merge 1.8 master.cfg changes from staging to production

r+. Should've diffed this against staging/master.cfg earlier, so much less scary :-)

Note to release monkeys: we'll need to revisit the required disk space for the linux slave since a bunch of jobs moved there from the master.
Attachment #313749 - Flags: review?(nrthomas) → review+
(Reporter)

Comment 20

10 years ago
Comment on attachment 313749 [details] [diff] [review]
[checked in] merge 1.8 master.cfg changes from staging to production

Tagged rev 1.24 as RELEASE_AUTOMATION_M8_1 before landing:

Checking in buildbot-configs/automation/production/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production/master.cfg,v  <--  master.cfg
new revision: 1.25; previous revision: 1.24
done
Attachment #313749 - Attachment description: merge 1.8 master.cfg changes from staging to production → [checked in] merge 1.8 master.cfg changes from staging to production
(Reporter)

Comment 21

10 years ago
Ok, this just needs to be rolled out on the production servers. Specifically:

* tag the master.cfg files

On the master:

* remove bootstrap-configs/ checkout
* checkout mozilla/tools/buildbotcustom to buildbotcustom/ in master's dir
* update buildbot-configs/ checkout to new tag

The BuildNotes should be updated as well, to reflect that this is tagged now. 
Whiteboard: waiting for review → needs rollout
(Reporter)

Comment 22

10 years ago
I pushed these changes to production-1.9-master like so:

(as buildbot user):
cd /builds/buildbot/trunk-automation
mv bootstrap-configs bootstrap-configs.old
cvs -d stgbld@cvs.mozilla.org:/cvsroot co -d buildbotcustom mozilla/tools/buildbotcustom
cd buildbot-configs
cvs up -A
# merge the password changes in.. will probably conflict because of the whitespace changes
cd ../
buildbot stop .
buildbot start `pwd`

Note that I did not update the master.cfg to a tag because no tag has been applied :) I will leave that to be done as part of the setup steps for the release.. feel free to go ahead and do this now if you like. It should work in any case.

If you need to roll this back, then update master.cfg to RELEASE_AUTOMATION_M8_1, move the bootstrap-configs.old back to bootstrap-configs, and restart the master. All should be the same as it was before I touched it today.
(Reporter)

Comment 23

10 years ago
Also, I noticed that the linux64 builder doesn't work unless the cvsroot uses the "stgbld" key; I put in a workaround (overrode that cvsroot for that builder), but this should probably be fixed.
(Reporter)

Comment 24

10 years ago
This broke the nightly stuff (still in test mode for 1.9); I landed bustage fixes in the tinder-config.pl files to do s/externalStaging/ftpServer/

Also, I made a local patch to master.cfg so automation_tag="HEAD", to verify bug 415180 using a previous release.
(Assignee)

Comment 25

10 years ago
I'm happy to herd this through rollout.
Assignee: robert → nrthomas
Status: ASSIGNED → NEW
Priority: P2 → P3
(Assignee)

Comment 26

10 years ago
Looks like we handled this in assorted other bugs.
Status: NEW → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED
Whiteboard: needs rollout
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.