457721 - create a production master.cfg for doing mercurial based releases

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[[e-mail] (:bhearsum)]

Gotta land an initial one before 3.1b1 starts.

Comment 1

[[e-mail] (:bhearsum)]

Attachment: initial production configs

This is largely based on the staging config, of course. The RepositorySetupFactory was removed for obvious reasons. I've added the ReleaseUpdatesFactory, which will *hopefully* be working before Beta 1. I did not add any Schedulers for it since it depends on things we haven't automated yet. We can still kick it off with 'Force Build' from the waterfall, however.

I've added a basic l10n-changesets file - this must be updated before we start the release. (It gets generated by the get-l10n-revs script in hg.m.o/build/tools which someone from L10n should be running post-freeze.)

release_config.py is virutally the same, with the unnecessary hgHost removed and the repositories switched with their production equivalents.

I've got a big factory.py patch that will accompany this :(.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: factory.py updates so far

This is all of the work I've done in buildbotcustom/ so far. Here's a run-down:
* Support building a specific revision in MercurialBuildFactory, if passed.
* Allow post-build clean-up to be optional (not sure if we should necessarily avoid it in release builds - but it was extremely useful when testing)
* Use createSnippet rather than 'nightly' to decide whether to create updates or not.
* Bug fixes + new method in ReleaseFactory - you've seen that part of the patch before, I think.
* Initial import of StagingRepositorySetupFactory
* Allow hgUsername and hgSshKey to be passed into ReleaseTaggingFactory - so we can actually push changes out.
* Bump timeouts on all hg clones to 30 minutes.
* Initial import of ReleaseUpdatesFactory - this has been mostly tested. Everything works, except that the patcher config bump script doesn't do long dir format yet and obviously, pushing snippets and mars out.

If it helps, I can make a point to be around when you're reviewing this. Let me know.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 340960 [details] [diff] [review]
factory.py updates so far

>Index: factory.py
>-        self.addCleanupSteps()
>+        if self.doCleanup:
>+            self.addCleanupSteps()

It looks like you're making doCleanup the master variable for this, but then addCleanupSteps also has a test on self.nightly. Was that intentional ?

>+    def getRepoPath(self, hgHost, repo):
>+        repo = repo.replace(hgHost, '').strip('/')
>+        if repo.find('central') == -1:
>+            repo = 'l10n-central/%s' % repo
>+        return repo.replace(hgHost, '').strip('/')

That doesn't seem quite right.

>+class ReleaseUpdatesFactory(ReleaseFactory):
>+        self.addStep(CVS,
>+         cvsroot=cvsroot,
>+         cvsmodule='mozilla/tools/patcher'
>+        )

Need to do -r patcherToolsRev for Bootstrap parity. Is the branch param of the CVS class useful here ?

>+        self.addStep(ShellCommand,
>+         command=['cvs', '-d', cvsroot, 'co', '-d', 'MozBuild',
>+                  'mozilla/tools/release/MozBuild'],
>+         haltOnFailure=True
>+        )

-r patcherToolsRev again.

>+        self.addStep(ShellCommand,
>+         command=['cvs', '-d', cvsroot, 'co', '-d' 'Bootstrap',
>+                  'mozilla/tools/release/Bootstrap'],
>+        )

What's this for ? Superseded by patcher-config-bump.pl in hg:build/tools ?

>+        self.addStep(ShellCommand,
>+         command=['perl', 'tools/release/patcher-config-bump.pl',

Checkin to be done manually this time ?

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 340950 [details] [diff] [review]
initial production configs

>diff -r bf8a8755da81 mozilla2/l10n-changesets
>+af 80d99ed1b0cc
>+be c440822548d3

Nit: Perhaps we could commit an empty file if these are going to be replaced ? Bogus values could cause confusion later. Or if you're trying to help l10n-drivers then add a comment that means - "this is only an example, never used this ammunition in anger"

>diff -r bf8a8755da81 mozilla2/master.cfg
>+from buildbotcustom.misc import isHgPollerTriggered

This needs to land in CVS still ? Couldn't see it in attachment 340960 [details] [diff] [review] but would like to before granting review of this file.

>diff -r bf8a8755da81 mozilla2/release_config.py
>+patcherToolsRev   = 'UPDATE_PACKAGING_R5'

Once patcherToolsRev is used, this will get you rev 1.34 of patcher2.pl; cvs log tells me you need 1.36 for --build-tools-hg support. Please be creating an _R6 tag to resolve this.

>+useBetaChannel    = '1'

You want this to be 0 I think, since enabling to the beta channel occurs at release time for 3.1b* and not a week earlier.

>diff -r bf8a8755da81 mozilla2/release_master.py
>+builders.append({
>+    'name': 'tag',
>+    'slavenames': ['moz2-linux-slave1', 'moz2-linux-slave02',
>+                   'moz2-linux-slave03', 'moz2-linux-slave05',
>+                   'moz2-linux-slave06'],

Disk space prep could be interesting given a random slave allocation. Have you any thoughts on this ?

>+    build_factory = MercurialBuildFactory(
...
>+        uploadPackages=False,

This will be a manual step for b1 ?

Comment 5

[[e-mail] (:bhearsum)]

(In reply to comment #3)
> It looks like you're making doCleanup the master variable for this, but then
> addCleanupSteps also has a test on self.nightly. Was that intentional ?
> 

Hmmmm...That's a good point...this is rather icky. I'm going to do a more thorough look at that, because I think assuming nightly == release is bad....
As for cleanup...no matter what we should be removing mac crap after all builds. We should be removing the builddir post-build for nightlies but not dep builds or releases, I think. I'll go through and make sure any 'if self.nightly' statements are actually things we *only* want to be done on nightlies, then we can set nightly=False for releases.

> >+    def getRepoPath(self, hgHost, repo):
> >+        repo = repo.replace(hgHost, '').strip('/')
> >+        if repo.find('central') == -1:
> >+            repo = 'l10n-central/%s' % repo
> >+        return repo.replace(hgHost, '').strip('/')
> 
> That doesn't seem quite right.

Hm, yeah. No kidding. Should just be 'return repo'.

> Need to do -r patcherToolsRev for Bootstrap parity. Is the branch param of the
> CVS class useful here ?
> 

Yep, absolutely

> >+        self.addStep(ShellCommand,
> >+         command=['cvs', '-d', cvsroot, 'co', '-d' 'Bootstrap',
> >+                  'mozilla/tools/release/Bootstrap'],
> >+        )
> 
> What's this for ? Superseded by patcher-config-bump.pl in hg:build/tools ?
> 

That script depends on some functions in Bootstrap::Util - maybe I should just check-out exclusively that?

> >+        self.addStep(ShellCommand,
> >+         command=['perl', 'tools/release/patcher-config-bump.pl',
> 
> Checkin to be done manually this time ?

Yeah - but I should add a commented out check-in step nonetheless.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to comment #4)
> Nit: Perhaps we could commit an empty file if these are going to be replaced ?
> Bogus values could cause confusion later. Or if you're trying to help
> l10n-drivers then add a comment that means - "this is only an example, never
> used this ammunition in anger"
>

Yeah, I'll just empty it. I reminded Axel yesterday that he needs to generate this file before we start builds.

> >diff -r bf8a8755da81 mozilla2/master.cfg
> >+from buildbotcustom.misc import isHgPollerTriggered
> 
> This needs to land in CVS still ? Couldn't see it in attachment 340960 [details] [diff] [review] but
> would like to before granting review of this file.
> 

It's coming in the factory.py patch.

> >diff -r bf8a8755da81 mozilla2/release_config.py
> >+patcherToolsRev   = 'UPDATE_PACKAGING_R5'
> 
> Once patcherToolsRev is used, this will get you rev 1.34 of patcher2.pl; cvs
> log tells me you need 1.36 for --build-tools-hg support. Please be creating an
> _R6 tag to resolve this.
> 

Will do.

> >+useBetaChannel    = '1'
> 
> You want this to be 0 I think, since enabling to the beta channel occurs at
> release time for 3.1b* and not a week earlier.
> 

Yeah, duh.

> Disk space prep could be interesting given a random slave allocation. Have you
> any thoughts on this ?
>

Hmmm, I'd thought the slaves had pretty even disk usage - it seems not though. 9GB - 15GB free on the various slaves. I'm a little worried about this since Linux slaves end up with a *lot* of check-outs (tag, source, build, repack, updates). I think we should simply add larger disks to these slaves to work around it. Gotta talk with John/IT about it, too. Maybe we can have an NFS mount where we do all the release work? That might be better for archiving purposes too.

For Beta 1, I'm going to keep an eye on disk usage while things run along. Since we have 5 slaves I don't think it'll be a problem - unless 1 slaves gets saddled with 3 or 4 jobs. We need to solve this before Beta 2 though.
 
> >+    build_factory = MercurialBuildFactory(
> ...
> >+        uploadPackages=False,
> 
> This will be a manual step for b1 ?

Yep. Waiting on 454594 for this :(.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: [checked in] factory.py + misc.py again

Comment 8

[[e-mail] (:bhearsum)]

Attachment: [checked in] release master configs, again

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 341102 [details] [diff] [review]
[checked in] factory.py + misc.py again

>+#        self.addStep(ShellCommand,
>+#         command=['cvs', 'commit',
>+#                  'Automated configuration bump: %s, from %s to %s' % \
>+#                    (patcherConfig, oldVersion, appVersion)

Need a '-m' after 'commit'. r+ with that fix on checkin.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 341103 [details] [diff] [review]
[checked in] release master configs, again

r+

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 341102 [details] [diff] [review]
[checked in] factory.py + misc.py again

checked in w/ '-m':
Checking in misc.py;
/cvsroot/mozilla/tools/buildbotcustom/misc.py,v  <--  misc.py
new revision: 1.2; previous revision: 1.1
done
Checking in process/factory.py;
/cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v  <--  factory.py
new revision: 1.15; previous revision: 1.14
done

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 341103 [details] [diff] [review]
[checked in] release master configs, again

changeset:   378:bcd7fd97afaa

Comment 13

[[e-mail] (:bhearsum)]

Attachment: update production to work with latest version of factory.py

This is basically the same patch as the last one in bug 454205. s/patcherToolsRev/patcherToolsTag/ and add the extra required parameters.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 344530 [details] [diff] [review]
update production to work with latest version of factory.py

r+ with minor fixes on checkin:

>diff -r 2e9314acb5cd mozilla2/release_config.py
> useBetaChannel    = '0'

Please make this a plain 0

>diff -r 2e9314acb5cd mozilla2/release_master.py
>+    stageSshKey=nightly_config.STAGE_SSH_KEY

Need a comma at the end here, thanks diff.

Comment 15

[[e-mail] (:bhearsum)]

Attachment: [checked in] patch as landed

changeset:   461:69448cdc3935

Comment 16

[[e-mail] (:bhearsum)]

Attachment: [checked in] update production release configs for beta 2, add final verification

Hopefully I'll get update verify ready before we need it but until then......

The update verify configs referenced here are for the Final Verification test. They're currently only in CVS but we intend to use hg.m.o/build/tools for that going forward. There's a patch to add those configs there incoming.

Comment 17

[[e-mail] (:bhearsum)]

Attachment: [checked in] moz191 update verify configs

Not sure if we should keep the CVS ones or not. There's not much history...what do you think Nick?

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 346455 [details] [diff] [review]
[checked in] moz191 update verify configs

This looks fine for a1 & a2, are you going to add 3.1b1 info though ? We'll need to do update verify this time round, and all the info needed for 3.1b1 is already known, so I think you could add that now.

I'm not worried about keeping the small amount of cvs history for these files.

Comment 19

[[e-mail] (:bhearsum)]

(In reply to comment #18)
> (From update of attachment 346455 [details] [diff] [review])
> This looks fine for a1 & a2, are you going to add 3.1b1 info though ? We'll
> need to do update verify this time round, and all the info needed for 3.1b1 is
> already known, so I think you could add that now.
> 

I'm hoping to have an UpdateVerifyFactory before we need to run it actually, and planning to let it do the bumping for us. It'll be a good test :).

> I'm not worried about keeping the small amount of cvs history for these files.

Alright, I'll delete them then.

Comment 20

[[e-mail] (:bhearsum)]

Comment on attachment 346453 [details] [diff] [review]
[checked in] update production release configs for beta 2, add final verification

changeset:   490:3de208c0efcb

Comment 21

[[e-mail] (:bhearsum)]

Comment on attachment 346455 [details] [diff] [review]
[checked in] moz191 update verify configs

changeset:   17:c2f9920f9334

Comment 22

[[e-mail] (:bhearsum)]

...and removed the moz191 configs from CVS:
Removing moz191-firefox-linux.cfg;
/cvsroot/mozilla/testing/release/updates/moz191-firefox-linux.cfg,v  <--  moz191-firefox-linux.cfg
new revision: delete; previous revision: 1.1
done
Removing moz191-firefox-mac.cfg;
/cvsroot/mozilla/testing/release/updates/moz191-firefox-mac.cfg,v  <--  moz191-firefox-mac.cfg
new revision: delete; previous revision: 1.1
done
Removing moz191-firefox-win32.cfg;
/cvsroot/mozilla/testing/release/updates/moz191-firefox-win32.cfg,v  <--  moz191-firefox-win32.cfg
new revision: delete; previous revision: 1.2
done

Comment 23

[[e-mail] (:bhearsum)]

Attachment: [checked in] add update verify builders to production release automation

Almost identical to the staging patch in bug 454205.

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 346925 [details] [diff] [review]
[checked in] add update verify builders to production release automation

The comments on attachment 346924 [details] [diff] [review] apply here too, the baseTag is a mustfix on checkin.

Comment 25

[[e-mail] (:bhearsum)]

Comment on attachment 346925 [details] [diff] [review]
[checked in] add update verify builders to production release automation

changeset:   495:5ce10d495c75

Comment 26

[[e-mail] (:bhearsum)]

Now that the initial config is setup I'm going to close this bug and we can take care of actually adding builders in the other dependent bugs -- that's what seems to be happening anyways.