Closed Bug 377929 Opened 15 years ago Closed 14 years ago

mozilla2 buildbot master setup

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ted, Assigned: bhearsum)

References

Details

Attachments

(7 files, 10 obsolete files)

3.87 KB, text/plain
rhelmer
: review+
Details
12.83 KB, patch
rhelmer
: review+
Details | Diff | Splinter Review
11.46 KB, patch
rhelmer
: review+
Details | Diff | Splinter Review
924 bytes, patch
benjamin
: review+
Details | Diff | Splinter Review
5.46 KB, patch
nthomas
: review+
Details | Diff | Splinter Review
1.82 KB, patch
Details | Diff | Splinter Review
1.02 KB, patch
nthomas
: review+
Details | Diff | Splinter Review
Setup a buildbot building from the Moz2 Hg repo.  In order of importance, we need:
1) Repeating build reporting to tinderbox server
2) Nightly builds pushed to FTP
3) Nightly updates
Actually we don't need to report to a tinderbox server, we could use a buildbot report directly.
(In reply to comment #1)
> Actually we don't need to report to a tinderbox server, we could use a buildbot
> report directly.

I'd like to do both, if possible, for a number of reasons.

We've been putting the buildbot masters in more-secured areas for now.
(In reply to comment #0)
> Setup a buildbot building from the Moz2 Hg repo.  In order of importance, we
> need:
> 1) Repeating build reporting to tinderbox server

Do we want to only build on checkin? I would say yes, the only drawback I can think of is that Tinderbox will drop the column if the machine hasn't reported back in a while. I think robcee came up with something to work around this for the unit test boxes, like having a time-based scheduler that runs every so many hours.

Looks like there isn't a bbot poller for hg, but there is a script in buildbot/contrib/hg_buildbot.py that can be used as an "incoming" hook script and generates Changes (using sendchange). It looks simple enough, requires buildbot to be installed on the repository machine (it doesn't need to be running there, just installed).

I'll see how much trouble it'd be to contribute an hg poller to buildbot. It looks like mercurial supports RSS feeds for checkins, that might be a nice place to plug into it.
Attached file master.cfg (obsolete) —
My master.cfg
Attached file hgpoller.py (obsolete) —
hgpoller.py, still needs some work
The timestamp parsing logic wasn't correct, this fixes it.
Attachment #268524 - Attachment is obsolete: true
Attachment #268610 - Flags: review?(rhelmer)
Comment on attachment 268523 [details]
master.cfg

Let's get this moving!
Attachment #268523 - Flags: review?(rhelmer)
Comment on attachment 268523 [details]
master.cfg

Looks good; where do you want to land, maybe mozilla/tools/buildbot-configs/moz2 ?
Attachment #268523 - Flags: review?(rhelmer) → review+
Comment on attachment 268610 [details]
fixed stupid timezone logic

Please get this upstream as well; create a ticket on buildbot.net
Attachment #268610 - Flags: review?(rhelmer) → review+
Blocks: 390789
Filed the HgPoller in buildbot: http://buildbot.net/trac/ticket/76
Checked in the master.cfg and related files to tools/buildbot-configs/mozilla2 (as ironic as it is to have Moz2 stuff in CVS), and the hgpoller.py into tools/buildbot/changes/.  Should we keep this bug open for the actual work of setting up a master and slaves to use this?
Morphing this over to be about setting up a buildmaster.
Assignee: ted.mielczarek → build
Summary: mozilla2 buildbot setup → mozilla2 buildbot master setup
Depends on: 391246
Depends on: 391248
Depends on: 391251
I have said this over in the buildbot ticket, let's restate this here, too:

- we should avoid urlopen, and use twisted's getPage instead, if we want to do remote calls.

- it might be a good idea to actually just pull/update a repository on the master and do the change magic locally, I can see several good reasons to have a working copy on the master. I haven't tested it, but twisted.internet.threads.deferToThread might be the right thing to pull a repository in-process by using hg's python api. See http://twistedmatrix.com/projects/core/documentation/howto/threading.html, too.
Assignee: build → nobody
QA Contact: mozpreed → build
Assignee: nobody → rhelmer
Assignee: rhelmer → bhearsum
(In reply to comment #13)
> Set up here:
> http://staging-build-console.build.mozilla.org:2004/

This is no longer true, but the master.cfg attached to this bug should be all you need Ben.
It's checked in to buildbot-configs/mozilla2, from comment 11.
Depends on: 414479
Depends on: 414614
Priority: P3 → P2
I spent some time this morning trying to get botrunner up and running. I've decided against using it for this, it still seems pretty buggy :(.
No longer depends on: 414614
Depends on: 414937
Botrunner would have made things easier to maintain... sorry to hear it needs work, but thanks for the update. 
This is a bit of a hacky class but I think it's generic enough to be useful to many people. I don't like that it's using a ShellCommand but there is simply _no_ other easy of running multiple commands in Buildbot. I could've done this as a bunch of BuildSteps but I think this is more self-contained and maintainable.

So far I've only tested it on my Macbook. I'm going to test it on the Mozilla2 Buildbot today, which will provide win32 and linux coverage.
Attachment #301717 - Flags: review?(rhelmer)
Attached patch same thing, diffed properly (obsolete) — Splinter Review
Sorry, I diffed this dumbly.
Attachment #301717 - Attachment is obsolete: true
Attachment #301720 - Flags: review?(rhelmer)
Attachment #301717 - Flags: review?(rhelmer)
So, I've got this up and running on the Mozilla2 Buildbot right now. If you logon to moz2-linux-slave1 as cltbld and look in ~/stage you'll see the upload area. The first win32 build is yet to complete but so far I'd say it's looking good.
Comment on attachment 301720 [details] [diff] [review]
same thing, diffed properly

This does not work on win32. I think I'll have to call out to bash to make it work. I'll get a new patch up tomorrow.
Attachment #301720 - Attachment is obsolete: true
Attachment #301720 - Flags: review?(rhelmer)
This took me awhile to get going. Because it (sadly) relies on a big string of commands there was lots of pain in getting it working on win32 and linux. It seems to work well but there's a few things that I don't like, but couldn't find a workaround for:
1) Globbing to get the package names.
2) Using 'ln -fs' in symlinkDateDirCommand() - this makes this code work differently than Tinderbox does and may introduce a race condition (the directory may not exist between unlinking + relinking).
3) Relying on a big giant string. I tried to not do this by making a new SlaveCommand and kicking off many different buildbot.slave.commands.ShellCommand's but simply could not get this to work.

I don't know if 2) is a real issue. If it's not I think this is OK for now, though.
Attachment #301942 - Flags: review?(rhelmer)
Attachment #301942 - Flags: review?(nrthomas)
Attachment #301942 - Attachment is obsolete: true
Attachment #301980 - Flags: review?(rhelmer)
Attachment #301980 - Flags: review?(nrthomas)
Attachment #301942 - Flags: review?(rhelmer)
Attachment #301942 - Flags: review?(nrthomas)
Comment on attachment 301980 [details] [diff] [review]
don't let tinderbox-builds be rsynced

>Index: steps/__init__.py
>Index: steps/transfer.py
>--- /dev/null
>+++ steps/transfer.py

>+    def symlinkDateDirCommand(self, datedDir):
>+        # Tinderbox client tests for existence rather than forcibly re-linking.
>+        # Because ShellCommand's get passed through cmd.exe, which has
>+        # weird and awful quoting rules we cannot do anything that requires
>+        # quotes, ie. no 'if [ ! -h ... ]'
>+        return self._getBaseCommand(ssh=True) + ' ' + self.remoteHost + \
>+               ' ln -fs ' + datedDir + ' ' + path.join(self.remoteBasePath,
>+                                                       'nightly')


We discussed this a bit... I'm surprised that you can't use quotes here, but you could have a shell command on the server to do this for you as last resort I guess..

This looks pretty good to me though, especially compared to http://mxr.mozilla.org/mozilla/source/tools/tinderbox/post-mozilla-rel.pl#1013 The comments around the weirder parts certainly help, and it's nice having all of these rules in one place.

Having an "uploadpackage" target in the build system (bug 404413) would make the hairier parts in here go away I think.
Attachment #301980 - Flags: review?(rhelmer) → review+
I'm wondering if should be using tinderbox-builds/$buildername rather than tinderbox-builds/$buildslavename since we'll definitely have multiple slaves in the future. I think it'd be extra confusing to have something like:
tinderbox-builds/moz2-linux-slave1
tinderbox-builds/moz2-linux-slave2
etc.

Having multiple directories would also mess-up Talos.

It'd be easy enough to have an option to swap between them, though. What do you guys think?
(In reply to comment #26)
> I'm wondering if should be using tinderbox-builds/$buildername rather than
> tinderbox-builds/$buildslavename since we'll definitely have multiple slaves in
> the future. I think it'd be extra confusing to have something like:
> tinderbox-builds/moz2-linux-slave1
> tinderbox-builds/moz2-linux-slave2
> etc.
> 
> Having multiple directories would also mess-up Talos.
> 
> It'd be easy enough to have an option to swap between them, though. What do you
> guys think?

I would just do builder name, no reason for buildslave name to be exposed outside of buildbot imho.
Attachment #301980 - Attachment is obsolete: true
Attachment #302154 - Flags: review?(nrthomas)
Attachment #301980 - Flags: review?(nrthomas)
(In reply to comment #25)
> We discussed this a bit... I'm surprised that you can't use quotes here, but
> you could have a shell command on the server to do this for you as last resort
> I guess..

Yeah, I've struggled with the multiple layers of quotes on windows too. In the end, I think it's easier to just wrap your stuff in a shell/python script and execute that.

(In reply to comment #26)
> Having multiple directories would also mess-up Talos.

It'll complicate matters, but not terribly. We're dealing with something like that for the talos try servers already.
Attachment #302154 - Attachment is obsolete: true
Attachment #302617 - Flags: review?(nrthomas)
Attachment #302154 - Flags: review?(nrthomas)
Comment on attachment 302617 [details] [diff] [review]
at support for overriding the chmod mode

Liking this alot, but one major issue and a handful of nits.

>Index: steps/transfer.py
>+    def __init__(self, objdir, username, milestone, remoteHost, platform,
>+                 remoteBasePath, group=None, chmodMode=755, sshKey=None,
>+                 releaseToDated=True, releaseToLatest=True,
>+                 releaseToTinderboxBuilds=True, **kwargs):
>+        """

Nit: missing docs for objdir,
Uber-Nit: ordering of arguments varies between init definition, docs, assignment

>+        @type  remoteBasePath: string
>+        @param remoteBasePath: The directory on the server used as a base path
>+                               for these builds. eg:
>+                               /home/ftp/pub/mozilla.org/firefox

Nit: There's no mozilla.org in the path on the disk, it only appears in URLs.

>+        @type  releaseToTinderboxBuilds: bool
>+        @param releaseToTinderboxBuilds: If True, builds will be pushed to
>+                'remoteBasePath'/tinderbox-builds/$hostname. This should
>+                generally be set to True for all builds.

Nit: This (and the previous two) default to True but not documented

>+        ShellCommand.__init__(self, **kwargs)
>+        self.addFactoryArguments(objdir=objdir,
>+                                 platform=platform,
>+                                 username=username,
>+                                 milestone=milestone,
>+                                 remoteHost=remoteHost,
>+                                 remoteBasePath=remoteBasePath,
>+                                 group=group,
>+                                 sshKey=sshKey,
>+                                 releaseToDated=releaseToDated,
>+                                 releaseToLatest=releaseToLatest,
>+                                 releaseToTinderboxBuilds=releaseToTinderboxBuilds)

Missed chmodMode here and in assignment, which you already fixed.

>+    def getBuildID(self):
>+        # extract the start time from the status object so we can figure out
>+        # the buildid
>+        startTime = self.step_status.build.getTimes()[0]
>+        return strftime("%Y-%m-%d-%H", localtime(startTime))

Looks like it would be possible to end up with a different timestamp for directory and BuildID, eg if the buildbot job started a few minutes before the hour, the checkout carried you over, then the build process generates the BuildID. It's pretty convenient to have the directory name match the BuildID, so it'd be worth looking up application.ini/platform.ini out of the objdir instead.

>+    def syncCommand(self, src, dst):
>+        # rsync needs trailing slashes
>+        src += '/'
>+        dst += '/'
>+        return self._getBaseCommand(ssh=True) + ' ' + self.remoteHost + \
>+               ' rsync -avz ' + src + ' ' + dst 

Nit: No need to use -z here, if rsync honours it then it adds overhead when doing a local copy.

>+        if self.releaseToLatest:
>+            # 1) Create the directory on the staging server.
>+            # 2) If there was a dated release, symlink those files to the

Nit: s/symlink/rsync/
Attachment #302617 - Flags: review?(nrthomas) → review-
(In reply to comment #31)
> (From update of attachment 302617 [details] [diff] [review])

You're right about all of the nits, I'll correct them.

As for the BuildID, I had it in my head that it was Tinderbox which generated it -- which you say is wrong. It's going to be a little tricky to get it out a file, since this is all one big ShellCommand, but it should be alright.

I'll post an updated patch later.
Attached patch addresses Nick's nits + concern (obsolete) — Splinter Review
Alright, I've addressed all of your concerns, Nick. I added a new class, 'GetBuildID', as this is a pretty generic function (and should be useful to pretty much anyone building Mozilla on Buildbot). I decided not to omit the 'Mozilla' from the name since it lives in buildbotcustom, which should be distinction enough.
Attachment #302617 - Attachment is obsolete: true
Attachment #302823 - Flags: review?(rhelmer)
Attachment #302823 - Flags: review?(nrthomas)
I propose that these be commited back to http://hg.mozilla.org/build/buildbot-configs/mozilla2. I don't have this _exact_ master.cfg running right now because I'm waiting on bug 417326. Once that's resolved I'll update the master.
Attachment #268523 - Attachment is obsolete: true
Attachment #303109 - Flags: review?(rhelmer)
Comment on attachment 302823 [details] [diff] [review]
addresses Nick's nits + concern

I'm going to say r+ for the overall approach and transfer.py, and let Rob weigh in on misc.py.
Attachment #302823 - Flags: review?(nrthomas) → review+
Comment on attachment 302823 [details] [diff] [review]
addresses Nick's nits + concern

Changes seem ok, not sure if platform.ini is the best place to get the build ID or not, but it's ok for now. Might want to ask Ted or Benjamin about that.
Attachment #302823 - Flags: review?(rhelmer) → review+
Comment on attachment 303109 [details] [diff] [review]
mozconfigs, master.cfg + associated files

>+        # TODO: does all of this have to be repeated? the only change is 'mode'

I think a custom Factory is probably the best way to go, but duplication is ok for now so we can get this going.

I like the more generic approach with the BRANCH dictionary, should make it much easier to add and remove feature branch coverage, etc. It might be worth splitting out the configurable stuff from the build instructions (e.g. "branches.py" include file or something) to make it easier for people to modify it directly.

No need to do that right now though, just thoughts for future refactoring.
Attachment #303109 - Flags: review?(rhelmer) → review+
 $(PYTHON) $(srcdir)/config/printconfigsetting.py $(DIST)/bin/application.ini App BuildID

http://lxr.mozilla.org/mozilla/source/Makefile.in#207
the command in GetBuildID is the only thing changed in this patch. I already did a test run on the Moz2 Buildbot and it works just the same as before.
Attachment #302823 - Attachment is obsolete: true
Attachment #303335 - Flags: review?(rhelmer)
Attachment #303335 - Flags: review?(rhelmer) → review+
Comment on attachment 303335 [details] [diff] [review]
[checked in] call out to printconfigsetting.py to get buildid

RCS file: /cvsroot/mozilla/tools/buildbotcustom/steps/__init__.py,v
done
Checking in steps/__init__.py;
/cvsroot/mozilla/tools/buildbotcustom/steps/__init__.py,v  <--  __init__.py
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/tools/buildbotcustom/steps/misc.py,v
done
Checking in steps/misc.py;
/cvsroot/mozilla/tools/buildbotcustom/steps/misc.py,v  <--  misc.py
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/tools/buildbotcustom/steps/transfer.py,v
done
Checking in steps/transfer.py;
/cvsroot/mozilla/tools/buildbotcustom/steps/transfer.py,v  <--  transfer.py
initial revision: 1.1
done
Attachment #303335 - Attachment description: call out to printconfigsetting.py to get buildid → [checked in] call out to printconfigsetting.py to get buildid
Comment on attachment 304229 [details] [diff] [review]
[checked in] add linux optimizations; mac UB support

r=me with the "you'll probably need to update the buildbot code to look in the right place for the UB package" proviso
Attachment #304229 - Flags: review?(benjamin) → review+
Comment on attachment 304229 [details] [diff] [review]
[checked in] add linux optimizations; mac UB support

changeset:   3:7d818c3a6624
tag:         tip
user:        bhearsum@mozilla.com
date:        Tue Feb 19 10:19:57 2008 -0500
files:       mozilla2/linux/mozconfig mozilla2/macosx/mozconfig
Attachment #304229 - Attachment description: add linux optimizations; mac UB support → [checked in] add linux optimizations; mac UB support
This update does the following:
* Use a new slave port (to stay in-line with the staging-master scheme)
* Fix config pulling by adding CONFIG_SUBDIR variable; updating BuildStep
* Add descriptions to the ShellCommand BuildSteps
* Use MozillaStageUpload 'dependToDated'
* Update the buildbotURL (moz2-master -> staging-master)
Attachment #304234 - Flags: review?(nrthomas)
Comment on attachment 304234 [details] [diff] [review]
[checked in] last round of master.cfg updates for moz2

Seems fine to me.
Attachment #304234 - Flags: review?(nrthomas) → review+
Comment on attachment 304234 [details] [diff] [review]
[checked in] last round of master.cfg updates for moz2

changeset:   3:7b01640de0e3
tag:         tip
user:        bhearsum@mozilla.com
date:        Tue Feb 19 11:55:30 2008 -0500
files:       mozilla2/master.cfg
Attachment #304234 - Attachment description: last round of master.cfg updates for moz2 → [checked in] last round of master.cfg updates for moz2
The lock creation has been moved outside of the loop; it turns out that we get a lock per branch when it's inside -- which is obviously not useful (and breaks the config by causing duplicate lock names). I've also added in a step to remove old builds before doing anything else. Nick, this should avoid any duplicate uploads after future version bumps.
Attachment #304286 - Flags: review?(nrthomas)
Comment on attachment 304286 [details] [diff] [review]
make locks work; remove old builds

Not sure why you need to remove old packages for the nightly. Doesn't the clobber remove the whole working directory ? Otherwise looks fine.
Attachment #304286 - Flags: review?(nrthomas) → review+
(In reply to comment #49)
> (From update of attachment 304286 [details] [diff] [review])
> Not sure why you need to remove old packages for the nightly. Doesn't the
> clobber remove the whole working directory ?
> 

Yeah, you're absolutely right.
changeset:   4:9d16335457f1
tag:         tip
user:        bhearsum@mozilla.com
date:        Wed Feb 20 07:35:51 2008 -0500
files:       mozilla2/master.cfg
Attachment #304286 - Attachment is obsolete: true
We don't have a script to clean up dependToDated builds yet (afaik), and within a few days these builds took up nearly 2GB. For now, let's leave them off.

I also bumped up the periodic build times to 2 hours. With 2 branches 20 minutes was way too low, and considering these branches are low activity right now there's no reason to build so often.
Attachment #304731 - Flags: review?(nrthomas)
Duplicate of this bug: 384943
Attachment #304731 - Flags: review?(nrthomas) → review+
Comment on attachment 304731 [details] [diff] [review]
[checked in] turn off dependToDated, bump Periodic builder to every 2 hours

changeset:   5:15f3a71c3e48
tag:         tip
user:        bhearsum@mozilla.com
date:        Thu Feb 21 11:02:28 2008 -0500
files:       mozilla2/master.cfg
Attachment #304731 - Attachment description: turn off dependToDated, bump Periodic builder to every 2 hours → [checked in] turn off dependToDated, bump Periodic builder to every 2 hours
Alright, I'd say this bug is complete. Getting Mac up will be tracked in bug 391251 and I'm going to file follow-ups about getting unit tests + talos for these builds.

When you want branches added/removed you can file a new bug about it.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Looks like I didn't mention this information anywhere yet:
I had to install Mercurial on the Windows slave, it's not currently part of the ref platform. Best way to solve this is getting it in MozillaBuild.

I also had to set MOZ_NO_RESET_PATH=1 in the windows environment variables.

Buildbot on win32 has to be started directly with twistd, this is due to a Buildbot bug. Here's a sample command-line:
/d/buildbot/python24/python /d/buildbot/python24/scripts/twistd.py --no_save --logfile=twistd.log --reactor=win32 --python=/e/builds/moz2-slave/buildbot.tac (executed from /e/builds/moz2-slave).
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.