508672 - Upgrade buildbot to 0.8.0

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

Newer is more better!

buildbot 0.7.11p1 has a few enhancements that would benefit us:
* Better management of event and change caches.  It actually cleans out changes.pck now!
* Show pending builds on builder pages, and allows us to cancel pending builds

Comment 1

[alice nodelman [:alice] [:anode]]

Futuring till we have time to work on this.

Comment 2

[[e-mail] (:bhearsum)]

(In reply to comment #0)
> * Show pending builds on builder pages, and allows us to cancel pending builds

Sexy.

Comment 3

[Rail Aliiev [:rail]]

BTW, 0.7.11p3 fixes some XSS vulnerabilities.

Comment 4

[Chris AtLee [:catlee]]

Running into problems with 0.7.12 on the server, and 0.7.10 on the slave:

2010/01/13 16:13 PST [Broker,client] Traceback (most recent call last):
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/python/log.py", line 53, in callWithLogger
            return callWithContext({"system": lp}, func, *args, **kw)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/python/log.py", line 38, in callWithContext
            return context.call({ILogContext: newCtx}, func, *args, **kw)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/python/context.py", line 59, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/python/context.py", line 37, in callWithContext
            return func(*args,**kw)
        --- <exception caught here> ---
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite
            why = getattr(selectable, method)()
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/internet/tcp.py", line 348, in doRead
            return self.protocol.dataReceived(data)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/banana.py", line 184, in dataReceived
            gotItem(item)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/banana.py", line 114, in gotItem
            self.callExpressionReceived(item)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/banana.py", line 81, in callExpressionReceived
            self.expressionReceived(obj)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/pb.py", line 559, in expressionReceived
            method(*sexp[1:])
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/pb.py", line 870, in proto_message
            self._recvMessage(self.localObjectForID, requestID, objectID, message, answerRequired, netArgs, netKw)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/pb.py", line 901, in _recvMessage
            self._sendError(f, requestID)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/pb.py", line 953, in _sendError
            self.sendCall("error", requestID, self.serialize(fail))
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/pb.py", line 807, in serialize
            return jelly(object, self.security, None, self)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/jelly.py", line 879, in jelly
            return _Jellier(taster, persistentStore, invoker).jelly(object)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/jelly.py", line 408, in jelly
            return obj.jellyFor(self)
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/flavors.py", line 308, in jellyFor
            sxp.extend([t, jellier.jelly(state)])
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/jelly.py", line 468, in jelly
            sxp.append([self.jelly(key), self.jelly(val)])
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/jelly.py", line 460, in jelly
            sxp.append(self.jelly(item))
          File "/tools/twisted-2.4.0/lib/python2.5/site-packages/twisted/spread/jelly.py", line 496, in jelly
            (objType, obj))
        <class 'twisted.spread.jelly.InsecureJelly'>: Type not allowed for object: <type 'type'> <class 'twisted.spread.flavors.NoSuchMethod'>

2010/01/13 16:13 PST [Broker,client] ReconnectingPBClientFactory.failedToGetPerspective
2010/01/13 16:13 PST [Broker,client] we lost the brand-new connection
2010/01/13 16:13 PST [Broker,client] <twisted.internet.tcp.Connector instance at 0xb7a0606c> will retry in 2 seconds
2010/01/13 16:13 PST [Broker,client] Stopping factory <buildbot.slave.bot.BotFactory instance at 0xb7ad7e6c>
2010/01/13 16:13 PST [-] Starting factory <buildbot.slave.bot.BotFactory instance at 0xb7ad7e6c>
2010/01/13 16:13 PST [Broker,client] ReconnectingPBClientFactory.failedToGetPerspective
2010/01/13 16:13 PST [Broker,client] Traceback from remote host -- Traceback (most recent call last):
          File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 194, in addCallback
            callbackKeywords=kw)
          File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 185, in addCallbacks
            self._runCallbacks()
          File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 323, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/tools/buildbot-catlee/buildbot/master.py", line 354, in requestAvatar
            d = defer.maybeDeferred(p.attached, mind)
        --- <exception caught here> ---
          File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/internet/defer.py", line 105, in maybeDeferred
            result = f(*args, **kw)
          File "/tools/buildbot-catlee/buildbot/buildslave.py", line 171, in attached
            d = self.disconnect()
          File "/tools/buildbot-catlee/buildbot/buildslave.py", line 286, in disconnect
            return self._disconnect(self.slave)
          File "/tools/buildbot-catlee/buildbot/buildslave.py", line 301, in _disconnect
            slave.notifyOnDisconnect(_disconnected)
          File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/spread/pb.py", line 297, in notifyOnDisconnect
            self.broker.notifyOnDisconnect(self._disconnected)
          File "/tools/twisted-8.0.1/lib/python2.5/site-packages/twisted/spread/pb.py", line 618, in notifyOnDisconnect
            self.disconnects.append(notifier)
        exceptions.AttributeError: 'NoneType' object has no attribute 'append'


Maybe need a newer version of twisted?

Comment 5

[Chris AtLee [:catlee]]

Both Talos and Build slaves require upgrades as well.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Chris, /tools/buildbot points to a buildbot-catlee on mv-moz2-linux-ix-slave01, but returns this

$ buildbot start /builds/slave/
'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "/tools/buildbot/bin/buildbot", line 4, in <module>
    from pkg_resources import require; require('buildbot==latest')
ImportError: No module named pkg_resources

Left as is, ack'd in nagios.

Comment 7

[Chris AtLee [:catlee]]

(In reply to comment #6)
> Chris, /tools/buildbot points to a buildbot-catlee on mv-moz2-linux-ix-slave01,
> but returns this
> 
> $ buildbot start /builds/slave/
> 'import site' failed; use -v for traceback
> Traceback (most recent call last):
>   File "/tools/buildbot/bin/buildbot", line 4, in <module>
>     from pkg_resources import require; require('buildbot==latest')
> ImportError: No module named pkg_resources
> 
> Left as is, ack'd in nagios.

Yes, that's because PYTHONHOME and PYTHONPATH are set up in ~/.bash_profile, I'll comment those out for now.

Comment 9

[Chris AtLee [:catlee]]

Attachment: Various changes required for buildbot 0.8.0

Most affected are the schedulers and l10n code.

This code is incompatible with previous versions of buildbot, so this needs to land in another branch or separate repo until everything is ported over.

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 439595 [details] [diff] [review]
Various changes required for buildbot 0.8.0

I didn't line-by-line this, but based on what you showed me the other week and a high level look at this, it seems fine.

Comment 12

[Axel Hecht [:Pike]]

Glancing at the patch, I see mostly removal of l10n infra, is that intended?

Comment 13

[Chris AtLee [:catlee]]

(In reply to comment #12)
> Glancing at the patch, I see mostly removal of l10n infra, is that intended?

In a way, yes.  A lot of the code isn't required since change properties came into play.  The rest of the code needed to be ported to the new way that schedulers have to behave.

I believe that the new code is functionally equivalent.

Comment 14

[Chris AtLee [:catlee]]

Comment on attachment 439595 [details] [diff] [review]
Various changes required for buildbot 0.8.0

changeset:   705:4fc214d9e3e4
branch:      buildbot-0.8.0

Comment 15

[Axel Hecht [:Pike]]

(In reply to comment #13) 
> I believe that the new code is functionally equivalent.

I don't think so, fwiw.

Comment 16

[[e-mail] (:bhearsum)]

(In reply to comment #15)
> (In reply to comment #13) 
> > I believe that the new code is functionally equivalent.
> 
> I don't think so, fwiw.

Can you be more specific?

Comment 17

[Axel Hecht [:Pike]]

You stopped parsing l10n.ini files, right? So whatever depends on that changed.

Comment 18

[Chris AtLee [:catlee]]

Attachment: Python 2.6 manitests for linux

Comment 19

[[e-mail] (:bhearsum)]

(In reply to comment #17)
> You stopped parsing l10n.ini files, right? So whatever depends on that changed.

That's not a helpful answer. What depends on the l10n.ini files?

Comment 20

[Axel Hecht [:Pike]]

l10n.ini controls which applications to kick off for which directories on which branches, en-US or l10n-wise.

Comment 21

[[e-mail] (:bhearsum)]

(In reply to comment #20)
> l10n.ini controls which applications to kick off for which directories on which
> branches, en-US or l10n-wise.

Okay, we should move that to config.py, then.

Comment 22

[[e-mail] (:bhearsum)]

Comment on attachment 441883 [details] [diff] [review]
Python 2.6 manitests for linux

I don't think you want to subscribe to /tools/python for this version. I think 2.5/2.6 would stomp on each other if you did that.

Comment 23

[Axel Hecht [:Pike]]

I don't intend to keep track on where that data ends up, I'm maintaining l10n.inis.

Comment 24

[[e-mail] (:bhearsum)]

(In reply to comment #23)
> I don't intend to keep track on where that data ends up, I'm maintaining
> l10n.inis.

What's the difference between editing a value in an ini and editing one in a .py?

Comment 25

[Armen [:armenzg]]

(In reply to comment #20)
> l10n.ini controls which applications to kick off for which directories on which
> branches, en-US or l10n-wise.

If I am not mistaken this can be done with isFileIsImportant.

Comment 26

[Chris AtLee [:catlee]]

Attachment: Python 2.6 manifests for linux

Comment 27

[Axel Hecht [:Pike]]

The l10n.ini information is shared between compare-locales and the scheduler, so instead of having to edit the configuration in multiple places, that only needs to be done, in the main source repo, together with the change that added the directory structure.

Comment 28

[Chris AtLee [:catlee]]

(In reply to comment #20)
> l10n.ini controls which applications to kick off for which directories on which
> branches, en-US or l10n-wise.

Ok, the way I'm handling this is specifying which directories to monitor in config.py:
http://hg.mozilla.org/build/buildbot-configs/file/daec2af2f26b/mozilla/config.py#l388

Then using isFileImportant on the scheduler to look for these directories:
http://hg.mozilla.org/build/buildbotcustom/file/b15567df30f7/misc.py#l549

which uses isImportantL10nFile here:
http://hg.mozilla.org/build/buildbotcustom/file/b15567df30f7/misc.py#l102

Am I missing anything?

Comment 29

[Armen [:armenzg]]

(In reply to comment #27)
> The l10n.ini information is shared between compare-locales and the scheduler,
> so instead of having to edit the configuration in multiple places, that only
> needs to be done, in the main source repo, together with the change that added
> the directory structure.

After catlee's changes only compare-locales will be using this information.
The info with regards the modules is moved to:
BRANCHES['mozilla-central']['l10n_modules'] = ['browser','dom','toolkit']
The removal in l10n.py will not affect compare-locale since it already has its own L10nConfigParser in:
http://hg.mozilla.org/build/compare-locales/file/844d4ed299a8/lib/Mozilla/Paths.py#l46

Am I missing anything?

catlee we are going to need this entry in mobile_config.py as well (even if it is currently disabled).

Comment 30

[Axel Hecht [:Pike]]

You're missing the point how you're getting changes to that data. See bug 561032 for an upcoming example.

Comment 31

[Axel Hecht [:Pike]]

Attachment: test to move l10n.ini parsing into a builder

Here's a little play stuff I did this morning.

The unused RevMercurial step is kinda nice, because you can pass in an explicit revision. It should probably add a render(self.revision) and render(self.property) still to support WithProperties. Anyway, that source step should enable us to use real source steps instead of our hacky shell scripts for the source checkouts. Just in there because it was part of the game.

The TreeLoader step is actually more interesting, it moves parsing of l10n.ini and it's children into a builder. There's no code yet to pass that info back to a scheduler, but that should be rather trivial. Benefits: We keep a single config, the load of the master.cfg is sync again, and a scheduler can trigger reruns of that build on changes in l10n.ini's, removing the need to reconfig or restart for those. I also succeeded in forcing a build via the web ui.

All tested on a vanilla 0.7.12 buildbot.

Comment 32

[Chris AtLee [:catlee]]

Attachment: Update for .bash_profile

Comment 34

[Chris AtLee [:catlee]]

Attachment: Update buildbot to '0.8.0pre'

Comment 35

[[e-mail] (:bhearsum)]

Comment on attachment 445114 [details] [diff] [review]
Update buildbot to '0.8.0pre'

If we need a 2nd buildbot 0.8.0pre we can just use 0.8.0pre-1, so this seems like a good improvement.

Comment 36

[Chris AtLee [:catlee]]

Attachment: Updates to profile and buildbot start script for darwin10

Comment 37

[Chris AtLee [:catlee]]

Attachment: Updates to profile and buildbot start script for darwin9

Comment 38

[Chris AtLee [:catlee]]

Comment on attachment 445114 [details] [diff] [review]
Update buildbot to '0.8.0pre'

changeset:   142:dd3399f33200

Comment 39

[Chris AtLee [:catlee]]

Attachment: Update buildbot to 0.8.0 (final)

Comment 40

[Chris AtLee [:catlee]]

Attachment: Port changes from mozilla2, mozilla2-staging/ to mozilla/

Comment 41

[[e-mail] (:bhearsum)]

Comment on attachment 447612 [details] [diff] [review]
Update buildbot to 0.8.0 (final)

woo woo!

Comment 42

[Chris AtLee [:catlee]]

Comment on attachment 447617 [details] [diff] [review]
Port changes from mozilla2, mozilla2-staging/ to mozilla/

changeset:   2444:842e9f53c3e8

Comment 43

[Chris AtLee [:catlee]]

Comment on attachment 447612 [details] [diff] [review]
Update buildbot to 0.8.0 (final)

changeset:   162:1df6be038ade

Comment 44

[Chris AtLee [:catlee]]

Attachment: Update fedora/fedora64 slaves to buildbot 0.8.0 via puppet

Comment 45

[Chris AtLee [:catlee]]

Comment on attachment 448077 [details] [diff] [review]
Update fedora/fedora64 slaves to buildbot 0.8.0 via puppet

changeset:   168:eb96e7cda1cc

Comment 46

[Chris AtLee [:catlee]]

Attachment: Install simplejson for buildbot on windows

Comment 47

[Chris AtLee [:catlee]]

Attachment: Staging configs for sm02 on buildbot-080, some fixes for setup-master

Comment 48

[[e-mail] (:bhearsum)]

Comment on attachment 449324 [details] [diff] [review]
Staging configs for sm02 on buildbot-080, some fixes for setup-master


>+L10N_SLAVES = {
>+    'linux': SLAVES['linux'],
>+    'win32': SLAVES['win32'],
>+    'macosx': MAC_MINIS + XSERVES,
>+}

64-bit variants are going to be added to this list in mozilla2/ soon, may as well do that now here. (https://bug559057.bugzilla.mozilla.org/attachment.cgi?id=449236)


>+for branch in ACTIVE_BRANCHES:
>+    branchObjects = generateBranchObjects(BRANCHES[branch], branch)
>+    # No change sources here please!
>+    c['builders'].extend(branchObjects['builders'])
>+    c['status'].extend(branchObjects['status'])
>+    c['schedulers'].extend(branchObjects['schedulers'])
>+    c['change_source'].extend(branchObjects['change_source'])

The comment here looks wrong.

Everything else looks fine, and feel free to fix the above on landing.

Comment 49

[Chris AtLee [:catlee]]

Comment on attachment 449324 [details] [diff] [review]
Staging configs for sm02 on buildbot-080, some fixes for setup-master

changeset:   2482:4971ce1423ad

Comment 50

[Chris AtLee [:catlee]]

Comment on attachment 449271 [details] [diff] [review]
Install simplejson for buildbot on windows

changeset:   51:7a7d4b5d5b8f

Comment 51

[Chris AtLee [:catlee]]

Attachment: Don't use --old-and-unmanageable to install buildbot

This fixes the case where buildbot was originally installed without --old-and-unmanageable and so has an entry in easy_install.pth.  Subsequent installations using --old-and-unmanageable have no affect.

Now we blow away all buildbot directories in site-packages before installing, and then install normally.

Comment 52

[Chris AtLee [:catlee]]

Comment on attachment 451334 [details] [diff] [review]
Don't use --old-and-unmanageable to install buildbot

52:721b78460763

Comment 53

[Chris AtLee [:catlee]]

I'm going to resolve this fixed.  We've deployed 0.8.0 to most of our masters and slaves.

The major pieces left on older versions of buildbot are the release builders (bug 553300) and mobile (bug 574729).