428307 - tracking bug for upgrading buildbot installations to 0.7.7

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[[e-mail] (:bhearsum)]

Comment 1

[[e-mail] (:bhearsum)]

Huh, not sure how wanted1.9.0.x? got set

Comment 2

[[e-mail] (:bhearsum)]

Good news, Buildbot 0.7.7 works (out-of-box) with Twisted 2.5/Python 2.5 on Windows. I had to do a bit of hackery to get Twisted and Pywin32 installed with the MozillaBuild Python - because it doesn't install any registry keys.

I'll be attaching a .reg file that needs to be installed before Twisted will work.

python25 needs to be added to the path before Pywin32 will install. On the reference platform, this is d:\mozilla-build\python25.

I hit the 'ping builder' problem (http://buildbot.net/trac/ticket/155) where a builder simply doesn't work until the master is restarted.

That's all for now, more to come on this later.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: python25 registry key

Comment 4

[[e-mail] (:bhearsum)]

Rob pointed out that he had problems with Twisted 2.5 + Buildbot 0.7.5 in the past - namely, logging stopped after the log was rotated. We should find out if this is still a problem.

Comment 5

[[e-mail] (:bhearsum)]

Another note: Anything using the 'categories' feature of the Waterfall will have to change a bit. Instead of having different ports for different categories you can filter WebStatus as a whole like so:
http://localhost:8010/waterfall?category=two

Comment 6

[[e-mail] (:bhearsum)]

I've confirmed that Buildbot 0.7.7 + Python 2.5 + Twisted 2.5 == broken logging on the master. This is not the case when using Twisted 2.4 or Twisted 8.0.1. How do people want to proceed here? IMHO, the simplest solution is to use Python 2.5 + Twisted 2.4 on Linux and Mac (this is what we already do - requires no work) and use Python 2.5 + Twisted 2.5 on Windows (Twisted 2.4 doesn't have binary package for Python 2.5, and we can't compile it without an old compiler). This would put Windows on a different version of Twisted than everything else but AFAICT that's not going to hurt anything -- I know we've run Buildbots w/ mixed Twisted's in the past. If there's concern over mixed versions of Twisted we could try upgrading all of them to Twisted 8.0.1.

Comment 7

[Rob Campbell [:rc] (:robcee)]

I'm reluctant to make the jump to Twisted 8.0.1 without some additional testing on it. That said, I think running different versions of Twisted on our Windows bots is a bit dicey as well. On the current production unittest boxes, we're using Twisted 2.4 from http://tmrc.mit.edu/mirror/twisted/Twisted/2.4/ on Python 2.4 which has some other limitations.

So, what can we do here? Since I'm upgrading the moz2 unittest boxes, I could install Twisted 8.0.1 in there and see how that works out. Did we ever get any comment from the buildbot dev community about what version of Twisted is preferred?

Comment 8

[[e-mail] (:bhearsum)]

(In reply to comment #7)
> So, what can we do here? Since I'm upgrading the moz2 unittest boxes, I could
> install Twisted 8.0.1 in there and see how that works out. Did we ever get any
> comment from the buildbot dev community about what version of Twisted is
> preferred?
> 

I see two options (assuming mismatched Twisteds is out of the question):
1) Try out Twisted 8.0.1, see how that goes. If you're OK with trialing it on Moz2 unit tests that's cool - I'm also happy to give it a try on the staging try server.
2) Use python24/twisted24. This is probably more safe, but I feel that if we're going to upgrade, now may be a good time. Also worth noting: this option is less work since this is what we run now.

I forgot to ask in my mailing list post about the preferred version of Twisted. FWIW, Buildbot claims to be compatible with 2.0 or higher (this was written before 8.0 was released, however). (source: http://buildbot.net/repos/release/NEWS)

Comment 9

[Rob Campbell [:rc] (:robcee)]

(In reply to comment #8)
> (In reply to comment #7)
> 
> I see two options (assuming mismatched Twisteds is out of the question):
> 1) Try out Twisted 8.0.1, see how that goes. If you're OK with trialing it on
> Moz2 unit tests that's cool - I'm also happy to give it a try on the staging
> try server.

I'll add that to my list of stuff to do. See bug 429332.

> 2) Use python24/twisted24. This is probably more safe, but I feel that if we're
> going to upgrade, now may be a good time. Also worth noting: this option is
> less work since this is what we run now.

More safe but we might be limiting our options down the road if we end up needing some better patches to help us with our windows process killing problems.

> I forgot to ask in my mailing list post about the preferred version of Twisted.

I replied to your post with a follow-up. It's sitting in moderation.

> FWIW, Buildbot claims to be compatible with 2.0 or higher (this was written
> before 8.0 was released, however). (source:
> http://buildbot.net/repos/release/NEWS)

Some good detail in there, particularly here in the 0.7.6 notes:

> Buildbot is tested on a regular basis (http://buildbot.buildbot.net) against
> nearly a full matrix of Python-(2.3,2.4,2.5) * Twisted-(2.0,2.1,2.2,2.4,2.5).

I wonder if we're just hitting something their tests aren't covering?

Comment 10

[Rob Campbell [:rc] (:robcee)]

quick update, I've been using python 2.5.2 + twisted 8.01 + buildbot 0.7.7 on the moz2 unittest machines for the past week without any noticeable problems on mac and linux. I'm going to see how that works on windows, but so far I'm recommending this configuration for our current standard.

Comment 11

[[e-mail] (:bhearsum)]

Given that, I'm going to set the try server up with Twisted 8.01 and see how that goes.

Comment 12

[Rob Campbell [:rc] (:robcee)]

naturally, there are issues.

I've been using zope.interface 3.4.1 on the linux and mac machines, but there is a compilation required for windows using an incompatible MSVC.

http://pypi.python.org/pypi/zope.interface/3.4.1

Ben also found http://twistedmatrix.com/trac/ticket/3144 and is looking for a build.

Comment 13

[[e-mail] (:bhearsum)]

Found one here: http://www.zope.org/Members/saffe/zope_interface/zope.interface-3.3.0.win32-py2.5.exe

Comment 14

[[e-mail] (:bhearsum)]

The package installed fine, Buildbot started fine...I'll kick off a build just to make sure nothing explodes.

Comment 15

[[e-mail] (:bhearsum)]

Alright, the try server is all upgraded to Twisted 8.0.1.

Comment 16

[[e-mail] (:bhearsum)]

My part here is done.

Comment 17

[[e-mail] (:bhearsum)]

This has been done for ages.

Comment 18

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.