425390 - (buildbot-0.7.7) tracking bug for importing buildbot 0.7.7 into our tree

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[[e-mail] (:bhearsum)]

The release of Buildbot 0.7.7 is imminent. This bug will track all tasks related to getting it in our tree.

I've done a little bit of work over in bug 425204 in getting better parity between our tree and the upstream one but there are a few differences still. We've preemptively landed many 0.7.7 fixes in our tree so it's difficult to determine exactly what's in our tree that's not in the upstream one. I'm going to try and figure out what we have in our tree that's not upstream. I'll attach a diff later.

Comment 1

[[e-mail] (:bhearsum)]

Alright, it's more complicated than I thought so a diff isn't going to be appropriate. Because we import a tarball, which contains some files that the buildbot cvs/darcs repos do not I had to do some filtering by hand. Here's the differences I've found though:
* buildbot/changes/freshcvsmail.py - exists in our repository, will be removed as of 0.7.7. This file needs to be manually deleted after the import.
* buildbot/changes/hgpoller.py - exists in our repository, does not exist upstream. This is slated to go into the 0.7.8 release; we should leave this file alone. (http://buildbot.net/trac/ticket/76)
* buildbot/changes/tinderboxpoller.py - exists in our repository, does not exist upstream. I filed bug 425408 on getting this moved to buildbotcustom - it should really live there if we aren't pushing it upstream.
* contrib/checkconfig.py - this file is getting moved to buildbot/scripts/checkconfig.py and integrated with the command-line (http://buildbot.net/trac/ticket/37). As long as this gets in 0.7.7 this file should be removed after the import.
* buildbots/scripts/runner.py - This is where the checkconfig patch mentioned above is being integrated with the command-line. There are currently differences between us and upstream but it is expected that the this patch will get into the 0.7.7 release. Assuming that, there will be no changes between us and upstream in this file in Buildbot 0.7.7.
* buildbot/changes/bonsaipoller.py - contains two patches that never made it upstream. I've filed a ticket on them but I'm not sure if they'll make it into 0.7.7. If they do, this file will be the same as upstream, if they don't it will be a clean merge - so either way we're OK. (http://buildbot.net/trac/ticket/216)
* buildbot/status/tinderbox.py - there are currently 3 changes in this file. One is an odd formatting differences between us and upstream. I have a patch to land that fixes this (bug 425204). The second is the date munging (useChangeTime) patch which never made it upstream. I filed http://buildbot.net/trac/ticket/213 to get it upstream. The final one is a scope fix that never made it upstream, filed in http://buildbot.net/trac/ticket/214. I do not know if these two patches will make it into 0.7.7, but it will be a clean merge if they do not.
* buildbot/steps/shell.py - This file has a one line conflict because of a patch we landed post-0.7.6 (see bug 403619). The final patch that got into Buildbot upstream differed a bit from it. It turns out that I landed that patch in bug 414479, which obsoletes the original patch. I've filed bug 425418 to get this removed.

There are some minor differences in a few files that appear to have their current CVS version in them -- I'm not clear what's happening there but they are non-code and I'm unconcerned.


All in all I'm very confident that the 0.7.7 import will go smoothly.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #1)
> Alright, it's more complicated than I thought so a diff isn't going to be
> appropriate. Because we import a tarball, which contains some files that the
> buildbot cvs/darcs repos do not I had to do some filtering by hand. Here's the
> differences I've found though:

If this tarball is the official one downloaded from the builbot.net, maybe we should we file a buildbot trac bug to have them fixup the tarball?

Comment 3

[[e-mail] (:bhearsum)]

(In reply to comment #2)
> (In reply to comment #1)
> > Alright, it's more complicated than I thought so a diff isn't going to be
> > appropriate. Because we import a tarball, which contains some files that the
> > buildbot cvs/darcs repos do not I had to do some filtering by hand. Here's the
> > differences I've found though:
> 
> If this tarball is the official one downloaded from the builbot.net, maybe we
> should we file a buildbot trac bug to have them fixup the tarball?
> 

The problem here is that there is not Buildbot 0.7.7 tarball yet - I'm working off of the current trunk. If there's still missing files when it's released we can see if that's intentional or not. I suspect this won't be the case though. IIRC the files in question are documentation related and generated at release time.

Comment 4

[[e-mail] (:bhearsum)]

Buildbot 0.7.7 was released yesterday. I'm doing a test import of the tarball to confirm my assumptions above. Here's the status:
* status/tinderbox.py patches didn't make it. There's a conflict in this file - easy to resolve.
* Still have to land config file tester parity patch. Looking to get that reviewed today.
* steps/shell.py - It looks like this got branched wrong when I did BUILDBOT_0_7_6_BRANCH. It's sticking off HEAD instead of BUILDBOT_0_7_6_RELEASE. Not sure we can do anything about this now but it's causing a bit of confusion with the import. It's easy to work around and manually fix. I'll make sure the BUILDBOT_0_7_7_BRANCH gets done properly.

Probably won't get to this for a week or two.

Comment 5

[[e-mail] (:bhearsum)]

Attachment: [checked in] make our config file tester like upstream

Hey Rob, no code changes here - just making our config file tester like upstream. Sorry to ask you to review this *again*.

Comment 6

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 313214 [details] [diff] [review]
[checked in] make our config file tester like upstream

no problem. Your detailed background makes this an easy review.

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 313214 [details] [diff] [review]
[checked in] make our config file tester like upstream

RCS file: /cvsroot/mozilla/tools/buildbot/buildbot/scripts/Attic/checkconfig.py,v
done
Checking in buildbot/scripts/checkconfig.py;
/cvsroot/mozilla/tools/buildbot/buildbot/scripts/Attic/checkconfig.py,v  <--  checkconfig.py
new revision: 1.1.2.1; previous revision: 1.1
done
Checking in buildbot/scripts/runner.py;
/cvsroot/mozilla/tools/buildbot/buildbot/scripts/runner.py,v  <--  runner.py
new revision: 1.2.2.2; previous revision: 1.2.2.1
done

on BUILDBOT_0_7_6_BRANCH

Comment 8

[[e-mail] (:bhearsum)]

Unless anyone has an objection, I will import 0.7.7 today. BUILDBOT_0_7_5_BRANCH and BUILDBOT_0_7_6_BRANCH will not be changed at all as a result of this import so our existing installations will not be affected even if a 'cvs up' is done on them.

While doing a final test import I realized that since the 0.7.5 and 0.7.6 branches were created we haven't landed anything on the trunk. I'm going to have to go back and re-commit some of this stuff to it after the import. For this reason I think we should explicitly _not_ create branches like this in the future. I was the one who created these branches in the first place -- I realize now I didn't understand the possible side effects.

Once we migrate all of our Buildbots to 0.7.7 we will not have any need to land fixes on old branches, so this should become unnecessary. If we feel we need to, we can move around the BUILDBOT_MOZILLA_PRODUCTION tag to wherever suits us, though.

Comment 9

[[e-mail] (:bhearsum)]

Alright, the import is done. It went smoothly and without surprises. I'll be attaching logs momentarily.

Comment 10

[[e-mail] (:bhearsum)]

Attachment: import log

Comment 11

[[e-mail] (:bhearsum)]

Attachment: conflict resolution checkout

Highlights here are:
* tinderbox.py conflict (expected)
* scripts.py conflict (expected)
* shell.py doesn't already contain the differences between 0.7.6 and 0.7.7 - this happened because shell.py was branched wrong when BUILDBOT_0_7_6_BRANCH was created.

Comment 12

[[e-mail] (:bhearsum)]

Attachment: diff of merges and conflict resolution

No surprise here, contains merges + resolution in runner.py and tinderbox.py. It's got a lot of changes for shell.py because of the aforementioned bad branching.

Also important: I said 'scripts.py' when I meant 'runner.py' in my previous comment.

Comment 13

[[e-mail] (:bhearsum)]

Attachment: unit test log

All of the unit tests pass. Arch, bzr, and git unit tests were skipped because I do not have them installed.

Comment 14

[[e-mail] (:bhearsum)]

Alright, this is done.