Closed Bug 322260 (cvs-branch-mirroring) Opened 18 years ago Closed 18 years ago

Implement server-side CVS mirror-to-branch-or-trunk feature

Categories

(mozilla.org :: CVS: Administration, task)

Product:
mozilla.org
Component:
CVS: Administration
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: brendan, Unassigned)

References

(
URL
)

Details

Attachments

(2 files)

client-side script
19.20 KB, patch
Details | Diff | Splinter Review
Patch that would merge MOZILLA_1_8_BRANCH with HEAD in mozilla/browser/
571.82 KB, patch
Details | Diff | Splinter Review 
Keyed by directory in repository (recursive by default: browser and all its subdirs, not just browser), with file-wise exception list per directory.

See http://wiki.mozilla.org/Global:1.9_Trunk_1.8_Branch_Plan for motivation.

The point is to get something done on the server side that benefits everyone hacking over the next ~6 months on Firefox 2 and Gecko 1.9/Firefox 3 code, saving people the error-prone and too-avoidable hassles of checking Firefox front end and some new (Places, e.g.) back end work twice.

myk has a client-side solution for those who need something sooner, but it would be best if we could revive the server solution linked from this bug's URL and make it righteous, soon.

Polvi suggested I file this under server ops, which I took to mean mozilla.org, not webtools.  If we get something truly righteous, I'm sure tarah et al. will be happy to consider it.

/be
Here's the client-side script.  I propose we check it in as mozilla/webtools/cross-commit.
I saw mail showing the old meme of cmd-line stuff in tools, CGIs and other web services in webtools being upheld.  So mozilla/tools/cross-commit.  Just noting that fact here.

/be
Brendan/Mike:

Couple of questions for ya:

I've skimmed through the tools we have and was thinking about the possible server-side solutions to this problem and I wanted to know what you were expecting the usage model to be for developers.

I'm mostly interested in the front-end input to the system and the (mostly backend) feedback mechanism. I'm also curious what about the timing of merges.

I've seen systems that do this sort of "patch moving" implement the request in a couple of ways: a) part of the checkin message; the pro here is that the request is included directly in the checkin message for post-mortem purposes; the con is that it requires some futzing at CVS checkin-time on the client-side (basically, this is a special case of the venerable CVS template). b) An external interface, (likely Bonsai for us) where after a developer checks in a patch, they select it via Bonsai to have it checked in there.

In terms of feedback (due to a merge conflict, etc.), were you expecting a webpage to display this type of information, or email to the developer with results (or both)?

Also, I've seen two implementations of such systems, whereby changes are globbed together (conceptually) and submitted at certain points in time (noon, say); the reasoning behind this was ensuring that when/if changes broke branches, people were available to look at it and fix it before the nightly builds. The other option is to have merges to other branches occur throughout the day, as developers make checkins. I'd think we'd lean towards the latter, given the decentralized nature of checkins, but it increases the possibility that target-branch breakages due to "automerged" code would go up, so it's a tradeoff.

If you can give me an idea of the "use cases" you were envisioning for such a tool, that'll be helpful in figuring out more specifically where to focus effort.

-preed
(In reply to comment #3)
> Brendan/Mike:
> 
> Couple of questions for ya:

Sure, glad you are on the case (are you gonna take the bug from justdave?  He'll be relieved ;-).

> I've seen systems that do this sort of "patch moving" implement the request in
> a couple of ways: a) part of the checkin message; the pro here is that the
> request is included directly in the checkin message for post-mortem purposes;
> the con is that it requires some futzing at CVS checkin-time on the client-side
> (basically, this is a special case of the venerable CVS template).

What I proposed in http://wiki.mozilla.org/Global:1.9_Trunk_1.8_Branch_Plan would be more implicit: every commit under mozilla/browser, mozilla/toolkit and probably some subdirs of mozilla/xpfe would be mirrored automatically.  There may be need for a file-wise exception list.  Not sure how to organize, but the upshot is admin files on the server, I think.

Note that due to the #ifdef MOZILLA_1_8_BRANCH usage, mirrored revisions must match between trunk and branch.  So no merging, so no possibility of conflict.  This may be a simplifying assumption for the fix to this bug, but it's actually intented to avoid trunk/branch API skew, minimizing it where it must happen.

> b) An
> external interface, (likely Bonsai for us) where after a developer checks in a
> patch, they select it via Bonsai to have it checked in there.

This seems less desirable to me.  People forget, or there's a delay and more room for conflicts or bugs in one tree but not the other causing wasted time.

> In terms of feedback (due to a merge conflict, etc.), were you expecting a
> webpage to display this type of information, or email to the developer with
> results (or both)?

No merging, no conflicts, fail commit up front if anything violates this constraint.

> Also, I've seen two implementations of such systems, whereby changes are
> globbed together (conceptually) and submitted at certain points in time (noon,
> say); the reasoning behind this was ensuring that when/if changes broke
> branches, people were available to look at it and fix it before the nightly
> builds. The other option is to have merges to other branches occur throughout
> the day, as developers make checkins. I'd think we'd lean towards the latter,

Yes.  Really, I'd rather simultaenous mirroring.

> given the decentralized nature of checkins, but it increases the possibility
> that target-branch breakages due to "automerged" code would go up, so it's a
> tradeoff.

Tinderbox, peer pressure, the usual feedback loops should keep things working and fix regressions in either trunk or branch quickly.

> If you can give me an idea of the "use cases" you were envisioning for such a
> tool, that'll be helpful in figuring out more specifically where to focus
> effort.

I hope this helps.  Ask anything.  Thanks,

/be
So I went through and re-read the 1.8/1.9 branching plan, and I *think* we're kinda talking about two different models to do code movement here:

This bug describes mirroring a certain set of subdirectories; changes would get merged immediately at checkin time and if the merge failed for some reason, then the commit would be rejected on all branches; is my understand correct?

I think I was reading this as a model whereby a checkin would go in on a target branch, and AFTER it's committed, be mirrored around the tree as appropriate; that model requires some front- and back-end notification/control, hence the supposition on which my initial set of questions were based.

Fundamentally, I'm a little worried about the a model that blindly mirrors certain directories, for a couple of reasons:

1. There's no (simple) way for a developer to say "No, really... I don't want this change on the trunk." It's an all-or-nothing model. ifdefs were proposed to solve that sort of problem, but I'm thinking of the more generalized case whereby a developer wants to check in a certain feature/fix on the 1.8.0 branch, and it really isn't supposed to go into the trunk (for whatever reason). I understand the reason for ifdefs, but in some sense, I think they're orthogonal to the issue of moving code around the tree; it's an implementation detail of this particular development cycle.

2. If we're doing this by directory levels, I could imagine a situation where a related set of changes gets split up and "goes missing." This may be a non-issue, given how the tree is setup (that's information I'm not completely clear on yet), but I could see someone checking in something that may have a configure flag or otherwise-dependent change outside the sets of directories that get mirrored.

I'd hate for someone to spend a bunch of time chasing a bug that exists merely because part of their change didn't get mirrored when it should have (because those directories weren't in the set for consideration).

I think it would be of more longterm use to provide developers with machinery that would mirror a CVS checkin (in its entireity) to another (related) branch. 

I *think* that solves this problem, but it also solves future problems that fit into this category (and even current problems; I would imagine it would be of use for 1_8_0_BRANCH fixes that need to get to the 1_8_BRANCH).

The particular devil here is the front/back-end notification details. There are a number of ways of handling this problem, but I totally hear you on the developers-forgetting problem. This is *somewhat* solvable by posing the question to the developer at checkin time... but that has its own issues.

Anyway, let me know what you think about all this; if you think doing "dumb mirroring," as it were, is the best way to go to accomplish the goal in the timeframe we're working with, we can focus on that and extend it to be smarter (if we need it) later. I'm just a bit skeptical, given that I've never seen development work so cleanly in practice (where all the constraints that are set out in that plan are enforceable all the time). Maybe we're more well-behaved than other teams, and there is precedent for this to work that way. :-)

If you think there's value in adding some smarts now and using it for this, as well as other plans, let me know, and we can talk about the simplest/easist-for-the-developer way to do that.

In terms of actual development time, I don't think we're talking all that much difference (some, yes, but I don't think it should be a huge factor in the decision).

Taking, per conversations with be and justdave.
Assignee: justdave → preed
(In reply to comment #5)
> This bug describes mirroring a certain set of subdirectories; changes would get
> merged immediately at checkin time and if the merge failed for some reason,
> then the commit would be rejected on all branches; is my understand correct?

Yes.

> I think I was reading this as a model whereby a checkin would go in on a target
> branch, and AFTER it's committed, be mirrored around the tree as appropriate;

You mean the revision tree?  Really, we want the Firefox front end files, and some new middle- or back-end files, but not most back end files, to be mirrored and never merged.

API compatibility, and #ifdef MOZILLA_1_8_BRANCH where API incompatibility is unavoidable, will keep the mirrors clean.

What you thought this was about could be handy too, but it requires dozens or more hackers to get their merge-commit operations correct at scale in time and (file tree) space.  Firefox 2 developers I've talked to have no desire to do this by hand when most who are on the trunk now will be moving to the 1.8 branch shortly.

> Fundamentally, I'm a little worried about the a model that blindly mirrors
> certain directories, for a couple of reasons:
> 
> 1. There's no (simple) way for a developer to say "No, really... I don't want
> this change on the trunk."

Sure there is -- #ifdef MOZILLA_1_8_BRANCH (in C++ or JS -- we preprocess the latter; also in Makefile.in sources if necessary).

> It's an all-or-nothing model. ifdefs were proposed
> to solve that sort of problem, but I'm thinking of the more generalized case
> whereby a developer wants to check in a certain feature/fix on the 1.8.0
> branch, and it really isn't supposed to go into the trunk (for whatever
> reason).

This bug is about managing the Firefox 2 work in the trunk and the 1.8 branch, so that people don't have to commit twice, and so there is pressure to conserve APIs.  It's not about general merging.

Although the 1.8.0-only fix case you cite is more about merging, and I'd love a tool to automate the stupid cvs log and update -j commands I've been typing too much lately, that's not this bug.

> I understand the reason for ifdefs, but in some sense, I think they're
> orthogonal to the issue of moving code around the tree;

Code is not moving around the source tree, and the only movement in the revision tree is merge-free because of the mirroring.  This model is intentionally simple to avoid missed checkins and by-hand effort, even if semi-automated.

> 2. If we're doing this by directory levels, I could imagine a situation where a
> related set of changes gets split up and "goes missing." This may be a
> non-issue, given how the tree is setup (that's information I'm not completely
> clear on yet),

Front end directories do not contain back end code, and vice versa.  Although I mentioned an exception list, I'd be happy to start without it.

> but I could see someone checking in something that may have a
> configure flag or otherwise-dependent change outside the sets of directories
> that get mirrored.

That change would have to be checked into the "other tree" (trunk or branch) by hand.  But that's the less likely scenario with this tool, compared to the more likely one without it: that people don't update trunk for too long, even a day but worse, a month.

> I'd hate for someone to spend a bunch of time chasing a bug that exists merely
> because part of their change didn't get mirrored when it should have (because
> those directories weren't in the set for consideration).

This can happen without the mirroring tool, of course.  If everyone has to think through every change, and redo it (without some degree of automation) against the other tree, they'll still have the opportunity to forget one file.  But they have more chances to mess up in general, or just postpone.  Postponing means other people see a different merge target and may get conflicts, or just bugs (unfixed when they thought they were by the pending change) or subtler problems due to the skew between the two Firefox front ends.

> I think it would be of more longterm use to provide developers with machinery
> that would mirror a CVS checkin (in its entireity) to another (related) branch. 

This is useful too, but the human factors are bad for a case where we have good front- vs. back-end separation and zero desire to redo any front end checkins on the other tree.

/be
(In reply to comment #6)
> This is useful too, but the human factors are bad for a case where we have good
> front- vs. back-end separation and zero desire to redo any front end checkins
> on the other tree.

By "redo" I mean "redo as a separate step, with whatever automation to minimize effort, but still with a conscious separate step."  The perfectly automated mirroring of front end source is the goal of this bug, as opposed to other bugs wanting different tools useful in other cases.

Again (from the wiki doc), we don't want to merge every day, week or month using automated CVS joins that dead-reckon from moving static tags, because we need tinderbox-granularity builds and regression commit windows on both trees.

/be
FWIW, I don't want this to happen in any subdirectory of toolkit *except* for toolkit/components/places (doesn't exist yet, but will contain the places backend) and toolkit/components/history (which will end up gone or mostly-gone). mconnor, do you have any other directories that are going to get significant revision and will need trunk<->branch automation?
For toolkit, places or whatever it'll be called in the tree may be the only mirrored subtree, but that's still a substantial source subset to keep in synch by hand, however semi-automated.

The subtree under mozilla/browser, OTOH, should be all-mirrored.  It isn't just the Places work, but all Firefox 2 browser source, that we want to mirror.

Having a general mirror-this-trunk-checkin-to-that-branch tool would be handy for sure, and don't let me stop one from being built.  If we had that, then hooking up the "last mile" from that tool to mirroring by source tree subdirectory would be easy -- or so I hope.

/be
Benjamin, what's the concern with fully automating what's under toolkit and using #ifdefs?  I realize that its more work, but its ifdefs or a lot of manual porting to branch.

components, content, locales, mozapps, and themes will all see enough traffic that I would vastly prefer to automate all of this.  If not, someone would have to ride shotgun on the excluded directories and make sure the right patches get in the right place.  With the number of patches that touch both browser and toolkit, if we're automating browser but not toolkit, we're really going to run into problems with partial commits, given the human error factor.  Unless I get a couple more helper monkeys, I don't have cycles to play enforcer.
(In reply to comment #10)
> With the number of patches that touch both browser and
> toolkit, if we're automating browser but not toolkit, we're really going to run
> into problems with partial commits, given the human error factor.  Unless I get
> a couple more helper monkeys, I don't have cycles to play enforcer.

It's worse than that.  To keep trunk Firefox working without a second front end team, we need its source to be close to Firefox 2 alpha -- that's the whole idea of the branch plan: that we are not doing concurrent development, rather front end for Firefox 2, but built and QA'd against the trunk back end as well as against the 1.8 branch back end.

I'd like to know why toolkit is mostly going to evolve independently on the trunk, and if so, how API compatibility will be maintained.  Or is the plan to #ifdef MOZILLA_1_8_BRANCH a bunch of uses under browser of toolkit APIs?

API compatibility, with exceptions, remains critical to our integrated development methods and the odds of success in the next year.

/be
The API compat issue I thought has been beaten to death, so I didn't go into it.  

I have no plans of moving toolkit in an diverging direction on the trunk, and if Benjamin has such plans, I'm refreshingly uninformed.  I know we want to move forward on the trunk with xulrunner and the toolkit, but I think we need to accept (and communicate clearly) that major changes in toolkit can't really get to trunk before Firefox 2 is done.  That's a sacrifice that'll unfortunately hurt Benjamin more than anyone else, but I think its necessary for the next six months.
A loop over a linked list 300 elements is not going to hang anything.  Maybe the parent chain was corrupted and there's an iloop?  Can you get more data from gdb?

/be
Sorry, wrong bug.

/be
toolkit/library is already well beyond any chance of merging it back to the 1.8 branch (we've merged a lot of extensions/ into libxul): in toolkit/xre we have significantly reworked how the directory provider and other pieces fit together and are going to be working on restart-within-process. None of that work can really land on branch. Pieces such as toolkit/components/autocomplete have been changed to facilitate bug 263042, and toolkit/components/startup has taken "scary" changes that aren't branch-compatible. None of these changes significantly impact the firefox FE.

I don't have significant plans that will change the toolkit XUL/JS/XBL bindings (except in extension manager). I'm planning on holding off 305746 until after ff2 is finished. And I think that XBL2 development will have to be on a branch anyway and can land after ff2.

But, I don't see any *need* to auto-port toolkit/: after all it is relatively stable backend code which is unlikely to see a lot of churn. And what churn it does experience is more likely to be caused by gecko-layer changes (cocoa widgets) than by the FE.
Benjamin: if toolkit is truly back end, great.  If mostly, then what is places doing as a toolkit component, if it's really particular to the Firefox front end?

/be
Toolkit is shades of grey: some parts are clearly backend (xre, library, components/startup), some clearly frontend (components/console, content/fullScreen.js), some straddle (content/widgets, mozapps/extensions, mozapps/update). I have a strong desire to avoid auto-merging of the clearly-backend pieces which are undergoing significant build and/or architectural changes which don't ifdef well (auto-merging would basically involve conditionally building entirely forked files on MOZILLA_1_8_BRANCH).

The places backend will live in the toolkit to serve as the replacement history impl for the toolkit/components/history implementation which is being removed.

If someone (i.e. Benjamin, since he's the chief beneficiary in this case) is going to be the bonsai-watcher, I'm fine with auto-syncing a smaller subset of toolkit.  That said, there's probably something to be said for splitting up toolkit into separate chunks for backendy platform bits (i.e. library/xre) and shared frontendy bits (i.e. content/mozapps).
 (In reply to comment #6)
> (In reply to comment #5)
> > This bug describes mirroring a certain set of subdirectories; changes would get
> > merged immediately at checkin time and if the merge failed for some reason,
> > then the commit would be rejected on all branches; is my understand correct?
> 
> Yes.

The implementation looks like it will be *slightly* different than this description, but not by much; minor details to make some corner cases go away.

Anyway, one of your described invariants is that the two branches will be exactly the same when mirroring is turned on.

To test this out, I ran:

cvs di -w -u -rHEAD -rMOZILLA_1_8_BRANCH browser/

and it's very not-empty (to the tune of 6200 lines; without -w, it's worse...), so I'm wondering: is there going to be a huge syncup of the two branches at some point to fulfill that initial condition, or?
Status: NEW → ASSIGNED
Does that "cvs diff" count all the lines in files that are new on the trunk and not yet on the 1.8 branch, like browser/components/places?  Those are not as troublesome as files that are on both branches but have differences, IMO.
(In reply to comment #20)
> Does that "cvs diff" count all the lines in files that are new on the trunk and
> not yet on the 1.8 branch, like browser/components/places?  Those are not as
> troublesome as files that are on both branches but have differences, IMO.

No, I didn't run it with -N (but I probably should have; good catch); they're real diffs.

Some of them are trivial (like:

--- browser/app/nsBrowserApp.cpp        3 Jan 2006 04:19:02 -0000       1.32
+++ browser/app/nsBrowserApp.cpp        23 Aug 2005 20:09:45 -0000      1.31.2.2
@@ -51,7 +51,7 @@
   NS_STRINGIFY(APP_VERSION),
   NS_STRINGIFY(BUILD_ID),
   "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}",
-  "Copyright (c) 1998 - 2006 mozilla.org",
+  "Copyright (c) 1998 - 2005 mozilla.org",
   NS_XRE_ENABLE_PROFILE_MIGRATOR |
   NS_XRE_ENABLE_EXTENSION_MANAGER
 };
and some that have only semantic differences) but others will require more pondering.
We've had the 1.8 branch under approval-only, so naturally the trunk has run ahead. The sync'up will be a mini-big-bang, but after that we should get the desired fine-grained tinderbox-based regression windowing.

Cc'ing darin, bryner and ben for their thoughts on how best to get the 1.8 branch and trunk mozilla/browser (and whatever else might be needed) in sync.  This could be spun out to another bug, but it seems worth going a bit further here -- thanks, preed.

/be
There are 34 fixed Firefox bugs requesting approval1.8.1:
https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&product=Firefox&resolution=FIXED&field0-0-0=flagtypes.name&type0-0-0=equals&value0-0-0=approval1.8.1%3F

Why don't you grant these approvals and ask the bug owner for prompt branch checkin?
(In reply to comment #23)
> There are 34 fixed Firefox bugs requesting approval1.8.1:
> https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&product=Firefox&resolution=FIXED&field0-0-0=flagtypes.name&type0-0-0=equals&value0-0-0=approval1.8.1%3F
> 
> Why don't you grant these approvals and ask the bug owner for prompt branch
> checkin?

Because we are not going to require approvals for 1.8 branch checkins shortly, and until within a few weeks of beta.  So not only is there no point in manually rubberstamping, that make-work would not suffice to sync the trunk browser tree to the branch, or keep it synced.  Your comments really don't fit in this bug.

/be

I did some research following comment 19 to see what would need to be done to get the 1.8 branch and HEAD in sync.

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot diff -R --unified=8 --show-c-function --new-file -r MOZILLA_1_8_BRANCH -r HEAD mozilla/browser/

It lists 147 files. Many changes are trivial, and could be checked in without any adjustments (there are a lot of copyright changes, as well as Gerv's licensing changes) but some would need #ifdef changes.

Some stats on directories mentioned in this bug:

mozilla/toolkit/ has 239 files that differ.
mozilla/browser/components/places/ has 0.
mozilla/toolkit/components/history/ has 3.
mozilla/toolkit/library/ has 5.
mozilla/toolkit/xre/ has 11.
mozilla/toolkit/components/autocomplete/ has 2.
mozilla/toolkit/components/startup/ has 2.
mozilla/toolkit/content/widgets/ has 17.
mozilla/toolkit/mozapps/extensions/ has 5.
mozilla/toolkit/mozapps/update/ has 11.

Of course, this doesn't count how large the differences are in each file, but it's a rough idea.

I hope this information was a help not a hinderance.
Alias: cvs-branch-mirroring
So what happened to this idea? Is anybody still working (or planning to work) on it? Have trunk and branch diverged so much that it's not feasible? Or is it not worth bothering because manual simultaneous checkins work good enough?
(In reply to comment #26)
> So what happened to this idea? Is anybody still working (or planning to work)
> on it?

I'm going to re-assign to build, since I'm not working on it anymore, and unfortunately, don't think I'll have time to right now. :-/
Assignee: preed → build
Status: ASSIGNED → NEW
Mass re-assign of bugs that aren't on the build team radar, so bugs assigned to build@mozilla-org.bugs reflects reality.

If there is a bug you really think we need to be looking at, please *email* build@mozilla.org with a bug number and explanation.
Assignee: build → nobody
Experience shows that manual mirroring isn't that bad. WONTFIX
Status: NEW → RESOLVED
Closed: 18 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.