Closed Bug 465521 Opened 16 years ago Closed 16 years ago

client.py needs a way to update the default repo for m-c

Categories

(MailNews Core :: Build Config, defect)

defect
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED
mozilla1.9.1b2

People

(Reporter: Callek, Assigned: Callek)

References

Details

Attachments

(2 files, 3 obsolete files)

I just thought of this, if our trunk comm-central will follow m-c's branch (divergence to a new repo url) then we'll need a way to ensure our client.py [can] update[s] the "pull from" url for m-c. Basically both for, "when we pull m-c branch the first time" and when trunk moves to pulling m-c trunk again... I'm not sure the best way to do this, so I'd value input. My best thought is to sed into hgrc (and if the m-c stable url is found, replace it, otherwise do nothing) and update with -C if something is done. (or error/warn that you'll need to use -C to switch branches, loosing work under mozilla/). In the end, programatically it would be easier if we just branched c-c and changed the default clone of c-c then, rather than worry about this on trunk. But learning-curve-wise (and buildbot wise) that method is more work.
Flags: blocking-thunderbird3?
Flags: blocking-seamonkey2?
(In reply to comment #0) > I just thought of this, if our trunk comm-central will follow m-c's branch > (divergence to a new repo url) then we'll need a way to ensure our client.py > [can] update[s] the "pull from" url for m-c. > > Basically both for, "when we pull m-c branch the first time" and when trunk > moves to pulling m-c trunk again... client.py supports a --mozilla-repo=http:// argument. > I'm not sure the best way to do this, so I'd value input. My best thought is to > sed into hgrc (and if the m-c stable url is found, replace it, otherwise do > nothing) and update with -C if something is done. (or error/warn that you'll > need to use -C to switch branches, loosing work under mozilla/). Why is all this necessary? To allow migration of existing checkouts when the branching happens ? > In the end, programatically it would be easier if we just branched c-c and > changed the default clone of c-c then, rather than worry about this on trunk. Yes, and that's effectively what we did for Aplha 3, except instead of branching, we just cloned comm-central, but essentially the same idea. > But learning-curve-wise (and buildbot wise) that method is more work. Why do you think this is? I suspect I am missing the original problem this is trying to solve.
The problem with branching/cloning/whatever c-c early is that we then get the issue of maintaining a trunk and a branch. If we just track m-c 1.9.1 then we can branch later and sort out re-merging with trunk when we are ready to do it.
(In reply to comment #1) > > Basically both for, "when we pull m-c branch the first time" and when trunk > > moves to pulling m-c trunk again... > > client.py supports a --mozilla-repo=http:// argument. Yes, but if we are already cloned (from default trunk) its either "educate people to pull correct m-c branch" or "have potentialy self-compiling testers be on an unsupported configuration" (which will likely break due to CORE changes) > > I'm not sure the best way to do this, so I'd value input. My best thought is to > > sed into hgrc (and if the m-c stable url is found, replace it, otherwise do > > nothing) and update with -C if something is done. (or error/warn that you'll > > need to use -C to switch branches, loosing work under mozilla/). > > Why is all this necessary? To allow migration of existing checkouts when the > branching happens ? Exactly. That is "when m-c branching happens". > > In the end, programatically it would be easier if we just branched c-c and > > changed the default clone of c-c then, rather than worry about this on trunk. > > Yes, and that's effectively what we did for Aplha 3, except instead of > branching, we just cloned comm-central, but essentially the same idea. err, typo.. I'd be changing the default clone url of m-c from our branch. (we can do this on trunk, but it breaks in the "switch back" for people who start compiling between now and when we do branch. and it breaks for the "switch to" for those of us who compile now. > > But learning-curve-wise (and buildbot wise) that method is more work. > > Why do you think this is? I suspect I am missing the original problem this is > trying to solve. We can always through people time at buildbot updating as far as urls, etc. But as far as educating people already compiling, thats the hard part. (In reply to comment #2) > The problem with branching/cloning/whatever c-c early is that we then get the > issue of maintaining a trunk and a branch. If we just track m-c 1.9.1 then we > can branch later and sort out re-merging with trunk when we are ready to do it. I definitely agree here, and if we can come up with a solution to this bug, we can definitely avoid branching earlier than needed.
We could require people who aren't using the m-c branch we default to to specify the mozilla repo URL every time they run client.py
(In reply to comment #4) > We could require people who aren't using the m-c branch we default to to > specify the mozilla repo URL every time they run client.py That might work, but I'm not sure if that is conducive to productivity. As it should definitely be possible for people to pull from a separate local clone, or even a friend without problems. (think proxy in a company situation). Though, I'd rs+ on the concept since it does sound the simplest.
(In reply to comment #2) > The problem with branching/cloning/whatever c-c early is that we then get the > issue of maintaining a trunk and a branch. If we just track m-c 1.9.1 then we > can branch later and sort out re-merging with trunk when we are ready to do it. Every time I've seen this tried at Mozilla, it's been a disaster. Things diverge too much, and the re-merge either doesn't happen (MailNews Nova, NVu) or is extremely painful and some stuff doesn't really get re-merged correctly (Aviary 1.0). As painful as it is, the only things I've ever seen work is "develop on trunk; bake; land on branch thereafter" or "develop mostly on trunk; cross-commit".
Oh, hmm, I partially misunderstood your suggestion. I haven't seen the "branch this part of the tree later" thing tried; that might work, but we still need to pay some attention to both trunk and branch of mozilla-central in that case: the failures always happen when one branch of the tree gets ignored.
Depends on: 464640
(In reply to comment #7) > the failures always happen when one branch of the tree gets ignored. we can always land in sequence to client.py before reopening the tree for "trunk" work, specifying varying m-c revs to build against... and correcting problems in "shifts" if we want ;-) That is a general tree-wide solution that would likely work.
I have been thinking on this and it seems we have a few options... the first I will list was approved in concept by the TB drivers and at least KaiRo from SeaMonkey. Though it does cause problems for Neil, (who is the only person I know currently using local separate clones rather than pulling directly from http) 1) Force all users of separate local clones to specify their repo location always on client.py invocation. ** We can do an additional command-line arg, and/or an environ arg to make this less painful for those who need the functionality. 2) Do nothing. Force people to re-clone when we switch to the stable-branch[es] 3) rename mozilla to mozilla.trunk and re-pull moz1.9 to mozilla, and when we switch back, rename mozilla to mozilla.stable and switch back. ** This makes things much more complicated especially in the "local clone" sense, but should work near-flawlessly. Though will convolute things in the future when the 1.9.2 branch happens. I'm in favor of just doing #1 as it will be easiest, and we need a solution by monday. We can revisit #3 or other options before we even branch c-c. Any further thoughts of something we can do in this timeframe?
Target Milestone: --- → mozilla1.9.1b2
It seems to me that you're going to need to detect that you've already switched to 1.9.1 to avoid doing an update -C by mistake so what's so hard about making the test "!=mozilla-central" instead of "=releases/mozilla-1.9.1"?
Re fixing this by Monday: worst case we could change to just pulling the FF beta 2 tag for mozilla-central builds seeing as they seem to be doing that in the mozilla-central repos. That would give us a couple of days breathing space.
(In reply to comment #11) > Re fixing this by Monday: worst case we could change to just pulling the FF > beta 2 tag for mozilla-central builds seeing as they seem to be doing that in > the mozilla-central repos. That would give us a couple of days breathing space. That would work for a few extra days, sure. I'd rather that be just "a few extra days" rather than prolong it. And certainly I would rather have the choice of which way to go for the 1.9.1 branching before monday, even if not implemented by then.
I don't think we can do that, since the JS-in-mail changes we want for b1 are partly in Gecko, and there's no way Gecko would accept between now and branch time.
So the current thinking from #maildev is that we should: - Make it possible for c-c to be built with m-c trunk and m-c 1.9.1 - Use #ifdefs where required to keep the code compiling with both versions of m-c. (whilst this is practical, there may be some point where it gets too complicated to do this). We still need to get client.py to move us to the m-c 1.9.1 branch, and a way to pull m-c trunk. To clone c-c with m-c trunk afresh we could use a argument to client.py, but then how about getting it to write a file somewhere and then when doing updates, if that file exists it doesn't try to switch to m-c 1.9.1 branch.
> We still need to get client.py to move us to the m-c 1.9.1 branch, and a way to > pull m-c trunk. To clone c-c with m-c trunk afresh we could use a argument to > client.py, but then how about getting it to write a file somewhere and then > when doing updates, if that file exists it doesn't try to switch to m-c 1.9.1 > branch. So which method of actually moving should I/we shoot for in the short term? Can we get some sort of mini-design-doc. I don't mind implementing this, it just will help if we have a direction before I really write too much of the code ;-)
heres what I propose... * If patch is not done/reviewed by monday morning (aka: official m-c branch) set client.py to pull the BASE tag of m-c, temporarily (anyone should push that with an rs=me) * parse the |mozilla/.hg/hgrc| file with pythons INI parser, if it exists * if the default path == m-c default, mv/rename the |mozilla| dir to |.mozilla.trunk| (!note the . at front, to prevent most FS's, including MXR from showing it) * clone |mozilla| locally from |.mozilla.trunk| (will be much faster for most, and no network latency insued) * push a word-for-word hgrc to the new |mozilla/.hg/hgrc| with updated-to-branch values for default and default-push * create a "we're done" file of some sort to check at start of this execution path to prevent any of this checking if we've been done before (this "we're done" file will be listed in .hgrc) * continue co "as normal" ... then in a followup, we can get a --use-trunk and a --use-stable option to client.py which will switch the two tree's around, (rename |mozilla| to |.mozilla.stable|) and (|.mozilla.trunk| to |mozilla|) in said followup, if the .trunk or .stable's are missing clone from scratch (as likely at that point the history diverged somehow) and in said followup, that "we're done" file can maintain the system's state (as in, "are we using .stable or .trunk") ;-) Anyone disagrees with this plan, please comment in here ASAP
(In reply to comment #16) > then in a followup, we can get a --use-trunk and a --use-stable option to > client.py which will switch the two tree's around, (rename |mozilla| to > |.mozilla.stable|) and (|.mozilla.trunk| to |mozilla|) in said followup, if the > .trunk or .stable's are missing clone from scratch (as likely at that point the > history diverged somehow) I think the steps you propose are fine, but please make the options --use-mc-trunk and --use-mc-stable to save folks getting confused as to what the options relate to.
Attached patch temp change (obsolete) — Splinter Review
KaiRo, if this is ok (which it seems it is, given no voiced objections) can you [or really anyone] land this in AM? I'll be unable to land myself and I have at most one more night to get my patch up to speed.
Assignee: nobody → bugspam.Callek
Status: NEW → ASSIGNED
Attachment #349699 - Flags: review?(kairo)
(In reply to comment #18) > Created an attachment (id=349699) [details] > temp change > > KaiRo, if this is ok (which it seems it is, given no voiced objections) can you > [or really anyone] land this in AM? I object to landing this patch at the moment: According to http://oduinn.com/2008/11/24/branch-diagram-including-the-new-ff31-project-branch/ FF 3.1b2 *will* come off the current mozilla-central build. Therefore we should temporarily be using that flag (when it happens) rather than going back to the base tag which was about 4/5 days ago.
The base tag will be moved to being the base of when the actual divergence of the branch happens, at least that was what they said all along - and I guess we wouldn't check this in before they actually do that move.
fwiw, from last night before I added this: <Callek> nthomas: since I'm not finding the info atm, are we updating GECKO_1_9_1_BASE to be the correct point of divergance, or are we just going to use the b2 tag as that point? <nthomas> Bug 464640 comment 21 implies the BASE tag will be updated
Attached patch move_to_stable (obsolete) — Splinter Review
This actually updates us to use m-c stable fairly easily (local clone) Note with this ver of patch we re-clone DOMi, and re-pull all CVS files, is that ok? This is not too pretty but client.py will need to be cleaned up a bit anyway... I also wrote out the treestatus file as an INI-syntax (with ConfigParser) because I figured that would be the best way to set it up for future possible changes
Attachment #350418 - Flags: review?(kairo)
Comment on attachment 350418 [details] [diff] [review] move_to_stable david can you skim my python to see if there is anything glaring back at me that I may have missed as far as correctness changes go?
Attachment #350418 - Flags: review?(david.ascher)
Comment on attachment 350418 [details] [diff] [review] move_to_stable Mark, I'd also value your input, but I won't hold landing this up. Whichever of you or KaiRo r+ this first I'll take. I can fix any issues in followup landings (or even bugs)
Attachment #350418 - Flags: review?(bugzilla)
Comment on attachment 350418 [details] [diff] [review] move_to_stable After a quick look, I'm 99% sure this doesn't affect full source clobber builds (e.g. nightlies) or developers pulling from scratch. They will currently remain on mozilla-central if I've read the patch correctly.
Attachment #350418 - Flags: review?(bugzilla) → review-
Attachment #350418 - Flags: review?(kairo) → review-
Comment on attachment 350418 [details] [diff] [review] move_to_stable >diff --git a/client.py b/client.py > # URL of the default hg repository to clone for Mozilla. > DEFAULT_MOZILLA_REPO = 'http://hg.mozilla.org/mozilla-central/' > DEFAULT_MOZILLA_REV = "GECKO_1_9_1_BASE" >+MOZILLA_BASE_REV = "GECKO_1_9_1_BASE" Shouldn't we change DEFAULT_MOZILLA_REPO as well (to branch)? And, of course, the patch probably doesn't apply to current trunk as we don't have that DEFAULT_MOZILLA_REV in yet (and we probably don't want that to stay there, right? >>+ #Compile the m-c regex >+ m_c_regex = re.compile("(ssh|http|https):\/\/hg\.mozilla\.org\/mozilla-central\/$", re.I) [...] >+ config.set('paths', 'default', >+ "%s://hg.mozilla.org/releases/mozilla-1.9.1/" % match.group(1) ) Could we make those repo URLs vars being placed at the top of the .py, where the other repo URLs are? e.g. something like MOZILLA_TRUNK_REPO(_REGEXP?) and either using the DEFAULT_MOZILLA_REPO or a separate var for branch. BTW, I want at least one person from the Thunderbird team to sign off on this approach - your choice of reviewers sounds good in that perspective ;-)
(In reply to comment #25) > (From update of attachment 350418 [details] [diff] [review]) > After a quick look, I'm 99% sure this doesn't affect full source clobber builds > (e.g. nightlies) or developers pulling from scratch. They will currently remain > on mozilla-central if I've read the patch correctly. Your right, thanks... :/ (I forgot to update the default url at top)(In reply to comment #26) > (From update of attachment 350418 [details] [diff] [review]) > >diff --git a/client.py b/client.py > > # URL of the default hg repository to clone for Mozilla. > > DEFAULT_MOZILLA_REPO = 'http://hg.mozilla.org/mozilla-central/' > > DEFAULT_MOZILLA_REV = "GECKO_1_9_1_BASE" > >+MOZILLA_BASE_REV = "GECKO_1_9_1_BASE" > > Shouldn't we change DEFAULT_MOZILLA_REPO as well (to branch)? And, of course, > the patch probably doesn't apply to current trunk as we don't have that > DEFAULT_MOZILLA_REV in yet (and we probably don't want that to stay there, > right? Yes re REPO to branch. The tag is currently "there" (aka: on trunk and branch") and I'm assured its being moved. as far as DEFAULT_.._REV thats just an artifact of my previous patch here that I did not remove from my MQ, it will stay 'tip'. > >>+ #Compile the m-c regex > >+ m_c_regex = re.compile("(ssh|http|https):\/\/hg\.mozilla\.org\/mozilla-central\/$", re.I) > [...] > >+ config.set('paths', 'default', > >+ "%s://hg.mozilla.org/releases/mozilla-1.9.1/" % match.group(1) ) > > Could we make those repo URLs vars being placed at the top of the .py, where > the other repo URLs are? e.g. something like MOZILLA_TRUNK_REPO(_REGEXP?) and > either using the DEFAULT_MOZILLA_REPO or a separate var for branch. We can, but considering that this is the only place to touch it, I'm not sure the exact benefit. but if you insist, sure. > BTW, I want at least one person from the Thunderbird team to sign off on this > approach - your choice of reviewers sounds good in that perspective ;-)
(In reply to comment #27) > > Could we make those repo URLs vars being placed at the top of the .py, where > > the other repo URLs are? e.g. something like MOZILLA_TRUNK_REPO(_REGEXP?) and > > either using the DEFAULT_MOZILLA_REPO or a separate var for branch. > > We can, but considering that this is the only place to touch it, I'm not sure > the exact benefit. but if you insist, sure. 1) I don't think this will be the last time we see Mozilla branch 2) Having all relevant repo URLs at the top gives us a nice overview of what trees we are relating to.
Attached patch v2 move_to_stable (obsolete) — Splinter Review
attempt 2, I purposely did not update the %s redoing of the stable-branch stuff, I *will* do that in a followup if I can (which will include some general client.py cleanup) Ignore the |DEFAULT_MOZILLA_REV = "GECKO_1_9_1_BASE"| in there please, I'll correct that on checkin (didn't feel like redoing my MQ order atm)
Attachment #350418 - Attachment is obsolete: true
Attachment #350550 - Flags: review?(kairo)
Attachment #350418 - Flags: review?(david.ascher)
Attachment #350550 - Flags: review?(bugzilla)
Comment on attachment 350550 [details] [diff] [review] v2 move_to_stable david, mainly requesting for python ok-ing
Attachment #350550 - Flags: review?(david.ascher)
Attachment #350550 - Flags: review?(kairo) → review-
Comment on attachment 350550 [details] [diff] [review] v2 move_to_stable >-DEFAULT_MOZILLA_REPO = 'http://hg.mozilla.org/mozilla-central/' >+DEFAULT_MOZILLA_REPO = 'http://hg.mozilla.org/mozilla-1.9.1/' It's /releases/mozilla.1.9.1/ ;-) >+# REGEX to match against, $1 should refer to protocol scheme >+MOZILLA_TRUNK_REPO_REGEXP = "(ssh|http|https):\/\/hg\.mozilla\.org\/mozilla-central\/$" Erm, nice to have this here, but why not use it down below in the code then? r- though because an upgrade pull break when trying to pull DOMi (as reported on IRC).
fix all of the above (discussed on IRC too)
Attachment #350550 - Attachment is obsolete: true
Attachment #350578 - Flags: review?(kairo)
Attachment #350550 - Flags: review?(david.ascher)
Attachment #350550 - Flags: review?(bugzilla)
Attachment #350578 - Flags: review?(bugzilla)
Attachment #350578 - Flags: review?(david.ascher)
Comment on attachment 350550 [details] [diff] [review] v2 move_to_stable >+MOZILLA_TRUNK_REPO_REGEXP = "(ssh|http|https):\/\/hg\.mozilla\.org\/mozilla-central\/$" >+def repo_config(): >+ import ConfigParser import statements should be hoisted to the top, so they don't get executed on each time that function gets called. It may not matter in this case, but it's a good habit. >+ import ConfigParser, re same thing. >+ config = ConfigParser.ConfigParser() >+ config.read([os.path.join(mozilla_path, '.hg', 'hgrc')]) >+ >+ if not config.has_option('paths', 'default'): >+ return >+ >+ #Compile the m-c regex >+ m_c_regex = re.compile("(ssh|http|https):\/\/hg\.mozilla\.org\/mozilla-central\/$", re.I) That regex, not depending on vars, should be compiled at the top and stored once (global, whatever). regex compilation is slow. Also, it looks like you defined it in the first line above but didn't use it. so, move m_c_regex = re.compile(MOZILLA_...) to the top-level. --da
Attachment #350550 - Flags: review-
Version: unspecified → Trunk
(In reply to comment #33) > (From update of attachment 350550 [details] [diff] [review]) > > >+MOZILLA_TRUNK_REPO_REGEXP = "(ssh|http|https):\/\/hg\.mozilla\.org\/mozilla-central\/$" > > >+def repo_config(): > >+ import ConfigParser > > import statements should be hoisted to the top, so they don't get executed on > each time that function gets called. It may not matter in this case, but it's > a good habit. > In this case repo_config is only used once per client.py invoke... I'd prefer to leave it scoped > >+ import ConfigParser, re > > same thing. Only used once on both, only repeat is ConfigParser, and the bottleneck here is definately not |import| statements when its needed. > >+ #Compile the m-c regex > >+ m_c_regex = re.compile("(ssh|http|https):\/\/hg\.mozilla\.org\/mozilla-central\/$", re.I) > > That regex, not depending on vars, should be compiled at the top and stored > once (global, whatever). regex compilation is slow. Imo, I only want to compile it once I know it will be needed, as I know that here... (in a clean checkout its not needed, also if .treestate already exists) > Also, it looks like you defined it in the first line above but didn't use it. > > so, move m_c_regex = re.compile(MOZILLA_...) to the top-level. Yea, I fixed that in v3, but did not move to top-level.
Attachment #350578 - Flags: review?(david.ascher) → review+
Comment on attachment 350578 [details] [diff] [review] v3 move_to_stable dealt w/ my comments.
Attachment #350578 - Flags: review?(kairo) → review+
Comment on attachment 350578 [details] [diff] [review] v3 move_to_stable Looks good for me functionality-wise, thanks for this work!
Comment on attachment 350578 [details] [diff] [review] v3 move_to_stable In general this looks fine. However I'm having a bit of trouble testing it. Hg has been complaining about "abort: empty or missing revlog for xpinstall/wizard/unix/src2/.LIBSREQD.list". I guess hg isn't happy with my repo for some reason. Therefore I'm going to reclone mozilla and then test the patch again. Hence when we check this patch in, I'd like to make sure gozer and KaiRo aren't too far away, just in case we get issues on the tinderboxes. As that's not going to happen until tomorrow afternoon my time, I'll finish looking at this in the morning.
Attachment #350578 - Flags: review?(bugzilla) → review+
Comment on attachment 350578 [details] [diff] [review] v3 move_to_stable >diff --git a/.hgignore b/.hgignore >@@ -22,3 +25,4 @@ > # directories we check out from other repositories > ^directory/c-sdk$ > ^mozilla$ >+^.mozilla-trunk$ >\ No newline at end of file Please fix this. >+def move_to_stable(): >+ mozilla_path = os.path.join(topsrcdir, 'mozilla') >+ if not os.path.exists(mozilla_path): >+ return >+ if os.path.exists(TREE_STATE_FILE): >+ return >+ nit: there are various blank lines with whitespace on, please fix. This looks fine. I have a blog post in progress, but due to the possible problems please co-ordinate landing of this with KaiRo/gozer and myself.
Attachment #349699 - Attachment is obsolete: true
Attachment #349699 - Flags: review?(kairo)
Pushed as http://hg.mozilla.org/comm-central/rev/87c1dc594c9b but we're running into a bad error with python versions that are high enough (like 2.6 on my machine): Traceback (most recent call last): File "client.py", line 249, in <module> repo_config() File "client.py", line 60, in repo_config config.add_section('default') File "/usr/lib/python2.6/ConfigParser.py", line 242, in add_section raise ValueError, 'Invalid section name: %s' % section ValueError: Invalid section name: default See http://mail.python.org/pipermail/python-checkins/2008-February/066304.html - this is built right into python itself, though I don't understand their reasoning...
(In reply to comment #39) > Pushed as http://hg.mozilla.org/comm-central/rev/87c1dc594c9b but we're running > into a bad error with python versions that are high enough (like 2.6 on my > machine): > > Traceback (most recent call last): > File "client.py", line 249, in <module> > repo_config() > File "client.py", line 60, in repo_config > config.add_section('default') > File "/usr/lib/python2.6/ConfigParser.py", line 242, in add_section > raise ValueError, 'Invalid section name: %s' % section > ValueError: Invalid section name: default > > See http://mail.python.org/pipermail/python-checkins/2008-February/066304.html > - this is built right into python itself, though I don't understand their > reasoning... rs=me to name the section 'mozilla_central' instead if that suites you. As that *is* what this refers to. Anyone who ran with this with earlier python vers will have that 'default' section, but imo it won't hurt anything.
As so far we're only checking if the file exists at all, it's pretty easy to just change us to write out a different section name that is not reserved by python 2.6 :)
Attachment #350811 - Flags: review?(bugzilla)
Attachment #350811 - Flags: review?(bugzilla) → review+
Pushed the followup fix as http://hg.mozilla.org/comm-central/rev/4e0ccdb5b84f - I hope this bug is fixed now.
Status: ASSIGNED → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
Flags: blocking-thunderbird3?
Flags: blocking-seamonkey2?
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: