Last Comment Bug 417635 - import nspr/nss to hg
: import nspr/nss to hg
Status: RESOLVED FIXED
: dev-doc-complete
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: unspecified
: All All
: -- normal (vote)
: ---
Assigned To: Benjamin Smedberg [:bsmedberg]
:
: Gregory Szorc [:gps]
Mentors:
Depends on: 438682
Blocks:
  Show dependency treegraph
 
Reported: 2008-02-14 17:07 PST by Robert Helmer [:rhelmer]
Modified: 2008-06-11 13:31 PDT (History)
25 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Proposed approach to resolving this bug (1.50 KB, text/plain)
2008-05-16 05:15 PDT, Sergey Yanovich
no flags Details
client.py and .hgignore changes for NSPR/NSS (5.74 KB, patch)
2008-06-05 11:59 PDT, Benjamin Smedberg [:bsmedberg]
ted: review+
Details | Diff | Splinter Review

Description Robert Helmer [:rhelmer] 2008-02-14 17:07:01 PST
We currently automatically import browser and xulrunner to hg, why not nspr/nss? I understand that the "upstream" for these projects is CVS, but it would make it easier if developers just had to check out hg, configure, make, and not use client.py to check these out from CVS.
Comment 1 Benjamin Smedberg [:bsmedberg] 2008-02-14 17:15:59 PST
They are independent projects... importing them can make it harder to deal with patches independently and get them in the correct review stream. It also means we'd have to take on additional repository size for them.

We know we're going to have to deal with multiple repositories in any case, for Tamarin and eventually separate repos for Suite/Tbird/etc... so we don't gain much by importing NSPR/NSS now.

Recommend WONTFIX, or bring this up in the newsgroups.
Comment 2 Vladimir Vukicevic [:vlad] [:vladv] 2008-02-14 17:42:45 PST
Well, I think we need a solution for single-checkout -- right now the only things holding us back from having this is nss/nspr.  I'd vote we import (without history), and just update as new revisions come out.  This is no different than what we're doing with cairo, libpng, sqlite, etc.  The suite/tbird/other moz-based projects will all require nss/nspr, so I don't think that's an issue.
Comment 3 Robert O'Callahan (:roc) (email my personal email if necessary) 2008-02-14 19:11:55 PST
The goal here is to make it as easy as possible for someone to pull and build trunk and make patches against trunk. So I want a copy of configure in there as well. We can have one of these all-in-one repos per product, ultimately, but starting with one for Firefox would satisfy 95% of the potential need.

I don't see how this makes it harder to "get patches in the correct review stream". Making users pull from multiple hg repositories isn't going to help them file bugs in the right Bugzilla component or choose the right reviewers for their patches.
Comment 4 Jason Orendorff [:jorendorff] 2008-04-10 15:15:27 PDT
(In reply to comment #3)
> I don't see how this makes it harder to "get patches in the correct review
> stream". Making users pull from multiple hg repositories isn't going to help
> them file bugs in the right Bugzilla component or choose the right reviewers
> for their patches.

True.  The balance here is, the good we can do for the thousands of Mozilla hackers who never touch NSS/NSPR vs. the harm we can do to the few people who will.

The good is obvious:  no one has to run "python client.py checkout".  I can testify that this is no big deal... until I forget to do it.  Then a browser build mysteriously dies halfway through and I have no idea why, until 20 minutes later I realize it's because I AM STUPID.  I hate that, and surely we can do something to spare everyone this pain.

Now the bad news.  How are NSS/NSPR hackers are supposed to work in such a world?  I see a few ways, all bad.  (For example: rm -rf nsprpub security/nss; then use cvs co.  This would confuse hg.  My fear is that when hg gets confused, work gets lost.)  (Another example:  Make a central hg repo that contains trunk, plus the full history of NSS/NSPR instead of periodic snapshots.  Work against that; to commit changes, manually apply your patch to a separate CVS tree.  Setting this up wouldn't be hard, but it's still kinda baffling and painful.)

I don't work on NSS or NSPR, but I do work on Tamarin.  I hope there's a way to fix this without making hackers like me suffer.
Comment 5 Robert O'Callahan (:roc) (email my personal email if necessary) 2008-04-10 15:23:09 PDT
It's really simple for NSS/NSPR developers. Option 1: they make their fix based on the hg repo, create a patch, get that patch reviewed, and check the patch into CVS. Option 2: check out NSS/NSPR from CVS, and check out Firefox (or another app) from an anonymous CVS mirror of the hg repo.
Comment 6 Jason Orendorff [:jorendorff] 2008-04-11 08:02:55 PDT
If I understand the proposal correctly, option 1 doesn't work if you ever do work that builds on other recent work.

But option 2 sounds just right.  Very nice.

It doesn't help me with Tamarin, which is in hg... but that is outside the scope of this bug.
Comment 7 Robert O'Callahan (:roc) (email my personal email if necessary) 2008-04-11 14:44:41 PDT
Not sure what you mean by "builds on other recent work".
Comment 8 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-05-13 21:47:00 PDT
We need this already.  The NSS and NSPR hackers are experts, they will have no trouble moving directories aside, or adding some build goop for --with-nspr-srcdir=../nspr-trunk .
Comment 9 Vladimir Vukicevic [:vlad] [:vladv] 2008-05-14 16:05:48 PDT
(In reply to comment #5)
> It's really simple for NSS/NSPR developers. Option 1: they make their fix based
> on the hg repo, create a patch, get that patch reviewed, and check the patch
> into CVS. Option 2: check out NSS/NSPR from CVS, and check out Firefox (or
> another app) from an anonymous CVS mirror of the hg repo.

Option 3: Add --with-nspr=... --with-nss=... options to our configure, to look in those places for nss/nspr.

Comment 10 Ted Mielczarek [:ted.mielczarek] 2008-05-14 19:10:51 PDT
That sounds like a totally reasonable idea.
Comment 11 Boris Zbarsky [:bz] (still a bit busy) 2008-05-14 20:02:20 PDT
Is there a reason that "make -f client.mk checkout" doesn't just do the client.py thing?  Or does it?
Comment 12 Sergey Yanovich 2008-05-15 02:25:18 PDT
As a developer of a xul application, I think this bug is wider than just nss/nspr. Current mozilla/ tree is optimized for cvs access. Specifically, it all works with the assumption, that each single file can be checked out independently. That's no longer true with hg, which operates in terms of trees, not files.

The possible solutions maybe to turn current mozilla/ tree into a mozilla/ forest (or stack). There is already a precedent for this in Mozilla. Taras has been running hg stack for oink for a long time.

The top level repository will contain only the build system (bug 381902). All other modules will have their own subrepositories:

* libnspr;
* libnss;
* libxul;
* firefox; (or libxul + firefox);
* every xul appilcation
* extensions:
** every extension (or group of them).

Pluses for firefox development:
# no merging
# no client.py
# make -f client.mk should work just like before

But this layout will *considerably* simplify extension/xulapp development:
1. Checkout the top level repo
2. Create a directory for my extension under extensions.
3. Create .mozconfig with appropriate --enable-application
4. Run make -f client.mk build

These changes will require some wholesale movement of code, but the results may well be worth it.
Comment 13 Nick Thomas [:nthomas] 2008-05-15 06:02:40 PDT
CC'ing NSPR and NSS folk.
Comment 14 Mike Shaver (:shaver -- probably not reading bugmail closely) 2008-05-15 07:53:53 PDT
(In reply to comment #12)
> As a developer of a xul application, I think this bug is wider than just
> nss/nspr.

No, this bug is very much about just nspr and nss.  If you want to propose a more sweeping change to how the tree is organized, please do so in one of the many threads on this topic in dev.planning.
Comment 15 Benjamin Smedberg [:bsmedberg] 2008-05-15 08:02:09 PDT
The mercurial folk are discussing and implementing a newer/better forest. We are not going to consider forests until that work is complete and available in a release.

I'm fine with importing flat tags of NSPR/NSS, I suppose. I don't think we should do the same thing with tamarin, but since we don't need tamarin in mozilla-central yet, we should just ignore it for the time being.
Comment 16 Sergey Yanovich 2008-05-16 05:15:55 PDT
Created attachment 321240 [details]
Proposed approach to resolving this bug

(In reply to comment #14)
> (In reply to comment #12)
> > As a developer of a xul application, I think this bug is wider than just
> > nss/nspr.
> 
> No, this bug is very much about just nspr and nss.

On second thought, there is no need for big changes to make nspr and nss working from separate Mercurial repositories. New proposal is attached.
Comment 17 Nelson Bolyard (seldom reads bugmail) 2008-05-16 11:02:55 PDT
(In reply to comment #16)

> On second thought, there is no need for big changes to make nspr and nss
> working from separate Mercurial repositories. New proposal is attached.

The attached proposal includes these changes:

> 2. Move security/manager out of security (i am not sure to what location), 
> and adjust toolkit/toolkit-tiers.mk
> 3. Move dbm to security/dbm-real, and adjust references in security/dbm as 
> appropriate

Those aren't big changes?
Comment 18 Sergey Yanovich 2008-05-16 11:09:42 PDT
ATM, #2 looks being limited to one line in toolkit/toolkit-tiers.mk, #3 is limited to security/dbm. So I think the answer is negative.
Comment 19 Nelson Bolyard (seldom reads bugmail) 2008-05-16 11:47:38 PDT
The NSS & NSPR team members discussed this in a conference call yesterday.
At this time, we do not wish to convert to hg, but prefer instead to stay
with cvs for the foreseeable future.  (We wonder how long the existing 
cvs-related infrastructure, such as lxr, mxr, bonsai, the cvs mirrors, etc.)
We plan to keep using the cvs repository as the primary (most upstream) 
source repository for NSS & NSPR.

Provided that the structure of the source tree under hg is not a major 
change from the structure of the source tree under cvs, it's not clear that 
pulling part from one repository and part from the other would be a big 
problem.  It would still be pulling all the source into a single build area, 
and then doing a single build.  

But if MoCo wants to keep downstream snapshots of NSS & NSPR sources in the 
hg repository, the team members are OK with that, PROVIDED that those 
snapshots are exact copies of the cvs sources from static cvs tags created 
by the NSS/NSPR team itself (which Benjamin called "flat tags" in comment 15).  
As in the past, the team's willingness to support mozilla's build of NSS 
continues to depend on the build coming from sources that match those tested 
and approved by the team itself.  

Presently, checkins to the NSS cvs tree are controlled, restricted to module peers.  We would recommend that the NSS portion of the hg repository be 
similarly controlled to avoid deviation of that snapshot from the sources as 
they appear in the cvs tags.
Comment 20 Sergey Yanovich 2008-05-16 12:25:15 PDT
(In reply to comment #19)
> Provided that the structure of the source tree under hg is not a major 
> change from the structure of the source tree under cvs, it's not clear that 
> pulling part from one repository and part from the other would be a big 
> problem.  It would still be pulling all the source into a single build area, 
> and then doing a single build.  

There is a catch. Mercurial allows going back and forth between branches without network access, but cvs don't. So the whole system won't work without network access.

> But if MoCo wants to keep downstream snapshots of NSS & NSPR sources in the 
> hg repository, the team members are OK with that, PROVIDED that those 
> snapshots are exact copies of the cvs sources from static cvs tags created 
> by the NSS/NSPR team itself (which Benjamin called "flat tags" in comment 15).  
> As in the past, the team's willingness to support mozilla's build of NSS 
> continues to depend on the build coming from sources that match those tested 
> and approved by the team itself.  
> 
> Presently, checkins to the NSS cvs tree are controlled, restricted to module
> peers.  We would recommend that the NSS portion of the hg repository be 
> similarly controlled to avoid deviation of that snapshot from the sources as 
> they appear in the cvs tags.

The proposal seems to fall in line with these requirements, right?

IIU Benjamin correctly, he proposed to merge NSS &  NSPR  tags into the main repository. That introduces a potential problem for NSS/NSPR devs. When there are changes to NSS/NSPR sources, which are not merged into the main hg tree, hg update will complain about them.

The very first time this happens, it will be possible to merge the changes locally. After that, options will be limited to restoring the last imported version, reverting all changes since that. If NSS/NSPR source contain any local (not committed to cvs) changes, they will need to be committed first or lost. The only practical workaround is to use an anonymous CVS mirror of the hg repo, as Robert proposed in comment 5.

NSPR can be converted to use hg without extra work any time. But when NSS people decide to switch to Mercurial, something similar to my proposal will be required.
Comment 21 Nelson Bolyard (seldom reads bugmail) 2008-05-16 12:41:07 PDT
Sergey, I think you imagine that NSS developers pull and build full browser
trees, and modify their NSS sources inside those trees.  With the possible
exception of Kai Engert, who works on both PSM and NSS, I doubt that any 
(other) NSS developer does that.  I believe that most of us pull and build
NSS and NSPR by themselves, stand-alone, and then download a "nightly" build 
of the browser and replace the NSS/NSPR shared libs in that build with the
shared libs we built ourselves.  So, we won't be experiencing any complains
from hg.  As long as browser developers are only using the officially tagged
snapshots, they should have no problem, either.  
Comment 22 Sergey Yanovich 2008-05-16 12:49:57 PDT
Comment on attachment 321240 [details]
Proposed approach to resolving this bug

(In reply to comment #21)
In this case, plain merging is the best solution. Thanks for explaining :)
Comment 23 Vladimir Vukicevic [:vlad] [:vladv] 2008-06-02 10:48:44 PDT
Hrm, so are we ok to grab a tagged snapshot and import it into hg then?
Comment 24 Benjamin Smedberg [:bsmedberg] 2008-06-02 10:52:28 PDT
Yeah... should we remove client.py while we're at it?
Comment 25 Ted Mielczarek [:ted.mielczarek] 2008-06-02 10:58:47 PDT
That leaves us without Tamarin, but that's probably ok, we should just import a static snapshot of it when we get there. We should probably leave client.py for the moment, so bhearsum can fix the Moz2 buildbot not to run it first. Just make it a no-op for the time being, I guess.
Comment 26 Axel Hecht [:Pike] 2008-06-02 10:59:38 PDT
Moving security/manager would be rather big, as that implies moving all localizations with it, which aren't even hooked up to 1.9.1. If moving security/manager is still in peoples heads.
Comment 27 Kai Engert (:kaie) 2008-06-02 11:02:15 PDT
It would be good to hear what your plans are wrt moving security/manager
Is it necessary?

I'd say it's not part of this bug, because
  security/manager != nspr/nss
Comment 28 Sergey Yanovich 2008-06-02 11:09:40 PDT
(In reply to comment #27)
> It would be good to hear what your plans are wrt moving security/manager
> Is it necessary?

Moving security/manager only needed for nss to live in a separate tree. Since the decision is to import the a tagged snapshot, that's not necessary.
Comment 29 Benjamin Smedberg [:bsmedberg] 2008-06-05 11:59:42 PDT
Created attachment 323904 [details] [diff] [review]
client.py and .hgignore changes for NSPR/NSS

This patch changes client.py to do nothing for "checkout"... it adds pull_nspr and pull_nss targets for people who need to update to new versions of NSPR/NSS
Comment 30 Ted Mielczarek [:ted.mielczarek] 2008-06-05 12:17:23 PDT
Comment on attachment 323904 [details] [diff] [review]
client.py and .hgignore changes for NSPR/NSS

 def do_cvs_checkout(modules, tag, cvsroot, cvs):
-    """Check out a CVS directory.
+    """Check out a CVS directory without CVS metadata, using "export"

Should probably call this "do_cvs_export" then, no?

+elif action in ('update_nspr'):
+    tag, = args[1:]
+    do_cvs_checkout(NSPR_DIRS, tag, options.cvsroot, options.cvs)
+elif action in ('update_nss'):
+    tag, = args[1:]
+    do_cvs_checkout(NSS_DIRS, tag, options.cvsroot, options.cvs)

You need to mention in the help output that you're expecting a tag name on the commandline, or make it an option with a value or something.

Otherwise looks good, one less hurdle for people to jump. Please blog or post to dev.something or both to let people know they're going to need to rm parts of their tree before pulling again. Actually, it's going to be sort of a pain to rm NSS, isn't it, since you can't just rm security/ ?
Comment 31 Wan-Teh Chang 2008-06-05 13:38:46 PDT
Comment on attachment 323904 [details] [diff] [review]
client.py and .hgignore changes for NSPR/NSS

If hg has static tags, can you continue to check out
NSPR and NSS from hg with static tags?  This will
provide additional assurance that you're using
NSPR and NSS snapshots blessed by the NSPR and NSS
teams.

Nice to have: is it possible to import NSPR into hg
in a way that won't preclude moving the NSPR source
code repository to hg in the future?
Comment 32 Benjamin Smedberg [:bsmedberg] 2008-06-05 13:53:04 PDT
wtc, this patch export NSPR/NSS from CVS static tags into mercurial, so that it's not a separate checkout.

Nothing precludes anything. If we end up with NSPR/NSS in mercurial and a usable forest implementation in the future, we can remove the imported NSPR and use the separate repo.
Comment 33 Benjamin Smedberg [:bsmedberg] 2008-06-06 13:13:36 PDT
Pushed to mozilla-central, revisions

3599b2ccdf57 - client.py and .hgignore changes
4231ddd5fd1a - import NSPR_4_7_1_RTM
437dcecc6377 - import NSS_3_12_RC4
Comment 34 Nickolay_Ponomarev 2008-06-08 12:48:24 PDT
I removed the client.py references from the Mercurial FAQ and the Getting the source code docs, but there are two less important documents left that mention it: 
http://developer.mozilla.org/en/docs/Special:Nutch?language=en&start=0&hitsPerPage=10&query=client.py&fulltext=Search

When it gets removed, those documents probably need to be updated.
Comment 35 Robert Kaiser 2008-06-09 05:00:08 PDT
Comment on attachment 323904 [details] [diff] [review]
client.py and .hgignore changes for NSPR/NSS

>-    if not options.skip_tamarin:
>-        do_hg_pull('js/tamarin', options.tamarin_repo, options.hg)
>-
>-    if not options.skip_mozilla:
>-        do_hg_pull('.', options.mozilla_repo, options.hg)

grr, why did you remove those, esp. the second? It would be so nice to just be able to run client.py and have it just update the mozilla-central repo instead of needing to remember the best pull options to use for that with hg directly.
Comment 36 Benjamin Smedberg [:bsmedberg] 2008-06-09 06:43:32 PDT
Robert, I made "client.py checkout" an obsolete option which doesn't do anything. Why is it better to run "python client.py checkout" than "hg pull -u"?
Comment 37 Robert Kaiser 2008-06-09 07:09:02 PDT
what I like better about doing a "python client.py checkout" is that I can do the same for a mailsuite repo (which will not be able to do without client.py) and just mozilla-central and I don't have to remember that -u is the option I probably miss when doing it manually ;-)
Comment 38 Benjamin Smedberg [:bsmedberg] 2008-06-09 07:16:53 PDT
I fixed up the docs.

Note You need to log in before you can comment on or make changes to this bug.