853775 - Update client.py to pull NSPR and NSS from Mercurial instead of CVS

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Attachment: Update client.py to pull NSPR and NSS from Mercurial instead of CVS

Luckily, somebody had already written the logic for importing Mercurial projects into mozilla-central, so this was easy.

However, there are some complications:

1. Mercurial does not support tagging parts of a repository or cloning/pulling part of a repository. Consequently, it is problematic to continue to support the importing of just updates to NSSCKBI into mozilla-central. Instead, if we want to mix/match NSSCKBI with other parts of NSS, then we should create a branch in the NSS Mercurial repository that contains the exact configuration we want, and then tag that branch with a release tag, and then import in that branch. That actually makes a lot more sense anyway, because NSSCKBI isn't really independent of the rest of NSS, since it shares the build system with NSS, and it is possible that a correct build of NSSCKBI depends on particular changes to the build system that are outside of the nss/lib/ckfw/builtins directory. Also, nobody ever modified the Gecko build configuration system (configure.in, et al.) so that we could specify a minimum version of NSSCKBI. In the past, we've run into cases where Linux distros have shipped Firefox with old NSSCKBI because we didn't update the NSS version requirement, which caused their builds of Firefox to be vulnerable to CA compromises when regular Firefox builds weren't. Accordingly, I removed the support for importing NSSCKBI separately from the rest of NSS from the script. Note that I removed the TAG-INFO-CKBI file too, because the NSSCKBI version information will be encoded in the tag put in the TAG-INFO file if we ever create a mix-and-match branch for NSS+NSSCKBI as I described above, so TAG-INFO-CKBI became redundant.

2. Sadly, the way client.py imports Mercurial repositories is via "hg clone" and the clone is very slow compared to the CVS-based import. I think this is just something we will have to live with for now.

I hard-coded the path to the NSPR/NSS repositories to use ssh:// URLs to the repositories. This makes sense to me because, after all, we're working on critical security code here, so we should make sure that we are using a communication mechanism that tries to ensure integrity of the data transfer.

Comment 1

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

I forgot to add: The user interface for client_py did not change:

python client.py update_nspr <TAG>
python client.py update_nss <TAG>

However, we should update the wiki page at https://developer.mozilla.org/en/Updating_NSPR_or_NSS_in_mozilla-central to change the references from CVS-isms to Mercurial-isms.

Also, the patch assumes that the shuffling of the NSS directories has been done, something like:

hg mv security/dbm security/nss/lib/
hg mv dbm security/nss/lib/
hg mv security/coreconf security/nss/

My plan is to do these directory moves, create a beta tag of NSS and pull that beta tag into mozilla-central as part of the commit that makes this change. That NSS beta will fix the directory reshuffling issues. (I know the "hg mv" commands above are not sufficient on their own, but importing in NSS from the NSS repository will fix the oversimplification.)

Comment 2

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Brendan: FYI, this will move dbm/ from the top level to security/nss/lib/dbm.

When this change is made, we will need to update the "Modules" wiki page to reference the reflect the new directory structure.

Comment 3

[(not currently active) Ted Mielczarek]

A nice followup here would be to allow you to specify a local hg repo to update NSPR/NSS from, since I assume the people doing the updates will usually have a local clone anyway. That makes more sense than re-cloning the whole thing remotely.

Comment 4

[Kai Engert (:KaiE:)]

> 1. Mercurial does not support tagging parts of a repository or
> cloning/pulling part of a repository. Consequently, it is problematic to
> continue to support the importing of just updates to NSSCKBI into
> mozilla-central. Instead, if we want to mix/match NSSCKBI with other parts
> of NSS, then we should create a branch in the NSS Mercurial repository that
> contains the exact configuration we want, and then tag that branch with a
> release tag, and then import in that branch.

I agree this approach makes sense going forward.
In the future we should create a branch based on the last published NSS release, and apply the root list updates to it. But the details can be worked out elsewhere and later. I agree to removing the update_nssckbi command from client.py and remove TAG-INFO-CKBI from mozilla hg.


> 2. Sadly, the way client.py imports Mercurial repositories is via "hg clone"
> and the clone is very slow compared to the CVS-based import. I think this is
> just something we will have to live with for now.
>
> I hard-coded the path to the NSPR/NSS repositories to use ssh:// URLs to the
> repositories. This makes sense to me because, after all, we're working on
> critical security code here, so we should make sure that we are using a
> communication mechanism that tries to ensure integrity of the data transfer.

Using ssh:// took about 70-80 seconds for me, and https:// took about 50 seconds. I don't think that's an issue.

However, I personally think using https:// would be better.

I think everyone should be able to easily update a local tree to a different nspr/nss snapshot, and given that very few people have an ssh account at hg.mozilla.org, only https will work for everyone else.

I'd say that https is sufficiently secure.

If you insist on using ssh:// then please add a comment to the client.py, next to the ssh:// URLs, that mentions the alternative https:// urls.

Comment 5

[Kai Engert (:KaiE:)]

(In reply to Brian Smith (:bsmith) from comment #1)
> Also, the patch assumes that the shuffling of the NSS directories has been
> done, something like:
> 
> hg mv security/dbm security/nss/lib/
> hg mv dbm security/nss/lib/

I suspect this will result in some sort of conflict, after the first command, the dbm directory will already exist, and the second command might complain.

I personally wouldn't worry about the history of mozilla's copy of nss and simply remove all files from the old locations.

rm -rf mozilla/dbm mozilla/security/dbm mozilla/security/coreconf mozilla/security/nss
python client.py update_nss
hg addremove -a

If people want to learn about the history of NSS, they can refer to the NSS repository.

But up to you.


> My plan is to do these directory moves, create a beta tag of NSS and pull
> that beta tag into mozilla-central as part of the commit that makes this
> change.

I tested that the update commands also work with the hexadecimal hg revision IDs, which is good.

Comment 6

[Kai Engert (:KaiE:)]

Comment on attachment 728098 [details] [diff] [review]
Update client.py to pull NSPR and NSS from Mercurial instead of CVS

r=kaie

Should bug 853749 get resolved as wontfix, then please change 'nspr' to 'nsprpub' in the following line:

+    do_hg_replace('nspr', 'ssh://hg.mozilla.org/projects/nspr',

Comment 7

[Gregory Szorc [:gps]]

Can we take this opportunity to move client.py out of the root directory? It's only used by like 2.5 people and it's presence in the root directory can only confuse new developers (ooo - a script - that must be relevant).

Comment 8

[Kai Engert (:KaiE:)]

(In reply to Gregory Szorc [:gps] from comment #7)
> Can we take this opportunity to move client.py out of the root directory?
> It's only used by like 2.5 people and it's presence in the root directory
> can only confuse new developers (ooo - a script - that must be relevant).

Instead of moving the file, 
how about adding a comment to the top of the script, like:

# This script isn't part of the regular build.
# It's used for importing new snapshots of dependent libraries.

Comment 9

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/b20eab2be2dc

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #3)
> A nice followup here would be to allow you to specify a local hg repo to
> update NSPR/NSS from, since I assume the people doing the updates will
> usually have a local clone anyway. That makes more sense than re-cloning the
> whole thing remotely.

Now bug 856883.

(In reply to Kai Engert (:kaie) from comment #6)
> Should bug 853749 get resolved as wontfix, then please change 'nspr' to
> 'nsprpub' in the following line:
> 
> +    do_hg_replace('nspr', 'ssh://hg.mozilla.org/projects/nspr',

I followed this suggestion to s/nspr/nsprpub/ since bug 853749 did not not get fixed (yet).

(In reply to Gregory Szorc [:gps] from comment #7)
> Can we take this opportunity to move client.py out of the root directory?
> It's only used by like 2.5 people and it's presence in the root directory
> can only confuse new developers (ooo - a script - that must be relevant).

I am happy to have the script live elsewhere. Doing so is now bug 856884.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/b20eab2be2dc