282610 - Move SVRCORE functionality into NSS

Status: RESOLVED FIXED

(NSS :: Libraries, enhancement)

Description

[Rich Megginson]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.2) Gecko/20040805 Netscape/7.2
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.2) Gecko/20040805 Netscape/7.2

Internal product builds we do for Red Hat (and prior to that Netscape,
and probably still at Sun) use a library called SVRCORE.  It provides two
services, only one of which is still used - the pin management
services.  These are defined in a header file called svrcore.h.  The other
service is server side SSL set up, in servssl.h, which is old, crufty, and
apparently hasn't been used in a long time.

I propose to move the pin management functionality into NSS.  I have a build
working already that creates an nss/lib/svrcore directory, builds libsrvcore.a,
exports svrcore.h, and puts these functions into libnss3.so.

The other files, servssl.c and servssl.h, would require some amount of work to
clean them up and "modernize" them.  They don't seem to be necessary for any
product I know of, so I don't think we need to keep them.  We could put them in
a README or something like that for historical/reference purposes.

Reproducible: Always

Steps to Reproduce:

Comment 1

[Rich Megginson]

See also LDAP C SDK bug https://bugzilla.mozilla.org/show_bug.cgi?id=282608

Comment 2

[Julien Pierre]

I think the directory team at Sun still uses SVRCORE. They are pulling it as a
static library . However, the code hasn't been built in 2 years. There is no
question that the source is very outdated. I don't think the NSS team at Sun
wants anything to do with this code .

I would strongly object to it being made part of the NSS public API exported
from libnss3.so. It will cause problems for products that have their own static
copy of SVRCORE. Bad things happen when the same symbols exist in multiple
libraries.

In addition, it isn't possible for us at Sun to coordinate the removal of
SVRCORE from directory and the addition of SVRCORE to NSS, because the NSS
library is part of the OS, and the NSS libraries can be updated without the
products.

Thus, my vote goes to keep SVRCORE as a separate static library, maintained and
built outside of NSS .

Comment 3

[Rich Megginson]

The Directory Server team still uses svrcore, pulled as a static library.  It's
possible that other server groups (Messaging, Calendar, ?) still use it as well.

The svrcore.h code is not outdated - the APIs used appear to be up to date.  
And it does perform a useful function that is not available anywhere else - why
reinvent the wheel again?  That's the code I'd like to keep. The stuff in
servssl.h is very outdated.  I'm proposing to drop it.

I don't think it will be a problem if the functions are defined in an executable
that dynamically links against libnss3.so.  What usually happens in that case is
that the functions in the shared library are not loaded because they already
exist in the executable.  I can write a test program to demonstrate this if you
like and attach it to this bug.

If my above assertion is true, no coordination needs to be done, unless svrcore
is being actively developed at Sun.  The DS team and other teams can remove the
static version of svrcore whenever they get around to it.  I can certainly
coordinate with Ludo and Regis in Grenoble if necessary.

Do you agree that the svrcore.h (not the servssl.h) functionality is useful?  Do
you agree that it should be open sourced?  If so, what do you think is the best
vehicle for releasing the code?

Comment 4

[Robert Relyea]

I agree with Julien's objection. libsrvcore.a should not be part of libnss3.so,
if  for no other reason than it's of zero value to half of the NSS users (client
applications). Julien's other objections are also quite valid as well. I think
it prudent not to push the varying semantics of everyone's loaders. Even if we
support our products on a limitted number of platforms, that's not true of all
users of NSS.

I do think the notion of moving svrcore to some part of the mozilla security
tree is needed, however. It should live in either:

security/nss/lib/svrcore
security/nss/svrcore
security/srvcore

I believe the last option is a non-starter, as srvcore is very tightly bound to
NSS, and has no real meaning outside of NSS (on the other hand PSM is also
tightly bound to NSS and lives in the security directory). It does make sense
because is supposedly represents current best practices in integrating NSS to a
stand alone "unattended" server.

svrcore should build it's own library. Currently I would expect that library to
be a .a, possibly transitioning to a shared library if we think the API is
stable enough.

We have an additional requirement that we need to build it as part of our nss
builds. That is most easily accomplished if it lives under NSS, but it's not a
strict requirement.

bob

Comment 5

[Wan-Teh Chang]

I suggest that we add it to security/nss/lib/svrcore,
preferrably with a more descriptive name, and release
it as a static library, just like how lib/crmf is released
now.

Comment 6

[Rich Megginson]

In my local build environment, I already have nss/lib/svrcore which builds
libsvrcore.a, and exports the svrcore.h include file.  That sounds good to me. 
So, what are the next steps?  Do I post the source files and the makefile diffs
as attachments to this bug?

Comment 7

[Robert Relyea]

yes,

have either Wan-Teh or I review it 
  and
 either nelson or julien

We use review and super-view flags in the NSS bugs to get two reviews.

bob

Comment 8

[Rich Megginson]

I think I would prefer to leave it as libsvrcore.a for now, instead of libpin.a
or something like that.  All of the API functions begin with SVRCORE_, and I
think it would require a lot more work in both the library and in the callers.

Comment 9

[Rich Megginson]

Attachment: Diffs for nss/lib/manifest.nm to add svrcore to build

This is just the diffs to add svrcore to the NSS build.  The svrcore code will
be in nss/lib/svrcore.

Comment 10

[Rich Megginson]

Attachment: zip file containing all of the new source code, makefiles, etc.

File is in zip format.	I thought this would be better than attaching each
individual file to the bug.  It was zipped using "zip" on RHEL3.

Comment 11

[Wan-Teh Chang]

Rich,

You can generate a patch containing new files (or
removed files) as follows:

1. Do "cvs add" on the new files.  Do "cvs remove"
on the removed files.

2. Generate the patch using "cvs diff -uN".  The
-N flag is key. 

Comment 12

[Rich Megginson]

I cannot do "cvs add" to add the svrcore directory.

cvs [server aborted]: "add" requires write access to the repository

Comment 13

[Nelson Bolyard (seldom reads bugmail)]

When we were at AOL, Terry Hayes (author of svrcore) told us that AOL's 
server groups that used NSS all used svrcore, and that THEY perceived 
svrcore to be an intergral part of NSS, even though the NSS developers
had not developed it, did not maintain it, and know almost nothing about 
it.  The problem (as I recall Terry explaining it) was that the server
groups expected someone other than themselves to build it, and it had
become orphaned.  Terry proposed moving the source to the NSS source
tree and building it with NSS, but not making it part of NSS's shared
libraries, IIRC.  

Now at Sun, I see that Sun's servers have individually either 
(a) stopped using it alltogether, or 
(b) have taken their own copies of the source, which have diverged.
So, I agree with Julian in opposing adding svrcore to NSS's shared libs.

I do not object very strongly to having the source code in the NSS source
tree in its own seeparate directory, or even to building it with NSS, but 
I oppose putting it into the existing shared libs, and having to maintain 
it.  

It seems to me the real issue is ongoing maaintenance.  Placing the 
code into NSS burden's NSS's developers to maintain it.  I cannot think
of any other example of code that NSS has adopted that NSS does not use.
I believe the reponsibility for ongoing maintenance should live with the
users of that code.  And if they do maintain it, then one wonders why it
should be put into the NSS tree.  

Comment 14

[Nelson Bolyard (seldom reads bugmail)]

svrcore's makefile needs to put svrcore into a different MODULE than nss.
It can set "REQUIRES" to nss, but the module (directory) into which its 
files are installed (under mozilla/dist) should not be NSS.

Comment 15

[Rich Megginson]

Nelson, about comment 13 - So, basically what happened is that Terry wrote
something useful as a layer on top of NSS and no one claimed ownership?  If it
doesn't go into NSS, where might an appropriate place be?  Mozilla is out -
svrcore is not useful to any GUI product.  But the problem as I see it is that
it doesn't really fit anywhere else except NSS.  Suggestions?

I think if it were in NSS, I would not have a problem helping to maintain it,
and I think other Red Hat and Sun server developers would prefer to do it this
way as well.

About comment 14 - I'm attaching the svrcore/manifest.nm diffs.  These diffs put
svrcore.h into dist/public/svrcore/svrcore.h and libsvrcore.a into
dist/<platform>/lib.  Is that OK?

Comment 16

[Rich Megginson]

Attachment: Diffs for svrcore/manifest.nm based on Nelson's comments #14

Comment 17

[Wan-Teh Chang]

The only SVRCORE maintenance we did in the five
years I work on NSS was to update it to use the
open source NSS shared libraries when NSS 3.2
was released.  Bob Relyea had to update SVRCORE
to use the approved API exported by our shared
libraries.  We haven't done anything with it
since thanks to NSS 3.x shared libraries'
backward compatibility.

The only future maintenance work I anticipate
is to build a binary release, once, from sources
pulled from its new home.

Let's find it a new home -- mozilla/security/svrcore,
mozilla/security/nss/lib/svrcore, or inside the
DS.

If the LDAP command-line tools on mozilla still use
SVRCORE, that's a good reason to put it in mozilla.
Otherwise, it may make sense to put it in the
DS.

Comment 18

[Rich Megginson]

The LDAP command line tools on Mozilla _can_ use svrcore.  The builds done for
Mozilla/Tbird have that option turned off since it's not open source code and
it's not useful for those applications.

I would rather not make it part of the DS because it's useful to a range of
applications.

mozilla/security/svrcore or mozilla/security/nss/lib/svrcore or wherever, it
makes no difference to me.

Comment 19

[Rich Megginson]

One more comment about LDAP SDK and SVRCORE - svrcore is only useful to the
command line programs, not to the core ldap libraries.  Mozilla/Tbird don't need
the LDAP SDK command line programs (like ldapsearch, etc.) either.

Comment 20

[Nelson Bolyard (seldom reads bugmail)]

mozilla/security/svrcore seems good to me.  
Rich's small patch above (referring to comment 14) seems good.

Comment 21

[Rich Megginson]

I moved svrcore under security/.  I only had to edit the manifest.nm to change
the CORE_DEPTH appropriately.  I added a build_svrcore target to the
nss/Makefile.  It is built after the "all" target.  I'll add a new diff for
that.  What follows is what the directory structure looks like:

mozilla/security/svrcore
mozilla/security/svrcore/svrcore.h
mozilla/security/svrcore/Linux2.4_x86_glibc_PTH_DBG.OBJ/libsvrcore.a
mozilla/dist/Linux2.4_x86_glibc_PTH_DBG.OBJ/lib/libsvrcore.a
mozilla/dist/public/svrcore
mozilla/dist/public/svrcore/svrcore.h

Comment 22

[Rich Megginson]

Attachment: Diffs to add svrcore to nss/Makefile

Comment 23

[Robert Relyea]

The way this is addes, svrcore is not built as part of most NSS builds, but it
can easily be included if the builder wants it.

Most notable it won't be build for mozilla, or for any build system which uses
binary pulls by default.

I think this is the right semantic. (particularly mozilla shouldn't build it by
default).

The only ones that will build svrcore without explicitly asking for it would be
people pulling NSS source and building it standalone. I think we should probably
remove the build_svrcore from the nss_build_all line (which is only invoked in
this particular case). We can have anthony add it to his nightly NSS build
script, and the Sun build team can decide if they want to include it in their
builds. I don't think this is a show-stopper.

The directory layout looks good to me.

bob

Comment 24

[Nelson Bolyard (seldom reads bugmail)]

I object to changing NSS's Makefile so that the target nss_build_all builds
svrcore.  This will break existing build scripts.

I do not object to adding a new target that builds svrcore.

Comment 25

[Rich Megginson]

Attachment: Diff for nss/Makefile

This one removes the build_svrcore dependency on nss_build_all.  Instead, the
build_svrcore target has a dependency on nss_build_all, because svrcore depends
on nss.  This way, if you want svrcore, you just gmake build_svrcore and it
also builds nss and everything else.  gmake nss_build_all works as before.  No
scripts should be broken.

Comment 26

[Mark Smith [:mcs]]

Will it be possible to pull and build svrcore without pulling and building NSS?
 For LDAP C SDK builds, I'd like to encourage people to use pre-built NSS shared
libraries when possible.

Comment 27

[Robert Relyea]

Right now, I don't think Coreconf knows how to pull binary builds from mozilla
(probably a good thing to add). If you do that, then we can just add the
following line to svrcore/maniest.mn:

IMPORTS =     nss/NSS_3_9_3_RTM \
              $(NULL)

(we may also need to add nspr as well)

bob

Comment 28

[Christopher Blizzard (:blizzard)]

We've had a long standing process in the rpm world of distributing static
libraries used for development with our -devel packages.  Would this work for
you, Mark?

Comment 29

[Mark Smith [:mcs]]

r.e. comment #28, that kind of approach would work... but only if someone made
such packages available.

Comment 30

[Rich Megginson]

Mark, I'm not sure I understand your comment <a
href="https://bugzilla.mozilla.org/show_bug.cgi?id=282610#c26">26> - svrcore
depends on NSS in order to build it, or at least the NSS header files, and
possibly the NSPR header files.

Comment 31

[Mark Smith [:mcs]]

Let me re-phrase my comment #26:  Suppose I pick up a binary NSS release
(headers and libraries) from
ftp://ftp.mozilla.org/pub/mozilla.org/security/nss/releases/ and the correct
NSPR release to go with it.  Will I then be able to pull and build the svrcore
code and create the svrcore library needed to build something else that uses it
such as the LDAP C SDK?  Or will I need to pull and build NSS from source?

Comment 32

[Rich Megginson]

Mark, yes.  In fact, I think you only need the nss and nspr header files in
order to build the svrcore library from source.  You definitely do not need to
build NSS from scratch.

Comment 33

[Nelson Bolyard (seldom reads bugmail)]

I received email from cvsmailfilter saying 

> Directory /cvsroot/mozilla/security/svrcore added to the repository
> --> Using per-directory sticky tag `DBM_1_61_RTM'

I'll bet you didn't really want to use that tag.  
That's a static tag corresponding to an old release of DBM, 
of which this directory is not a part.  

Comment 34

[Rich Megginson]

Hmm - I don't think it was me - I don't have write permission in mozilla/security?

Comment 35

[Wan-Teh Chang]

(Nelson, thanks for pointing that out.  I saw that, too
and had to add the directory again from scratch.)

I added the svrcore files to mozilla/security/svrcore.

The files I added are based on the files in ns/svrcore
with the cvs tag SVRCORE_3_5_RTM.

I found that Rich made the following changes to those
files in the zip file he attached to this bug.

1. He fixed compiler warnings.
2. He removed from svrcore.h two functions that were marked
as deprecated (SVRCORE_SetPinObj and SVRCORE_Pk11PasswordFunc).
3. He removed declarations of NLS and UTF8 related functions
from svrcore.h.

I made the following changes.

1. I added MPL/GPL/LGPL triple license headers to the
files.  The copyright years are the same as the copyright
years in the original files (mostly 1998).  I describe the
Original Code as "Netscape svrcore library".

2. I didn't use a config.mk file.

3. I didn't check in key.ico, logo.ico, ntgetpin.rc,
and ntresource.h because they don't seem to be built.
Let me know if the LDAP command-line tools or any
Netscape servers need them.

4. I changed "HCL" to "NSS" in a comment in svrcore.h.

5. I didn't add the build_svrcore makefile target to
nss/Makefile.  Instead, I added the RelEng_bld makefile
target to svrcore/Makefile.  We can change these later.

You can build mozilla/security/svrcore as follows.

1. If you have /share/builds/components, just do
the following:

  cvs co mozilla/security/coreconf mozilla/security/svrcore
  cd mozilla/security/svrcore
  gmake RelEng_bld

2. If you don't have /share/builds/components, you
also need to check out the sources of NSPR, DBM,
and NSS.

  cvs co mozilla/nsprpub
  cvs co mozilla/dbm mozilla/security/dbm
  cvs co mozilla/security/nss
  cvs co mozilla/security/coreconf mozilla/security/svrcore

Then do the following:

  cd mozilla/security/nss
  gmake nss_build_all
  cd ../svrcore
  gmake

Comment 36

[Wan-Teh Chang]

Re: Mark Smith's question in comment 26 (which he
clarified in comment 31)

Right now you can't do that.  I'd like to address
that in a separate bug, because right now you
can't build NSS without pulling and building its
dependencies (NSPR and DBM) either.  We should
solve that problem for NSS and SVRCORE at the
same time.

Comment 37

[Wan-Teh Chang]

I added the following files, which are used in
conjunction with ntgetpin.c on Windows:
  ntgetpin.rc
  ntresource.h
  key.ico
  logo.ico

I mistakenly thought they were not being built
on Windows (see the second #3 in my comment 35).

I added a README file that describes what svrcore
is and how it is being used.  Most of the contents
was written by Terry Hayes.  I'd appreciate it if
Mark and Rich can contribute to the last paragraph in
the README file, which describes how svrcore is
being used.

At this point, I'd like to mark this bug fixed.
It builds on Linux and Windows.  If you can
build the LDAP C SDK tools or the Directory Server
against it, please mark the bug verified.

Comment 38

[Wan-Teh Chang]

I just wanted to note that I cvs added key.ico
and logo.ico with the -kb flag because they are
binary files.