Closed Bug 52990 Opened 24 years ago Closed 23 years ago

NSS should use autoconf

Categories

(NSS :: Libraries, enhancement, P3)

Product:
NSS
Component:
Libraries
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: lennox, Assigned: rrelyea)

References

Details

I'm in the process of trying to port NSS to run on FreeBSD (3.5), and it's
very quickly becoming obvious that the current build system is going to
be a portability nightmare.  I'm having to go through and add #ifdef(FREEBSD)
in far far far too many places.

It's also pretty clear that mozilla/security/coreconf is descended from
the same source that gave us mozilla/nsprpub/config, but hasn't been touched
for anything but the core platforms.

Things would be a immeasurably easier if NSS were using feature tests rather
than platform tests, and if these feature tests were generated automatically
rather than having to be set by hand in coreconf/*.mk.
Target Milestone: --- → 3.2
Target Milestone: 3.2 → 3.3
Status: NEW → ASSIGNED
Assignee: wtc → relyea
Status: ASSIGNED → NEW
Assigned the bug to Bob.

Please refrain from starting a religious debate in this
bug report.  That should be done in the mozilla.crypto
newsgroup.

My proposal is a hybrid solution.
1. Convert all the Makefile's in NSS to Makefile.in's.
2. Add a configure shell or Perl script for NSS that
   does the following.
   - Translate the configure options (e.g., --enable-optimize)
     to coreconf's make variables (e.g., BUILD_OPT=1).
   - Generate Makefile's from Makefile.in's in either the
     source tree or an object build tree (created by the
     configure script).
   - The generated Makefile's still use coreconf.

This proposal solves two problems.
1. It presents a familiar configure interface for
   specifying build options.
2. It allows building NSS from a read-only source tree.

It does not solve the following problems.
1. It may not allow you to specify an arbitrary compiler
   by saying
       CC=cc ./configure
   Each compiler (e.g., native compiler or gcc) must
   be specified by a configure option.  For example,
       ./configure --with-gcc
2. Each platform still requires a <platform>.mk in
   coreconf.
I think the majority of the people who build NSS from
source can live with and won't even notice these two
problems.
Unfortunately, this proposal doesn't seem to address the original reporter's
concerns of using feature tests instead of hardcoding the values per platform. 
It will alieviate some of the build issues we've had though.


So I have a question. Is the problem that NSS does not have feature tests or
that NSS does not use autoconf. These are two very different and separable problems.

The first problem should be a big which enumerates where NSS incorrectly uses
platforms rather than feature tests. These places do exist but reside mostly in
the following areas: depricated commands, pkcs #11, and FORTEZZA. Of these only
pkcs #11 should be an issue.

The second problem is providing an autoconf front end so the existing scripts.
because 1) the current title implies this problem, and 2) the bug does not
contain the specific areas to fix feature based verse platform based issues,
I'll treat the bug as if it was for this second problem. Fixes for specific
ifdef problems should be filed under a specific bug.

Discussions on this should be initiated in the mozilla.crypto newsgroup.
bob
I'm going to skip this round of "post issue to ng, have just a handful of people
post an opinion _once_ to the thread and then watch the issue be stalled out for
9 more weeks" and skip straight to the chase.  We need to figure out what we are
going to do about nss/psm and whatever other modules that mozilla depends upon
*now*.  

In answer to your question, Bob: The original bug report was on the fact that
NSS does not use feature tests but makes assumptions about based upon the
platform.  A separate issue (and more essential to the big picture) is the fact
that NSS as it builds today cannot be integrated into the mozilla build system
without breaking several of our supported build configurations.  As I pointed
out in bug 60912, the key building issues are:
* building in parallel (-j)
* multiple objtree builds
* cross-compiling

The easiest (most common?) way for most people to solve the 2 issues
(feature-tests & extended build features) is to use autoconf.  It handles
building outside of the source tree natively and without having to worry about
generating .OBJ subdirs on the fly, we fix the parallel build problem as well.  
As with bug 60912, if you can figure out how to accomplish this without using
autoconf, more power to you.  I don't have time to search for that alternative
so I'll stick with what I know. 

This simple build issue sparks so many Mozilla policy discussions that it's near
sickening.  And probably off-topic, so I'll skip it.

This bug is 6 months old.  How much more time are we going to waste discussing
the simple matter of whether or not to autoconf the build when we could have
implemented it and had it working in production by now?  If the NSS team does
not want to provide a usable (to someone other than NSS) build system, then why
can we (read Mozilla developers) create our own copy of one of their releases
(3.2?) and modify it to our needs?  Although I still disagree with creating such
forks in principle, we've done it in the past with several libraries developed
by an external group (jpeg, zlib, png, mng).  And if the Mozilla team decides
not to fork for whatever reason, how do we resolve this?  Make it an external
dependency for all platforms?  Last I (mis)remembered, no feature is enabled on
a platform by default unless it is enabled on all platforms.   And is NSS in a
state where it can be installed on a platform as a system lib (win32, linux &
mac) or does 'make install' just populate dist/?
Blocks: 60912
I think the best way to handle this is by creating an NSS_CLIENT_BRANCH, so we
can make modifications to the files we need to support building in the client
build configurations.
Chris,

Please post your discussion to netscape.public.mozilla.crypto as Wan-Teh
suggested. I can respond there. As it stands I'm treating this bug as a request
to implement Wan-Teh's proposal. "Fully autoconf'ing" NSS has *NOT* been
accepted by the team because the preconditions we layed out for that has not
been met. We can discuss this in the news group.

bob
Bob, when did you ever lay out these preconditions for moving to autoconf?  Or
are you referring to wtc's proposal?  If this bug is going to be about wtc's
proposal as opposed to the orignal bug report, then we're going to be switching
things over to use the autoconf framework anyways so let's get to it. 
 
Leaf or bryner, create the NSS_CLIENT_BRANCH and let's get going.  I'm guessing
that we'll want it branched from the 3.2 branch rather than the tip.
Wan-Teh laid these preconditions out long ago.
The Mozilla browser is not the only product that uses NSS.
It is one of many.
The NSS group's products have many customers, some paying, some not.
We absolutely must meet the needs of our paying customers first.
The iPlanet servers are paying customers.  They have their own build
group with their own standards for build systems.
So, having a build system that meets their needs is an absolute
precondition.  It is not acceptable to replace the existing 
build system with one that warms the hearts of lots of open source
people, but doesn't meet the needs of our paying customers.

If you branch the tree, I believe there will be little or no new 
crypto development in that branch.  You'll have makefiles to your 
liking however.

Finally, I'll opine that NSS needs many things done to it, mostly
difficult things that have to do with cryptography and certificates.
Since we _are_ able to build working code for mozilla at the present
time, I believe the need for a new build system is minor compared
to the other needs.  
nelsonb:

Saying that the build system must meet the needs of your paying customers is 
pretty vague, especially when you give us no idea of what is required for that 
to happen.  Surely the existing build system is not the only possible build 
system that meets these needs.

If we were to branch the tree, we would land stable NSS releases on the branch 
as they happen.  This is the strategy currently being used for our NSPR branch.

Yes, NSS needs crypto work done on it.  Very few people working on Mozilla know 
the intricacies of this as well as the NSS team does.  However, there *are* 
people outside the NSS team who are ready and willing to put in the time to 
implement a new build system.  The NSS team is denying them the opportunity to 
contribute this on the NSS tip.  We (Mozilla developers) have been laying out 
for months now the shortcomings of the existing build system, but no one seems 
to be listening.

If NSS wasn't hosted on cvs.mozilla.org we wouldn't have hesitated at all to 
check it into our tree and convert it to the Mozilla build system (see libjpeg, 
libpng, zlib, and others).

By branching the tree we would actually be taking the responsibility for a 
large number of client build integration issues off the hands of the NSS team, 
which would let you (as you said) focus on improving crypto functionality and 
delivering builds for your paying customers.  And if the NSS team ever wanted 
to land the branch back onto the tip, that would always be an option.

So here's my concern:

You branch the tree and but in *mozilla* autoconf (which is presumably what we
are really talking about when we say we want to 'autoconf' nss).

Because mozilla autoconf is not yet mature in handling components, the NSS team
have the bandwidth to support the autoconf build system, and continue on it's
development with it's existing system. Official NSS releases, and instructions
to build NSS will still be coreconf and mozilla will always be working with an
old, and unsupported version of NSS.

I'm not against using autoconf. My problem is we can't even get autoconf to
correctly and reliably build NSPR and DBM on all of our platforms as components.
Sure, you can pull all of mozilla and get them to work, but if you want to build
just a single component, autoconf just isn't there yet. Right now it's actually
easier for us to walk through each of Chris's requirements and back fill them
into coreconf than it would be for us to try to maintain a mozilla autoconf system.

I don't think this state of affairs will continue forever, as binary components
will become more and more important to the mozilla autoconf and a lot of the
unnecessary baggage and complications it adds to people just trying to build a
component will get ironned out, but it's just not there yet, and it doesn't yet
seem to be a priority. Currently I can build on all my platforms (Unix, Windows,
Open/VMS, and OS/2, with just the most basic tools (native compilers and the
small set of nstools) using all the same makefiles. The current build
instructions (including machine setup) for NSS is half a page long. Trying to
build DBM for NSS is several pages with dozens on unecessary environment
variables, and dozens of exceptions for platforms like AIX and HP.

When we can build DBM and NSPR as binary components with autoconf on all
platforms as smoothly as we can build them with coreconf, then we can start
looking at converting NSS.

So what do we do in the meantime?

First if Chris' concerns are real concerns, and not just excuses to move to
autoconf, Chris needs to right bugs that say 'NSS does not support XXXXXXX' so
we can prioritize these features appropriately. Given the current state of
affairs, the NSS team would most likely support these features by modifying
coreconf.

If these features are just a red herring and religion dictates that mozilla must
have it's monolithic autoconf system, then perhaps a setup similar to what NSS
did to fix our problems with DBM. We created our own DBM directory, which
contains our own makefiles. These makefiles link the real mozilla dbm code, but
uses it's own makefiles to build dbm. It took Nelson one day to get this
working. This code could live on the tip, and get tagged when appropriate
branches are made. There is still the problem of keeping the parallel build
system in sync, but it sounds like the mozilla community is willing to carry
this burden.

bob

By "autoconf'ing NSS", we mean convert the NSS build system over to use
autoconf.  Full stop.  Additional dependencies upon the mozilla build system is
neither required nor desired.  NSS would continue to be a standalone package
like NSPR. 

Your problems with building dbm are not problems with autoconf.  They are
organizational problems with Mozilla's build system.  Pull the pristine sources
from sleepycat.com and see how they use autoconf to build it for many platforms
without a hitch. Having several pages of instructions to do:
./configure --enable-modules=dbm; make 
seems to indicate a severe lack of understanding about the build system.

As I said, if you can resolve the build issues I've pointed out *and* the
original reporter's problem without switching to autoconf, go for it.  My
impression is that fixing these problems is not a priority for the NSS team but
they are quickly becoming so for the Mozilla team as we get daily requests to
pull & build nss/psm by default.  If we have to solve this problem, then we'll
do it as we know how.  If the NSS team wants to solve it as they know how, then
they need to step up and do it.

The statement about using an "unsupported" release of NSS confuses me and brings
up another issue.  Are you saying that you will *only* support the _latest_
release of NSS?  If that's true (which I really don't believe it is), then what
makes Mozilla any different than any other client that you have using NSS?  Are
they forced to upgrade whenever a new release comes out or are they free to
upgrade at their leisure?


Weren't the original reporters problems solved long ago?
Doesn't NSS build on freeBSD now? 

The NSS team doesn't support branches other than those we create.  This
is a policy of long standing.  If other groups create branches, they
have the full burden of support for those branches.  If the mozilla team
makes its own branch, and there are problems with the code on that
branch, problems that are not also seen on the supported branches, we
don't have time to deal with those problems.  I think that's what Bob
was mentioning about "unsupported versions of NSS".

We don't have the cycles to support two build systems in parallel, 
nor to support one whose requirements are constantly changing. 

Any build system that we use (and are expected to maintain) must
understand all the build targets and environment variables that are now
understood and supported by coreconf.  The ability to build packaged jar
files, for example, must be supported.  The ability to use iPlanet's
conventions for object directory names must be supported.

We cannot impose new work on the iPlanet build team, or on our paying
customers, to convert to a new build system.  Any replacement build
system must be a "drop in" replacement, as far as the iPlanet build
people are concerned, and the outputs of those builds must not require
changes by the server groups that use them.

The performance sensitive code (mostly in nss/lib/freebl and
subdirectories thereof) must continue to perform as well or better than
it does now.  It isn't enough to produce code that merely successfully
compiles on all platforms.  It must run optimally, too.

mozilla may not be very performance sensitive, but our paying customers
all are, without exception.  Our paying customers won't stand for poorer
performance brought about by new build systems and/or conversion from
system-based #ifs to feature-based #ifs.

Some platforms (e.g.  Solaris, HPUX, and possibly AIX) support 32-bit
CPUs, 64-bit CPUs running 32-bit ABI (32-bit pointers and libs), and
64-bit CPUs running 64-bit ABI (64-bit pointers and libs).  NSS must
continue to produce optimal code for all those combinations.  Today, NSS
allows a single application running with 32-bit code to get optimal
performance on both 32-bit CPUs and 64-bit CPUs.  It must continue to do
so.

Anyone who wants to tackle converting NSS MUST seriously evaluate the
work to be done on freebl.  Any proposals to convert NSS that don't
include details of how to convert freebl are incomplete.

This is not an exhaustive list of the requirements for the NSS build
system, but it's a good start, I think.  And I believe all the 
things above are motivated by _business_ requirements, not mere 
personal preferences of style.

IMO, if someone were to produce a new build system that is a drop in
replacement for coreconf, meeting all the requirements of our iPlanet
customers, and also mozilla, that would allow all existing build scripts
to continue to work, I think such a proposal would not be rejected out
of hand.  But so far, I've seen no proposals that addressed any of the
requirements of anything but mozilla itself, and proposals that ignore
the needs of our supporting customers probably won't receive more
serious consideration than they have in the past, IMO.

In trying to get PSM1 or PSM2 to compile on IRIX6.5 (when using a separate 
object tree) NSS uses IRIX6_mips_DBG.OBJ and IRIX6.5_DBG.OBJ inconsistently; 
resulting in build problems.  It also trys to link n32 and o32 objects 
together.  While this is probably a different bug, NSS using autoconf would 
sure help.  While it does not block 68591, it makes the build process very 
tedious to automate.

John
It's not NSS being inconsistant. It's very component having it's own idea about
what OBJDIR needs to be and which compilier flags to use. These choices are made
based on different criteria (Servers want to use compilers and ABI's supported
by the vendors and run on the latest version of the OS. Mozilla prefers
compilers freely available (gcc) and ABI's that run on the oldest reasonable
platform). Going to Autoconf won't help that. NSS still has to supply both types
of binaries.

BTW most of the inconsistancies go away with the latest NSS build scheme
(independent of autoconf).
Your comment on different criteria for servers vs mozilla is not accurate in 
this case, as NSS was building O32 (which is extremely old) whereas 
mozilla/autoconf choose N32(SGIs prefered ABI to distribute).  I think it would 
be accurate for most unix vendors that mozilla requires more optimisation than 
a 'server', because it is quite large, and hence needs to be optimised more 
than most 'applications', added to the fact that there could be 100 copies of 
mozilla running on a server, whereas few systems would run 100 copies of any 
server.

Using the instructions posted to n.p.m.crypto for building PSM2.0 Milestone 1.5 
against the latest trunk, I was able to get PSM2 to build after creating 
symbolic links from dist/IRIX6_mips_DBG.OBJ to dist/IRIX6.5_DBG.OBJ.  
IRIX6_mips_DBG.OBJ is created in the build process, however both are utilised, 
even in the same compilation commands.

---
cd crmf; gmake libs
gmake[3]: Entering directory 
`/projects/sise/mozilla/devel/workpits/moz/latest_debug/workarea/security/nss/li
b/crmf'
cc -o IRIX6.5_DBG.OBJ/crmfenc.o -c -g -dollar -fullwarn -xansi -n32 -mips3 -
exceptions -DSVR4 -DIRIX  -multigot -D_SGI_MP_SOURCE -MDupdate 
IRIX6.5_DBG.OBJ/.md -DIRIX6_5 -mips3 -DXP_UNIX -DDEBUG -UNDEBUG -DDEBUG_johnv -
I../../../../dist/IRIX6.5_DBG.OBJ/include  -
I/projects/sise/mozilla/devel/workpits/moz/latest_debug/workarea/dist/IRIX6_mips
_DBG.OBJ/public/security -
I/projects/sise/mozilla/devel/workpits/moz/latest_debug/workarea/dist/IRIX6_mips
_DBG.OBJ/private/security -
I/projects/sise/mozilla/devel/workpits/moz/latest_debug/workarea/dist/IRIX6_mips
_DBG.OBJ/public/dbm  crmfenc.c
cc WARNING:  -I../../../../dist/IRIX6.5_DBG.OBJ/include does not refer to a 
valid directory
cc WARNING:  -
I/projects/sise/mozilla/devel/workpits/moz/latest_debug/workarea/dist/IRIX6_mips
_DBG.OBJ/public/dbm does not refer to a valid directory
cc-1005 cc: ERROR File 
= /projects/sise/mozilla/devel/workpits/moz/latest_debug/workarea/dist/IRIX6_mip
s_DBG.OBJ/public/security/seccomon.h, Line = 47
  The source file "prtypes.h" is unavailable.

  #include "prtypes.h"
                      ^

1 catastrophic error detected in the compilation of "crmfenc.c".
Compilation terminated.
---

I had the exact same problem with PSM1 on 0.8.  How is it that "It's not NSS 
being inconsistant. ".  Within NSS it is obviously having problems working out 
what "... it's own idea about what OBJDIR needs to be ..." .

Why cant we have two build systems?  Netscape retain coreconf & co for servers 
that require it, and mozilla start the process of creating an autoconf build.  
Should it not have all that netscape requires, they dont need to use it.  
Should it become developed enough to produce optimal binaries on all platforms, 
Netscape could re-assess their use of coreconf & co.
In building NSS, the PSM makefiles alter some of NSS's makefile variables so
that NSS can be built in the Mozilla environment.  It appears to me that IRIX
breaks that layer.  The above error doesn't show an inconsistency in the NSS
build, but an inconsistency in the named OBJ directories between mozilla and
NSS.

What needs to be looked at here is the PSM makefile which has never been tested
on IRIX, not the NSS build mechanism.
I think you're right, Javi.  There are two more environment variables 
that are relevant to builds of NSS on IRIX.  They are:

USE_N32  - set to 1 if you want the build to generate n32 binaries.  
	default is to generate o32 binaries.
	This is mutually exclusive with USE_64 (which is documented).

USE_PTHREADS - set to 1 if you want to use IRIX's pthreads implementation
	for threads.  As I recall, the default is to use IRIX's sprocs
	for threads.  

I just sucesfully built NSS 3.2 on an IRIX 6.5.5 machine with the above two 
variables set, using the instructions Sean posted.

I have built and tested PSM2 M1.5/NSS 3.2; the build dump was to show why I 
assumed NSS was breaking due to differently named OBJDIRs.  As suggested this 
could be a PSM problem, but the best fix for this problem would be to enable 
NSS to compile its object in a separate tree (without copying source/make 
files).
I've made my decision: NSS will continue to use its current
build system.

Many Mozilla developers have requested that the NSS build
system be converted to use autoconf.  In fact, some have
already started doing that on the unofficial, experimental
NSS_AUTOCONF_BRANCH.

In a meeting in February with mozilla.org, I agreed to have
the NSPR build system (which is similar to the NSS build
system) converted to use autoconf (the "NSPR experiment")
and then decide whether the same should be done for NSS.
Below is a summary of my findings and decisions.

1. Cost.

   The NSPR experiment gave us a realistic estimate of
   how much work will be required to convert NSS to
   use autoconf.  Our requirement is that the NSS
   autoconf build system be 100% equivalent to the
   current NSS coreconf build system on the platforms
   that the NSS team supports.  Assuming that the new
   autoconf build system will be implemented by volunteers
   and the NSS team is only responsible for the verification
   (including making the necessary fixes), I estimate that
   it requires six working days of one person to do the
   first round of verification, and five man-days to do
   the second round of verification.  Note that the first
   verification pass is best done by the same person; the
   second round can be parallelized.

   Then, additional time is needed to modify our binary
   release and nightly build scripts to work with the new
   build procedure and build tree structure.  Unfortunately
   the NSPR experiment did not offer any insight into how
   much time is required to complete this work.  This could
   just require minor tweaking as is the case with NSPR.  A
   rough estimate would be four days.

   We would also need to convert JSS to use autoconf as it
   shares the "coreconf" build configuration system with
   NSS.  Since the volunteers are only interested in building
   the Mozilla client, which does not use JSS, the NSS team
   would need to both implement the autoconf build system for
   JSS and verify its equivalence to the current build system.

2. Benefits.

   This change will affect three kinds of people who need
   to build NSS.

   For the NSS team, the current build system works just
   fine.  There is little benefit to be derived from
   ditching a working system and rewriting it in another
   language.

   For developers new to NSS who need to use NSS in their
   products, it is necessary to read the build instructions
   to learn how to specify the build options in the current
   NSS build system.  But it is not rocket science.
   Moreover, our experience has been that most NSS users
   would happily use our binary distributions once they
   learned that the binary distributions exist.
   ftp://ftp.mozilla.org/pub/security/nss/releases/
   ftp://ftp.mozilla.org/pub/nspr/releases/

   For people building the Mozilla client, the NSS build
   system has been a continual source of pain due to the
   lack of support of certain Mozilla build features.  Some
   of the complaints are a matter of taste, but there are
   more substantive issues (e.g., the ability to build from
   a read-only source tree and support for cross-compilation).
   Our team has taken steps to address them.  We have a
   solution ready that should meet the needs of the majority
   of the people building the Mozilla client (see the patches
   attached to bug #83225 and bug #82324).

With the cost-benefit analysis above, I have decided that
NSS should continue to use its current build system.  We
will meet the requirements of the majority of the NSS users
and people building Mozilla client by providing better NSS
binary distributions and better integration with the Mozilla
client build system.  To those who have requirement that
are not met, I ask for your understanding and your cooperation
in the best interests of the Mozilla and NSS projects.  We
did not ignore your requests.  They have all been carefully
considered and rejected because there is not a big demand
for them or easy workarounds are present
Status: NEW → RESOLVED
Closed: 23 years ago
Resolution: --- → WONTFIX
If you could host & link to a forum/wiki with subsections for
the different build targets & cross compilation, that would
help the open source community help itself.  Better internal 
documentation of the NSS build would also be nice.

Building NSS on a Linux native compiler was a breeze, but
cross-compiling for Win32 on Linux using mingw was such
a pain, I ultimately gave up and used a pre-build Win32
binary (and did the necessary dance with nm, sed, and dlltool
to create the proper .dll.a import library in order to link
with all the other mingw-compiled stuff).

The pre-built msvc NSS binary isn't a very good substitute
for being able to build from source, because it limits my
ability to follow things in a debugger.   While it's true
that some libraries are more difficult to cross-compile than
others (e.g.: zlib is also kind of cantankerous), I was able
to get everything to work _except_ for NSS.  

If there were an official wiki/forum on mozilla.org for the 3rd tier,
someone more familiar with NSS than I am would probably have posted
an answer & patches by now. :)  Would you consider hosting such a wiki?


                Cheers,
                -Jon
                                                                           
(In reply to comment #19)
> I've made my decision: NSS will continue to use its current
> build system.

So, 10 years later...one thing that isn't obvious to me from this discussion is what issues would be involved in simply having "one and a half" build systems.  More precisely, add a plain autoconf configure script, and a Makefile that supports the automake conventions (even if not actually being automake).

In that vein, I wanted to link to the current nss patch we're using in GNOME:

http://git.gnome.org/browse/jhbuild/tree/patches/nss.pkgconfig-and-configure.patch

The impact as you can see on the nss tree is small - it's not a large patch.  The biggest cost is having to reimplement the "install" logic so we get support for "make install DESTDIR=", which is crucial for us.
You need to log in before you can comment on or make changes to this bug.