Closed Bug 52990 Opened 20 years ago Closed 19 years ago
NSS should use autoconf
I'm in the process of trying to port NSS to run on FreeBSD (3.5), and it's very quickly becoming obvious that the current build system is going to be a portability nightmare. I'm having to go through and add #ifdef(FREEBSD) in far far far too many places. It's also pretty clear that mozilla/security/coreconf is descended from the same source that gave us mozilla/nsprpub/config, but hasn't been touched for anything but the core platforms. Things would be a immeasurably easier if NSS were using feature tests rather than platform tests, and if these feature tests were generated automatically rather than having to be set by hand in coreconf/*.mk.
Assigned the bug to Bob. Please refrain from starting a religious debate in this bug report. That should be done in the mozilla.crypto newsgroup. My proposal is a hybrid solution. 1. Convert all the Makefile's in NSS to Makefile.in's. 2. Add a configure shell or Perl script for NSS that does the following. - Translate the configure options (e.g., --enable-optimize) to coreconf's make variables (e.g., BUILD_OPT=1). - Generate Makefile's from Makefile.in's in either the source tree or an object build tree (created by the configure script). - The generated Makefile's still use coreconf. This proposal solves two problems. 1. It presents a familiar configure interface for specifying build options. 2. It allows building NSS from a read-only source tree. It does not solve the following problems. 1. It may not allow you to specify an arbitrary compiler by saying CC=cc ./configure Each compiler (e.g., native compiler or gcc) must be specified by a configure option. For example, ./configure --with-gcc 2. Each platform still requires a <platform>.mk in coreconf. I think the majority of the people who build NSS from source can live with and won't even notice these two problems.
Unfortunately, this proposal doesn't seem to address the original reporter's concerns of using feature tests instead of hardcoding the values per platform. It will alieviate some of the build issues we've had though.
So I have a question. Is the problem that NSS does not have feature tests or that NSS does not use autoconf. These are two very different and separable problems. The first problem should be a big which enumerates where NSS incorrectly uses platforms rather than feature tests. These places do exist but reside mostly in the following areas: depricated commands, pkcs #11, and FORTEZZA. Of these only pkcs #11 should be an issue. The second problem is providing an autoconf front end so the existing scripts. because 1) the current title implies this problem, and 2) the bug does not contain the specific areas to fix feature based verse platform based issues, I'll treat the bug as if it was for this second problem. Fixes for specific ifdef problems should be filed under a specific bug. Discussions on this should be initiated in the mozilla.crypto newsgroup. bob
I'm going to skip this round of "post issue to ng, have just a handful of people post an opinion _once_ to the thread and then watch the issue be stalled out for 9 more weeks" and skip straight to the chase. We need to figure out what we are going to do about nss/psm and whatever other modules that mozilla depends upon *now*. In answer to your question, Bob: The original bug report was on the fact that NSS does not use feature tests but makes assumptions about based upon the platform. A separate issue (and more essential to the big picture) is the fact that NSS as it builds today cannot be integrated into the mozilla build system without breaking several of our supported build configurations. As I pointed out in bug 60912, the key building issues are: * building in parallel (-j) * multiple objtree builds * cross-compiling The easiest (most common?) way for most people to solve the 2 issues (feature-tests & extended build features) is to use autoconf. It handles building outside of the source tree natively and without having to worry about generating .OBJ subdirs on the fly, we fix the parallel build problem as well. As with bug 60912, if you can figure out how to accomplish this without using autoconf, more power to you. I don't have time to search for that alternative so I'll stick with what I know. This simple build issue sparks so many Mozilla policy discussions that it's near sickening. And probably off-topic, so I'll skip it. This bug is 6 months old. How much more time are we going to waste discussing the simple matter of whether or not to autoconf the build when we could have implemented it and had it working in production by now? If the NSS team does not want to provide a usable (to someone other than NSS) build system, then why can we (read Mozilla developers) create our own copy of one of their releases (3.2?) and modify it to our needs? Although I still disagree with creating such forks in principle, we've done it in the past with several libraries developed by an external group (jpeg, zlib, png, mng). And if the Mozilla team decides not to fork for whatever reason, how do we resolve this? Make it an external dependency for all platforms? Last I (mis)remembered, no feature is enabled on a platform by default unless it is enabled on all platforms. And is NSS in a state where it can be installed on a platform as a system lib (win32, linux & mac) or does 'make install' just populate dist/?
I think the best way to handle this is by creating an NSS_CLIENT_BRANCH, so we can make modifications to the files we need to support building in the client build configurations.
Chris, Please post your discussion to netscape.public.mozilla.crypto as Wan-Teh suggested. I can respond there. As it stands I'm treating this bug as a request to implement Wan-Teh's proposal. "Fully autoconf'ing" NSS has *NOT* been accepted by the team because the preconditions we layed out for that has not been met. We can discuss this in the news group. bob
Bob, when did you ever lay out these preconditions for moving to autoconf? Or are you referring to wtc's proposal? If this bug is going to be about wtc's proposal as opposed to the orignal bug report, then we're going to be switching things over to use the autoconf framework anyways so let's get to it. Leaf or bryner, create the NSS_CLIENT_BRANCH and let's get going. I'm guessing that we'll want it branched from the 3.2 branch rather than the tip.
Wan-Teh laid these preconditions out long ago. The Mozilla browser is not the only product that uses NSS. It is one of many. The NSS group's products have many customers, some paying, some not. We absolutely must meet the needs of our paying customers first. The iPlanet servers are paying customers. They have their own build group with their own standards for build systems. So, having a build system that meets their needs is an absolute precondition. It is not acceptable to replace the existing build system with one that warms the hearts of lots of open source people, but doesn't meet the needs of our paying customers. If you branch the tree, I believe there will be little or no new crypto development in that branch. You'll have makefiles to your liking however. Finally, I'll opine that NSS needs many things done to it, mostly difficult things that have to do with cryptography and certificates. Since we _are_ able to build working code for mozilla at the present time, I believe the need for a new build system is minor compared to the other needs.
nelsonb: Saying that the build system must meet the needs of your paying customers is pretty vague, especially when you give us no idea of what is required for that to happen. Surely the existing build system is not the only possible build system that meets these needs. If we were to branch the tree, we would land stable NSS releases on the branch as they happen. This is the strategy currently being used for our NSPR branch. Yes, NSS needs crypto work done on it. Very few people working on Mozilla know the intricacies of this as well as the NSS team does. However, there *are* people outside the NSS team who are ready and willing to put in the time to implement a new build system. The NSS team is denying them the opportunity to contribute this on the NSS tip. We (Mozilla developers) have been laying out for months now the shortcomings of the existing build system, but no one seems to be listening. If NSS wasn't hosted on cvs.mozilla.org we wouldn't have hesitated at all to check it into our tree and convert it to the Mozilla build system (see libjpeg, libpng, zlib, and others). By branching the tree we would actually be taking the responsibility for a large number of client build integration issues off the hands of the NSS team, which would let you (as you said) focus on improving crypto functionality and delivering builds for your paying customers. And if the NSS team ever wanted to land the branch back onto the tip, that would always be an option.
So here's my concern: You branch the tree and but in *mozilla* autoconf (which is presumably what we are really talking about when we say we want to 'autoconf' nss). Because mozilla autoconf is not yet mature in handling components, the NSS team have the bandwidth to support the autoconf build system, and continue on it's development with it's existing system. Official NSS releases, and instructions to build NSS will still be coreconf and mozilla will always be working with an old, and unsupported version of NSS. I'm not against using autoconf. My problem is we can't even get autoconf to correctly and reliably build NSPR and DBM on all of our platforms as components. Sure, you can pull all of mozilla and get them to work, but if you want to build just a single component, autoconf just isn't there yet. Right now it's actually easier for us to walk through each of Chris's requirements and back fill them into coreconf than it would be for us to try to maintain a mozilla autoconf system. I don't think this state of affairs will continue forever, as binary components will become more and more important to the mozilla autoconf and a lot of the unnecessary baggage and complications it adds to people just trying to build a component will get ironned out, but it's just not there yet, and it doesn't yet seem to be a priority. Currently I can build on all my platforms (Unix, Windows, Open/VMS, and OS/2, with just the most basic tools (native compilers and the small set of nstools) using all the same makefiles. The current build instructions (including machine setup) for NSS is half a page long. Trying to build DBM for NSS is several pages with dozens on unecessary environment variables, and dozens of exceptions for platforms like AIX and HP. When we can build DBM and NSPR as binary components with autoconf on all platforms as smoothly as we can build them with coreconf, then we can start looking at converting NSS. So what do we do in the meantime? First if Chris' concerns are real concerns, and not just excuses to move to autoconf, Chris needs to right bugs that say 'NSS does not support XXXXXXX' so we can prioritize these features appropriately. Given the current state of affairs, the NSS team would most likely support these features by modifying coreconf. If these features are just a red herring and religion dictates that mozilla must have it's monolithic autoconf system, then perhaps a setup similar to what NSS did to fix our problems with DBM. We created our own DBM directory, which contains our own makefiles. These makefiles link the real mozilla dbm code, but uses it's own makefiles to build dbm. It took Nelson one day to get this working. This code could live on the tip, and get tagged when appropriate branches are made. There is still the problem of keeping the parallel build system in sync, but it sounds like the mozilla community is willing to carry this burden. bob
By "autoconf'ing NSS", we mean convert the NSS build system over to use autoconf. Full stop. Additional dependencies upon the mozilla build system is neither required nor desired. NSS would continue to be a standalone package like NSPR. Your problems with building dbm are not problems with autoconf. They are organizational problems with Mozilla's build system. Pull the pristine sources from sleepycat.com and see how they use autoconf to build it for many platforms without a hitch. Having several pages of instructions to do: ./configure --enable-modules=dbm; make seems to indicate a severe lack of understanding about the build system. As I said, if you can resolve the build issues I've pointed out *and* the original reporter's problem without switching to autoconf, go for it. My impression is that fixing these problems is not a priority for the NSS team but they are quickly becoming so for the Mozilla team as we get daily requests to pull & build nss/psm by default. If we have to solve this problem, then we'll do it as we know how. If the NSS team wants to solve it as they know how, then they need to step up and do it. The statement about using an "unsupported" release of NSS confuses me and brings up another issue. Are you saying that you will *only* support the _latest_ release of NSS? If that's true (which I really don't believe it is), then what makes Mozilla any different than any other client that you have using NSS? Are they forced to upgrade whenever a new release comes out or are they free to upgrade at their leisure?
Weren't the original reporters problems solved long ago? Doesn't NSS build on freeBSD now? The NSS team doesn't support branches other than those we create. This is a policy of long standing. If other groups create branches, they have the full burden of support for those branches. If the mozilla team makes its own branch, and there are problems with the code on that branch, problems that are not also seen on the supported branches, we don't have time to deal with those problems. I think that's what Bob was mentioning about "unsupported versions of NSS". We don't have the cycles to support two build systems in parallel, nor to support one whose requirements are constantly changing. Any build system that we use (and are expected to maintain) must understand all the build targets and environment variables that are now understood and supported by coreconf. The ability to build packaged jar files, for example, must be supported. The ability to use iPlanet's conventions for object directory names must be supported. We cannot impose new work on the iPlanet build team, or on our paying customers, to convert to a new build system. Any replacement build system must be a "drop in" replacement, as far as the iPlanet build people are concerned, and the outputs of those builds must not require changes by the server groups that use them. The performance sensitive code (mostly in nss/lib/freebl and subdirectories thereof) must continue to perform as well or better than it does now. It isn't enough to produce code that merely successfully compiles on all platforms. It must run optimally, too. mozilla may not be very performance sensitive, but our paying customers all are, without exception. Our paying customers won't stand for poorer performance brought about by new build systems and/or conversion from system-based #ifs to feature-based #ifs. Some platforms (e.g. Solaris, HPUX, and possibly AIX) support 32-bit CPUs, 64-bit CPUs running 32-bit ABI (32-bit pointers and libs), and 64-bit CPUs running 64-bit ABI (64-bit pointers and libs). NSS must continue to produce optimal code for all those combinations. Today, NSS allows a single application running with 32-bit code to get optimal performance on both 32-bit CPUs and 64-bit CPUs. It must continue to do so. Anyone who wants to tackle converting NSS MUST seriously evaluate the work to be done on freebl. Any proposals to convert NSS that don't include details of how to convert freebl are incomplete. This is not an exhaustive list of the requirements for the NSS build system, but it's a good start, I think. And I believe all the things above are motivated by _business_ requirements, not mere personal preferences of style. IMO, if someone were to produce a new build system that is a drop in replacement for coreconf, meeting all the requirements of our iPlanet customers, and also mozilla, that would allow all existing build scripts to continue to work, I think such a proposal would not be rejected out of hand. But so far, I've seen no proposals that addressed any of the requirements of anything but mozilla itself, and proposals that ignore the needs of our supporting customers probably won't receive more serious consideration than they have in the past, IMO.
In trying to get PSM1 or PSM2 to compile on IRIX6.5 (when using a separate object tree) NSS uses IRIX6_mips_DBG.OBJ and IRIX6.5_DBG.OBJ inconsistently; resulting in build problems. It also trys to link n32 and o32 objects together. While this is probably a different bug, NSS using autoconf would sure help. While it does not block 68591, it makes the build process very tedious to automate. John
It's not NSS being inconsistant. It's very component having it's own idea about what OBJDIR needs to be and which compilier flags to use. These choices are made based on different criteria (Servers want to use compilers and ABI's supported by the vendors and run on the latest version of the OS. Mozilla prefers compilers freely available (gcc) and ABI's that run on the oldest reasonable platform). Going to Autoconf won't help that. NSS still has to supply both types of binaries. BTW most of the inconsistancies go away with the latest NSS build scheme (independent of autoconf).
Your comment on different criteria for servers vs mozilla is not accurate in this case, as NSS was building O32 (which is extremely old) whereas mozilla/autoconf choose N32(SGIs prefered ABI to distribute). I think it would be accurate for most unix vendors that mozilla requires more optimisation than a 'server', because it is quite large, and hence needs to be optimised more than most 'applications', added to the fact that there could be 100 copies of mozilla running on a server, whereas few systems would run 100 copies of any server. Using the instructions posted to n.p.m.crypto for building PSM2.0 Milestone 1.5 against the latest trunk, I was able to get PSM2 to build after creating symbolic links from dist/IRIX6_mips_DBG.OBJ to dist/IRIX6.5_DBG.OBJ. IRIX6_mips_DBG.OBJ is created in the build process, however both are utilised, even in the same compilation commands. --- cd crmf; gmake libs gmake: Entering directory `/projects/sise/mozilla/devel/workpits/moz/latest_debug/workarea/security/nss/li b/crmf' cc -o IRIX6.5_DBG.OBJ/crmfenc.o -c -g -dollar -fullwarn -xansi -n32 -mips3 - exceptions -DSVR4 -DIRIX -multigot -D_SGI_MP_SOURCE -MDupdate IRIX6.5_DBG.OBJ/.md -DIRIX6_5 -mips3 -DXP_UNIX -DDEBUG -UNDEBUG -DDEBUG_johnv - I../../../../dist/IRIX6.5_DBG.OBJ/include - I/projects/sise/mozilla/devel/workpits/moz/latest_debug/workarea/dist/IRIX6_mips _DBG.OBJ/public/security - I/projects/sise/mozilla/devel/workpits/moz/latest_debug/workarea/dist/IRIX6_mips _DBG.OBJ/private/security - I/projects/sise/mozilla/devel/workpits/moz/latest_debug/workarea/dist/IRIX6_mips _DBG.OBJ/public/dbm crmfenc.c cc WARNING: -I../../../../dist/IRIX6.5_DBG.OBJ/include does not refer to a valid directory cc WARNING: - I/projects/sise/mozilla/devel/workpits/moz/latest_debug/workarea/dist/IRIX6_mips _DBG.OBJ/public/dbm does not refer to a valid directory cc-1005 cc: ERROR File = /projects/sise/mozilla/devel/workpits/moz/latest_debug/workarea/dist/IRIX6_mip s_DBG.OBJ/public/security/seccomon.h, Line = 47 The source file "prtypes.h" is unavailable. #include "prtypes.h" ^ 1 catastrophic error detected in the compilation of "crmfenc.c". Compilation terminated. --- I had the exact same problem with PSM1 on 0.8. How is it that "It's not NSS being inconsistant. ". Within NSS it is obviously having problems working out what "... it's own idea about what OBJDIR needs to be ..." . Why cant we have two build systems? Netscape retain coreconf & co for servers that require it, and mozilla start the process of creating an autoconf build. Should it not have all that netscape requires, they dont need to use it. Should it become developed enough to produce optimal binaries on all platforms, Netscape could re-assess their use of coreconf & co.
In building NSS, the PSM makefiles alter some of NSS's makefile variables so that NSS can be built in the Mozilla environment. It appears to me that IRIX breaks that layer. The above error doesn't show an inconsistency in the NSS build, but an inconsistency in the named OBJ directories between mozilla and NSS. What needs to be looked at here is the PSM makefile which has never been tested on IRIX, not the NSS build mechanism.
I think you're right, Javi. There are two more environment variables that are relevant to builds of NSS on IRIX. They are: USE_N32 - set to 1 if you want the build to generate n32 binaries. default is to generate o32 binaries. This is mutually exclusive with USE_64 (which is documented). USE_PTHREADS - set to 1 if you want to use IRIX's pthreads implementation for threads. As I recall, the default is to use IRIX's sprocs for threads. I just sucesfully built NSS 3.2 on an IRIX 6.5.5 machine with the above two variables set, using the instructions Sean posted.
I have built and tested PSM2 M1.5/NSS 3.2; the build dump was to show why I assumed NSS was breaking due to differently named OBJDIRs. As suggested this could be a PSM problem, but the best fix for this problem would be to enable NSS to compile its object in a separate tree (without copying source/make files).
I've made my decision: NSS will continue to use its current build system. Many Mozilla developers have requested that the NSS build system be converted to use autoconf. In fact, some have already started doing that on the unofficial, experimental NSS_AUTOCONF_BRANCH. In a meeting in February with mozilla.org, I agreed to have the NSPR build system (which is similar to the NSS build system) converted to use autoconf (the "NSPR experiment") and then decide whether the same should be done for NSS. Below is a summary of my findings and decisions. 1. Cost. The NSPR experiment gave us a realistic estimate of how much work will be required to convert NSS to use autoconf. Our requirement is that the NSS autoconf build system be 100% equivalent to the current NSS coreconf build system on the platforms that the NSS team supports. Assuming that the new autoconf build system will be implemented by volunteers and the NSS team is only responsible for the verification (including making the necessary fixes), I estimate that it requires six working days of one person to do the first round of verification, and five man-days to do the second round of verification. Note that the first verification pass is best done by the same person; the second round can be parallelized. Then, additional time is needed to modify our binary release and nightly build scripts to work with the new build procedure and build tree structure. Unfortunately the NSPR experiment did not offer any insight into how much time is required to complete this work. This could just require minor tweaking as is the case with NSPR. A rough estimate would be four days. We would also need to convert JSS to use autoconf as it shares the "coreconf" build configuration system with NSS. Since the volunteers are only interested in building the Mozilla client, which does not use JSS, the NSS team would need to both implement the autoconf build system for JSS and verify its equivalence to the current build system. 2. Benefits. This change will affect three kinds of people who need to build NSS. For the NSS team, the current build system works just fine. There is little benefit to be derived from ditching a working system and rewriting it in another language. For developers new to NSS who need to use NSS in their products, it is necessary to read the build instructions to learn how to specify the build options in the current NSS build system. But it is not rocket science. Moreover, our experience has been that most NSS users would happily use our binary distributions once they learned that the binary distributions exist. ftp://ftp.mozilla.org/pub/security/nss/releases/ ftp://ftp.mozilla.org/pub/nspr/releases/ For people building the Mozilla client, the NSS build system has been a continual source of pain due to the lack of support of certain Mozilla build features. Some of the complaints are a matter of taste, but there are more substantive issues (e.g., the ability to build from a read-only source tree and support for cross-compilation). Our team has taken steps to address them. We have a solution ready that should meet the needs of the majority of the people building the Mozilla client (see the patches attached to bug #83225 and bug #82324). With the cost-benefit analysis above, I have decided that NSS should continue to use its current build system. We will meet the requirements of the majority of the NSS users and people building Mozilla client by providing better NSS binary distributions and better integration with the Mozilla client build system. To those who have requirement that are not met, I ask for your understanding and your cooperation in the best interests of the Mozilla and NSS projects. We did not ignore your requests. They have all been carefully considered and rejected because there is not a big demand for them or easy workarounds are present
Status: NEW → RESOLVED
Closed: 19 years ago
Resolution: --- → WONTFIX
If you could host & link to a forum/wiki with subsections for the different build targets & cross compilation, that would help the open source community help itself. Better internal documentation of the NSS build would also be nice. Building NSS on a Linux native compiler was a breeze, but cross-compiling for Win32 on Linux using mingw was such a pain, I ultimately gave up and used a pre-build Win32 binary (and did the necessary dance with nm, sed, and dlltool to create the proper .dll.a import library in order to link with all the other mingw-compiled stuff). The pre-built msvc NSS binary isn't a very good substitute for being able to build from source, because it limits my ability to follow things in a debugger. While it's true that some libraries are more difficult to cross-compile than others (e.g.: zlib is also kind of cantankerous), I was able to get everything to work _except_ for NSS. If there were an official wiki/forum on mozilla.org for the 3rd tier, someone more familiar with NSS than I am would probably have posted an answer & patches by now. :) Would you consider hosting such a wiki? Cheers, -Jon
(In reply to comment #19) > I've made my decision: NSS will continue to use its current > build system. So, 10 years later...one thing that isn't obvious to me from this discussion is what issues would be involved in simply having "one and a half" build systems. More precisely, add a plain autoconf configure script, and a Makefile that supports the automake conventions (even if not actually being automake). In that vein, I wanted to link to the current nss patch we're using in GNOME: http://git.gnome.org/browse/jhbuild/tree/patches/nss.pkgconfig-and-configure.patch The impact as you can see on the nss tree is small - it's not a large patch. The biggest cost is having to reimplement the "install" logic so we get support for "make install DESTDIR=", which is crucial for us.
You need to log in before you can comment on or make changes to this bug.