Closed Bug 60912 Opened 24 years ago Closed 23 years ago

PSM must build on Linux for Mozilla users

Categories

(Core :: Security: PSM, defect, P3)

Product:
Core
Component:
Security: PSM
Version:
1.0 Branch
Platform:
x86
Linux
Type:
defect

Tracking

()

Status:
VERIFIED FIXED

People

(Reporter: ddrinan0264, Assigned: javi)

References

Details

Attachments

(13 files)

This patch builds PSM as part of Mozilla Linux build if configure is run with the --with-psm flag.
158.30 KB, patch
Details | Diff | Splinter Review
This patch uses --enable-psm to enable PSM
158.35 KB, patch
Details | Diff | Splinter Review
This patch works with a separate objdir and without creating new files in the source tree.
160.42 KB, patch
Details | Diff | Splinter Review
OS/2 version of configure.in
1.81 KB, patch
Details | Diff | Splinter Review
Updated patch including OS/2 patch with some instructions.
11.91 KB, text/plain
Details
Patch incorporating wtc's suggestions.
12.17 KB, patch
Details | Diff | Splinter Review
So I didn't save the change to client.mk incorporating wtc's suggestion. Here is the patch for real.
12.17 KB, patch
Details | Diff | Splinter Review
Inform standalone build system of psm
5.34 KB, patch
Details | Diff | Splinter Review
last portion of javi's attach 20918 that was not already checked in
2.62 KB, patch
Details | Diff | Splinter Review
Updated "stage1" patch. Includes psm-glue this time
6.97 KB, patch
Details | Diff | Splinter Review
Updated "stage2" patch. Fixes the objs in srcdir problem at cost of size of nss src tree
4.16 KB, patch
Details | Diff | Splinter Review
Do not pull PSM/NSS by default.
502 bytes, patch
Details | Diff | Splinter Review
if we touch mozilla/security, the checkout of mozilla/security fails (as CVS dirs are missing) shouldn't touch unless the build module has psm included
439 bytes, patch
Details | Diff | Splinter Review 
PSM must build as part of the regular mozilla build for Linux.
That should be --enable-psm instead of --with-psm. Although enable-* and with-*
work identically, there is a convention on which one to use in which situation
(unfortunately not always followed on other occasions too, see bug 61907)

Turns out the current patch does not work if you do the different objdir trick.

Fixing this will require a patch to NSS.  I'll work on getting that fix in tomorrow.
Before I post the next patch, I'd like to know what are the requirements for the
separate objdir feature?  Is it that no mozilla/dist appear?  I've got it so
that NSS/PSM build the way they used to (by putting everything in mozilla/dist),
and then installing the necessary pieces in mozilla/objdir/dist, but I'm
wondering if I need to make NSS/PSM put all of it's build pieces in
mozilla/objdir/dist.
> Before I post the next patch, I'd like to know what are the requirements for the
> separate objdir feature?

I can see several:
- Building several platforms from the same tree
- Not making massive writes to the source tree in order to save the filesystem
(same reason why /home and /tmp are usually on different filesystems - /home has
few writes, but very important data, while /tmp has the opposite charateristics;
same is true for source modifications vs. object files).

The requirements are that all generated files are built in the separate objdir.
 In essence, the user could build from a CD (or read-only partition) if need be.
Depends on: 62267
CC'ing myself on this bug as it will be intresting to see how this pans out in
Solaris.
Javi,
I am having a problem getting this to patch correctly on Solaris 8, x86, I
followed cls's instructions for doing the client.mk patch first and even that
wont patch correctly. any ideas?
In the future, you will want to break up a patch such as this.  You need to
apply the client.mk chunk first,  pull again to get the new security directories
then apply the rest of the patch.  You also want to remove the configure portion
of the patch as well as the slightest change to that file usually creates a huge
diff that usually doesn't apply cleanly to another tree.  

That said, once the patch was applied, a simple build worked.  However, parallel
builds do not.  They failed creating one of the Linux2.2_x86_glibc_PTH_OPT.OBJ
subdirs.  The usual if-test-mkdir race condition.

I do not like the prospect of making a separate copy of the various security/
subdirs for each separate object tree that we use.  This creates an additional
penalty for building in the objtree.  I'm sure that not everyone has the 6
objtrees that I use and it's only an additional 20 megs per tree but I feel that
it creates a bad precedent.

Is there a way to make the NSS build directly use the include and lib
directories from the main tree, rather than doing a moz_import, which takes up
unnecessary space?
I am trying to get this working on the OS/2 build, which uses the UNIX
makefiles.  However, I had problems with the configure.in:  for some reason, it
would not define MOZ_ENABLE_PSM when I passed in --enable-psm.  I finally got
this working by moving the enable-psm code further down in the file.  I am
attaching that patch below.  Does this work on the other platforms?

That step mirrors the instructions at
http://www.mozilla.org/projects/security/pki/nss/buildnss_31.html#unix

cls, I tried to make it so that it wasn't necessary to copy over the source for
separate objdir builds, but that diff was huge and was quite nasty.  Don't think
reviewers would like it because it was huge and not autoconf'ing (just making
coreconf do just enough), and the NSS team didn't like mucking too much with
their build system.

NSS also poses the problem that souce files are copied within the tree as part
of the build process, and if I understood correctly, I should assume a separate
objdir build could be on a read-only partition.  That kind of forced my hand
into copying the NSS source over.
Adding members of the NSS team to cc list, since this bug depends on integrating
NSS into Mozilla builds.
When looking through the tree, I noticed NSPR does create files in the source
tree (.o and .a files) so I went to not copying source files around, but only
over-riding the dist so that PSM/NSS files wind up in the new dist for separate
ojdir directories.

This will follow the same practice as NSPR, so there will be no new precedent
set.  And there won't be a big penalty for people with lots of objdir trees.
Then you will be landing an autoconf build system as nspr has as well?  The plan
(at least from this side) has always been to move to making the nspr autoconf
system the default while building Mozilla.  Right now, that's blocked by one
major bug, bug 58804, which is more logistical than technical in nature.  Also,
the classic nspr build does not allow users to have more than one objdir of
either an optimized or debug build because it uses the same names for the .OBJ
dirs in the srcdir.  See previous note about bad precedent.  

Once these modules are integrated with the Mozilla build, they will have to
conform to the same build standards as the rest of the modules.  This goes
beyond the vanilla build configuration that the tinderboxes on the main page
run.  Right now, there are certain "advanced" build features that we support for
the unix build.  These include but are not limited to: parallel builds, multiple
objdir builds & cross-compiling.  Before a major change like adding a new module
to the build, we should make sure that all of this feature set is present for
the new module or at least, a significant portion is present with plans to
implement the rest.  I'm not seeing that here. 


Let's move the NSS/PSM build system discussion
to the mozilla.crypto newsgroup.  As far as this
bug is concerned, Javier's patch meets the key
requirement: PSM can be built as part of Mozilla.
I disagree.  The end goal is that PSM builds with the Mozilla build.  The key
requirement is that it is integrated properly with the Mozilla build system.
This patch does not fulfill the key requirement.
Blocks: 62855
Blocks: 62856
I reviewed javi's latest patch (attachment id=20585).
I have one suggested change: in mozilla/client.mk,
move mozilla/security/Makefile.in from NSS_CO_MODULE
to PSM_CO_MODULE.  I also noticed a #if 0 in
mozilla/security/psm/server/serv.h that looks like a
temporary quick fix.  Other than these two, this
patch is fine.  r=wtc@netscape.com.

Note: this patch contains the changes to not build
NSS command-line utilities on Unix and Windows.
cls, do you sign off on this patch?
No.

Adding message sent by wtc to cls and brendan.

Brendan, Chris,

We are planning to check in Javi's latest patch
in Bug #60912 (http://bugzilla.mozilla.org/show_bug.cgi?id=60912)
for integrating PSM with the Mozilla build on Unix
this Wednesday morning.  We realize that some developers
don't consider this patch the ideal solution, but it
works and will allow us to make progress.  Discussion
on build systems tends to get emotional and drag on,
and I don't want that to prevent us from achieving the
important goal of integrating PSM with the Mozilla build
on all platforms.  (It's already done for Mac and Windows.)

Could you please review Javi's patch and give your
blessing?

Chris: let's meet face-to-face next time you come up
here.  I have some proposals of how to solve the NSPR
and NSS/PSM build system issues in the long term that
I'd like to run by you.  In particular I'd like to
hear more about your idea of using a Perl configure
script on Windows to fill in pre-determined values
in the *.in files so that we don't need to do the full
configure thing.

Thanks,
Wan-Teh
Hi,
this patch includes a modification of mozilla/security/nss/makefile.win, which
probably doesn't belong into this one, right?

Axel
Axel: this patch also contains a change to build
just the NSS libs, not the commands, on both Unix
and Windows. This is why mozilla/security/nss/makefile.win
was also modified.
is there any other place i can/will be able to get the command-line tools?  i would dearly like to, for example, add the Rice root ca cert to the database...am i stuck downloading the whole mozilla tree and building it just for a few dinky binaries?
To test these changes:

cvs -z3 co mozilla/client.mk
make -f client.mk pull_all BUILD_MODULES=
env POSIXLY_CORRECT=1 patch -p0 < ~/psm-stage1.patch
make -f client.mk pull_all BUILD_MODULES=psm
env POSIXLY_CORRECT=1 patch -p0 < ~/psm-stage2.patch
./configure --enable-modules=psm
make

I'm still testing the "integration" of the standalone psm build with the default
build but it should be as simple as doing:
make BUILD_MODULES=psm
either before or after the default build.

What are stage1 and stage2 patches, your last two attachments?

Can I do "make BUILD_MODULES=psm" from an object tree, or do I have to be
in the source tree?

Does the above command build NSS and well as PSM?
Also, what happened to "--enable-psm"?
Colin, I'm not sure how much of the meeting you caught as it appears that we
lost you somewhere in the middle.  Due to resource issues and the mutual desire
to avoid a prolonged deadlock, it was decided to land the changes into the tree
as soon as possible but to keep psm/nss as an external module (ie, outside of
the normal mozilla build process).  But this is kinda hard to do since a portion
of psm, libnlslayer.a, depends upon Mozilla's libintl component & uses xpcom to
talk to it.  And nss requires dbm.  So, with those additional requirements, it
made sense to just add psm to the standalone module setup that we are already
using.  That's what the stage1 patch does.  The inappropriately named stage2
patch just applies the last bit of javi's patch that was not already in the
tree. --enable-psm is no longer needed as psm is not part of the Mozilla build
(as a configure option would imply).  Building in a separate objtree works. 
However, psm & nss are still putting objs inside the srcdir so whatever hack
javi was previously using to get around this needs to be reapplied.

My default build (in objdir, of course) finished and mozilla appears to work
fine.  It just sits and spins when I try to go to https://www.verisign.com/
though. After doing a |make BUILD_MODULES=psm|, I discovered that I forgot to
add psm-glue to the psm module.  So, after adding it and building the psm module
again, ssl sites work.


I had to duck out of the meeting at 3pm (my time), but it looked like that
was the direction you were heading. So once your last two attachments are 
checked in, I will need to:

pull the code as usual (will pull PSM/NSS too)
./configure # as I used to
gmake       # build Mozilla as I used to
gmake BUILD_MODULES=psm  # build NSS/PSM

That's it?
Yes.

Will there be PSM in the nightlies with this, or do I have to download the whole
source tree?
cls,

The patches dont seem to be going in smootly on solaris, but I have applied what
I could and im continuing on with your instructions...Ill let you know how it
goes...im compiling now. 
The patches have been checked in.  Unfortunately, after the fact, we discovered
that people building in the srcdir will end up with psm Makefile conflicts at
least the first time that they pull.  Parallel builds do not work when building
psm and neither do cross-compile builds.  Leaving the bug open until those
issues have been resolved (at least the conflict problem).

r=leaf on the removal of default pull of psm/nss
Nice catch. r=cls

cls, bryner: any objections to marking this bug FIXED?
I would vehemently object.  The key requirement of being properly integrated has
not been accomplished.

Last I checked. there are 4 Makefile - Makefile.in overlaps under security/psm
which will cause cvs conflicts for unix users if those Makefiles are ever
modified.  This occurred during the landing right before x-mas.  The only way to
fix those conflicts on the tinderboxes is to manually remove them once they
occur or to remove the Makefile(s) now. Until this problem is fixed, it is not
in our interests to even pull psm by default, much less build it.

Wrt building, there is still the key issues of:
* building in parallel (-j)
* multiple objtree builds
* cross-compiling




I only ask because QA asks....
*** Bug 66532 has been marked as a duplicate of this bug. ***
Depends on: 63832
Depends on: 52990
cls: whose job is it to fix these makefile overlaps?

Gerv
It was my understanding that the module owner is ultimately responsible for
everything that occurs in their module.  Javi removed the static Makefiles once
and the win32 builds broke so we're back to square one.  
Since we've switched to PSM2 and we have it and nss building (with various
issues) in the tree using --enable-crypto, I think this bug can be marked fixed.

i changed --enable--modules=psm2 to --enable-crypto in ~/.mozmyconfig.sh, and
this works fine. tested with a clean tree and subsequent depend builds.
PSM2 now ships with mozilla nightlies.  So I'm marking this one as fixed.
Marking fixed.
Status: NEW → RESOLVED
Closed: 23 years ago
Resolution: --- → FIXED
Verified.
Status: RESOLVED → VERIFIED
--enable-crypto is great. But what about the mentioned missing build features?
Is the spearation of objdir / srcdir strict? Does it waste diskspace? Does make
-j x work?
Ben Bucksch wrote:
> Is the spearation of objdir / srcdir strict? Does it waste diskspace?

This will be fixed when the patches for bug 83225 and bug 82324 are
checked in.  The new solution does not waste diskspace.

> Does make -j x work?

NSS does not support make -j x.  However, you can issue make -j x from
the top level of mozilla without problem.  The PSM makefile overrides
-j x by -j 1 before descending into NSS.
Product: PSM → Core
Version: psm1.4 → 1.0 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: