PSM must build on Linux for Mozilla users

VERIFIED FIXED

Status

()

Core
Security: PSM
P3
normal
VERIFIED FIXED
17 years ago
10 years ago

People

(Reporter: David P. Drinan, Assigned: Javier Delgadillo)

Tracking

1.0 Branch
x86
Linux
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(13 attachments)

158.30 KB, patch
Details | Diff | Splinter Review
158.35 KB, patch
Details | Diff | Splinter Review
160.42 KB, patch
Details | Diff | Splinter Review
1.81 KB, patch
Details | Diff | Splinter Review
11.91 KB, text/plain
Details
12.17 KB, patch
Details | Diff | Splinter Review
12.17 KB, patch
Details | Diff | Splinter Review
5.34 KB, patch
Details | Diff | Splinter Review
2.62 KB, patch
Details | Diff | Splinter Review
6.97 KB, patch
Details | Diff | Splinter Review
4.16 KB, patch
Details | Diff | Splinter Review
502 bytes, patch
Details | Diff | Splinter Review
439 bytes, patch
Details | Diff | Splinter Review
(Reporter)

Description

17 years ago
PSM must build as part of the regular mozilla build for Linux.
(Assignee)

Comment 1

17 years ago
Created attachment 20069 [details] [diff] [review]
This patch builds PSM as part of Mozilla Linux build if configure is run with the --with-psm flag.

Comment 2

17 years ago
That should be --enable-psm instead of --with-psm. Although enable-* and with-*
work identically, there is a convention on which one to use in which situation
(unfortunately not always followed on other occasions too, see bug 61907)
(Assignee)

Comment 3

17 years ago
Created attachment 20162 [details] [diff] [review]
This patch uses --enable-psm to enable PSM
(Assignee)

Comment 4

17 years ago
Turns out the current patch does not work if you do the different objdir trick.

Fixing this will require a patch to NSS.  I'll work on getting that fix in tomorrow.
(Assignee)

Comment 5

17 years ago
Before I post the next patch, I'd like to know what are the requirements for the
separate objdir feature?  Is it that no mozilla/dist appear?  I've got it so
that NSS/PSM build the way they used to (by putting everything in mozilla/dist),
and then installing the necessary pieces in mozilla/objdir/dist, but I'm
wondering if I need to make NSS/PSM put all of it's build pieces in
mozilla/objdir/dist.

Comment 6

17 years ago
> Before I post the next patch, I'd like to know what are the requirements for the
> separate objdir feature?

I can see several:
- Building several platforms from the same tree
- Not making massive writes to the source tree in order to save the filesystem
(same reason why /home and /tmp are usually on different filesystems - /home has
few writes, but very important data, while /tmp has the opposite charateristics;
same is true for source modifications vs. object files).

Comment 7

17 years ago
The requirements are that all generated files are built in the separate objdir.
 In essence, the user could build from a CD (or read-only partition) if need be.
(Assignee)

Updated

17 years ago
Depends on: 62267

Comment 8

17 years ago
CC'ing myself on this bug as it will be intresting to see how this pans out in
Solaris.
(Assignee)

Comment 9

17 years ago
Created attachment 20502 [details] [diff] [review]
This patch works with a separate objdir and without creating new files in the source tree.

Comment 10

17 years ago
Javi,
I am having a problem getting this to patch correctly on Solaris 8, x86, I
followed cls's instructions for doing the client.mk patch first and even that
wont patch correctly. any ideas?

Comment 11

17 years ago
In the future, you will want to break up a patch such as this.  You need to
apply the client.mk chunk first,  pull again to get the new security directories
then apply the rest of the patch.  You also want to remove the configure portion
of the patch as well as the slightest change to that file usually creates a huge
diff that usually doesn't apply cleanly to another tree.  

That said, once the patch was applied, a simple build worked.  However, parallel
builds do not.  They failed creating one of the Linux2.2_x86_glibc_PTH_OPT.OBJ
subdirs.  The usual if-test-mkdir race condition.

I do not like the prospect of making a separate copy of the various security/
subdirs for each separate object tree that we use.  This creates an additional
penalty for building in the objtree.  I'm sure that not everyone has the 6
objtrees that I use and it's only an additional 20 megs per tree but I feel that
it creates a bad precedent.
Is there a way to make the NSS build directly use the include and lib
directories from the main tree, rather than doing a moz_import, which takes up
unnecessary space?
I am trying to get this working on the OS/2 build, which uses the UNIX
makefiles.  However, I had problems with the configure.in:  for some reason, it
would not define MOZ_ENABLE_PSM when I passed in --enable-psm.  I finally got
this working by moving the enable-psm code further down in the file.  I am
attaching that patch below.  Does this work on the other platforms?
Created attachment 20528 [details] [diff] [review]
OS/2 version of configure.in
(Assignee)

Comment 15

17 years ago
That step mirrors the instructions at
http://www.mozilla.org/projects/security/pki/nss/buildnss_31.html#unix

cls, I tried to make it so that it wasn't necessary to copy over the source for
separate objdir builds, but that diff was huge and was quite nasty.  Don't think
reviewers would like it because it was huge and not autoconf'ing (just making
coreconf do just enough), and the NSS team didn't like mucking too much with
their build system.

NSS also poses the problem that souce files are copied within the tree as part
of the build process, and if I understood correctly, I should assume a separate
objdir build could be on a read-only partition.  That kind of forced my hand
into copying the NSS source over.
(Assignee)

Comment 16

17 years ago
Adding members of the NSS team to cc list, since this bug depends on integrating
NSS into Mozilla builds.
(Assignee)

Comment 17

17 years ago
Created attachment 20585 [details]
Updated patch including OS/2 patch with some instructions.
(Assignee)

Comment 18

17 years ago
When looking through the tree, I noticed NSPR does create files in the source
tree (.o and .a files) so I went to not copying source files around, but only
over-riding the dist so that PSM/NSS files wind up in the new dist for separate
ojdir directories.

This will follow the same practice as NSPR, so there will be no new precedent
set.  And there won't be a big penalty for people with lots of objdir trees.

Comment 19

17 years ago
Then you will be landing an autoconf build system as nspr has as well?  The plan
(at least from this side) has always been to move to making the nspr autoconf
system the default while building Mozilla.  Right now, that's blocked by one
major bug, bug 58804, which is more logistical than technical in nature.  Also,
the classic nspr build does not allow users to have more than one objdir of
either an optimized or debug build because it uses the same names for the .OBJ
dirs in the srcdir.  See previous note about bad precedent.  

Once these modules are integrated with the Mozilla build, they will have to
conform to the same build standards as the rest of the modules.  This goes
beyond the vanilla build configuration that the tinderboxes on the main page
run.  Right now, there are certain "advanced" build features that we support for
the unix build.  These include but are not limited to: parallel builds, multiple
objdir builds & cross-compiling.  Before a major change like adding a new module
to the build, we should make sure that all of this feature set is present for
the new module or at least, a significant portion is present with plans to
implement the rest.  I'm not seeing that here. 

Comment 20

17 years ago
Let's move the NSS/PSM build system discussion
to the mozilla.crypto newsgroup.  As far as this
bug is concerned, Javier's patch meets the key
requirement: PSM can be built as part of Mozilla.

Comment 21

17 years ago
I disagree.  The end goal is that PSM builds with the Mozilla build.  The key
requirement is that it is integrated properly with the Mozilla build system.
This patch does not fulfill the key requirement.

Updated

17 years ago
Blocks: 62855

Updated

17 years ago
Blocks: 62856

Comment 22

17 years ago
I reviewed javi's latest patch (attachment id=20585).
I have one suggested change: in mozilla/client.mk,
move mozilla/security/Makefile.in from NSS_CO_MODULE
to PSM_CO_MODULE.  I also noticed a #if 0 in
mozilla/security/psm/server/serv.h that looks like a
temporary quick fix.  Other than these two, this
patch is fine.  r=wtc@netscape.com.

Note: this patch contains the changes to not build
NSS command-line utilities on Unix and Windows.
(Assignee)

Comment 23

17 years ago
Created attachment 20910 [details] [diff] [review]
Patch incorporating wtc's suggestions.
(Assignee)

Comment 24

17 years ago
Created attachment 20918 [details] [diff] [review]
So I didn't save the change to client.mk incorporating wtc's suggestion.  Here is the patch for real.
(Assignee)

Comment 25

17 years ago
cls, do you sign off on this patch?

Comment 26

17 years ago
No.
(Assignee)

Comment 27

17 years ago
Adding message sent by wtc to cls and brendan.

Brendan, Chris,

We are planning to check in Javi's latest patch
in Bug #60912 (http://bugzilla.mozilla.org/show_bug.cgi?id=60912)
for integrating PSM with the Mozilla build on Unix
this Wednesday morning.  We realize that some developers
don't consider this patch the ideal solution, but it
works and will allow us to make progress.  Discussion
on build systems tends to get emotional and drag on,
and I don't want that to prevent us from achieving the
important goal of integrating PSM with the Mozilla build
on all platforms.  (It's already done for Mac and Windows.)

Could you please review Javi's patch and give your
blessing?

Chris: let's meet face-to-face next time you come up
here.  I have some proposals of how to solve the NSPR
and NSS/PSM build system issues in the long term that
I'd like to run by you.  In particular I'd like to
hear more about your idea of using a Perl configure
script on Windows to fill in pre-determined values
in the *.in files so that we don't need to do the full
configure thing.

Thanks,
Wan-Teh

Comment 28

17 years ago
Hi,
this patch includes a modification of mozilla/security/nss/makefile.win, which
probably doesn't belong into this one, right?

Axel

Comment 29

17 years ago
Axel: this patch also contains a change to build
just the NSS libs, not the commands, on both Unix
and Windows. This is why mozilla/security/nss/makefile.win
was also modified.

Comment 30

17 years ago
is there any other place i can/will be able to get the command-line tools?  i would dearly like to, for example, add the Rice root ca cert to the database...am i stuck downloading the whole mozilla tree and building it just for a few dinky binaries?

Comment 31

17 years ago
Created attachment 21171 [details] [diff] [review]
Inform standalone build system of psm

Comment 32

17 years ago
Created attachment 21172 [details] [diff] [review]
last portion of javi's attach 20918 that was not already checked in

Comment 33

17 years ago
To test these changes:

cvs -z3 co mozilla/client.mk
make -f client.mk pull_all BUILD_MODULES=
env POSIXLY_CORRECT=1 patch -p0 < ~/psm-stage1.patch
make -f client.mk pull_all BUILD_MODULES=psm
env POSIXLY_CORRECT=1 patch -p0 < ~/psm-stage2.patch
./configure --enable-modules=psm
make

I'm still testing the "integration" of the standalone psm build with the default
build but it should be as simple as doing:
make BUILD_MODULES=psm
either before or after the default build.

Comment 34

17 years ago
What are stage1 and stage2 patches, your last two attachments?

Can I do "make BUILD_MODULES=psm" from an object tree, or do I have to be
in the source tree?

Does the above command build NSS and well as PSM?

Comment 35

17 years ago
Also, what happened to "--enable-psm"?

Comment 36

17 years ago
Colin, I'm not sure how much of the meeting you caught as it appears that we
lost you somewhere in the middle.  Due to resource issues and the mutual desire
to avoid a prolonged deadlock, it was decided to land the changes into the tree
as soon as possible but to keep psm/nss as an external module (ie, outside of
the normal mozilla build process).  But this is kinda hard to do since a portion
of psm, libnlslayer.a, depends upon Mozilla's libintl component & uses xpcom to
talk to it.  And nss requires dbm.  So, with those additional requirements, it
made sense to just add psm to the standalone module setup that we are already
using.  That's what the stage1 patch does.  The inappropriately named stage2
patch just applies the last bit of javi's patch that was not already in the
tree. --enable-psm is no longer needed as psm is not part of the Mozilla build
(as a configure option would imply).  Building in a separate objtree works. 
However, psm & nss are still putting objs inside the srcdir so whatever hack
javi was previously using to get around this needs to be reapplied.

My default build (in objdir, of course) finished and mozilla appears to work
fine.  It just sits and spins when I try to go to https://www.verisign.com/
though. After doing a |make BUILD_MODULES=psm|, I discovered that I forgot to
add psm-glue to the psm module.  So, after adding it and building the psm module
again, ssl sites work.

Comment 37

17 years ago
I had to duck out of the meeting at 3pm (my time), but it looked like that
was the direction you were heading. So once your last two attachments are 
checked in, I will need to:

pull the code as usual (will pull PSM/NSS too)
./configure # as I used to
gmake       # build Mozilla as I used to
gmake BUILD_MODULES=psm  # build NSS/PSM

That's it?

Comment 38

17 years ago
Yes.

Comment 39

17 years ago
Created attachment 21184 [details] [diff] [review]
Updated "stage1" patch. Includes psm-glue this time

Comment 40

17 years ago
Created attachment 21185 [details] [diff] [review]
Updated "stage2" patch.  Fixes the objs in srcdir problem at cost of size of nss src tree
Will there be PSM in the nightlies with this, or do I have to download the whole
source tree?

Comment 42

17 years ago
cls,

The patches dont seem to be going in smootly on solaris, but I have applied what
I could and im continuing on with your instructions...Ill let you know how it
goes...im compiling now. 

Comment 43

17 years ago
The patches have been checked in.  Unfortunately, after the fact, we discovered
that people building in the srcdir will end up with psm Makefile conflicts at
least the first time that they pull.  Parallel builds do not work when building
psm and neither do cross-compile builds.  Leaving the bug open until those
issues have been resolved (at least the conflict problem).

Comment 44

17 years ago
Created attachment 21226 [details] [diff] [review]
Do not pull PSM/NSS by default.

Comment 45

17 years ago
r=leaf on the removal of default pull of psm/nss

Comment 46

17 years ago
Created attachment 21308 [details] [diff] [review]
if we touch mozilla/security, the checkout of mozilla/security fails (as CVS dirs are missing) shouldn't touch unless the build module has psm included

Comment 47

17 years ago
Nice catch. r=cls
(Assignee)

Comment 48

17 years ago
cls, bryner: any objections to marking this bug FIXED?

Comment 49

17 years ago
I would vehemently object.  The key requirement of being properly integrated has
not been accomplished.

Last I checked. there are 4 Makefile - Makefile.in overlaps under security/psm
which will cause cvs conflicts for unix users if those Makefiles are ever
modified.  This occurred during the landing right before x-mas.  The only way to
fix those conflicts on the tinderboxes is to manually remove them once they
occur or to remove the Makefile(s) now. Until this problem is fixed, it is not
in our interests to even pull psm by default, much less build it.

Wrt building, there is still the key issues of:
* building in parallel (-j)
* multiple objtree builds
* cross-compiling



(Assignee)

Comment 50

17 years ago
I only ask because QA asks....

Comment 51

17 years ago
*** Bug 66532 has been marked as a duplicate of this bug. ***

Updated

17 years ago
Depends on: 63832

Updated

17 years ago
Depends on: 52990
cls: whose job is it to fix these makefile overlaps?

Gerv

Comment 53

17 years ago
It was my understanding that the module owner is ultimately responsible for
everything that occurs in their module.  Javi removed the static Makefiles once
and the win32 builds broke so we're back to square one.  

Comment 54

17 years ago
Since we've switched to PSM2 and we have it and nss building (with various
issues) in the tree using --enable-crypto, I think this bug can be marked fixed.
i changed --enable--modules=psm2 to --enable-crypto in ~/.mozmyconfig.sh, and
this works fine. tested with a clean tree and subsequent depend builds.
(Assignee)

Comment 56

17 years ago
PSM2 now ships with mozilla nightlies.  So I'm marking this one as fixed.

Comment 57

17 years ago
Marking fixed.
Status: NEW → RESOLVED
Last Resolved: 17 years ago
Resolution: --- → FIXED

Comment 58

17 years ago
Verified.
Status: RESOLVED → VERIFIED

Comment 59

17 years ago
--enable-crypto is great. But what about the mentioned missing build features?
Is the spearation of objdir / srcdir strict? Does it waste diskspace? Does make
-j x work?

Comment 60

17 years ago
Ben Bucksch wrote:
> Is the spearation of objdir / srcdir strict? Does it waste diskspace?

This will be fixed when the patches for bug 83225 and bug 82324 are
checked in.  The new solution does not waste diskspace.

> Does make -j x work?

NSS does not support make -j x.  However, you can issue make -j x from
the top level of mozilla without problem.  The PSM makefile overrides
-j x by -j 1 before descending into NSS.

Updated

13 years ago
Component: Security: PSM → Security: PSM
Product: PSM → Core

Updated

10 years ago
Version: psm1.4 → 1.0 Branch
You need to log in before you can comment on or make changes to this bug.