858231 - Update Mozilla 23 to use NSS 3.15 (uses a new directory layout)

Status: RESOLVED FIXED

(Core :: Security: PSM, defect)

Description

[Kai Engert [:KaiE:]]

Attachment: patch to adjust dbm build rules

The NSS team is working on NSS version 3.15, which will use a new layout for its directories. The client.py script has already been updated to pull from the new location in bug 853775.

As a first step for this task, mozilla-central should pull the NSS_3_15_BETA1 tag.

I already did some local experimentation, and I can propose steps to perform the landing. Also, it seems (luckily) the beta1 tag makes the currently local patches in mozilla-central obsolete.

I propose:
- rm -rf dbm security/dbm security/coreconf
- rm security/patches/*.patch
- hg addremove

HG discovers that some of the deletions are moves.

But even for the files where hg doesn't detect the move, I don't think it matters. History for NSS is contained in the upstream NSS repository anyway.

On top of above, I used a small patch to security/build that allowed my local Linux build to pass the NSS directory. I'll attach that patch. I don't know if it's sufficient. You might want to test on the other platforms prior to landing this patch.

Comment 1

[Mike Hommey [:glandium]]

Comment on attachment 733530 [details] [diff] [review]
patch to adjust dbm build rules

Review of attachment 733530 [details] [diff] [review]:
-----------------------------------------------------------------

::: security/build/Makefile.in
@@ +298,5 @@
>  endif
>  
>  NSS_DIRS =
>  ifndef NSS_DISABLE_DBM
> +NSS_DIRS += nss/lib/dbm

I think you can remove this, although you probably need to do something for DBM not to be built with NSS_DISABLE_DBM is set (which it is on android)

Comment 2

[Wan-Teh Chang]

Comment on attachment 733530 [details] [diff] [review]
patch to adjust dbm build rules

Review of attachment 733530 [details] [diff] [review]:
-----------------------------------------------------------------

::: security/build/Makefile.in
@@ +298,5 @@
>  endif
>  
>  NSS_DIRS =
>  ifndef NSS_DISABLE_DBM
> +NSS_DIRS += nss/lib/dbm

Yes, we should simply remove this (lines 301-303).

NSS_DISABLE_DBM is now handled by nss/lib/{Makefile,manifest.mn}.

@@ -440,5 @@
>  libs-nss/lib/softoken: $(NSPR_IMPORT_LIBS) $(SQLITE_IMPORT_LIB)
>  libs-nss/lib/softoken: libs-nss/lib/freebl
> -ifndef NSS_DISABLE_DBM
> -libs-nss/lib/softoken: libs-dbm
> -endif

Did you delete this because libs-dbm isn't defined?

http://mxr.mozilla.org/mozilla-central/search?string=libs-dbm

Comment 3

[Mike Hommey [:glandium]]

Attachment: Adapt security/build rules to new NSS layout

This should work.

Comment 4

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/0857f2bc8f8a
https://hg.mozilla.org/integration/mozilla-inbound/rev/c4bac10ee49e

In addition to following Kai's instructions and merging glandium's changes, I updated configure.in to require NSS 3.15 and I updated security/patches/README to remove the references to the no-longer-needed private patches that we previously had in mozilla-central.

Thanks for the help with this!

Comment 5

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/0857f2bc8f8a
https://hg.mozilla.org/mozilla-central/rev/c4bac10ee49e

Comment 6

[Kai Engert [:KaiE:]]

Reopening, because that was only a beta, but this bug is track the full landing including the final release of NSS 3.15 (pending).

Comment 7

[Wan-Teh Chang]

Comment on attachment 736404 [details] [diff] [review]
Adapt security/build rules to new NSS layout

Review of attachment 736404 [details] [diff] [review]:
-----------------------------------------------------------------

::: security/build/Makefile.in
@@ +302,5 @@
>  NSS_DIRS += nss/lib
> +else
> +ifndef NSS_DISABLE_DBM
> +NSS_DIRS += nss/lib/dbm
> +endif

I think it is better to add nss/lib/dbm here:

336 # Remaining nss/lib directories
337 NSS_DIRS += nss/lib/freebl nss/lib/softoken nss/lib/jar nss/lib/crmf nss/lib/ckfw nss/lib/libpkix

Comment 8

[Ginn Chen]

In security/nss/lib/freebl/Makefile, HAVE_ABI32_INT32, USE_ABI32_INT32 sections under SPARC 32bit are removed.

Is it intentional?

But in security/build/Makefile.in, it still have
HAVE_FREEBL_LIBS_32 = 1

It caused build failure.

Comment 9

[Wan-Teh Chang]

Ginn: yes, we dropped Solaris SPARC v8 support in NSS 3.15 (bug 841664).

Could you create a patch for security/build/Makefile.in? You may just
need to delete the line
  HAVE_FREEBL_LIBS_32 = 1

I have another question for you: which versions of the Solaris Studio
compiler do you use? There are a lot of compiler warnings when NSS is
compiled with the latest version of the Solaris Studio compiler. It
will be easier to fix those warnings if I only need to support recent
versions of Solaris Studio.

Comment 10

[Wan-Teh Chang]

Ginn: I looked into the security/build/Makefile.in problem. I think
HAVE_FREEBL_LIBS_32 needs to be refined into two variables:

  HAVE_FREEBL_LIBS_32INT32
  HAVE_FREEBL_LIBS_32FPU

and

  ifdef HAVE_FREEBL_LIBS_32INT32
  NSS_EXTRA_DLLS += freebl_32int_3
  endif
  ifdef HAVE_FREEBL_LIBS_32FPU
  NSS_EXTRA_DLLS += freebl_32fpu_3
  endif

Comment 11

[Wan-Teh Chang]

Attachment: Refine HAVE_FREEBL_LIBS_32 into HAVE_FREEBL_LIBS_32INT32 and HAVE_FREEBL_LIBS_32FPU

Ginn: please review and test this patch. If it works, please check it in
for me. Thanks.

Description: Refine HAVE_FREEBL_LIBS_32 into HAVE_FREEBL_LIBS_32INT32 and
HAVE_FREEBL_LIBS_32FPU, and set only HAVE_FREEBL_LIBS_32FPU to 1 for
32-bit Solaris SPARC builds.

Comment 12

[Ginn Chen]

(In reply to Wan-Teh Chang from comment #9)

> I have another question for you: which versions of the Solaris Studio
> compiler do you use? There are a lot of compiler warnings when NSS is
> compiled with the latest version of the Solaris Studio compiler. It
> will be easier to fix those warnings if I only need to support recent
> versions of Solaris Studio.

I'm using Solaris Studio 12.2 and 12.3 for Firefox.
I guess another team is compiling NSS with Studio 12.1.

IMO, you only need to consider 12.3.
If it breaks 12.1 or 12.2, they should apply workarounds on their local workspace.

Comment 13

[Ginn Chen]

Pushed:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a1a2af3cebff

Comment 14

[Kai Engert [:KaiE:]]

I've tagged NSS_3_15_BETA2.
Could you please pick it up in mozilla-central?

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/a1a2af3cebff

Comment 16

[Wan-Teh Chang]

Attachment: Upgrade to NSS_3_15_BETA2

Kai, this is the patch to push NSS_3_15_BETA2 to m-c.

I noticed that Brian did not add the .cvsignore files when he pushed
NSS_3_15_BETA1, so I didn't add those files either.

https://hg.mozilla.org/integration/mozilla-inbound/rev/08e868f22c98

Comment 17

[Wan-Teh Chang]

Comment on attachment 743333 [details] [diff] [review]
Upgrade to NSS_3_15_BETA2

Kai, I saw these in the diffs:

>diff --git a/security/nss/cmd/pk11gcmtest/Makefile b/security/nss/cmd/pk11gcmtest/Makefile
>old mode 100644
>new mode 100755

>diff --git a/security/nss/tests/libpkix/certs/make-ca-u50-u51 b/security/nss/tests/libpkix/certs/make-ca-u50-u51
>old mode 100644
>new mode 100755

Do you know why? I wonder if they are caused by files being
pushed on Windows vs. on Linux.

Comment 18

[Kai Engert [:KaiE:]]

Comment on attachment 743333 [details] [diff] [review]
Upgrade to NSS_3_15_BETA2

I didn't review this long patch, I think it's not necessary. I assume you simply used the "python client.py update_nss" script.

Comment 19

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Wan-Teh Chang from comment #17)
> Comment on attachment 743333 [details] [diff] [review]
> Upgrade to NSS_3_15_BETA2
> 
> Kai, I saw these in the diffs:
> 
> >diff --git a/security/nss/cmd/pk11gcmtest/Makefile b/security/nss/cmd/pk11gcmtest/Makefile
> >old mode 100644
> >new mode 100755
> 
> >diff --git a/security/nss/tests/libpkix/certs/make-ca-u50-u51 b/security/nss/tests/libpkix/certs/make-ca-u50-u51
> >old mode 100644
> >new mode 100755
> 
> Do you know why? I wonder if they are caused by files being
> pushed on Windows vs. on Linux.

I did a little searching and I found that this does indeed seem to be caused by me doing the update on Windows. khuey (who just updated NSPR in mozilla-beta) and I both use Windows as our primary dev. platform. Until we can find a real solution, I will do the updates of NSS and NSPR in my Linux VM.

Probably we should just change all the test scripts so that the executable bit does not need to be set (i.e. explicitly invoke the interpreter when running any script). Anybody disagree? If not, I will file the bug in NSS.

Comment 20

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/08e868f22c98

Comment 21

[Wan-Teh Chang]

NSS_3_15_BETA3 (tagged this morning) pushed to mozilla-inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0314d200873a

Comment 22

[Wan-Teh Chang]

(In reply to Brian Smith (:bsmith) from comment #19)
>
> Probably we should just change all the test scripts so that the executable
> bit does not need to be set (i.e. explicitly invoke the interpreter when
> running any script). Anybody disagree? If not, I will file the bug in NSS.

This seems like an over-reaction. Do you know why nss/tests/all.sh, which is
marked as executable, doesn't have the permission mode problem when you pushed
it to mozilla-central on Windows?

Comment 23

[Ryan VanderMeulen [:RyanVM]]

(In reply to Wan-Teh Chang from comment #21)
> NSS_3_15_BETA3 (tagged this morning) pushed to mozilla-inbound:
> https://hg.mozilla.org/integration/mozilla-inbound/rev/0314d200873a

Backed out for Windows build bustage.
https://hg.mozilla.org/integration/mozilla-inbound/rev/05275490b9c1

https://tbpl.mozilla.org/php/getParsedLog.php?id=22468935&tree=Mozilla-Inbound

Comment 24

[Wan-Teh Chang]

NSS_3_15_BETA3 with two local patches (bug-835919.patch and
bug-832272.patch) pushed to mozilla-inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c6f5c1bbcf76

Comment 25

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/c6f5c1bbcf76

Comment 26

[Wan-Teh Chang]

NSS_3_15_BETA4 pushed to mozilla-inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/06e3d7e19a09

Comment 27

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

https://hg.mozilla.org/mozilla-central/rev/06e3d7e19a09

Comment 28

[Wan-Teh Chang]

NSS_3_15_BETA5 pushed to mozilla-inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/cc8bbdd0533c

Comment 29

[Wan-Teh Chang]

I had to revert to NSS_3_15_BETA4 in mozilla-inbound
because NSS_3_15_BETA5 broke three Mozilla build bots:
https://hg.mozilla.org/integration/mozilla-inbound/rev/cd4d5caefa88

I filed bug 874261 for the Mozilla build system problem.

Comment 30

[Wan-Teh Chang]

Second attempt at pushing NSS_3_15_BETA5 to mozilla-inbound, without
Kai's SECITEM_ReallocItemV2 patch this time to work around bug 874261:
https://hg.mozilla.org/integration/mozilla-inbound/rev/011a86ed18fe

Comment 31

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/011a86ed18fe

Comment 32

[Wan-Teh Chang]

NSS_3_15_BETA6 pushed to mozilla-inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/84985c77b141

To work around bug 874261, I applied a private patch to avoid using
the new SECITEM_ReallocItemV2 function.

Comment 33

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/84985c77b141

Comment 34

[Kai Engert [:KaiE:]]

undoing:
  status-firefox24: --- → fixed
because the main change hasn't landed yet

Comment 35

[Kai Engert [:KaiE:]]

Final versions of NSS 3.15 and NSPR 4.10 have been released.

We must upgrade mozilla-central (24) and mozilla-aurora (23) to pick them up.

Comment 36

[Kai Engert [:KaiE:]]

Attachment: final for mozilla-central

mozilla-central currently uses
NSPR (portable runtime) version: NSPR_4_10_BETA2
NSS (security library) version: NSS_3_15_BETA6

This patch was produced using:

python client.py update_nspr NSPR_4_10_RTM
python client.py update_nss NSS_3_15_RTM

Comment 37

[Kai Engert [:KaiE:]]

Attachment: final for mozilla-aurora 23

Comment 38

[Kai Engert [:KaiE:]]

Comment on attachment 758585 [details] [diff] [review]
final for mozilla-aurora 23

approval required because of the rule
  "must ship final versions of nspr/nss in releases"

Comment 39

[Wan-Teh Chang]

Comment on attachment 758583 [details] [diff] [review]
final for mozilla-central

Review of attachment 758583 [details] [diff] [review]:
-----------------------------------------------------------------

r=wtc.

IMPORTANT: please remove the local patch security/patches/revert-bug-808217.patch
and remove the description of that patch in security/patches/README.

Watch out for linker errors (unresolved symbol SECITEM_ReallocItemV2) on the
Android and B2G build bots.

Comment 40

[Wan-Teh Chang]

Comment on attachment 758585 [details] [diff] [review]
final for mozilla-aurora 23

Review of attachment 758585 [details] [diff] [review]:
-----------------------------------------------------------------

r=wtc.

IMPORTANT: change nsprpub/config/prdepend.h.

Comment 41

[Kai Engert [:KaiE:]]

(In reply to Wan-Teh Chang from comment #39)
> IMPORTANT: please remove the local patch
> security/patches/revert-bug-808217.patch
> and remove the description of that patch in security/patches/README.

https://hg.mozilla.org/integration/mozilla-inbound/rev/6486354e07e4

I read your comment, but then forgot to do it at the same time of the landing.
And immediately after I checked in, mozilla-inbound was closed because of other bustage :)

This cleanup is for tracking purposes, only, and doesn't affect the build.
I'll do it after the tree opens.

Comment 42

[Kai Engert [:KaiE:]]

(In reply to Wan-Teh Chang from comment #40)
> 
> IMPORTANT: change nsprpub/config/prdepend.h.

I wonder why the client.py script didn't make that change on the aurora branch...

Comment 43

[Kai Engert [:KaiE:]]

> Watch out for linker errors (unresolved symbol SECITEM_ReallocItemV2) on the
> Android and B2G build bots.

We didn't get such errors after the landing. We are fine.

> IMPORTANT: please remove the local patch
> security/patches/revert-bug-808217.patch
> and remove the description of that patch in security/patches/README.

done:
https://hg.mozilla.org/integration/mozilla-inbound/rev/796a79fc347a


The work for mozilla-central is done.

It's OK to set this to fixed, after inbound has been merged into central.

Comment 44

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/34ad18786196
https://hg.mozilla.org/mozilla-central/rev/796a79fc347a

Comment 45

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/5cd24379d192

I forgot to set the author before pushing :(. Please make sure your attached patches have it in the future.

Comment 46

[Oleg Romashin (:romaxa)]

http://hg.mozilla.org/mozilla-central/rev/c4bac10ee49e#l1.12
Native NSS version check does not work 
AM_PATH_NSS - need version specified in format \D+.\D+.\D+
otherwise it would not detect and think 3.14.3 - as 3.15 - compatible