Closed Bug 1629553 Opened 4 years ago Closed 4 years ago

gmake build rework and cleanup

Categories

(NSS :: Build, enhancement, P1)

enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: glogow, Assigned: glogow)

References

Details

Attachments

(9 files, 1 obsolete file)

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0

Steps to reproduce:

Try working on the NSS gmake build system to fix the parallel build issue with NSS experienced in the LibreOffice CI, forcefully building NSS with -j1.

Actual results:

The system is split into too many separate files per directory. It is hard to follow the actual workflow of the finally parsed / merged Makefile and parts of it are definitely not working as intended, where later includes override some previous settings.
There are multiple parts, where different defaults can reduce the size of the Makefiles in general. And it seems there are quite a bit of unused rules.

Then there is the whole gigantic freebl Makefile, where I thing all the separate libraries should probably be build in different directories, or some better way to build multiple libraries from one Makefile.

Expected results:

Makefiles are smaller due to better defaults. In the end they probably just include config and rules from coreconf and are otherwise self contained. Some templating is use to reduce boilerplate Makefile code. Unused rules and variables are dropped. It's actually possible to read the freebl Makefiles without total confusion ;-)

Maybe even someone will implement and merge a non-recursive build system, or the new gyp+ninja one becomes more ubiquitous and usable on more platforms, so this gmake one can be finally dropped.

Depends on: 290526

In the manifest.mn the LIBRARY_VERSION is normally used to define
the major version of the build shared library. This ust works for
the pk11wrap case, because pk11wrap is a static library. But it's
still very confusing when reading the manifest.mn. Also the
referenced define in the code is just named SHLIB_VERSION.

So this prefixes the defines and the variables with NSS_, because
it tries to load the NSS library, just as the SOFTOKEN_.*_VERSION
is used to load the versioned softokn library.

Gets rid of a lot of boilerplate Makefile code.

  • Drop the WIN% "32" default DLL suffix
  • Add default export script handling => drop default MAPFILE
  • Add default resource file handling => drop default RES
  • Generate IMPORT_LIBRARY based on IMPORT_LIB_SUFFIX and
    SHARED_LIBRARY, so we can drop all the explicit empty
    IMPORT_LIBRARY lines

Depends on D70369

There is really no good reason to explicitly change the TARGET
variable. And the empty SHARED_LIBRARY variable should also be
in the manifest.mn to begin with.

All the other empty variables start empty or undefined, so there
is also no need to explicitly set them empty.

Depends on D70690

Attachment #9140143 - Attachment is obsolete: true

Depends on D70689

Attachment #9140141 - Attachment description: Bug 1629553 Use eval templates for export and freebl → Bug 1629553 Use an eval template for freebl libs

These pattern rules already had a comment to keep both in sync, so
just use an eval template to enforce this.

Depends on D69022

There aren't an Java sources in NSS, so just drop all the stuff
referencing java, jars, jni, etc.

I didn't try to remove it from tests.

Depends on D70691

Depends on D70986

Introduces a simple "%/d" rule to create directories using
$(MAKE_OBJDIR) and replace all explicit $(MAKE_OBJDIR) calls
with an order-only-prerequisites.

To expand the $(@D) prerequisite, this needs .SECONDEXPANSION.

Depends on D70988

Resetting severity to default of --.

I'll take this series in NSS 3.53, so we'll land them later this week / early next week after we branch.

Assignee: nobody → glogow
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Priority: -- → P1
Target Milestone: --- → 3.53

Try push failed for Windows make builds: https://treeherder.mozilla.org/#/jobs?repo=nss-try&revision=59e1b1970611b2ed08a3c7d2b0e270aa43584038

WIN954.0_DBG.OBJ/nssutil.lib
nsinstall -m 664 WIN954.0_DBG.OBJ/nssutil.lib ../../../dist/WIN954.0_DBG.OBJ/lib
rc.exe -DDEBUG -DWIN32 -D_X86_ -D_WINDOWS -DWIN95 -DNSS_DISABLE_AVX2 -DNSS_NO_INIT_SUPPORT -DUSE_UTIL_DIRECTLY -DNO_NSPR_10_SUPPORT -DSSL_DISABLE_DEPRECATED_CIPHER_SUITE_NAMES -I../../../dist/WIN954.0_DBG.OBJ/include -I../../../dist/public/nss -I../../../dist/private/nss -FoWIN954.0_DBG.OBJ/nssutil.res nssutil.rc
Microsoft (R) Windows (R) Resource Compiler Version 10.0.10011.16384

Copyright (C) Microsoft Corporation.  All rights reserved.


WIN954.0_DBG.OBJ/nssutil.res finished
cp nssutil.def WIN954.0_DBG.OBJ/nssutil.def
rm -f WIN954.0_DBG.OBJ/nssutil3.dll
link -nologo -DLL -SUBSYSTEM:WINDOWS -DYNAMICBASE -DEBUG -OUT:WIN954.0_DBG.OBJ/nssutil3.dll  -MAP  WIN954.0_DBG.OBJ\\quickder.obj WIN954.0_DBG.OBJ\\secdig.obj WIN954.0_DBG.OBJ\\derdec.obj WIN954.0_DBG.OBJ\\derenc.obj WIN954.0_DBG.OBJ\\dersubr.obj WIN954.0_DBG.OBJ\\dertime.obj WIN954.0_DBG.OBJ\\errstrs.obj WIN954.0_DBG.OBJ\\nssb64d.obj WIN954.0_DBG.OBJ\\nssb64e.obj WIN954.0_DBG.OBJ\\nssrwlk.obj WIN954.0_DBG.OBJ\\nssilock.obj WIN954.0_DBG.OBJ\\oidstring.obj WIN954.0_DBG.OBJ\\pkcs1sig.obj WIN954.0_DBG.OBJ\\portreg.obj WIN954.0_DBG.OBJ\\secalgid.obj WIN954.0_DBG.OBJ\\secasn1d.obj WIN954.0_DBG.OBJ\\secasn1e.obj WIN954.0_DBG.OBJ\\secasn1u.obj WIN954.0_DBG.OBJ\\secitem.obj WIN954.0_DBG.OBJ\\secload.obj WIN954.0_DBG.OBJ\\secoid.obj WIN954.0_DBG.OBJ\\sectime.obj WIN954.0_DBG.OBJ\\secport.obj WIN954.0_DBG.OBJ\\templates.obj WIN954.0_DBG.OBJ\\utf8.obj WIN954.0_DBG.OBJ\\utilmod.obj WIN954.0_DBG.OBJ\\utilpars.obj WIN954.0_DBG.OBJ\\pkcs11uri.obj   ..\\..\\..\\dist\\WIN954.0_DBG.OBJ\\lib\\plc4.lib ..\\..\\..\\dist\\WIN954.0_DBG.OBJ\\lib\\plds4.lib ..\\..\\..\\dist\\WIN954.0_DBG.OBJ\\lib\\nspr4.lib    WIN954.0_DBG.OBJ\\nssutil.res
if test -f WIN954.0_DBG.OBJ/nssutil3.dll.manifest; then \
		mt.exe -NOLOGO -MANIFEST WIN954.0_DBG.OBJ/nssutil3.dll.manifest -OUTPUTRESOURCE:WIN954.0_DBG.OBJ/nssutil3.dll\;2; \
		rm -f WIN954.0_DBG.OBJ/nssutil3.dll.manifest; \
	fi
nsinstall -m 775 WIN954.0_DBG.OBJ/nssutil3.dll ../../../dist/WIN954.0_DBG.OBJ/lib
nsinstall -m 775 WIN954.0_DBG.OBJ/nssutil3.lib ../../../dist/WIN954.0_DBG.OBJ/lib
nsinstall: WIN954.0_DBG.OBJ\nssutil3.lib: No such file or directory
make[4]: *** [WIN954.0_DBG.OBJ/nssutil3.lib] Error 3

Maybe the script is assuming it can make a symlink from nssutil.lib to nssutil3.lib but it actually has to copy on Windows?

Flags: needinfo?(glogow)

No. I made a mistake when handling the patches between my LibreOffice and the mozilla-build NSS setup. The linker command is missing the export script (-DEF:$(MAPFILE)). So I can't generate a default MAPFILE in the ruleset.mk, because that variable is already used earlier in arch.mk to modify the linker command. And I don't see an easy way to fix that, so I reverted all the MAPFILE cleanup changes and just left them in manifest.mn.

Then I saw the broken incremental build... and had to move more stuff ... and gave up on using the LibreOffice automatic CI to test my changes in a much more parallel environment then my laptop, as I always have to patch the old NSS release. I'm missing something like git rerere when transfering the patches from hg to git...

Flags: needinfo?(glogow)
You need to log in before you can comment on or make changes to this bug.