486141 - Build NSS with pymake, even on Windows with windows-style paths

Status: VERIFIED FIXED

(Firefox Build System :: General, defect)

Description

[Benjamin Smedberg]

For the moment we're not building NSS with pymake primarily because it doesn't understand Windows-style paths. I had a couple patches which made things better (using forward slashes instead of backslashes in some places, IIRC), but there were additional issues.

Comment 1

[Mike Hommey [:glandium]]

I wonder how things got since bug 736066.

Comment 2

[Ed Morley [:emorley]]

Using glandium's suggestion of forcing |NSSMAKE = $(MAKE)| in http://mxr.mozilla.org/mozilla-central/source/security/manager/Makefile.in#161, I get:

{
make.py[5]: Entering directory 'd:\obj-inbound\security/manager'
make.py[6]: Entering directory 'd:/srcdirs/inbound/security/dbm'
cd include; c;C:\mozilla-build\msys\mozilla-build\python\python.exe d;C:\mozilla-build\msys\srcdirs\inbound\build\pymake\pymake\..\make.py -j1 export
/usr/bin/sh: c: command not found
d:\srcdirs\inbound\security\coreconf\rules.mk:96:0: command 'for directory in include src ; do if test -d $directory; then set -e; echo "cd $directory; c;C:\mozilla-build\msys\mozilla-build\python\python.exe d;C:\mozilla-build\msys\srcdirs\inbound\build\pymake\pymake\..\make.py -j1 export"; c;C:\mozilla-build\msys\mozilla-build\python\python.exe d;C:\mozilla-build\msys\srcdirs\inbound\build\pymake\pymake\..\make.py -j1 -C $directory export; set +e; else echo "Skipping non-directory $directory..."; fi; true; done' failed, return code 127
evaluation from d:\obj-inbound\security\manager\Makefile:337:26:2:0$ c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -C d:/srcdirs/inbound/security/dbm MAKE="c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -j1" -j1 CC=" cl" SOURCE_MD_DIR=d:/obj-inbound/dist SOURCE_MDHEADERS_DIR=d:/obj-inbound/dist/include/nspr DIST=d:/obj-inbound/dist NSPR_INCLUDE_DIR=d:/obj-inbound/dist/include/nspr NSPR_LIB_DIR=d:/obj-inbound/dist/lib MOZILLA_CLIENT=1 NO_MDUPDATE=1 NSS_ENABLE_ECC=1 NSINSTALL="d:/obj-inbound/config/nsinstall.exe" SQLITE_LIB_NAME=mozsqlite3 SQLITE_INCLUDE_DIR=d:/obj-inbound/dist/include BUILD='d:/obj-inbound/security/$(subst /d/srcdirs/inbound/security/,,$(CURDIR))' BUILD_TREE='$(BUILD)' OBJDIR='$(BUILD)' DEPENDENCIES='$(BUILD)/.deps' SINGLE_SHLIB_DIR='$(BUILD)' SOURCE_XP_DIR=d:/obj-inbound/dist BUILD_OPT=1 OPT_CODE_SIZE=1 NS_USE_GCC= OS_TARGET=WIN95 NSS_ENABLE_ZLIB= PROGRAMS= CHECKLOC= FREEBL_NO_DEPEND=0
evaluation from d:\obj-inbound\security\manager\Makefile:337:26:2:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -C d:/srcdirs/inbound/security/dbm MAKE="c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -j1" -j1 CC=" cl" SOURCE_MD_DIR=d:/obj-inbound/dist SOURCE_MDHEADERS_DIR=d:/obj-inbound/dist/include/nspr DIST=d:/obj-inbound/dist NSPR_INCLUDE_DIR=d:/obj-inbound/dist/include/nspr NSPR_LIB_DIR=d:/obj-inbound/dist/lib MOZILLA_CLIENT=1 NO_MDUPDATE=1 NSS_ENABLE_ECC=1 NSINSTALL="d:/obj-inbound/config/nsinstall.exe" SQLITE_LIB_NAME=mozsqlite3 SQLITE_INCLUDE_DIR=d:/obj-inbound/dist/include BUILD='d:/obj-inbound/security/$(subst /d/srcdirs/inbound/security/,,$(CURDIR))' BUILD_TREE='$(BUILD)' OBJDIR='$(BUILD)' DEPENDENCIES='$(BUILD)/.deps' SINGLE_SHLIB_DIR='$(BUILD)' SOURCE_XP_DIR=d:/obj-inbound/dist BUILD_OPT=1 OPT_CODE_SIZE=1 NS_USE_GCC= OS_TARGET=WIN95 NSS_ENABLE_ZLIB= PROGRAMS= CHECKLOC= FREEBL_NO_DEPEND=0' failed, return code 2
d:\srcdirs\inbound\config\makefiles\target_libs.mk:54:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -C security/manager libs' failed, return code 2
d:\srcdirs\inbound\config\rules.mk:709:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py libs_tier_platform' failed, return code 2
d:\srcdirs\inbound\config\rules.mk:674:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py  tier_platform' failed, return code 2
d:\srcdirs\inbound\client.mk:381:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -j12 -C d:/obj-inbound' failed, return code 2
d:\srcdirs\inbound\client.mk:181:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -f d:/srcdirs/inbound/client.mk realbuild' failed, return code 2
}

Comment 3

[Ed Morley [:emorley]]

Attachment: Patch for testing only

Using the attached patch (next iteration of glandium's suggestions), I get:
http://pastebin.mozilla.org/1641933

Comment 4

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

The NSS build system is quite complicated (see security/coreconf, and the build scripts for security/nss/lib/freebl in particular) and I think we've pretty much agreed that we're likely to rewrite the NSS build system more-or-less completely for performance reasons (to support parallel building, in particular). I'm not sure it's worthwhile to take the risk of whatever (undetected) problems will come up in the switch to pymake. It makes more sense to me to do the build system rewrite, which sould simplify the NSS build system considerably, and *then* switch to pymake.

Comment 5

[Benjamin Smedberg]

Unless rewriting the NSS build system is a 2-week project, we need some solution which allows the mozilla-central builders to build mozilla with pymake (for performance reasons). That could either be:

* switch to gmake only for NSS (which requires a lot more configuration on the builders)
* do the minimal coreconf/NSS fixup which allows us to use pymake across the board.

I tend to think that the second option is the better one.

Comment 6

[Mike Hommey [:glandium]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #5)
> * switch to gmake only for NSS (which requires a lot more configuration on
> the builders)

Nothing is actually required on the builders. That hybrid setup already works.

Comment 7

[Mike Hommey [:glandium]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #5)
> * do the minimal coreconf/NSS fixup which allows us to use pymake across the
> board.

Fixup in security/coreconf is actually pretty small, but fixing pymake is more involved.

Comment 8

[Mike Hommey [:glandium]]

Attachment: Stop building NSS with gmake when building m-c with pymake

Comment 9

[Mike Hommey [:glandium]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/d126ed851887

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/d126ed851887

Comment 11

[Jared Wein [:jaws] (please needinfo? me)]

This change has caused my local pymake builds to fail on Windows.

This is the error that I'm getting:

Creating d:/mc/obj-dir/dist/public/dbm
cd src; c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py export
cd include; c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py libs
cd src; c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py libs
nsinstall: failed to create directory d:\mc\obj-dir\security\d:\mc\security\dbm\src: [Error 123] The filename, directory name, or volume label syntax is incorrect: u'd:\\mc\\obj-dir\\security\\d:'
d:\mc\security\coreconf\rules.mk:389:0: command 'd:/mc/obj-dir/_virtualenv/Scripts/python.exe d:/mc/obj-dir/security/build/../../../config/nsinstall.py -D d:/mc/obj-dir/security/d:/mc/security/dbm/src' failed, return code 1
d:\mc\security\coreconf\rules.mk:101:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py -C src libs' failed, return code 2
evaluation from d:\mc\obj-dir\security\build\Makefile:296:26:2:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py -C ../../../security/dbm -j1 CC=" cl" SOURCE_MD_DIR=d:/mc/obj-dir/dist SOURCE_MDHEADERS_DIR=d:/mc/obj-dir/dist/include/nspr DIST=d:/mc/obj-dir/dist NSPR_INCLUDE_DIR=d:/mc/obj-dir/dist/include/nspr NSPR_LIB_DIR=d:/mc/obj-dir/dist/lib MOZILLA_CLIENT=1 NO_MDUPDATE=1 NSS_ENABLE_ECC=1 NSINSTALL="d:/mc/obj-dir/_virtualenv/Scripts/python.exe d:/mc/obj-dir/security/build/../../../config/nsinstall.py" SQLITE_LIB_NAME=mozsqlite3 SQLITE_INCLUDE_DIR=d:/mc/obj-dir/dist/include BUILD='d:/mc/obj-dir/security/$(subst d:/mc/obj-dir/security/build/../../../security/,,$(CURDIR))' BUILD_TREE='$(BUILD)' OBJDIR='$(BUILD)' DEPENDENCIES='$(BUILD)/.deps' SINGLE_SHLIB_DIR='$(BUILD)' SOURCE_XP_DIR=d:/mc/obj-dir/dist BUILD_OPT=1 OPT_CODE_SIZE=1 NS_USE_GCC= OS_TARGET=WIN95 NSS_ENABLE_ZLIB= PROGRAMS= CHECKLOC= FREEBL_NO_DEPEND=0' failed, return code 2
d:\mc\config\makefiles\target_libs.mk:18:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py -C security/build libs' failed, return code 2
d:\mc\config\rules.mk:583:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py libs_tier_platform' failed, return code 2
d:\mc\config\rules.mk:548:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py  tier_platform' failed, return code 2

What do I need to change to fix this?

Comment 12

[Jared Wein [:jaws] (please needinfo? me)]

Comment on attachment 665430 [details] [diff] [review]
Stop building NSS with gmake when building m-c with pymake

Review of attachment 665430 [details] [diff] [review]:
-----------------------------------------------------------------

::: security/build/Makefile.in
@@ +143,5 @@
>  DEFAULT_GMAKE_FLAGS += NSS_DISABLE_DBM=1
>  endif
>  ABS_topsrcdir   := $(call core_abspath,$(topsrcdir))
>  # Hack to force NSS build system to use "normal" object directories
> +DEFAULT_GMAKE_FLAGS += BUILD='$(MOZ_BUILD_ROOT)/security/$$(subst $(ABS_topsrcdir)/security/,,$$(CURDIR))'

If I were to take a wild guess I'd think that my build failure is the result of the change from $(topsrcdir) to $(ABS_topsrcdir). Why does that change need to be made?

Comment 13

[Mike Hommey [:glandium]]

(In reply to Jared Wein [:jaws] from comment #12)
> Comment on attachment 665430 [details] [diff] [review]
> Stop building NSS with gmake when building m-c with pymake
> 
> Review of attachment 665430 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: security/build/Makefile.in
> @@ +143,5 @@
> >  DEFAULT_GMAKE_FLAGS += NSS_DISABLE_DBM=1
> >  endif
> >  ABS_topsrcdir   := $(call core_abspath,$(topsrcdir))
> >  # Hack to force NSS build system to use "normal" object directories
> > +DEFAULT_GMAKE_FLAGS += BUILD='$(MOZ_BUILD_ROOT)/security/$$(subst $(ABS_topsrcdir)/security/,,$$(CURDIR))'
> 
> If I were to take a wild guess I'd think that my build failure is the result
> of the change from $(topsrcdir) to $(ABS_topsrcdir). Why does that change
> need to be made?

Because the path needs to be absolute, which $(topsrcdir) doesn't guarantee. Sadly, ABS_topsrcdir also doesn't normalize...
A corollary question is why isn't your topsrcdir absolute (which would work around the issue)?

Comment 14

[Jared Wein [:jaws] (please needinfo? me)]

(In reply to Mike Hommey [:glandium] from comment #13)
> A corollary question is why isn't your topsrcdir absolute (which would work
> around the issue)?

Probably because I had never set my TOPSRCDIR environment variable before :/

It worked before and so I never thought there was an issue. I couldn't find any documentation that says to actually set a TOPSRCDIR environment variable, but that it just references it. I was under the assumption that TOPSRCDIR was determined dynamically based on the build location.

Comment 15

[Mike Hommey [:glandium]]

(In reply to Jared Wein [:jaws] from comment #14)
> (In reply to Mike Hommey [:glandium] from comment #13)
> > A corollary question is why isn't your topsrcdir absolute (which would work
> > around the issue)?
> 
> Probably because I had never set my TOPSRCDIR environment variable before :/

I never set it, yet it always is an absolute path for me. How do you set your objdir?

Comment 16

[[e-mail]]

A common way to get a relative topsrcdir is to run configure from your objdir using a relative path to your srcdir e.g.
mkdir objdir
cd objdir
../configure
make
(This is an easy way to make multiple builds with similar configure options from the same srcdir, which would otherwise involve messing around with mozconfigs.)

Comment 17

[Makoto Kato [:m_kato]]

CURDIR becomes real path on my build env.  This depends on Python version?

http://mxr.mozilla.org/mozilla-central/source/build/pymake/pymake/data.py#1627
1623         if workdir is None:
1624             workdir = os.getcwd()
1625         workdir = os.path.realpath(workdir)
1626         self.workdir = workdir
1627         self.variables.set('CURDIR', Variables.FLAVOR_SIMPLE,
1628                            Variables.SOURCE_AUTOMATIC, workdir.replace('\\','/'))

Comment 18

[Makoto Kato [:m_kato]]

Attachment: fix for pymake

core_abspath returns abs path (/<drive>/path/name).  But CURDIR format is <drive>:/path/name on PyMake.  So subst cannot remove abs path.

Comment 19

[Benjamin Smedberg]

My topsrcdir is always relative because I configure like this:

../src/configure --enable-debug --disable-optimize

Comment 20

[Jared Wein [:jaws] (please needinfo? me)]

(In reply to Mike Hommey [:glandium] from comment #15)
> (In reply to Jared Wein [:jaws] from comment #14)
> > (In reply to Mike Hommey [:glandium] from comment #13)
> > > A corollary question is why isn't your topsrcdir absolute (which would work
> > > around the issue)?
> > 
> > Probably because I had never set my TOPSRCDIR environment variable before :/
> 
> I never set it, yet it always is an absolute path for me. How do you set
> your objdir?

(In reply to [e-mail] from comment #16)
> A common way to get a relative topsrcdir is to run configure from your
> objdir using a relative path to your srcdir e.g.
> mkdir objdir
> cd objdir
> ../configure
> make

Yes, this is how I always run configure.

Comment 21

[Mike Hommey [:glandium]]

Comment on attachment 690804 [details] [diff] [review]
fix for pymake

Review of attachment 690804 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Makoto Kato from comment #18)
> Created attachment 690804 [details] [diff] [review]
> fix for pymake
> 
> core_abspath returns abs path (/<drive>/path/name).  But CURDIR format is
> <drive>:/path/name on PyMake.  So subst cannot remove abs path.

Actually, that's not what the problem is, but this fix should work.

Comment 22

[Makoto Kato [:m_kato]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ba511a01e9c7

Comment 23

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/ba511a01e9c7

Comment 24

[Jared Wein [:jaws] (please needinfo? me)]

I imported the patch from inbound and it fixed my issue. Thanks!