Build NSS with pymake, even on Windows with windows-style paths

VERIFIED FIXED in mozilla20

Status

VERIFIED FIXED
10 years ago
8 months ago

People

(Reporter: benjamin, Assigned: glandium)

Tracking

unspecified
mozilla20
All
Windows 7
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [pymake][buildfaster:?])

Attachments

(2 attachments, 1 obsolete attachment)

(Reporter)

Description

10 years ago
For the moment we're not building NSS with pymake primarily because it doesn't understand Windows-style paths. I had a couple patches which made things better (using forward slashes instead of backslashes in some places, IIRC), but there were additional issues.
(Reporter)

Updated

9 years ago
Assignee: benjamin → nobody
Blocks: 485412

Updated

7 years ago
OS: Windows NT → Windows 7
Hardware: x86 → All
Whiteboard: [pymake] → [pymake][buildfaster:?]

Updated

7 years ago
Blocks: 755684
(Assignee)

Comment 1

7 years ago
I wonder how things got since bug 736066.

Comment 2

7 years ago
Using glandium's suggestion of forcing |NSSMAKE = $(MAKE)| in http://mxr.mozilla.org/mozilla-central/source/security/manager/Makefile.in#161, I get:

{
make.py[5]: Entering directory 'd:\obj-inbound\security/manager'
make.py[6]: Entering directory 'd:/srcdirs/inbound/security/dbm'
cd include; c;C:\mozilla-build\msys\mozilla-build\python\python.exe d;C:\mozilla-build\msys\srcdirs\inbound\build\pymake\pymake\..\make.py -j1 export
/usr/bin/sh: c: command not found
d:\srcdirs\inbound\security\coreconf\rules.mk:96:0: command 'for directory in include src ; do if test -d $directory; then set -e; echo "cd $directory; c;C:\mozilla-build\msys\mozilla-build\python\python.exe d;C:\mozilla-build\msys\srcdirs\inbound\build\pymake\pymake\..\make.py -j1 export"; c;C:\mozilla-build\msys\mozilla-build\python\python.exe d;C:\mozilla-build\msys\srcdirs\inbound\build\pymake\pymake\..\make.py -j1 -C $directory export; set +e; else echo "Skipping non-directory $directory..."; fi; true; done' failed, return code 127
evaluation from d:\obj-inbound\security\manager\Makefile:337:26:2:0$ c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -C d:/srcdirs/inbound/security/dbm MAKE="c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -j1" -j1 CC=" cl" SOURCE_MD_DIR=d:/obj-inbound/dist SOURCE_MDHEADERS_DIR=d:/obj-inbound/dist/include/nspr DIST=d:/obj-inbound/dist NSPR_INCLUDE_DIR=d:/obj-inbound/dist/include/nspr NSPR_LIB_DIR=d:/obj-inbound/dist/lib MOZILLA_CLIENT=1 NO_MDUPDATE=1 NSS_ENABLE_ECC=1 NSINSTALL="d:/obj-inbound/config/nsinstall.exe" SQLITE_LIB_NAME=mozsqlite3 SQLITE_INCLUDE_DIR=d:/obj-inbound/dist/include BUILD='d:/obj-inbound/security/$(subst /d/srcdirs/inbound/security/,,$(CURDIR))' BUILD_TREE='$(BUILD)' OBJDIR='$(BUILD)' DEPENDENCIES='$(BUILD)/.deps' SINGLE_SHLIB_DIR='$(BUILD)' SOURCE_XP_DIR=d:/obj-inbound/dist BUILD_OPT=1 OPT_CODE_SIZE=1 NS_USE_GCC= OS_TARGET=WIN95 NSS_ENABLE_ZLIB= PROGRAMS= CHECKLOC= FREEBL_NO_DEPEND=0
evaluation from d:\obj-inbound\security\manager\Makefile:337:26:2:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -C d:/srcdirs/inbound/security/dbm MAKE="c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -j1" -j1 CC=" cl" SOURCE_MD_DIR=d:/obj-inbound/dist SOURCE_MDHEADERS_DIR=d:/obj-inbound/dist/include/nspr DIST=d:/obj-inbound/dist NSPR_INCLUDE_DIR=d:/obj-inbound/dist/include/nspr NSPR_LIB_DIR=d:/obj-inbound/dist/lib MOZILLA_CLIENT=1 NO_MDUPDATE=1 NSS_ENABLE_ECC=1 NSINSTALL="d:/obj-inbound/config/nsinstall.exe" SQLITE_LIB_NAME=mozsqlite3 SQLITE_INCLUDE_DIR=d:/obj-inbound/dist/include BUILD='d:/obj-inbound/security/$(subst /d/srcdirs/inbound/security/,,$(CURDIR))' BUILD_TREE='$(BUILD)' OBJDIR='$(BUILD)' DEPENDENCIES='$(BUILD)/.deps' SINGLE_SHLIB_DIR='$(BUILD)' SOURCE_XP_DIR=d:/obj-inbound/dist BUILD_OPT=1 OPT_CODE_SIZE=1 NS_USE_GCC= OS_TARGET=WIN95 NSS_ENABLE_ZLIB= PROGRAMS= CHECKLOC= FREEBL_NO_DEPEND=0' failed, return code 2
d:\srcdirs\inbound\config\makefiles\target_libs.mk:54:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -C security/manager libs' failed, return code 2
d:\srcdirs\inbound\config\rules.mk:709:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py libs_tier_platform' failed, return code 2
d:\srcdirs\inbound\config\rules.mk:674:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py  tier_platform' failed, return code 2
d:\srcdirs\inbound\client.mk:381:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -j12 -C d:/obj-inbound' failed, return code 2
d:\srcdirs\inbound\client.mk:181:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -f d:/srcdirs/inbound/client.mk realbuild' failed, return code 2
}

Comment 3

7 years ago
Created attachment 624366 [details] [diff] [review]
Patch for testing only

Using the attached patch (next iteration of glandium's suggestions), I get:
http://pastebin.mozilla.org/1641933
(Assignee)

Updated

6 years ago
Depends on: 785694
The NSS build system is quite complicated (see security/coreconf, and the build scripts for security/nss/lib/freebl in particular) and I think we've pretty much agreed that we're likely to rewrite the NSS build system more-or-less completely for performance reasons (to support parallel building, in particular). I'm not sure it's worthwhile to take the risk of whatever (undetected) problems will come up in the switch to pymake. It makes more sense to me to do the build system rewrite, which sould simplify the NSS build system considerably, and *then* switch to pymake.
(Reporter)

Comment 5

6 years ago
Unless rewriting the NSS build system is a 2-week project, we need some solution which allows the mozilla-central builders to build mozilla with pymake (for performance reasons). That could either be:

* switch to gmake only for NSS (which requires a lot more configuration on the builders)
* do the minimal coreconf/NSS fixup which allows us to use pymake across the board.

I tend to think that the second option is the better one.
(Assignee)

Comment 6

6 years ago
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #5)
> * switch to gmake only for NSS (which requires a lot more configuration on
> the builders)

Nothing is actually required on the builders. That hybrid setup already works.
(Assignee)

Comment 7

6 years ago
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #5)
> * do the minimal coreconf/NSS fixup which allows us to use pymake across the
> board.

Fixup in security/coreconf is actually pretty small, but fixing pymake is more involved.
(Assignee)

Updated

6 years ago
Depends on: 794472
(Assignee)

Updated

6 years ago
Depends on: 794896
(Assignee)

Comment 8

6 years ago
Created attachment 665430 [details] [diff] [review]
Stop building NSS with gmake when building m-c with pymake
Attachment #665430 - Flags: review?(ted.mielczarek)
(Assignee)

Updated

6 years ago
Assignee: nobody → mh+mozilla
(Assignee)

Updated

6 years ago
Depends on: 794966
Attachment #665430 - Flags: review?(ted.mielczarek) → review+

Updated

6 years ago
Attachment #624366 - Attachment is obsolete: true
https://hg.mozilla.org/mozilla-central/rev/d126ed851887
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla20
This change has caused my local pymake builds to fail on Windows.

This is the error that I'm getting:

Creating d:/mc/obj-dir/dist/public/dbm
cd src; c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py export
cd include; c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py libs
cd src; c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py libs
nsinstall: failed to create directory d:\mc\obj-dir\security\d:\mc\security\dbm\src: [Error 123] The filename, directory name, or volume label syntax is incorrect: u'd:\\mc\\obj-dir\\security\\d:'
d:\mc\security\coreconf\rules.mk:389:0: command 'd:/mc/obj-dir/_virtualenv/Scripts/python.exe d:/mc/obj-dir/security/build/../../../config/nsinstall.py -D d:/mc/obj-dir/security/d:/mc/security/dbm/src' failed, return code 1
d:\mc\security\coreconf\rules.mk:101:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py -C src libs' failed, return code 2
evaluation from d:\mc\obj-dir\security\build\Makefile:296:26:2:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py -C ../../../security/dbm -j1 CC=" cl" SOURCE_MD_DIR=d:/mc/obj-dir/dist SOURCE_MDHEADERS_DIR=d:/mc/obj-dir/dist/include/nspr DIST=d:/mc/obj-dir/dist NSPR_INCLUDE_DIR=d:/mc/obj-dir/dist/include/nspr NSPR_LIB_DIR=d:/mc/obj-dir/dist/lib MOZILLA_CLIENT=1 NO_MDUPDATE=1 NSS_ENABLE_ECC=1 NSINSTALL="d:/mc/obj-dir/_virtualenv/Scripts/python.exe d:/mc/obj-dir/security/build/../../../config/nsinstall.py" SQLITE_LIB_NAME=mozsqlite3 SQLITE_INCLUDE_DIR=d:/mc/obj-dir/dist/include BUILD='d:/mc/obj-dir/security/$(subst d:/mc/obj-dir/security/build/../../../security/,,$(CURDIR))' BUILD_TREE='$(BUILD)' OBJDIR='$(BUILD)' DEPENDENCIES='$(BUILD)/.deps' SINGLE_SHLIB_DIR='$(BUILD)' SOURCE_XP_DIR=d:/mc/obj-dir/dist BUILD_OPT=1 OPT_CODE_SIZE=1 NS_USE_GCC= OS_TARGET=WIN95 NSS_ENABLE_ZLIB= PROGRAMS= CHECKLOC= FREEBL_NO_DEPEND=0' failed, return code 2
d:\mc\config\makefiles\target_libs.mk:18:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py -C security/build libs' failed, return code 2
d:\mc\config\rules.mk:583:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py libs_tier_platform' failed, return code 2
d:\mc\config\rules.mk:548:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py  tier_platform' failed, return code 2

What do I need to change to fix this?
Comment on attachment 665430 [details] [diff] [review]
Stop building NSS with gmake when building m-c with pymake

Review of attachment 665430 [details] [diff] [review]:
-----------------------------------------------------------------

::: security/build/Makefile.in
@@ +143,5 @@
>  DEFAULT_GMAKE_FLAGS += NSS_DISABLE_DBM=1
>  endif
>  ABS_topsrcdir   := $(call core_abspath,$(topsrcdir))
>  # Hack to force NSS build system to use "normal" object directories
> +DEFAULT_GMAKE_FLAGS += BUILD='$(MOZ_BUILD_ROOT)/security/$$(subst $(ABS_topsrcdir)/security/,,$$(CURDIR))'

If I were to take a wild guess I'd think that my build failure is the result of the change from $(topsrcdir) to $(ABS_topsrcdir). Why does that change need to be made?
(Assignee)

Comment 13

6 years ago
(In reply to Jared Wein [:jaws] from comment #12)
> Comment on attachment 665430 [details] [diff] [review]
> Stop building NSS with gmake when building m-c with pymake
> 
> Review of attachment 665430 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: security/build/Makefile.in
> @@ +143,5 @@
> >  DEFAULT_GMAKE_FLAGS += NSS_DISABLE_DBM=1
> >  endif
> >  ABS_topsrcdir   := $(call core_abspath,$(topsrcdir))
> >  # Hack to force NSS build system to use "normal" object directories
> > +DEFAULT_GMAKE_FLAGS += BUILD='$(MOZ_BUILD_ROOT)/security/$$(subst $(ABS_topsrcdir)/security/,,$$(CURDIR))'
> 
> If I were to take a wild guess I'd think that my build failure is the result
> of the change from $(topsrcdir) to $(ABS_topsrcdir). Why does that change
> need to be made?

Because the path needs to be absolute, which $(topsrcdir) doesn't guarantee. Sadly, ABS_topsrcdir also doesn't normalize...
A corollary question is why isn't your topsrcdir absolute (which would work around the issue)?
(In reply to Mike Hommey [:glandium] from comment #13)
> A corollary question is why isn't your topsrcdir absolute (which would work
> around the issue)?

Probably because I had never set my TOPSRCDIR environment variable before :/

It worked before and so I never thought there was an issue. I couldn't find any documentation that says to actually set a TOPSRCDIR environment variable, but that it just references it. I was under the assumption that TOPSRCDIR was determined dynamically based on the build location.
(Assignee)

Comment 15

6 years ago
(In reply to Jared Wein [:jaws] from comment #14)
> (In reply to Mike Hommey [:glandium] from comment #13)
> > A corollary question is why isn't your topsrcdir absolute (which would work
> > around the issue)?
> 
> Probably because I had never set my TOPSRCDIR environment variable before :/

I never set it, yet it always is an absolute path for me. How do you set your objdir?
A common way to get a relative topsrcdir is to run configure from your objdir using a relative path to your srcdir e.g.
mkdir objdir
cd objdir
../configure
make
(This is an easy way to make multiple builds with similar configure options from the same srcdir, which would otherwise involve messing around with mozconfigs.)
CURDIR becomes real path on my build env.  This depends on Python version?

http://mxr.mozilla.org/mozilla-central/source/build/pymake/pymake/data.py#1627
1623         if workdir is None:
1624             workdir = os.getcwd()
1625         workdir = os.path.realpath(workdir)
1626         self.workdir = workdir
1627         self.variables.set('CURDIR', Variables.FLAVOR_SIMPLE,
1628                            Variables.SOURCE_AUTOMATIC, workdir.replace('\\','/'))
Created attachment 690804 [details] [diff] [review]
fix for pymake

core_abspath returns abs path (/<drive>/path/name).  But CURDIR format is <drive>:/path/name on PyMake.  So subst cannot remove abs path.
Attachment #690804 - Flags: review?(mh+mozilla)
(Reporter)

Comment 19

6 years ago
My topsrcdir is always relative because I configure like this:

../src/configure --enable-debug --disable-optimize
(In reply to Mike Hommey [:glandium] from comment #15)
> (In reply to Jared Wein [:jaws] from comment #14)
> > (In reply to Mike Hommey [:glandium] from comment #13)
> > > A corollary question is why isn't your topsrcdir absolute (which would work
> > > around the issue)?
> > 
> > Probably because I had never set my TOPSRCDIR environment variable before :/
> 
> I never set it, yet it always is an absolute path for me. How do you set
> your objdir?

(In reply to neil@parkwaycc.co.uk from comment #16)
> A common way to get a relative topsrcdir is to run configure from your
> objdir using a relative path to your srcdir e.g.
> mkdir objdir
> cd objdir
> ../configure
> make

Yes, this is how I always run configure.
(Assignee)

Comment 21

6 years ago
Comment on attachment 690804 [details] [diff] [review]
fix for pymake

Review of attachment 690804 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Makoto Kato from comment #18)
> Created attachment 690804 [details] [diff] [review]
> fix for pymake
> 
> core_abspath returns abs path (/<drive>/path/name).  But CURDIR format is
> <drive>:/path/name on PyMake.  So subst cannot remove abs path.

Actually, that's not what the problem is, but this fix should work.
Attachment #690804 - Flags: review?(mh+mozilla) → review+
I imported the patch from inbound and it fixed my issue. Thanks!
Status: RESOLVED → VERIFIED
Depends on: 824004
Duplicate of this bug: 824004

Updated

8 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.