Closed Bug 486141 Opened 15 years ago Closed 12 years ago

Build NSS with pymake, even on Windows with windows-style paths

Categories

(Firefox Build System :: General, defect)

All
Windows 7
defect
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED
mozilla20

People

(Reporter: benjamin, Assigned: glandium)

References

Details

(Whiteboard: [pymake][buildfaster:?])

Attachments

(2 files, 1 obsolete file)

For the moment we're not building NSS with pymake primarily because it doesn't understand Windows-style paths. I had a couple patches which made things better (using forward slashes instead of backslashes in some places, IIRC), but there were additional issues.
Assignee: benjamin → nobody
OS: Windows NT → Windows 7
Hardware: x86 → All
Whiteboard: [pymake] → [pymake][buildfaster:?]
Blocks: 755684
I wonder how things got since bug 736066.
Using glandium's suggestion of forcing |NSSMAKE = $(MAKE)| in http://mxr.mozilla.org/mozilla-central/source/security/manager/Makefile.in#161, I get:

{
make.py[5]: Entering directory 'd:\obj-inbound\security/manager'
make.py[6]: Entering directory 'd:/srcdirs/inbound/security/dbm'
cd include; c;C:\mozilla-build\msys\mozilla-build\python\python.exe d;C:\mozilla-build\msys\srcdirs\inbound\build\pymake\pymake\..\make.py -j1 export
/usr/bin/sh: c: command not found
d:\srcdirs\inbound\security\coreconf\rules.mk:96:0: command 'for directory in include src ; do if test -d $directory; then set -e; echo "cd $directory; c;C:\mozilla-build\msys\mozilla-build\python\python.exe d;C:\mozilla-build\msys\srcdirs\inbound\build\pymake\pymake\..\make.py -j1 export"; c;C:\mozilla-build\msys\mozilla-build\python\python.exe d;C:\mozilla-build\msys\srcdirs\inbound\build\pymake\pymake\..\make.py -j1 -C $directory export; set +e; else echo "Skipping non-directory $directory..."; fi; true; done' failed, return code 127
evaluation from d:\obj-inbound\security\manager\Makefile:337:26:2:0$ c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -C d:/srcdirs/inbound/security/dbm MAKE="c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -j1" -j1 CC=" cl" SOURCE_MD_DIR=d:/obj-inbound/dist SOURCE_MDHEADERS_DIR=d:/obj-inbound/dist/include/nspr DIST=d:/obj-inbound/dist NSPR_INCLUDE_DIR=d:/obj-inbound/dist/include/nspr NSPR_LIB_DIR=d:/obj-inbound/dist/lib MOZILLA_CLIENT=1 NO_MDUPDATE=1 NSS_ENABLE_ECC=1 NSINSTALL="d:/obj-inbound/config/nsinstall.exe" SQLITE_LIB_NAME=mozsqlite3 SQLITE_INCLUDE_DIR=d:/obj-inbound/dist/include BUILD='d:/obj-inbound/security/$(subst /d/srcdirs/inbound/security/,,$(CURDIR))' BUILD_TREE='$(BUILD)' OBJDIR='$(BUILD)' DEPENDENCIES='$(BUILD)/.deps' SINGLE_SHLIB_DIR='$(BUILD)' SOURCE_XP_DIR=d:/obj-inbound/dist BUILD_OPT=1 OPT_CODE_SIZE=1 NS_USE_GCC= OS_TARGET=WIN95 NSS_ENABLE_ZLIB= PROGRAMS= CHECKLOC= FREEBL_NO_DEPEND=0
evaluation from d:\obj-inbound\security\manager\Makefile:337:26:2:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -C d:/srcdirs/inbound/security/dbm MAKE="c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -j1" -j1 CC=" cl" SOURCE_MD_DIR=d:/obj-inbound/dist SOURCE_MDHEADERS_DIR=d:/obj-inbound/dist/include/nspr DIST=d:/obj-inbound/dist NSPR_INCLUDE_DIR=d:/obj-inbound/dist/include/nspr NSPR_LIB_DIR=d:/obj-inbound/dist/lib MOZILLA_CLIENT=1 NO_MDUPDATE=1 NSS_ENABLE_ECC=1 NSINSTALL="d:/obj-inbound/config/nsinstall.exe" SQLITE_LIB_NAME=mozsqlite3 SQLITE_INCLUDE_DIR=d:/obj-inbound/dist/include BUILD='d:/obj-inbound/security/$(subst /d/srcdirs/inbound/security/,,$(CURDIR))' BUILD_TREE='$(BUILD)' OBJDIR='$(BUILD)' DEPENDENCIES='$(BUILD)/.deps' SINGLE_SHLIB_DIR='$(BUILD)' SOURCE_XP_DIR=d:/obj-inbound/dist BUILD_OPT=1 OPT_CODE_SIZE=1 NS_USE_GCC= OS_TARGET=WIN95 NSS_ENABLE_ZLIB= PROGRAMS= CHECKLOC= FREEBL_NO_DEPEND=0' failed, return code 2
d:\srcdirs\inbound\config\makefiles\target_libs.mk:54:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -C security/manager libs' failed, return code 2
d:\srcdirs\inbound\config\rules.mk:709:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py libs_tier_platform' failed, return code 2
d:\srcdirs\inbound\config\rules.mk:674:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py  tier_platform' failed, return code 2
d:\srcdirs\inbound\client.mk:381:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -j12 -C d:/obj-inbound' failed, return code 2
d:\srcdirs\inbound\client.mk:181:0: command 'c:/mozilla-build/python/python.exe d:/srcdirs/inbound/build/pymake/pymake/../make.py -f d:/srcdirs/inbound/client.mk realbuild' failed, return code 2
}
Attached patch Patch for testing only (obsolete) — Splinter Review
Using the attached patch (next iteration of glandium's suggestions), I get:
http://pastebin.mozilla.org/1641933
Depends on: 785694
The NSS build system is quite complicated (see security/coreconf, and the build scripts for security/nss/lib/freebl in particular) and I think we've pretty much agreed that we're likely to rewrite the NSS build system more-or-less completely for performance reasons (to support parallel building, in particular). I'm not sure it's worthwhile to take the risk of whatever (undetected) problems will come up in the switch to pymake. It makes more sense to me to do the build system rewrite, which sould simplify the NSS build system considerably, and *then* switch to pymake.
Unless rewriting the NSS build system is a 2-week project, we need some solution which allows the mozilla-central builders to build mozilla with pymake (for performance reasons). That could either be:

* switch to gmake only for NSS (which requires a lot more configuration on the builders)
* do the minimal coreconf/NSS fixup which allows us to use pymake across the board.

I tend to think that the second option is the better one.
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #5)
> * switch to gmake only for NSS (which requires a lot more configuration on
> the builders)

Nothing is actually required on the builders. That hybrid setup already works.
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #5)
> * do the minimal coreconf/NSS fixup which allows us to use pymake across the
> board.

Fixup in security/coreconf is actually pretty small, but fixing pymake is more involved.
Depends on: 794472
Depends on: 794896
Assignee: nobody → mh+mozilla
Depends on: 794966
Attachment #665430 - Flags: review?(ted.mielczarek) → review+
Attachment #624366 - Attachment is obsolete: true
https://hg.mozilla.org/mozilla-central/rev/d126ed851887
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla20
This change has caused my local pymake builds to fail on Windows.

This is the error that I'm getting:

Creating d:/mc/obj-dir/dist/public/dbm
cd src; c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py export
cd include; c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py libs
cd src; c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py libs
nsinstall: failed to create directory d:\mc\obj-dir\security\d:\mc\security\dbm\src: [Error 123] The filename, directory name, or volume label syntax is incorrect: u'd:\\mc\\obj-dir\\security\\d:'
d:\mc\security\coreconf\rules.mk:389:0: command 'd:/mc/obj-dir/_virtualenv/Scripts/python.exe d:/mc/obj-dir/security/build/../../../config/nsinstall.py -D d:/mc/obj-dir/security/d:/mc/security/dbm/src' failed, return code 1
d:\mc\security\coreconf\rules.mk:101:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py -C src libs' failed, return code 2
evaluation from d:\mc\obj-dir\security\build\Makefile:296:26:2:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py -C ../../../security/dbm -j1 CC=" cl" SOURCE_MD_DIR=d:/mc/obj-dir/dist SOURCE_MDHEADERS_DIR=d:/mc/obj-dir/dist/include/nspr DIST=d:/mc/obj-dir/dist NSPR_INCLUDE_DIR=d:/mc/obj-dir/dist/include/nspr NSPR_LIB_DIR=d:/mc/obj-dir/dist/lib MOZILLA_CLIENT=1 NO_MDUPDATE=1 NSS_ENABLE_ECC=1 NSINSTALL="d:/mc/obj-dir/_virtualenv/Scripts/python.exe d:/mc/obj-dir/security/build/../../../config/nsinstall.py" SQLITE_LIB_NAME=mozsqlite3 SQLITE_INCLUDE_DIR=d:/mc/obj-dir/dist/include BUILD='d:/mc/obj-dir/security/$(subst d:/mc/obj-dir/security/build/../../../security/,,$(CURDIR))' BUILD_TREE='$(BUILD)' OBJDIR='$(BUILD)' DEPENDENCIES='$(BUILD)/.deps' SINGLE_SHLIB_DIR='$(BUILD)' SOURCE_XP_DIR=d:/mc/obj-dir/dist BUILD_OPT=1 OPT_CODE_SIZE=1 NS_USE_GCC= OS_TARGET=WIN95 NSS_ENABLE_ZLIB= PROGRAMS= CHECKLOC= FREEBL_NO_DEPEND=0' failed, return code 2
d:\mc\config\makefiles\target_libs.mk:18:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py -C security/build libs' failed, return code 2
d:\mc\config\rules.mk:583:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py libs_tier_platform' failed, return code 2
d:\mc\config\rules.mk:548:0: command 'c:/mozilla-build/python/python.exe d:/mc/build/pymake/pymake/../make.py  tier_platform' failed, return code 2

What do I need to change to fix this?
Comment on attachment 665430 [details] [diff] [review]
Stop building NSS with gmake when building m-c with pymake

Review of attachment 665430 [details] [diff] [review]:
-----------------------------------------------------------------

::: security/build/Makefile.in
@@ +143,5 @@
>  DEFAULT_GMAKE_FLAGS += NSS_DISABLE_DBM=1
>  endif
>  ABS_topsrcdir   := $(call core_abspath,$(topsrcdir))
>  # Hack to force NSS build system to use "normal" object directories
> +DEFAULT_GMAKE_FLAGS += BUILD='$(MOZ_BUILD_ROOT)/security/$$(subst $(ABS_topsrcdir)/security/,,$$(CURDIR))'

If I were to take a wild guess I'd think that my build failure is the result of the change from $(topsrcdir) to $(ABS_topsrcdir). Why does that change need to be made?
(In reply to Jared Wein [:jaws] from comment #12)
> Comment on attachment 665430 [details] [diff] [review]
> Stop building NSS with gmake when building m-c with pymake
> 
> Review of attachment 665430 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: security/build/Makefile.in
> @@ +143,5 @@
> >  DEFAULT_GMAKE_FLAGS += NSS_DISABLE_DBM=1
> >  endif
> >  ABS_topsrcdir   := $(call core_abspath,$(topsrcdir))
> >  # Hack to force NSS build system to use "normal" object directories
> > +DEFAULT_GMAKE_FLAGS += BUILD='$(MOZ_BUILD_ROOT)/security/$$(subst $(ABS_topsrcdir)/security/,,$$(CURDIR))'
> 
> If I were to take a wild guess I'd think that my build failure is the result
> of the change from $(topsrcdir) to $(ABS_topsrcdir). Why does that change
> need to be made?

Because the path needs to be absolute, which $(topsrcdir) doesn't guarantee. Sadly, ABS_topsrcdir also doesn't normalize...
A corollary question is why isn't your topsrcdir absolute (which would work around the issue)?
(In reply to Mike Hommey [:glandium] from comment #13)
> A corollary question is why isn't your topsrcdir absolute (which would work
> around the issue)?

Probably because I had never set my TOPSRCDIR environment variable before :/

It worked before and so I never thought there was an issue. I couldn't find any documentation that says to actually set a TOPSRCDIR environment variable, but that it just references it. I was under the assumption that TOPSRCDIR was determined dynamically based on the build location.
(In reply to Jared Wein [:jaws] from comment #14)
> (In reply to Mike Hommey [:glandium] from comment #13)
> > A corollary question is why isn't your topsrcdir absolute (which would work
> > around the issue)?
> 
> Probably because I had never set my TOPSRCDIR environment variable before :/

I never set it, yet it always is an absolute path for me. How do you set your objdir?
A common way to get a relative topsrcdir is to run configure from your objdir using a relative path to your srcdir e.g.
mkdir objdir
cd objdir
../configure
make
(This is an easy way to make multiple builds with similar configure options from the same srcdir, which would otherwise involve messing around with mozconfigs.)
CURDIR becomes real path on my build env.  This depends on Python version?

http://mxr.mozilla.org/mozilla-central/source/build/pymake/pymake/data.py#1627
1623         if workdir is None:
1624             workdir = os.getcwd()
1625         workdir = os.path.realpath(workdir)
1626         self.workdir = workdir
1627         self.variables.set('CURDIR', Variables.FLAVOR_SIMPLE,
1628                            Variables.SOURCE_AUTOMATIC, workdir.replace('\\','/'))
Attached patch fix for pymakeSplinter Review
core_abspath returns abs path (/<drive>/path/name).  But CURDIR format is <drive>:/path/name on PyMake.  So subst cannot remove abs path.
Attachment #690804 - Flags: review?(mh+mozilla)
My topsrcdir is always relative because I configure like this:

../src/configure --enable-debug --disable-optimize
(In reply to Mike Hommey [:glandium] from comment #15)
> (In reply to Jared Wein [:jaws] from comment #14)
> > (In reply to Mike Hommey [:glandium] from comment #13)
> > > A corollary question is why isn't your topsrcdir absolute (which would work
> > > around the issue)?
> > 
> > Probably because I had never set my TOPSRCDIR environment variable before :/
> 
> I never set it, yet it always is an absolute path for me. How do you set
> your objdir?

(In reply to neil@parkwaycc.co.uk from comment #16)
> A common way to get a relative topsrcdir is to run configure from your
> objdir using a relative path to your srcdir e.g.
> mkdir objdir
> cd objdir
> ../configure
> make

Yes, this is how I always run configure.
Comment on attachment 690804 [details] [diff] [review]
fix for pymake

Review of attachment 690804 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Makoto Kato from comment #18)
> Created attachment 690804 [details] [diff] [review]
> fix for pymake
> 
> core_abspath returns abs path (/<drive>/path/name).  But CURDIR format is
> <drive>:/path/name on PyMake.  So subst cannot remove abs path.

Actually, that's not what the problem is, but this fix should work.
Attachment #690804 - Flags: review?(mh+mozilla) → review+
I imported the patch from inbound and it fixed my issue. Thanks!
Status: RESOLVED → VERIFIED
Depends on: 824004
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: