824004 - Build bustage: nsinstall: failed to create directory c:\Mozilla\src\mozilla-central\_obj-browser-debug\security\C:\Mozilla\src\mozilla-central\security\dbm\src: [Error 123]

Status: RESOLVED DUPLICATE of bug 846889

(Core :: Security: PSM, defect)

Description

[Honza Bambas (:mayhemer)]

m-c fb4836c277eb, pymake, mozilla-build 1.6, full clobbered build (client.mk) or pymake -C _obj/security/build ran after the full build failure.

Comment 1

[Honza Bambas (:mayhemer)]

Attachment: build log

Comment 2

[Honza Bambas (:mayhemer)]

Bug 486141 seems like a culprit here.

Comment 3

[Honza Bambas (:mayhemer)]

This is very strange.  I confirmed by examining the bash history I pulled two hours ago from m-c and updated (so a full fix for bug 486141 was in the tree).  However, clobbered build was failing.  After another pull that downloaded only one (1 !) changeset and update that modified just one file the build now works.  It could be that hg somehow didn't fully update.  

I really don't understand.

Sorry for spam...

Comment 4

[Honza Bambas (:mayhemer)]

Reopening, I'm experiencing this bug again after an update (first just non-clobbered build, then full clobbered rebuild).

It seems there is something left in the src dir that makes this bug manifest.

Going to investigate since I cannot work...

Comment 5

[Honza Bambas (:mayhemer)]

Attachment: builds but doesn't link

Build passes the 123 error but fails with unresolved externals for me with the patch (clobbered full build):

   Creating library c:/Mozilla/src/mozilla-central/_obj-browser-debug/security/build/nssdbm3.lib and object c:/Mozilla/src/mozilla-central/_obj-browser-debug/
security/build/nssdbm3.exp
keydb.obj : error LNK2019: unresolved external symbol _lg_nsslowkey_DestroyPublicKey referenced in function _nsslowkey_KeyForCertExists
lgattr.obj : error LNK2001: unresolved external symbol _lg_nsslowkey_DestroyPublicKey
lowcert.obj : error LNK2001: unresolved external symbol _lg_nsslowkey_DestroyPublicKey
keydb.obj : error LNK2001: unresolved external symbol _lg_nsslowkey_PrivateKeyInfoTemplate
keydb.obj : error LNK2001: unresolved external symbol _lg_nsslowkey_ECPrivateKeyTemplate
keydb.obj : error LNK2019: unresolved external symbol _lg_prepare_low_ec_priv_key_for_asn1 referenced in function _seckey_encrypt_private_key
keydb.obj : error LNK2001: unresolved external symbol _lg_nsslowkey_DHPrivateKeyTemplate
keydb.obj : error LNK2019: unresolved external symbol _lg_prepare_low_dh_priv_key_for_asn1 referenced in function _seckey_encrypt_private_key
keydb.obj : error LNK2001: unresolved external symbol _lg_nsslowkey_PQGParamsTemplate
keydb.obj : error LNK2019: unresolved external symbol _lg_prepare_low_pqg_params_for_asn1 referenced in function _seckey_encrypt_private_key
keydb.obj : error LNK2001: unresolved external symbol _lg_nsslowkey_DSAPrivateKeyTemplate
keydb.obj : error LNK2019: unresolved external symbol _lg_prepare_low_dsa_priv_key_for_asn1 referenced in function _seckey_encrypt_private_key
keydb.obj : error LNK2001: unresolved external symbol _lg_nsslowkey_RSAPrivateKeyTemplate
keydb.obj : error LNK2019: unresolved external symbol _lg_prepare_low_rsa_priv_key_for_asn1 referenced in function _seckey_encrypt_private_key
keydb.obj : error LNK2019: unresolved external symbol _LGEC_FillParams referenced in function _seckey_decrypt_private_key
lgcreate.obj : error LNK2001: unresolved external symbol _LGEC_FillParams
lowcert.obj : error LNK2001: unresolved external symbol _LGEC_FillParams
keydb.obj : error LNK2019: unresolved external symbol _lg_prepare_low_ecparams_for_asn1 referenced in function _seckey_decrypt_private_key
keydb.obj : error LNK2001: unresolved external symbol _lg_nsslowkey_RSAPrivateKeyTemplate2
keydb.obj : error LNK2019: unresolved external symbol _lg_nsslowkey_DestroyPrivateKey referenced in function _nsslowkey_FindKeyNicknameByPublicKey
lgattr.obj : error LNK2001: unresolved external symbol _lg_nsslowkey_DestroyPrivateKey
lgcreate.obj : error LNK2001: unresolved external symbol _lg_nsslowkey_DestroyPrivateKey
lgfind.obj : error LNK2001: unresolved external symbol _lg_nsslowkey_DestroyPrivateKey
lgattr.obj : error LNK2019: unresolved external symbol _lg_nsslowkey_ConvertToPublicKey referenced in function _lg_GetPublicKey
lowkey.obj : error LNK2019: unresolved external symbol _EC_CopyParams referenced in function _nsslowkey_ConvertToPublicKey
c:/Mozilla/src/mozilla-central/_obj-browser-debug/security/build/nssdbm3.dll : fatal error LNK1120: 18 unresolved externals



Output of the 'echo' target:

+echo::
+	@echo $(CURDIR)
+	@echo $(ABS_topsrcdir)
+	@echo $(MOZ_BUILD_ROOT)/security/$(subst $(ABS_topsrcdir)/security/,,$(CURDIR))
+	@echo $(MOZ_BUILD_ROOT)/security/$(CURDIR)

> $ cd /c/Mozilla/src/mozilla-central/_obj-browser-debug
> $ pymake -C security/build/ echo
> make.py[0]: Entering directory 'c:\Mozilla\src\mozilla-central\_obj-browser-debug\security/build/'
> creating Makefile
> c:/Mozilla/src/mozilla-central/_obj-browser-debug/security/build
> c:/Mozilla/src/mozilla-central
> c:/Mozilla/src/mozilla-central/_obj-browser-debug/security/c:/Mozilla/src/mozilla-central/_obj-browser-debug/security/build
> c:/Mozilla/src/mozilla-central/_obj-browser-debug/security/c:/Mozilla/src/mozilla-central/_obj-browser-debug/security/build
> make.py[0]: Leaving directory 'c:\Mozilla\src\mozilla-central\_obj-browser-debug\security/build/'


The subst in BUILD='$(MOZ_BUILD_ROOT)/security/$$(subst $(ABS_topsrcdir)/security/,,$$(CURDIR))' has actually no affect.


Going to try to build with make instead of pymake (no parallel build).

Comment 6

[Honza Bambas (:mayhemer)]

Building with make works.

I'm lost.

Comment 7

[Mike Hommey [:glandium]]

(In reply to Honza Bambas (:mayhemer) from comment #5)
> The subst in BUILD='$(MOZ_BUILD_ROOT)/security/$$(subst
> $(ABS_topsrcdir)/security/,,$$(CURDIR))' has actually no affect.

Note it's
$$(subst $(ABS_topsrcdir)/security/,,$$(CURDIR))
So it applies when building under security/nss, not security/build.

(In reply to Honza Bambas (:mayhemer) from comment #4)
> Reopening, I'm experiencing this bug again after an update (first just
> non-clobbered build, then full clobbered rebuild).

An update from what changeset to what changeset?

Comment 8

[Mike Hommey [:glandium]]

Also, are you using pymake from a clone of the standalone pymake, or are you using the version in the tree (build/pymake/make.py) ? If the former, try the latter.

Comment 9

[Honza Bambas (:mayhemer)]

(In reply to Mike Hommey [:glandium] from comment #7)
> (In reply to Honza Bambas (:mayhemer) from comment #5)
> > The subst in BUILD='$(MOZ_BUILD_ROOT)/security/$$(subst
> > $(ABS_topsrcdir)/security/,,$$(CURDIR))' has actually no affect.
> 
> Note it's
> $$(subst $(ABS_topsrcdir)/security/,,$$(CURDIR))
> So it applies when building under security/nss, not security/build.

Not sure I follow.

> 
> (In reply to Honza Bambas (:mayhemer) from comment #4)
> > Reopening, I'm experiencing this bug again after an update (first just
> > non-clobbered build, then full clobbered rebuild).
> 
> An update from what changeset to what changeset?

A common update of a tree.  I mean that I simply have updated the tree and that source has generally modified and rebuild was needed.  No particular changeset here is involved IMO.

(In reply to Mike Hommey [:glandium] from comment #8)
> Also, are you using pymake from a clone of the standalone pymake, or are you
> using the version in the tree (build/pymake/make.py) ? If the former, try
> the latter.

I'm using the in-tree pymake.

Comment 10

[Honza Bambas (:mayhemer)]

I cannot build b2g desktop now just because of this bug.  

Ted, I really need to ask you for urgent help with this.  What should I do?

Comment 11

[Honza Bambas (:mayhemer)]

For info, I'm building the same tree (mozilla-central) using different mozconfigs to have more then one _obj- dir under the sources for browser, mobile, b2g.

Could that somehow ruin the source tree that I cannot build?

(BTW: I'm pretty blocked by this bug!, CC'ing my team leader to let him know of this issue)

Comment 12

[Honza Bambas (:mayhemer)]

After renaming (effectively deleting) my other _obj dirs under my mozilla-central source tree, the build passes.  This could be a clue to the fix.

Comment 13

[Honza Bambas (:mayhemer)]

Hitting again while building an optimized Firefox having a debug build obj dir aside.  I'm sure this time, the debug obj dir is in a good health.

Deleting all _ob* dirs and updating DOESN'T HELP.

So I'm completely stuck!

Comment 14

[Honza Bambas (:mayhemer)]

Attachment: .mozconfig

Comment 15

[Honza Bambas (:mayhemer)]

I deleted all files like .pyc etc (that are generated during build) from the source tree.  Restarted the mingw prompt.  Tried my debug and opt config.  Compared fresh cone of m-c and my dev tree and synced even empty dirs to match.  Did anything possible.  Nothing of it helped to get over the error except using an obj dir in an upper level dir (../_obj-blabla) as brian pointed out on IRC.

However, that is not very acceptable workaround for me, since this simply used to work, and all VC++ projects and my years-lasting habits point to _obj dir under the source tree.  Which is BTW a default when no OBJ_DIR config has been provided.

Comment 16

[Mike Hommey [:glandium]]

(In reply to Honza Bambas (:mayhemer) from comment #15)
> Which is BTW a default when no OBJ_DIR config has been provided.

Note the default is obj-blabla, not _obj-blabla.

Did you try without an underscore?

Comment 17

[Honza Bambas (:mayhemer)]

Funny is that after a successful build with obj-dir out of the source tree, now in-tree obj-dir build works...

Really, absolutely no theory what this could be caused by?  No pymake expert here?

Comment 18

[Honza Bambas (:mayhemer)]

(In reply to Mike Hommey [:glandium] from comment #16)
> (In reply to Honza Bambas (:mayhemer) from comment #15)
> > Which is BTW a default when no OBJ_DIR config has been provided.
> 
> Note the default is obj-blabla, not _obj-blabla.
> 
> Did you try without an underscore?

Actually I never tried.  When this shows again, I'll try.  But I suspect that any change in the sub-dir name will make this work.

Comment 19

[Honza Bambas (:mayhemer)]

Since I experience this often (and when least suitable) I decided to re-clone m-c.  It's in the same folder, but freshly cloned.

I found one difference in .mozconfig.out file, the failing tree had:
Adding client.mk options from C:/Mozilla/src/mozilla-central/.mozconfig:

while the functioning tree had:
Adding client.mk options from c:/Mozilla/src/mozilla-central/.mozconfig:

The c: has different capitalization.  Could this (propagated to config files) be the cause?  No idea how that happens to become capital letter, tho.

Comment 20

[Honza Bambas (:mayhemer)]

Getting somewhere:

I've tried to play with the drive letter capitalization by hardcoding value of ABS_topsrcdir and here are the results:

ABS_topsrcdir='c:/Mozilla/src/mozilla-central' - DOESN'T WORK

ABS_topsrcdir='C:/Mozilla/src/mozilla-central' - WORKS


No idea how the letter becomes lower.

Comment 21

[Honza Bambas (:mayhemer)]

Attachment: v1 - workaround for 99% of people

- replace c: to C: ; I'm not aware of any make function to selectively upper-case first letter in a string

Comment 22

[Honza Bambas (:mayhemer)]

Comment on attachment 720052 [details] [diff] [review]
v1 - workaround for 99% of people

Hmm.. this may just revert the issue, when we create src path with C: and not c: (where c: is apparently correct) this patch will make builds fail...

How is the path obtained in our build system?  The patch should go there rather then here apparently.

Dropping request to feedback only.

Comment 23

[Mike Hommey [:glandium]]

Comment on attachment 720052 [details] [diff] [review]
v1 - workaround for 99% of people

Review of attachment 720052 [details] [diff] [review]:
-----------------------------------------------------------------

::: security/build/Makefile.in
@@ +136,5 @@
>  ABS_topsrcdir   := $(shell cd $(topsrcdir); pwd)
>  ifeq ($(HOST_OS_ARCH),WINNT)
>  ifdef .PYMAKE
>  ABS_topsrcdir   := $(shell cd $(topsrcdir); pwd -W)
> +ABS_topsrcdir   := $(subst c:, C:, $(ABS_topsrcdir))

How about the following?
ABS_topsrcdir := $(shell cd $(topsrcdir); $(PYTHON) -c 'import os; print os.path.abspath(os.curdir)')

Comment 24

[Honza Bambas (:mayhemer)]

(In reply to Mike Hommey [:glandium] from comment #23)
> How about the following?
> ABS_topsrcdir := $(shell cd $(topsrcdir); $(PYTHON) -c 'import os; print
> os.path.abspath(os.curdir)')

Doesn't work.  ABS_topsrcdir is: 'c:Mozillasrcmozilla-central'

Comment 25

[Mike Hommey [:glandium]]

(In reply to Honza Bambas (:mayhemer) from comment #24)
> (In reply to Mike Hommey [:glandium] from comment #23)
> > How about the following?
> > ABS_topsrcdir := $(shell cd $(topsrcdir); $(PYTHON) -c 'import os; print
> > os.path.abspath(os.curdir)')
> 
> Doesn't work.  ABS_topsrcdir is: 'c:Mozillasrcmozilla-central'

Then:
ABS_topsrcdir := $(shell cd $(topsrcdir); $(PYTHON) -c 'import os; print os.path.abspath(os.curdir).replace(os.sep, "/")')

Comment 26

[Mike Hommey [:glandium]]

or
ABS_topsrcdir := $(shell -c $(PYTHON) 'import buildconfig; print buildconfig.topsrcdir')

Comment 27

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #26)
> or
> ABS_topsrcdir := $(shell -c $(PYTHON) 'import buildconfig; print
> buildconfig.topsrcdir')

err
ABS_topsrcdir := $(shell $(PYTHON) 'import buildconfig; print buildconfig.topsrcdir')

Comment 28

[Honza Bambas (:mayhemer)]

Got ABS_topsrcdir 'c:/Mozilla/src/mozilla-central' but also my beloved '[Error 123]'.

Comment 29

[Mike Hommey [:glandium]]

So the problem is the other way around...

Try this instead (*really* awful hack):
DEFAULT_GMAKE_FLAGS += CURDIR=$$(shell pwd -W)

Comment 30

[Honza Bambas (:mayhemer)]

(In reply to Honza Bambas (:mayhemer) from comment #28)
> Got ABS_topsrcdir 'c:/Mozilla/src/mozilla-central' but also my beloved
> '[Error 123]'.

Disregard, this is reaction to comment 25 actually.

Applying 26:
ABS_topsrcdir := $(shell $(PYTHON) 'import buildconfig; print buildconfig.topsrcdir')

gives:

c:/Mozilla/src/mozilla-central/_obj-browser-debug/_virtualenv/Scripts/python.exe: can't open file 'import buildconfig; print buildconfig.topsrcdir': [Errno 2]
 No such file or directory

Python 2.7.2 from latest mozilla-build (unmodified).

Comment 31

[Honza Bambas (:mayhemer)]

(In reply to Mike Hommey [:glandium] from comment #29)
> So the problem is the other way around...
> 
> Try this instead (*really* awful hack):
> DEFAULT_GMAKE_FLAGS += CURDIR=$$(shell pwd -W)

Don't worry, doesn't work either:

/bin/sh: c: command not found
/bin/sh: c:Mozillamozilla-buildmsysMozillasrcmozilla-central_obj-browser-debug_virtualenvScriptspython.exe: command not found
/bin/sh: c:Mozillamozilla-buildmsysMozillasrcmozilla-centralconfignsinstall.py: command not found

Comment 32

[Mike Hommey [:glandium]]

Grah: DEFAULT_GMAKE_FLAGS += CURDIR='$$(shell pwd -W)'

Comment 33

[Honza Bambas (:mayhemer)]

(In reply to Mike Hommey [:glandium] from comment #32)
> Grah: DEFAULT_GMAKE_FLAGS += CURDIR='$$(shell pwd -W)'

This one works!

Thanks Mike for quick feedback.  Anyway, I presume this is not the proper fix to land, right?  Also, I still cannot get over bug 846857 that may have the same root cause (846889).

Comment 34

[Honza Bambas (:mayhemer)]

(In reply to Honza Bambas (:mayhemer) from comment #33)
> (In reply to Mike Hommey [:glandium] from comment #32)
> > Grah: DEFAULT_GMAKE_FLAGS += CURDIR='$$(shell pwd -W)'
> 
> This one works!
> 
> Thanks Mike for quick feedback.  Anyway, I presume this is not the proper
> fix to land, right?  Also, I still cannot get over bug 846857 that may have
> the same root cause (846889).

Hitting my favorite bug again :)

This patch seems to be useless...


Can, please, somebody answer me the following question:

How is the $(topsrcdir) variable determined and how is the $(CURDIR) variable determined?


Somewhere there is a dog buried and I really need to figure this out.  I am loosing so much time with this bug and never can say for certain how it is possible I can build suddenly OK again.

Comment 35

[Honza Bambas (:mayhemer)]

Looks like this time helped:

- apply patch from comment 32
- delete (rename) configure file in the src root

The newly generated configure file is identical to the old one.

This bug one mystery to me!

Comment 36

[(not currently active) Ted Mielczarek]

topsrcdir comes from $top_srcdir here:
http://mxr.mozilla.org/mozilla-central/source/build/autoconf/config.status.m4#57

which comes from $srcdir via the AC_INIT macro:
http://git.savannah.gnu.org/gitweb/?p=autoconf.git;a=blob;f=lib/autoconf/general.m4;h=ae971de139977f52d92f424f0890bbab3ede4093;hb=74b21b9b2ce75529e3d93df7367f4f4b0c6b6adf#l667

Autoconf simply takes $0 (the filename of configure) and strips off the filename. When you run a build via client.mk, it executes configure as $(TOPSRCDIR)/configure:
http://mxr.mozilla.org/mozilla-central/source/client.mk#296
where $(TOPSRCDIR) is likely to be $(CURDIR):
http://mxr.mozilla.org/mozilla-central/source/client.mk#50

Pymake's CURDIR comes via os.getcwd():
http://mxr.mozilla.org/mozilla-central/source/build/pymake/pymake/data.py#1624

Comment 37

[Mike Hommey [:glandium]]

topsrcdir comes from WIN_TOP_SRC on windows.
http://mxr.mozilla.org/mozilla-central/source/build/autoconf/config.status.m4#86

Comment 38

[Mike Hommey [:glandium]]

And WIN_TOP_SRC itself comes from
http://mxr.mozilla.org/mozilla-central/source/build/autoconf/config.status.m4#51

I was thinking to replace that by a python invokation.

Comment 39

[(not currently active) Ted Mielczarek]

I'm fairly confident that I'm correct and glandium is not. :) However, he points out that we also now normalize top_srcdir before substituting it:
http://mxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/backend/configenvironment.py#133

Comment 40

[(not currently active) Ted Mielczarek]

Honza: the next time this breaks for you, can you attach $objdir/config.status to a bug?

Comment 41

[Mike Hommey [:glandium]]

Honza: are you sometimes running the build from cmd.exe or do you have a cmd script wrapping pymake, or something involving cmd?

Comment 42

[Honza Bambas (:mayhemer)]

Thanks everybody!  We can hopefully move forward.

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #40)
> Honza: the next time this breaks for you, can you attach
> $objdir/config.status to a bug?

I will for sure.

(In reply to Mike Hommey [:glandium] from comment #41)
> Honza: are you sometimes running the build from cmd.exe or do you have a cmd
> script wrapping pymake, or something involving cmd?

Maybe I do, not sure, read on.  

I'm using the start-msvc10.bat from mozilla build (latest).  

However, slightly modified (all the years I'm using mozilla-build, and never had trouble before this bug):

* start-msvc10.bat:

removed |cd "%USERPROFILE%"|

changed |"%MOZILLABUILD%\msys\bin\bash" --login -i|
to |start "" /MAX "%MOZILLABUILD%"\msys\bin\bash --login -i|

* msys/etc/profile

removed |cd "$HOME"|

I run start-msvc10.bat via c:/windows/bash.bat file from Total Commander at the src tree (c:/mozilla/src/mozilla-central/).

I am adding these changes every time I update mozilla-build for two reasons:
- I hate to be moved to the home dir
- I want to start visual studio through this bat file to have all the path and lib env setup to build mozilla from VC++ and be able to run tests directly via Run command from VC++

start was the only way to make the bash window disappear after VC has started as I recall.

Next time I encounter this I'll try to remove using of the start command.

Comment 43

[Mike Hommey [:glandium]]

So, turns out there's something fishy with python on windows:

$ python -c 'import os;print os.getcwd()'
C:\Users\mh

$ cd mozilla-central/

$ python -c 'import os;print os.getcwd()'
c:\Users\mh\mozilla-central

$ cd ..

$ python -c 'import os;print os.getcwd()'
c:\Users\mh

Comment 44

[Honza Bambas (:mayhemer)]

(In reply to Mike Hommey [:glandium] from comment #43)
> So, turns out there's something fishy with python on windows:

I can confirm this is happening to me as well in my bash terminal.

Comment 45

[Honza Bambas (:mayhemer)]

Attachment: build log #2

And here we are again!

STR:
a working build
pop all pathces
update
push and merge some patches
create new patch
do some work
clobber
mach build
-> fail

attached build log first, config comes right on.

Notice the first line, there are C:'s in the paths.  I *think* the first command I've run after start of the bash was immediately mach build, so the python path print inconsistency had been inprinted to build config, maybe...

Comment 46

[Honza Bambas (:mayhemer)]

Attachment: config.status for build log #2

Comment 47

[Honza Bambas (:mayhemer)]

and, 

$ python -c 'import os;print os.getcwd()'
C:\Mozilla\src\mozilla-central

Comment 48

[Honza Bambas (:mayhemer)]

Mike, when we meet one day in person, I buy a beer, and not just one :)

doing:
- cd <any dir of a choice>
- cd ..
- rm -rf _obj...
- mach build

works!

Comment 49

[Honza Bambas (:mayhemer)]

I think we should track solution to this issue in bug 846889.