928961 - mozmake can't handle long path names

Status: NEW

(Firefox Build System :: General, defect)

Description

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

mozmake can't handle long path names, which need some special magic under windows -- you need to use the unicode versions of the IO functions, and prepend paths with the \\?\ UNC prefix.  (Note that also turns off automatic / -> \ conversion.)  So, \\?\C:\foo\bar\baz instead of C:/foo/bar/baz or similar.

STR:

cd c:/tmp   # or wherever

mkdir -p this-is-a-long-path-component-0/this-is-a-long-path-component-1/this-is-a-long-path-component-2/this-is-a-long-path-component-3/this-is-a-long-path-component-4/this-is-a-long-path-component-5

cd !$

echo hello > this-is-a-long-filename.txt

cat > Makefile

all: success

success: ../../../../../../this-is-a-long-path-component-0/this-is-a-long-path-component-1/this-is-a-long-path-component-2/this-is-a-long-path-component-3/this-is-a-long-path-component-4/this-is-a-long-path-component-5/this-is-a-long-filename.txt
        cp $< $@
^D


Using mozmake:

mozmake: *** No rule to make target '../../../../../../this-is-a-long-path-component-0/this-is-a-long-path-component-1/this-is-a-long-path-component-2/this-is-a-long-path-component-3/this-is-a-long-path-component-4/this-is-a-long-path-component-5/this-is-a-long-filename.txt', needed by 'success'.  Stop.

pymake works, as does msys make.

Comment 1

[Mike Hommey [:glandium]]

According to my #build log, that is what's blocking you in m-c:
mozmake[3]: *** No rule to make target 'test_window-null-names.html.json', needed by 
'../../../../../../../../_tests/testing/mochitest/tests/dom/imptests/failures/html/html/browsers/the-window-object/named-access-on-the-window-object/test_window-null-names.html.json'.  Stop.

The interesting thing is that this doesn't happen on my local build, and it doesn't happen on birch, where I enabled mozmake. Is your objdir path deep?

(But yes, I can reproduce with the testcase, and filed an upstream bug: https://savannah.gnu.org/bugs/?40344 )

Comment 2

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Not super deep.. objdir is in c:\proj\moz\fx64-debug

Could be right on the edge somewhere?  It's definitely not exceeding 259 though; in some of my earlier experiments with this when I was fixing winrm, I actually couldn't get path names longer than ~150 chars without using the wide apis and \\?\ full paths.

Comment 3

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Note for me or whoever pokes at this more -- the win32 wide-char stdio functions understand UNC \\?\ pathnames. The only issue is that those can't be relative, which is stupid and dumb.  There is code here: http://stackoverflow.com/questions/18580945/need-clarification-for-converting-paths-into-long-unicode-paths-or-the-ones-star to do the proper 8-bit to long-wide-path conversion.

The simple fix is to get make to shunt all pathnames through this before calling libc/stdio/native functions, and to use the wchar_t versions of those (doing a simple conversion before if necessary).

Comment 4

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

I just filed bug 950742 which is the main offender.  I suspect that this is so very close to the limit.  I think the way it's exceeding 259 is by appending the relative path to the cwd -- pymake might be making the path fully absolute before passing it to the win32 APIs, and gnu make is not.  That might be a useful approach for the upstream bug instead of converting it to use UNC pathnames.

One thing I discovered while doing winrm is that the Windows function GetFullPathNameW happily handles non-UNC path names and normalizes things properly; that would be the thing to use before passing things to stdio.

However, I've lost interest in fixing the upstream bug because they require FSF copyright assignment.  So, blah.