1177634 - error occurred while processing the following file: access denied to suite/app.mozbuild

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Edmund Wong (:ewong)]

js\src\ctypes\libffi> checking command to parse dumpbin -symbols output from /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh object... ok
js\src\ctypes\libffi> checking for sysroot... no
js\src\ctypes\libffi> checking for i686-pc-mingw32-mt... no
js\src\ctypes\libffi> checking for mt... mt
js\src\ctypes\libffi> checking if mt is a manifest tool... yes
js\src\ctypes\libffi> checking how to run the C preprocessor... cl -nologo -EP
js\src\ctypes\libffi> checking for ANSI C header files... yes
js\src\ctypes\libffi> checking for sys/types.h... yes
js\src\ctypes\libffi> checking for sys/stat.h... yes
js\src\ctypes\libffi> checking for stdlib.h... yes
js\src\ctypes\libffi> checking for string.h... yes
js\src\ctypes\libffi> checking for memory.h... yes
js\src\ctypes\libffi> checking for strings.h... no
js\src\ctypes\libffi> checking for inttypes.h... yes
js\src\ctypes\libffi> checking for stdint.h... yes
js\src\ctypes\libffi> checking for unistd.h... no
js\src\ctypes\libffi> checking for dlfcn.h... no
js\src\ctypes\libffi> checking for objdir... .libs
js\src\ctypes\libffi> checking for /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh option to produce PIC... -DDLL_EXPORT -DPIC
js\src\ctypes\libffi> checking if /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh PIC flag -DDLL_EXPORT -DPIC works... yes
js\src\ctypes\libffi> checking if /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh static flag  works... yes
js\src\ctypes\libffi> checking if /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh supports -c -o file.obj... yes
js\src\ctypes\libffi> checking if /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh supports -c -o file.obj... (cached) yes
js\src\ctypes\libffi> checking whether the /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh linker (link) supports shared libraries... yes
js\src\ctypes\libffi> checking dynamic linker characteristics... Win32 ld.exe
js\src\ctypes\libffi> checking how to hardcode library paths into programs... immediate
js\src\ctypes\libffi> checking whether stripping libraries is possible... no
js\src\ctypes\libffi> checking if libtool supports shared libraries... yes
js\src\ctypes\libffi> checking whether to build shared libraries... no
js\src\ctypes\libffi> checking whether to build static libraries... yes
js\src\ctypes\libffi> checking how to run the C++ preprocessor... cl -nologo -EP
js\src\ctypes\libffi> checking whether the /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh linker (link) supports shared libraries... no
js\src\ctypes\libffi> checking for /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh option to produce PIC... -DDLL_EXPORT -DPIC
js\src\ctypes\libffi> checking if /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh PIC flag -DDLL_EXPORT -DPIC works... yes
js\src\ctypes\libffi> checking if /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh static flag  works... yes
js\src\ctypes\libffi> checking if /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh supports -c -o file.obj... yes
js\src\ctypes\libffi> checking if /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh supports -c -o file.obj... (cached) yes
js\src\ctypes\libffi> checking whether the /c/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/msvcc.sh linker (link) supports shared libraries... no
js\src\ctypes\libffi> checking dynamic linker characteristics... Win32 ld.exe
js\src\ctypes\libffi> checking how to hardcode library paths into programs... immediate
js\src\ctypes\libffi> checking size of size_t... 4
js\src\ctypes\libffi> checking for C compiler vendor... microsoft
js\src\ctypes\libffi> 
js\src\ctypes\libffi> ********************************************************
js\src\ctypes\libffi> * WARNING: Don't know the best CFLAGS for this system  *
js\src\ctypes\libffi> * Use ./configure CFLAGS=... to specify your own flags *
js\src\ctypes\libffi> * (otherwise, a default of CFLAGS=-O3 will be used)    *
js\src\ctypes\libffi> ********************************************************
js\src\ctypes\libffi> 
js\src\ctypes\libffi> checking whether C compiler accepts -O3... yes
js\src\ctypes\libffi> checking CFLAGS for maximum warnings... -Wall
js\src\ctypes\libffi> checking whether to enable maintainer-specific portions of Makefiles... no
js\src\ctypes\libffi> checking sys/mman.h usability... no
js\src\ctypes\libffi> checking sys/mman.h presence... no
js\src\ctypes\libffi> checking for sys/mman.h... no
js\src\ctypes\libffi> checking for mmap... no
js\src\ctypes\libffi> checking for sys/mman.h... (cached) no
js\src\ctypes\libffi> checking for mmap... (cached) no
js\src\ctypes\libffi> checking for ANSI C header files... (cached) yes
js\src\ctypes\libffi> checking for memcpy... no
js\src\ctypes\libffi> checking for size_t... yes
js\src\ctypes\libffi> checking for working alloca.h... no
js\src\ctypes\libffi> checking for alloca... yes
js\src\ctypes\libffi> checking size of double... 8
js\src\ctypes\libffi> checking size of long double... 8
js\src\ctypes\libffi> checking whether byte ordering is bigendian... no
js\src\ctypes\libffi> checking assembler .cfi pseudo-op support... no
js\src\ctypes\libffi> checking assembler supports pc related relocs... yes
js\src\ctypes\libffi> checking assembler .ascii pseudo-op support... yes
js\src\ctypes\libffi> checking assembler .string pseudo-op support... yes
js\src\ctypes\libffi> checking for _ prefix in compiled symbols... yes
js\src\ctypes\libffi> configure: updating cache c:/builds/slave3/c-c/build/objdir/js/src/ctypes/libffi/config.cache
js\src\ctypes\libffi> checking that generated files are newer than configure... done
js\src\ctypes\libffi> configure: creating ./config.status
js\src\ctypes\libffi> config.status: creating include/Makefile
js\src\ctypes\libffi> config.status: creating include/ffi.h
js\src\ctypes\libffi> config.status: creating Makefile
js\src\ctypes\libffi> config.status: creating testsuite/Makefile
js\src\ctypes\libffi> config.status: creating man/Makefile
js\src\ctypes\libffi> config.status: creating libffi.pc
js\src\ctypes\libffi> config.status: creating fficonfig.h
js\src\ctypes\libffi> config.status: linking C:/builds/slave3/c-c/build/mozilla/js/src/ctypes/libffi/src/x86/ffitarget.h to include/ffitarget.h
js\src\ctypes\libffi> config.status: executing buildir commands
js\src\ctypes\libffi> config.status: skipping top_srcdir/Makefile - not created
js\src\ctypes\libffi> config.status: executing depfiles commands
js\src\ctypes\libffi> config.status: executing libtool commands
js\src\ctypes\libffi> config.status: executing include commands
js\src\ctypes\libffi> config.status: executing src commands

Reticulating splines...
Traceback (most recent call last):
  File "./config.status", line 971, in <module>
    config_status(**args)
  File "c:\builds\slave3\c-c\build\mozilla\python\mozbuild\mozbuild\config_status.py", line 149, in config_status
    summary = the_backend.consume(definitions)
  File "c:\builds\slave3\c-c\build\mozilla\python\mozbuild\mozbuild\backend\base.py", line 180, in consume
    for obj in objs:
  File "c:\builds\slave3\c-c\build\mozilla\python\mozbuild\mozbuild\frontend\emitter.py", line 147, in emit
    for out in output:
  File "c:\builds\slave3\c-c\build\mozilla\python\mozbuild\mozbuild\frontend\reader.py", line 1051, in read_mozbuild
    sys.exc_info()[2], sandbox_load_error=sle)
mozbuild.frontend.reader.BuildReaderError: 
==============================
ERROR PROCESSING MOZBUILD FILE
==============================

The error occurred while processing the following file:

    c:/builds/slave3/c-c/build/mozilla/moz.build

The underlying problem is an illegal file access. This is likely due to trying to access a file outside of the top source directory.

The path whose access was denied is:

    c:/builds/slave3/c-c/build/suite/app.mozbuild

Modify the script to not access this file and try again.

*** Fix above errors and then restart with\
               "c:/builds/slave3/c-c/build/mozmake.exe -f client.mk build"
C:/builds/slave3/c-c/build/client.mk:361: recipe for target 'configure' failed
mozmake.exe[1]: *** [configure] Error 1
mozmake.exe[1]: Leaving directory 'C:/builds/slave3/c-c/build'
client.mk:375: recipe for target 'C:/builds/slave3/c-c/build/objdir/Makefile' failed
mozmake.exe: *** [C:/builds/slave3/c-c/build/objdir/Makefile] Error 2

This is with the following mozconfig:

. "$topsrcdir/build/mozconfig.common"

ac_add_options --enable-application=suite
ac_add_options --enable-update-channel=${MOZ_UPDATE_CHANNEL}
ac_add_options --enable-update-packaging
ac_add_options --enable-jemalloc
ac_add_options --enable-tests
ac_add_options --enable-profiling

# Needed to enable breakpad in application.ini
export MOZILLA_OFFICIAL=1

# Bug 1155007 temporary workaround. Needs to be
# backed out once we get a Socorro token generated.
unset SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE

if test "$PROCESSOR_ARCHITECTURE" = "AMD64" -o "$PROCESSOR_ARCHITEW6432" = "AMD64"; then
  . $topsrcdir/build/win32/mozconfig.vs2013-win64
else
  . $topsrcdir/build/win32/mozconfig.vs2010
fi

# Set up mapi includes (must be done after visual studio setup)
export INCLUDE=$INCLUDE:/c/Office\ 2010\ Developer\ Resources/Outlook\ 2010\ MAPI\ Headers
mk_export_correct_style INCLUDE
mk_add_options MOZ_OBJDIR=c:/builds/slave3/c-c/build/objdir

Comment 1

[Edmund Wong (:ewong)]

Also, this isn't done in a mozbuild start.bat environment.  It's done
from a command prompt with environment variables copied from a working
Thunderbird nightly build.

Comment 2

[Edmund Wong (:ewong)]

Having done some testing, I have found out where the problem lies.

The problem I had was with the EXTERNAL_SOURCE_DIR as retrieved in
http://mxr.mozilla.org/comm-central/source/mozilla/python/mozbuild/mozbuild/backend/configenvironment.py#146

In the environment that I was building with, EXTERNAL_SOURCE_DIR was
C:/builds/slave/c-c/build.

whereas for the other checks in the system, specifically in
http://mxr.mozilla.org/comm-central/source/mozilla/python/mozbuild/mozbuild/frontend/reader.py#149,
the other paths were of the format "c:/builds/slave/c-c/build..."

So since C: != c:  in python but in Windows it is, this is where the
check fails.

I'm not exactly sure what the proper fix should be.

I tried:

1) for http://mxr.mozilla.org/comm-central/source/mozilla/python/mozbuild/mozbuild/backend/configenvironment.py#151,
   self.external_source_dir = mozpath.normpath(external).lower()

2) somehow even during the config.status processing, EXTERNAL_SOURCE_DIR
could be lowercased.

But this could potentially screw up case-sensitive paths.

So I'm going to guess that the proper way of doing this might be to
check if the first part of the path is a Windows drive, i.e. C:, and
just lowercase it. C: -> c:

glandium, what should be done?

Comment 3

[Mike Hommey [:glandium]]

Use os.path.normcase in is_read_allowed in reader.py.

Comment 4

[Edmund Wong (:ewong)]

Attachment: proposed patch (v1)

Comment 5

[Mike Hommey [:glandium]]

Comment on attachment 8629294 [details] [diff] [review]
proposed patch (v1)

Review of attachment 8629294 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/frontend/reader.py
@@ +147,5 @@
>          return True
>  
> +    if config.external_source_dir:
> +        external_dir = os.path.normcase(config.external_source_dir)
> +        if mozpath.basedir(path, [external_dir]):

you need to normcase path as well.

Comment 6

[Edmund Wong (:ewong)]

Attachment: proposed patch (v2)

Comment 7

[Cameron McCormack (:heycam)]

I'm running into this problem too, and unfortunately the v2 patch here doesn't help.  I get this error: https://pastebin.mozilla.org/8838761.  I also tried with my objdir explicitly set (to c:/comm-central/obj) but got the same error.

These are the variable values I get at this point:

norm_path: c:\comm-central\suite\app.mozbuild
external_dir: \c\comm-central

I was playing around with various os.path methods but couldn't find one that would normalize the Windows-style paths to msys ones or vice versa.

If I just force is_read_allowed to return True, I run into another error: https://pastebin.mozilla.org/8838763

Comment 8

[Cameron McCormack (:heycam)]

Oh, I realized I was using make rather than mozmake.  Using mozmake and the v2 patch here works.

Comment 9

[Edmund Wong (:ewong)]

Try push:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f015830da5f5

Comment 10

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6b30ec9f6580

Comment 11

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/6b30ec9f6580

Comment 12

[Edmund Wong (:ewong)]

Comment on attachment 8630288 [details] [diff] [review]
proposed patch (v2)

Approval Request Comment
[Feature/regressing bug #]:
[User impact if declined]: c-a, c-b, and c-r cannot be built.
[Describe test coverage new/current, TreeHerder]:
[Risks and why]: 
[String/UUID change made/needed]: none

Comment 13

[Mike Hommey [:glandium]]

That's NPOTB for m-* automation. That only affects seamonkey, thunderbird and comm-* related stuff.

Comment 14

[Kate Glazko]

Comment on attachment 8630288 [details] [diff] [review]
proposed patch (v2)

Approving for beta/aurora, it has been on m-c for several days with no known issues. For release, please on the train to arrive.

Comment 15

[Edmund Wong (:ewong)]

(In reply to Kate Glazko from comment #14)
> Comment on attachment 8630288 [details] [diff] [review]
> proposed patch (v2)
> 
> Approving for beta/aurora, it has been on m-c for several days with no known
> issues. For release, please on the train to arrive.

Thanks Kate!  As for m-r,  I also need it on it; but iiuc, I can
always push it to the SEAMONKEY_2_35_RELEASE_BRANCH, right?  That
shouldn't affect m-r proper, right?

Comment 16

[Sylvestre Ledru [:Sylvestre]]

Indeed, please use the seamonkey relbranch for release.

Comment 17

[Edmund Wong (:ewong)]

Setting c-n for the m-a and m-b version of the patches.

Comment 18

[Ryan VanderMeulen [:RyanVM]]

No need to set checkin-needed for uplifts if you've got the other flags set correctly ;)

Comment 19

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/1bdf823f1829

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/34bd36c88216

Comment 21

[Edmund Wong (:ewong)]

Comment on attachment 8630288 [details] [diff] [review]
proposed patch (v2)

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration:
User impact if declined: SeaMonkey 2.35 is to be based on mozilla-esr38.  This
patch is needed in order to get it built. (NPOTB since this patch only
is used by comm-release)
Fix Landed on Version:
Risk to taking this patch (and alternatives if risky): 
String or UUID changes made by this patch:  none

See https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for more info.

Comment 22

[Edmund Wong (:ewong)]

(In reply to Edmund Wong (:ewong) from comment #21)
> Comment on attachment 8630288 [details] [diff] [review]
> proposed patch (v2)
> 
> [Approval Request Comment]
> If this is not a sec:{high,crit} bug, please state case for ESR
> consideration:
> User impact if declined: SeaMonkey 2.35 is to be based on mozilla-esr38. 
> This
> patch is needed in order to get it built. (NPOTB since this patch only
> is used by comm-release)
Fix Landed on Version: 41
Risk to taking this patch (and alternatives if risky):  None
> String or UUID changes made by this patch:  none
> 
> See https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for more
> info.

Comment 23

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8630288 [details] [diff] [review]
proposed patch (v2)

This seems like a mozbuild related patch. Should be safe.

Comment 24

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr38/rev/41024098e3f6