libvpx failed to build with "too many sections" using -GL flag on VS2013

RESOLVED DUPLICATE of bug 934984

Status

()

RESOLVED DUPLICATE of bug 934984
5 years ago
4 years ago

People

(Reporter: xavier114fch, Unassigned)

Tracking

Trunk
x86_64
Windows 8.1
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

5 years ago
Using VS 2013 Ultimate on Windows 8.1 x64. If -GL flag is added to --enable-optimize, libvpx fails to build with "Too many sections".

mozmake[5]: Entering directory 'e:/mozilla-inbound/obj/media/libvpx'
Too many sections
Makefile:433: recipe for target 'asm_enc_offsets.asm' failed
mozmake[5]: *** [asm_enc_offsets.asm] Error 1
mozmake[5]: *** Deleting file 'asm_enc_offsets.asm'
mozmake[5]: Leaving directory 'e:/mozilla-inbound/obj/media/libvpx'
e:/mozilla-inbound/config/recurse.mk:99: recipe for target 'media/libvpx/compile
' failed
mozmake[4]: *** [media/libvpx/compile] Error 2

However, with --disable-debug and --disable-debug-symbols in MOZCONFIG, libvpx can be built with -GL enabled, but the resulting build cannot be launched with crash reporter comes up.
(Reporter)

Comment 1

5 years ago
I have tried to build with -GL again and it still fails.

 2:21.28 mozmake.EXE[5]: Entering directory 'e:/mozilla-inbound/obj/media/libvpx
'
 2:21.45 Too many sections
 2:21.47 Makefile:113: recipe for target 'vp8_asm_enc_offsets.asm' failed
 2:21.47 mozmake.EXE[5]: *** [vp8_asm_enc_offsets.asm] Error 1
 2:21.47 mozmake.EXE[5]: *** Deleting file 'vp8_asm_enc_offsets.asm'
 2:21.47 mozmake.EXE[5]: Leaving directory 'e:/mozilla-inbound/obj/media/libvpx'

 2:21.47 e:/mozilla-inbound/config/recurse.mk:100: recipe for target 'media/libv
px/compile' failed
 2:21.47 mozmake.EXE[4]: *** [media/libvpx/compile] Error 2
 2:21.47 mozmake.EXE[4]: Leaving directory 'e:/mozilla-inbound/obj'
 2:21.48 e:/mozilla-inbound/config/recurse.mk:39: recipe for target 'compile' fa
iled
 2:21.48 mozmake.EXE[3]: *** [compile] Error 2
 2:21.48 mozmake.EXE[3]: Leaving directory 'e:/mozilla-inbound/obj'
 2:21.48 e:/mozilla-inbound/config/rules.mk:589: recipe for target 'default' fai
led
 2:21.48 mozmake.EXE[2]: *** [default] Error 2
 2:21.49 mozmake.EXE[2]: Leaving directory 'e:/mozilla-inbound/obj'
 2:21.49 e:/mozilla-inbound/client.mk:398: recipe for target 'realbuild' failed
 2:21.49 mozmake.EXE[1]: *** [realbuild] Error 2
 2:21.49 mozmake.EXE[1]: Leaving directory 'e:/mozilla-inbound'
 2:21.49 client.mk:185: recipe for target 'build' failed
 2:21.49 mozmake.EXE: *** [build] Error 2
 2:21.52 271 compiler warnings present.
2
Component: Build Config → Video/Audio
Flags: needinfo?(j)
Version: unspecified → Trunk

Comment 2

5 years ago
would help to see the actual command that fails.
could be mozmake problem or HOST_SOURCES (build/make/obj_int_extract.c) fails to compile.
Flags: needinfo?(j)

Comment 3

5 years ago
I have confirmed the problem when building Firefox 28.0 with VS2013 Update 1.
(Reporter)

Comment 4

5 years ago
I went to obj/media/libvpx and run mozmake, it gave the following.

mozmake[1]: Entering directory 'e:/mozilla-inbound/obj/media/libvpx'
./host_obj_int_extract.exe gas vp8_asm_enc_offsets.obj \
     > vp8_asm_enc_offsets.asm
Too many sections
Makefile:113: recipe for target 'vp8_asm_enc_offsets.asm' failed
mozmake[1]: *** [vp8_asm_enc_offsets.asm] Error 1
mozmake[1]: *** Deleting file 'vp8_asm_enc_offsets.asm'

Comment 5

5 years ago
Confirmed, unable to build the new libvpx (and as a result all later Firefox/SM/TB/etc.) on VS2012 also.

It seems to me what is done there is: an .obj is built, and then things are extracted from the .obj again with a custom-built tool. That's a little backwards, isn't it? Wouldn't that make things too dependent on what kind of object code a compiler/linker spits out (which may change between compiler versions or even updates of the same VS version)?

Output (from Seamonkey comm-release):

q:\sm\mozilla\config\rules.mk:982:0$ mozbuild.action.cl main  cl -Fohost_obj_int_extract.obj -c -TC
-nologo -Fdhost_obj_int_extract.pdb -DXP_WIN32 -DXP_WIN -DWIN32 -D_WIN32 -DNO_X11 -D_CRT_SECURE_NO_W
ARNINGS -O2 -fp:precise -arch:SSE2 -GFLTs -GS- -Qfast_transcendentals -Qpar -MD           -Iq:/sm/mo
zilla/media/libvpx -I.  -I../../dist/include  -Iq:/sm/build-optimized-suite/mozilla/dist/include/nsp
r -Iq:/sm/build-optimized-suite/mozilla/dist/include/nss     -Iq:/sm/build-optimized-suite/mozilla/d
ist/include/nspr q:/sm/mozilla/media/libvpx/build/make/obj_int_extract.c
obj_int_extract.c
host_obj_int_extract.exe
q:\sm\mozilla\config\rules.mk:778:0$ q:/sm/build-optimized-suite/mozilla/_virtualenv/Scripts/python.
exe q:/sm/mozilla/config/expandlibs_exec.py --depend .deps/host_obj_int_extract.exe.pp --target host
_obj_int_extract.exe -- link -NOLOGO -OUT:host_obj_int_extract.exe -PDB:host_obj_int_extract.pdb hos
t_obj_int_extract.obj  -MACHINE:X86
host_obj_int_extract.obj : MSIL .netmodule or module compiled with /GL found; restarting link with /
LTCG; add /LTCG to the link command line to improve linker performance
Generating code
Finished generating code
q:\sm\mozilla\config\rules.mk:1506:0$ nsinstall nsinstall -t -m 755 'host_obj_int_extract.exe' '../.
./dist/host/bin'
q:\sm\build-optimized-suite\mozilla\media\libvpx\Makefile:150:0$ ./host_obj_int_extract.exe gas vp8_
asm_enc_offsets.obj \
     > vp8_asm_enc_offsets.asm
Too many sections
q:\sm\build-optimized-suite\mozilla\media\libvpx\Makefile:150:0: command './host_obj_int_extract.exe
 gas vp8_asm_enc_offsets.obj \
     > vp8_asm_enc_offsets.asm' failed, return code 1
<quantize_sse4.obj>: Found error
<quantize_ssse3.obj>: Found error
<media_libvpx.lib.desc>: Found error
<media_libvpx.lib.desc>: Found error
<libs>: Found error
evaluation from q:\sm\mozilla\config\recurse.mk:189:53:7:0: command 'e:/mozilla-build/python/python.
exe q:/sm/mozilla/build/pymake/pymake/../make.py -C ../../media/libvpx libs' failed, return code 2
evaluation from q:\sm\mozilla\config\recurse.mk:189:53:12:0: command 'e:/mozilla-build/python/python
.exe q:/sm/mozilla/build/pymake/pymake/../make.py ../../db/sqlite3/src_libs ../../media/libjpeg_libs
 ../../media/libvorbis_libs ../../media/libopus_libs ../../media/libnestegg_libs ../../media/libvpx_
libs ../../media/libogg_libs ../../media/libtheora_libs ../../media/libpng_libs ../../media/kiss_fft
_libs ../../media/libcubeb_libs ../../media/libspeex_resampler_libs ../../media/libsoundtouch_libs'
failed, return code 2
q:\sm\mozilla\config\recurse.mk:162:0: command 'e:/mozilla-build/python/python.exe q:/sm/mozilla/bui
ld/pymake/pymake/../make.py -C config/external libs' failed, return code 2
q:\sm\mozilla\config\rules.mk:602:0: command 'e:/mozilla-build/python/python.exe q:/sm/mozilla/build
/pymake/pymake/../make.py libs' failed, return code 2
q:\sm\build-optimized-suite\Makefile:52:0: command 'e:/mozilla-build/python/python.exe q:/sm/mozilla
/build/pymake/pymake/../make.py -C mozilla default' failed, return code 2
q:\sm\client.mk:391:0: command 'e:/mozilla-build/python/python.exe q:/sm/mozilla/build/pymake/pymake
/../make.py -j4 -C q:/sm/build-optimized-suite' failed, return code 2
(In reply to Mark Straver from comment #5)
> It seems to me what is done there is: an .obj is built, and then things are
> extracted from the .obj again with a custom-built tool. That's a little
> backwards, isn't it? Wouldn't that make things too dependent on what kind of
> object code a compiler/linker spits out (which may change between compiler
> versions or even updates of the same VS version)?

The COFF/PE format has been used by Windows since NT 3.1 (that's before Windows 95, or over two decades ago now).

media/libvpx/build/make/obj_int_extract.c:624 contains a check that the file has no more than 96 sections, matching an old limit of the Windows image loader. If you remove that check, does the build succeed?

Comment 7

5 years ago
(In reply to Timothy B. Terriberry (:derf) from comment #6)
> The COFF/PE format has been used by Windows since NT 3.1 (that's before
> Windows 95, or over two decades ago now).

I'm aware of that, but can you reliably trust object code as a source of extracted data? Obviously, as seen in this bug, the number of sections differs depending on compile flags (and compiler version, since the VS version used for official builds doesn't run into this build bustage)...

> media/libvpx/build/make/obj_int_extract.c:624 contains a check that the file
> has no more than 96 sections, matching an old limit of the Windows image
> loader. If you remove that check, does the build succeed?

I'll give that a try and report back. Would there be a potential problem when more than 96 sections are present, or is that simply a limit for loading the object file to extract data from that no longer matters?

Comment 8

5 years ago
Removing the check makes the tool crash when actually used:

Problem signature:
  Problem Event Name:	APPCRASH
  Application Name:	host_obj_int_extract.exe
  Application Version:	0.0.0.0
  Application Timestamp:	53296056
  Fault Module Name:	MSVCR110.dll
  Fault Module Version:	11.0.51106.1
  Fault Module Timestamp:	5098858e
  Exception Code:	c0000005
  Exception Offset:	000190ff
  OS Version:	6.1.7601.2.1.0.256.1
  Locale ID:	2057
  Additional Information 1:	0a9e
  Additional Information 2:	0a9e372d3b4ad19135b953a78882e789
  Additional Information 3:	0a9e
  Additional Information 4:	0a9e372d3b4ad19135b953a78882e789


I did a quick check and MS documentation (from 2013) still states 96 is the limit. There's no indication this has changed, so 96 is the limit for the image loader. Looks like you may have to change your approach to what this was written for.

See http://msdn.microsoft.com/library/windows/hardware/gg463125

output from build window:
host_obj_int_extract.exe
q:\sm\mozilla\config\rules.mk:778:0$ q:/sm/build-optimized-suite/mozilla/_virtualenv/Scripts/python.
exe q:/sm/mozilla/config/expandlibs_exec.py --depend .deps/host_obj_int_extract.exe.pp --target host
_obj_int_extract.exe -- link -NOLOGO -OUT:host_obj_int_extract.exe -PDB:host_obj_int_extract.pdb hos
t_obj_int_extract.obj  -MACHINE:X86
host_obj_int_extract.obj : MSIL .netmodule or module compiled with /GL found; restarting link with /
LTCG; add /LTCG to the link command line to improve linker performance
Generating code
Finished generating code
q:\sm\mozilla\config\rules.mk:1506:0$ nsinstall nsinstall -t -m 755 'host_obj_int_extract.exe' '../.
./dist/host/bin'
q:\sm\build-optimized-suite\mozilla\media\libvpx\Makefile:152:0$ ./host_obj_int_extract.exe gas vp8_
asm_enc_offsets.obj \
     > vp8_asm_enc_offsets.asm
*** CRASH ***
q:\sm\build-optimized-suite\mozilla\media\libvpx\Makefile:152:0: command './host_obj_int_extract.exe
 gas vp8_asm_enc_offsets.obj \
     > vp8_asm_enc_offsets.asm' failed, return code 5
<quantize_sse4.obj>: Found error
<quantize_ssse3.obj>: Found error
<media_libvpx.lib.desc>: Found error
<media_libvpx.lib.desc>: Found error
<libs>: Found error
evaluation from q:\sm\mozilla\config\recurse.mk:189:53:7:0: command 'e:/mozilla-build/python/python.
exe q:/sm/mozilla/build/pymake/pymake/../make.py -C ../../media/libvpx libs' failed, return code 2
evaluation from q:\sm\mozilla\config\recurse.mk:189:53:12:0: command 'e:/mozilla-build/python/python
.exe q:/sm/mozilla/build/pymake/pymake/../make.py ../../db/sqlite3/src_libs ../../media/libjpeg_libs
 ../../media/libvorbis_libs ../../media/libopus_libs ../../media/libnestegg_libs ../../media/libvpx_
libs ../../media/libogg_libs ../../media/libtheora_libs ../../media/libpng_libs ../../media/kiss_fft
_libs ../../media/libcubeb_libs ../../media/libspeex_resampler_libs ../../media/libsoundtouch_libs'
failed, return code 2
q:\sm\mozilla\config\recurse.mk:162:0: command 'e:/mozilla-build/python/python.exe q:/sm/mozilla/bui
ld/pymake/pymake/../make.py -C config/external libs' failed, return code 2
q:\sm\mozilla\config\rules.mk:602:0: command 'e:/mozilla-build/python/python.exe q:/sm/mozilla/build
/pymake/pymake/../make.py libs' failed, return code 2
q:\sm\build-optimized-suite\Makefile:52:0: command 'e:/mozilla-build/python/python.exe q:/sm/mozilla
/build/pymake/pymake/../make.py -C mozilla default' failed, return code 2
q:\sm\client.mk:391:0: command 'e:/mozilla-build/python/python.exe q:/sm/mozilla/build/pymake/pymake
/../make.py -j4 -C q:/sm/build-optimized-suite' failed, return code 2
(Reporter)

Comment 9

5 years ago
I have searched the Dear Web and it sounds like RyanVM has bumped into this on IRC earlier.
http://krijnhoetmer.nl/irc-logs/developers/20131105 (search for "host_obj_int_extract.exe")

And he has already filed Bug 934984 to turn off optimization for this part as PGO implies -GL.
Flags: needinfo?(ryanvm)
Why am I being needinfo'd here?
Flags: needinfo?(ryanvm)
Status: UNCONFIRMED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 934984
You need to log in before you can comment on or make changes to this bug.