791442 - Mac trunk build with clang and XCode 4.1 failing

Status: RESOLVED FIXED

(SeaMonkey :: Release Engineering, defect)

Description

[Justin Wood (:Callek)]

Hey Rafael, 

So SeaMonkey just updated one of our systems to XCode 4.1, OSX 10.6.8 (from OSX 10.6.2) and uninstalled XCode 3.x in order to be able to build with clang w/ tooltool.

All that said, the system was able to build current-beta just fine (used gcc there), but when it went to build trunk had a weird error. I can't see code-wise how it could get this, but it was a complete clobber.

I'm looking for any insight you can provide, along with the offer of us handing you this machine to reproduce.

rev's:
 m-c: http://hg.mozilla.org/mozilla-central/rev/e5af3d785252ceb68a9ea3033c6e4a7cbdf3a010
 c-c: http://hg.mozilla.org/comm-central/rev/4fac45745a7326a79390c09048214a35ea96b8c5

I set a dep of Bug 783154 which is where this code was added. I do not rule out a system issue, but want to figure this out asap.

/builds/slave/comm-cen-trunk-osx64/build/objdir/i386/mozilla/_virtualenv/bin/python /builds/slave/comm-cen-trunk-osx64/build/mozilla/config/pythonpath.py -I../../config /builds/slave/comm-cen-trunk-osx64/build/mozilla/config/expandlibs_exec.py --depend .deps/XUL.pp --target XUL --uselist --  /usr/local/bin/ccache /builds/slave/comm-cen-trunk-osx64/build/mozilla/../clang/bin/clang++ -arch i386 -Qunused-arguments  -Qunused-arguments -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Wtype-limits -Wempty-body -Wno-ctor-dtor-privacy -Wno-overlength-strings -Wno-invalid-offsetof -Wno-variadic-macros -Wno-c++0x-extensions -Wno-extended-offsetof -Wno-unknown-warning-option -Wno-return-type-c-linkage -Wno-mismatched-tags -gdwarf-2 -isysroot /Developer/SDKs/MacOSX10.6.sdk -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -std=gnu++0x -pthread -DNO_X11 -pipe  -DNDEBUG -DTRIMMED -gdwarf-2 -O3 -fomit-frame-pointer -fPIC  -o XUL  nsStaticXULComponents.o nsUnicharUtils.o nsBidiUtils.o nsSpecialCasingData.o nsUnicodeProperties.o nsRDFResource.o    -framework Cocoa -lobjc  -framework ExceptionHandling -Wl,-executable_path,/builds/slave/comm-cen-trunk-osx64/build/objdir/i386/mozilla/dist/bin -Wl,-dead_strip  ../../toolkit/components/osfile/libosfile_s.a ../../toolkit/xre/libxulapp_s.a  ../../staticlib/components/libnecko.a ../../staticlib/components/libuconv.a ../../staticlib/components/libi18n.a ../../staticlib/components/libchardet.a ../../staticlib/components/libjar50.a ../../staticlib/components/libstartupcache.a ../../staticlib/components/libpref.a ../../staticlib/components/libhtmlpars.a ../../staticlib/components/libidentity.a ../../staticlib/components/libimglib2.a ../../staticlib/components/libmediasniffer.a ../../staticlib/components/libgkgfx.a ../../staticlib/components/libgklayout.a ../../staticlib/components/libdocshell.a ../../staticlib/components/libembedcomponents.a ../../staticlib/components/libwebbrwsr.a ../../staticlib/components/libnsappshell.a ../../staticlib/components/libtxmgr.a ../../staticlib/components/libcommandlines.a ../../staticlib/components/libtoolkitcomps.a ../../staticlib/components/libpipboot.a ../../staticlib/components/libpipnss.a ../../staticlib/components/libappcomps.a ../../staticlib/components/libjsreflect.a ../../staticlib/components/libcomposer.a ../../staticlib/components/libtelemetry.a ../../staticlib/components/libjsinspector.a ../../staticlib/components/libjsdebugger.a ../../staticlib/components/libstoragecomps.a ../../staticlib/components/librdf.a ../../staticlib/components/libwindowds.a ../../staticlib/components/libjsctypes.a ../../staticlib/components/libjsperf.a ../../staticlib/components/libgkplugin.a ../../staticlib/components/libosxproxy.a ../../staticlib/components/libjsd.a ../../staticlib/components/libautoconfig.a ../../staticlib/components/libauth.a ../../staticlib/components/libcookie.a ../../staticlib/components/libpermissions.a ../../staticlib/components/libuniversalchardet.a ../../staticlib/components/libplaces.a ../../staticlib/components/libtkautocomplete.a ../../staticlib/components/libsatchel.a ../../staticlib/components/libpippki.a ../../staticlib/components/libimgicon.a ../../staticlib/components/libprofiler.a ../../staticlib/components/libwidget_mac.a ../../staticlib/components/libaccessibility.a ../../staticlib/components/libspellchecker.a ../../staticlib/components/libzipwriter.a ../../staticlib/components/libservices-crypto.a ../../staticlib/components/libxpautocomplete.a ../../staticlib/components/libmail.a ../../staticlib/components/libmsgsmime.a ../../staticlib/components/libimport.a ../../staticlib/components/libmozldap.a ../../staticlib/components/libmork.a ../../staticlib/libjsipc_s.a ../../staticlib/libdomipc_s.a ../../staticlib/libdomplugins_s.a ../../staticlib/libmozipc_s.a ../../staticlib/libmozipdlgen_s.a ../../staticlib/libipcshell_s.a ../../staticlib/libgfxipc_s.a ../../staticlib/libhal_s.a ../../staticlib/libdombindings_s.a ../../staticlib/libxpcom_core.a ../../staticlib/libucvutil_s.a ../../staticlib/libchromium_s.a ../../staticlib/libsnappy_s.a ../../staticlib/libthebes.a ../../staticlib/libgl.a ../../staticlib/libycbcr.a  -L../../dist/bin -L../../dist/lib /builds/slave/comm-cen-trunk-osx64/build/objdir/i386/mozilla/dist/lib/libjs_static.a -L../../dist/bin -L../../dist/lib -lcrmf -lsmime3 -lssl3 -lnss3 -lnssutil3  -L../../dist/bin -L../../dist/lib -lldap60 -lprldap60 -lldif60 ../../dist/lib/libmozsqlite3.a  /builds/slave/comm-cen-trunk-osx64/build/objdir/i386/mozilla/modules/zlib/src/libmozz.a ../../dist/lib/libgkmedias.a -L../../dist/bin -L../../dist/lib  -L/builds/slave/comm-cen-trunk-osx64/build/objdir/i386/mozilla/dist/lib -lnspr4 -lplc4 -lplds4 ../../dist/lib/libmozalloc.a -dynamiclib -install_name @executable_path/XUL -compatibility_version 1 -current_version 1 -single_module -L/builds/slave/comm-cen-trunk-osx64/build/objdir/i386/mozilla/dist/lib -lmozglue  -framework OpenGL -lcups -framework SystemConfiguration -framework QTKit -framework IOKit -F/System/Library/PrivateFrameworks -framework CoreUI -framework QuartzCore -framework Carbon -framework CoreAudio -framework AudioToolbox -framework AudioUnit -framework AddressBook -framework OpenGL  -framework Carbon -framework CoreAudio -framework AudioToolbox -framework AudioUnit -framework IOKit -framework Foundation -framework AppKit -framework Security   ../../build/unix/stdc++compat/libstdc++compat.a  
Undefined symbols for architecture i386:
  "mozilla::Telemetry::GetStackAndModules(std::vector<unsigned int, std::allocator<unsigned int> > const&, bool)", referenced from:
      (anonymous namespace)::ValidWriteAssert(bool) in mozPoisonWriteMac.o
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[6]: *** [XUL] Error 1

Comment 1

[Justin Wood (:Callek)]

Full log can be found at http://tbpl.drapostles.org/?rev=4fac45745a73  note that -02 is the only system upgraded this way so far.

Comment 2

[Justin Wood (:Callek)]

With manual bisection I found that https://hg.mozilla.org/mozilla-central/rev/3047a182724d is the exact rev that breaks us (as expected based on the error) and https://hg.mozilla.org/mozilla-central/rev/492ef59eb268 works fine (one cset earlier).

With staring at code alone, I can't come up with any reason we'd not be able to find the symbol though.

Comment 3

[Justin Wood (:Callek)]

Ok, further analysis, still clueless as to how to fix:

cb-sea-miniosx64-02:telemetry seabld$ pwd
/builds/slave/comm-cen-trunk-osx64/build/objdir/i386/mozilla/toolkit/components/telemetry
cb-sea-miniosx64-02:telemetry seabld$ nm Telemetry.o | grep GetStack
00000d10 T __ZN7mozilla9Telemetry18GetStackAndModulesERKSt6vectorImSaImEEb
00000a50 T __ZNK7mozilla9Telemetry14ProcessedStack12GetStackSizeEv
cb-sea-miniosx64-02:telemetry seabld$ nm Telemetry.o | c++filt -p -i | grep GetStack
00000d10 T mozilla::Telemetry::GetStackAndModules
00000a50 T mozilla::Telemetry::ProcessedStack::GetStackSize
cb-sea-miniosx64-02:telemetry seabld$ ls
Makefile                        TelemetryHistogramEnums.h       tests
Telemetry.o                     _xpidlgen
TelemetryHistogramData.inc      libtelemetry.a.desc
cb-sea-miniosx64-02:telemetry seabld$ ls ../../../staticlib/components/libtelemetry.a.desc
../../../staticlib/components/libtelemetry.a.desc
cb-sea-miniosx64-02:telemetry seabld$ cat ../../../staticlib/components/libtelemetry.a.desc
OBJS = /builds/slave/comm-cen-trunk-osx64/build/objdir/i386/mozilla/toolkit/components/telemetry/Telemetry.o
cb-sea-miniosx64-02:telemetry seabld$ ls /builds/slave/comm-cen-trunk-osx64/build/objdir/i386/mozilla/toolkit/co
mponents/telemetry/Telemetry.o
/builds/slave/comm-cen-trunk-osx64/build/objdir/i386/mozilla/toolkit/components/telemetry/Telemetry.o

Comment 4

[Justin Wood (:Callek)]

realizing that the func args matter:

cb-sea-miniosx64-02:telemetry seabld$ nm Telemetry.o | c++filt | grep GetStack
00000d10 T mozilla::Telemetry::GetStackAndModules(std::vector<unsigned long, std::allocator<unsigned long> > const&, bool)
00000a50 T mozilla::Telemetry::ProcessedStack::GetStackSize() const

cb-sea-miniosx64-02:build seabld$ nm mozPoisonWriteMac.o | c++filt | grep GetStack
         U mozilla::Telemetry::GetStackAndModules(std::vector<unsigned int, std::allocator<unsigned int> > const&, bool)
         U mozilla::Telemetry::ProcessedStack::GetStackSize() const

Comment 5

[Justin Wood (:Callek)]

Comparing against a Firefox build for the compile of mozPoisonWriteMac.cpp differences in compile command line:

Only in SeaMonkey Copy
-DMOZ_SUITE=1
-gdwarf-2  (twice)
-fomit-frame-pointer

Only in Firefox Copy
-g
-fno-omit-frame-pointer

Comparing against a Firefox build for the compile of Telemetry.cpp:

Only in SeaMonkey Copy
-DMOZ_SUITE=1
-gdwarf-2  (twice)
-fomit-frame-pointer

Only in Firefox Copy
-g
-fno-omit-frame-pointer

Comment 6

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #5)
> Comparing against a Firefox build for the compile of Telemetry.cpp:
> 
> Only in SeaMonkey Copy
> -fomit-frame-pointer
> 
> Only in Firefox Copy
> -fno-omit-frame-pointer

For clarity, I did a test Try build of Firefox with -fomit-frame-pointer (by disabling --enabe-profiling) and no build error.

Comment 7

[Justin Wood (:Callek)]

So with Neil[Away]'s urging, answering "where is uintptr_t defined"

make -C $(objdir)/toolkit/components/telemetry Telemetry.i

then a bit of manual grepping of the file:
# 109 "/Developer/SDKs/MacOSX10.6.sdk/usr/include/i386/types.h" 3 4
typedef unsigned long uintptr_t;

and

make -C $(objdir)/mozilla/xpcom/build mozPoisonWriteMac.i
# 176 "/builds/slave/comm-cen-trunk-osx64/build/clang/bin/../lib/clang/3.2/include/stdint.h" 3 4
...
typedef uint32_t uintptr_t;

Comment 8

[Justin Wood (:Callek)]

soooo.... failed a Firefox build on this same system with this same ver of clang....

cb-sea-miniosx64-02:build seabld$ env
MOZ_CRASHREPORTER_NO_REPORT=1
TERM=cygwin
SHELL=/bin/bash
SSH_CLIENT=<<<MANUALLY_STRIPPED_BY_CALLEK>>>
OLDPWD=/builds/slave/m-cen-osx64
MOZ_SYMBOLS_EXTRA_BUILDID=macosx64
MOZ_OBJDIR=objdir
SSH_TTY=/dev/ttys000
USER=seabld
CCACHE_DIR=/builds/ccache
SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_sea/
CCACHE_COMPRESS=1
mount_authenticator=
MAIL=/var/mail/seabld
PATH=/tools/buildbot/bin:/tools/python/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/b
in
PWD=/builds/slave/m-cen-osx64/build
SYMBOL_SERVER_SSH_KEY=/Users/seabld/.ssh/seabld_dsa
CCACHE_BASEDIR=/builds/slave/m-cen-osx64
SHLVL=1
HOME=/Users/seabld
TINDERBOX_OUTPUT=1
PYTHONPATH=/tools/buildbot/lib/python2.6/site-packages:/tools/twisted/lib/python2.6/site-packages/:/tools/twisted
-core/lib/python2.6/site-packages:/tools/zope-interface/lib/python2.6/site-packages
LOGNAME=seabld
CVS_RSH=ssh
SSH_CONNECTION=<<<MANUALLY_STRIPPED_BY_CALLEK>>>
SYMBOL_SERVER_HOST=symbolpush.mozilla.org
SYMBOL_SERVER_USER=seabld
_=/usr/bin/env


Firefox:
cb-sea-miniosx64-02:build seabld$ make -C objdir/i386/toolkit/components/telemetry Telemetry.i
--> http://people.mozilla.org/~jwood/bug_791442/fx_Telemetry.i

cb-sea-miniosx64-02:build seabld$ make -C objdir/i386/xpcom/build mozPoisonWriteMac.i
--> http://people.mozilla.org/~jwood/bug_791442/fx_mozPoisonWriteMac.i

SeaMonkey Copies:
--> http://people.mozilla.org/~jwood/bug_791442/sm_Telemetry.i
--> http://people.mozilla.org/~jwood/bug_791442/sm_mozPoisonWriteMac.i

Comment 9

[Justin Wood (:Callek)]

After remembering about the make target |showbuild| I ran that in these two directories (for firefox) as well... to help us narrow this down, since that also shows env vars grabbed by configure/the-build.

http://people.mozilla.org/~jwood/bug_791442/fx_toolkit-components-telemetry-showbuild.txt
http://people.mozilla.org/~jwood/bug_791442/fx_xpcom-build-showbuild.txt

I do not have more time tonight to investigate, so any/all help welcome

Comment 10

[[e-mail]]

(In reply to Justin Wood from comment #8)
> http://people.mozilla.org/~jwood/bug_791442/fx_Telemetry.i
Telemetry.cpp includes <algorithm> and the compiler finds
/Developer/SDKs/MacOSX10.6.sdk/usr/include/c++/4.2.1/algorithm
which triggers the include of types.h that makes uintptr_t unsigned long.

> http://people.mozilla.org/~jwood/bug_791442/fx_mozPoisonWriteMac.i
mozPoisonWriteMac.cpp includes mozilla/mozPoisonWriteMac.h includes mozilla/Types.h includes mozilla/StandardInteger.h which includes stdint.h
Interesting comment in clang's stdint.h:
/* prevent glibc sys/types.h from defining conflicting types */

Comment 11

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

(In reply to Justin Wood (:Callek) from comment #8)
> soooo.... failed a Firefox build on this same system with this same ver of
> clang....
>
 
Can you provide the .mozconfig?

Comment 12

[Justin Wood (:Callek)]

(In reply to Rafael Ávila de Espíndola (:espindola) from comment #11)
> (In reply to Justin Wood (:Callek) from comment #8)
> > soooo.... failed a Firefox build on this same system with this same ver of
> > clang....
> >
>  
> Can you provide the .mozconfig?

http://hg.mozilla.org/mozilla-central/file/e5af3d785252/browser/config/mozconfigs/macosx-universal/nightly

Comment 13

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

> > Can you provide the .mozconfig?
> 
> http://hg.mozilla.org/mozilla-central/file/e5af3d785252/browser/config/
> mozconfigs/macosx-universal/nightly

Works on try:

https://tbpl.mozilla.org/php/getParsedLog.php?id=15338048&tree=Try&full=1

something broken with your setup.

Comment 14

[Philip Chee]

> something broken with your setup.
Could you be less specific?

Comment 15

[Justin Wood (:Callek)]

After deeper investigation, it looks like we have files in the SDK that the MoCo machine doesn't, and vice versa. Even though we BOTH have the same XCode ver, just "for different versions of OSX". Which is very puzzling, but I guess this is not a common Apple use-case.

I've filed a bug to get the new macs installed with OSX 10.7, and this one overwrote back to a "clean" 10.6.2

For reference, a full list of files in /Developer sorted by md5 is at http://people.mozilla.org/~jwood/bug_791442/sea_vs_moco_dev_files.diff

Comment 17

[Frank Wein [:mcsmurf]]

Just for reference: The latest build issue on the new OS X builders looks like Bug 805091. The latest comment there says the bug has been fixed by the virtualenv.py upgrade (not for us), but that there is still open. So not sure about the status of that bug.

Comment 18

[Justin Wood (:Callek)]

Fixed with a few followup issues remaining, we migrated to the rev4 mini's we have with OSX10.7 instead of using OSX10.6 r3's.