Closed Bug 655339 Opened 13 years ago Closed 13 years ago

[10.7] error: expected initializer before ‘NS_ATTR_MALLOC’

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox6 affected, firefox7 affected)

Status:
RESOLVED FIXED
Milestone:
mozilla8
Tracking Flags:
Tracking Status
firefox6 --- affected
firefox7 --- affected

People

(Reporter: Nomis101, Assigned: smichaud)

References

(Blocks 1 open bug)

Details

(Whiteboard: rdar://9463556)

Attachments

(2 files, 1 obsolete file)

Patch for Gnu grep 2.5.1 (*not* a fix for this bug)
4.92 KB, patch
Details | Diff | Splinter Review
Follow Ted's suggestion
2.70 KB, patch
smichaud
: review+
Details | Diff | Splinter Review 
If I build FF trunk or TB trunk on 10.7 it stops with an error. The error is the same, witch 10.7 SDK and with 10.6 SDK. 
OS X 10.7 11A444d, Xcode 4.1 4B47f

The error is:

In file included from /Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc_oom.cpp:45:
../../dist/include/mozilla/mozalloc_abort.h:61: error: expected initializer before ‘NS_NORETURN’
/Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc_oom.cpp: In function ‘void mozalloc_handle_oom()’:
/Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc_oom.cpp:54: error: ‘mozalloc_abort’ was not declared in this scope
In file included from /Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc_abort.cpp:56:
../../dist/include/mozilla/mozalloc_abort.h:61: error: expected initializer before ‘NS_NORETURN’
/Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc_abort.cpp: In function ‘void mozalloc_abort(const char*)’:
/Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc_abort.cpp:88: error: ‘_exit’ was not declared in this scope
make[6]: *** [mozalloc_abort.o] Error 1
make[6]: *** Waiting for unfinished jobs....
make[6]: *** [mozalloc_oom.o] Error 1
In file included from /Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc.cpp:67:
../../dist/include/mozilla/mozalloc.h:109: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:113: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:117: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:120: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:124: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:127: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:131: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:134: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h: In function ‘void* operator new(size_t)’:
../../dist/include/mozilla/mozalloc.h:229: error: ‘moz_xmalloc’ was not declared in this scope
../../dist/include/mozilla/mozalloc.h: In function ‘void* operator new(size_t, const std::nothrow_t&)’:
../../dist/include/mozilla/mozalloc.h:235: error: ‘moz_malloc’ was not declared in this scope
../../dist/include/mozilla/mozalloc.h: In function ‘void* operator new [](size_t)’:
../../dist/include/mozilla/mozalloc.h:241: error: ‘moz_xmalloc’ was not declared in this scope
../../dist/include/mozilla/mozalloc.h: In function ‘void* operator new [](size_t, const std::nothrow_t&)’:
../../dist/include/mozilla/mozalloc.h:247: error: ‘moz_malloc’ was not declared in this scope
../../dist/include/mozilla/mozalloc.h: In function ‘void* operator new(size_t, const mozilla::fallible_t&)’:
../../dist/include/mozilla/mozalloc.h:303: error: ‘moz_malloc’ was not declared in this scope
../../dist/include/mozilla/mozalloc.h: In function ‘void* operator new [](size_t, const mozilla::fallible_t&)’:
../../dist/include/mozilla/mozalloc.h:309: error: ‘moz_malloc’ was not declared in this scope
make[6]: *** [mozalloc.o] Error 1
make[5]: *** [libs] Error 2
make[4]: *** [libs_tier_base] Error 2
make[3]: *** [tier_base] Error 2
make[2]: *** [default] Error 2
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2
Does your $objdir/dist/include/mozilla-config.h #define these macros?  Do you have other patches applied when you see these build errors?
(In reply to comment #1)
> Does your $objdir/dist/include/mozilla-config.h #define these macros?
This file is empty, it only includes this three lines:

#ifndef _MOZILLA_CONFIG_H_
#define _MOZILLA_CONFIG_H_

#endif /* _MOZILLA_CONFIG_H_ */

> you have other patches applied when you see these build errors?
Today I've tried it with fresh code with no patches. Same result.
(In reply to comment #2)
> (In reply to comment #1)
> > Does your $objdir/dist/include/mozilla-config.h #define these macros?
> This file is empty, it only includes this three lines:
> 
> #ifndef _MOZILLA_CONFIG_H_
> #define _MOZILLA_CONFIG_H_
> 
> #endif /* _MOZILLA_CONFIG_H_ */

That's bad.  Are you following these --- https://developer.mozilla.org/En/Simple_Firefox_build --- build instructions?

I'm pretty sure FF has been built on 10.7; maybe you need some patches that aren't in 10.7.  CC'ing folks who would know better.
I haven't tried to build FF on 10.7 yet. Too lazy and too little reason to install the requirements like autoconf213, libidl, hg, and yasm manually since MacPorts doesn't work on 10.7 yet. I'll probably have to figure it out in the next couple of weeks.

Also, you won't be able to make a proper release build on 10.7 since Xcode 4 doesn't support the 10.5 SDK and targeting the OS version isn't exactly the same (even if it should be in theory).
I haven't yet tried to build on 10.7, either -- out of laziness, and having (I think) more urgent things to do.

But I don't use MacPorts.  So I may end up trying to build on 10.7 before the new MacPorts comes out.  If/when I do, I'll comment here.

You do realize, I hope, that without MacPorts you'll need to install all the build requirements manually.
(In reply to comment #3)
> That's bad.  Are you following these ---
> https://developer.mozilla.org/En/Simple_Firefox_build --- build instructions?
Yes, I did it like it works on 10.6. And with a very simple and basic mozconfig.

(In reply to comment #4)
> I haven't tried to build FF on 10.7 yet. Too lazy and too little reason to
> install the requirements like autoconf213, libidl, hg, and yasm manually
> since MacPorts doesn't work on 10.7 yet. I'll probably have to figure it out
> in the next couple of weeks.
I've updated a copy of 10.6 to 10.7.  So everything of the requirements are still installed from 10.6.

But I remember that I was able to build FF and TB with the first DP of 10.7 (so I was able to fill Bug 643150). So the build must be broken by any of the updates.
In a somewhat masochistic mood, I've started trying to build the
various requirements (and prerequirements) for the Mozilla tree by
hand on the latest Lion DP (build 11A444d).

The very first component I tried to build, gettext, failed to build
with an utterly inexplicable error.  I got the same error building
both an older gettext distro (gettext-0.17) and the current distro
(gettext-0.18.1.1).

I strongly suspect this is due to a bug in the gcc 4.2.1 that's
included with build 11A444d:

$ gcc --version
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.14.00)
Copyright (C) 2007 Free Software Foundation, Inc.

Please try building one of those gettext distros, and see if you get
the same error (or one that seems close).  If you do, I'd bet the
error you report here is also due to a bug in build 11A444d's gcc
4.2.1.

Here's the error I got building gettext-0.18.1.1:

stpncpy.c:34: error: expected declaration specifiers or ‘...’ before numeric constant
stpncpy.c:34: error: expected ‘)’ before ‘!=’ token
stpncpy.c:34: error: expected ‘)’ before ‘?’ token
stpncpy.c:34: error: expected declaration specifiers or ‘...’ before numeric constant
stpncpy.c:34: error: expected ‘)’ before ‘!=’ token
stpncpy.c:34: error: expected ‘)’ before ‘?’ token
I get the same error building gettext-0.17 on the previous Lion DP
(build 11A430e):

gcc --version
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.11.00)
Copyright (C) 2007 Free Software Foundation, Inc.

I no longer have a copy of the first DP (build 11A390) ... though I
suppose I could reinstall it.
Actually, my errors are no longer so inexplicable -- it looks like the offending file redefined stpncpy.  I wish the error message had *said* this.

Sigh.
(In reply to comment #7)
> Please try building one of those gettext distros, and see if you get
> the same error (or one that seems close).  If you do, I'd bet the
> error you report here is also due to a bug in build 11A444d's gcc
> 4.2.1.

OK, I will try this after I have installed the new 10.7 and Xcode Update. Maybe this will fix some problems.
What I can say is, that I get the same error (from my initital post) with Apples gcc 4.2.1 and with Clang 2.9.
>> Please try building one of those gettext distros, and see if you
>> get the same error (or one that seems close).  If you do, I'd bet
>> the error you report here is also due to a bug in build 11A444d's
>> gcc 4.2.1.
>
> OK, I will try this ...

You don't need to bother.  What I first thought was a spurious error
turns out to be a real one (though the error messages were misleading
and mostly wrong).  So this isn't a gcc 4.2.1 (or clang) bug (aside
from the uselessness of the error messages).

Do install the appropriate XCode update for your DP, though, and let
us know if this makes a difference.  As I assume you're aware, each DP
seed has it's own (seperate) XCode download.

At some point I'll finish installing the build prerequisites by hand,
and be able to try a build myself.
Today I've also tried to build gettext 0.18.1.1 on 10.7 (11A459e) with Xcode 4.1 DP5. I get:

stpncpy.c:34: error: expected declaration specifiers or ‘...’ before numeric constant
stpncpy.c:34: error: expected ‘)’ before ‘!=’ token
stpncpy.c:34: error: expected ‘)’ before ‘?’ token
make[4]: *** [stpncpy.lo] Error 1
make[3]: *** [all] Error 2
make[2]: *** [all-recursive] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1


Than I've tried to build it with Clang 2.9 from http://llvm.org/releases/download.html#2.9 and I get a similar error:

stpncpy.c:34:1: error: expected parameter declarator
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:5: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
    ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
In file included from /usr/include/secure/_string.h:32:
/usr/include/secure/_common.h:38:63: note: instantiated from:
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                                                              ^
stpncpy.c:34:1: error: expected ')'
stpncpy.c:34:1: note: to match this '('
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:5: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
    ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
In file included from /usr/include/secure/_string.h:32:
/usr/include/secure/_common.h:38:54: note: instantiated from:
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                                                     ^
stpncpy.c:34:1: error: expected ')'
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:27: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
                          ^
stpncpy.c:34:1: note: to match this '('
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:4: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
   ^
stpncpy.c:34:1: error: expected ')'
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:111:4: note: instantiated from:
   ? __builtin___stpncpy_chk (dest, src, len, __darwin_obsz (dest))     \
   ^
stpncpy.c:34:1: note: to match this '('
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:3: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
  ^
stpncpy.c:34:1: error: conflicting types for '__builtin_object_size'
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:5: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
    ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
In file included from /usr/include/secure/_string.h:32:
/usr/include/secure/_common.h:38:32: note: instantiated from:
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                               ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:61:56: note: '__builtin_object_size' is a builtin
      with type 'unsigned long (void *, int)'
  return __builtin___memcpy_chk (__dest, __src, __len, __darwin_obsz0(__dest));
                                                       ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
In file included from /usr/include/secure/_string.h:32:
/usr/include/secure/_common.h:38:32: note: instantiated from:
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                               ^
stpncpy.c:34:1: error: definition of builtin function '__builtin_object_size'
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:5: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
    ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
In file included from /usr/include/secure/_string.h:32:
/usr/include/secure/_common.h:38:32: note: instantiated from:
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                               ^
stpncpy.c:39:7: error: use of undeclared identifier 'n'
  if (n >= 4)
      ^
stpncpy.c:41:19: error: use of undeclared identifier 'n'
      size_t n4 = n >> 2;
                  ^
stpncpy.c:45:16: error: use of undeclared identifier 'src'
          c = *src++;
               ^
stpncpy.c:49:16: error: use of undeclared identifier 'src'
          c = *src++;
               ^
stpncpy.c:53:16: error: use of undeclared identifier 'src'
          c = *src++;
               ^
stpncpy.c:57:16: error: use of undeclared identifier 'src'
          c = *src++;
               ^
stpncpy.c:64:7: error: use of undeclared identifier 'n'
      n -= dest - s;
      ^
stpncpy.c:69:3: error: use of undeclared identifier 'n'
  n &= 3;
  ^
stpncpy.c:70:7: error: use of undeclared identifier 'n'
  if (n == 0)
      ^
stpncpy.c:75:12: error: use of undeclared identifier 'src'
      c = *src++;
           ^
stpncpy.c:76:9: error: use of undeclared identifier 'n'
      --n;
        ^
stpncpy.c:80:11: error: use of undeclared identifier 'n'
      if (n == 0)
          ^
stpncpy.c:85:10: error: use of undeclared identifier 'n'
  while (n-- > 0)
         ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make[4]: *** [stpncpy.lo] Error 1
make[3]: *** [all] Error 2
make[2]: *** [all-recursive] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1
> At some point I'll finish installing the build prerequisites by
> hand, and be able to try a build myself.

I finished building all the prerequisites, then tried doing a
mozilla-central build ... and got exactly the same error that Nomis101
has reported.

I built on what's now the second-to-latest DP (build 11A444d), with
the appropriate XCode download also installed.  Later I'll try with
the current DP (build 11A459e), plus its own XCode download (I'm
downloading the bits now).  Assuming the error persists, next week
I'll start trying to figure out why it happens, and see what we can do
about it.

Installing the build prerequisites by hand is *not* straightforward.
But you can get a head start by downloading current Mac Ports source
distros, and checking 1) source changes they've made and 2) what
configure parameters they use.  At some point I'll document how I
build them ... but I may not have time for that for a while.

For the record, here are the prerequisites I installed, in the order I
installed them.  Except for autoconf, I always used the latest
versions.  (Autoconf 2.6.8 is needed for translating changes I make to
glib's configure.ac file.)

gettext
pkgconfig
autoconf 2.13
autoconf 2.68
glib
libIDL
docutils
mercurial
yasm
> autoconf 2.13
> autoconf 2.68

Just so that nobody asks:  I installed Autoconf 2.13 with '--program-suffix=213' and Autoconf 2.68 with '--program-suffix=268'.  The two different versions shouldn't get mixed up, or interfere with each other.
Today I've erased my Lion HD and installed everything new. Installing Macports and anything that is needed to build Mozilla on 10.7 seems easy.
First I've installed this unofficial version of MacPorts 1.9.2 http://blog.touchspot.at/2011/04/28/macports-1-9-2-for-mac-os-x-10-7-lion/ Than I've changed the macports.conf file, so that it builds x86_64 on 10.7 (don't know if this is necessary). After that I've installed everything in the way:
$ sudo port -v selfupdate
$ sudo port install libidl configure.cflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk" configure.cxxflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk"
$ sudo port install autoconf213 configure.cflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk" configure.cxxflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk"
$ sudo port install mercurial configure.cflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk" configure.cxxflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk"
$ sudo port install yasm configure.cflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk" configure.cxxflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk"

The only problem I've had was db46 (needed for mercurial), you need first to install the Java Runtime for 10.7. I've not tried to build FF with this new installed files for now.
This problem persists on the latest DP (build 11A459e).

I also have an empty mozilla-config.h file.

Looking through my build logs, I noticed two instances of the
following error around the time that mozilla-config.h gets built:

egrep: Regular expression too big

So for the hell of it I tried compiling and installing the current
(latest) version of GNU grep (which also includes fgrep and egrep) --
version 2.8.  And what do you know, my mozilla-config.h is no longer
empty, and the build gets a lot futher along before it errors out (on
something else that's probably unrelated)!

I haven't a clue why a newer version of egrep should make a
difference.  Boris and Ted, do you have any ideas?
Looks like we use egrep to build mozilla-config.h in configure:
http://mxr.mozilla.org/mozilla-central/source/configure.in#9197

and we build up a really big egrep pattern for some reason. (I didn't really analyze what it's doing there.)
I suspect the "egrep: Regular expression too big" error message is
spurious: OS X 10.5, 10.6 and 10.7 all have the same version (2.51) of
Gnu grep. And at the following link someone talks about making the
error go away by changing his LC_CTYPE from "de_AT.UTF-8" to "de_AT".

So maybe it's some kind of encoding issue. But playing with the
LC_CTYPE, LC_ALL, LC_MESSAGES and LANG environment variables before
building didn't make any difference ... at least not the changes I
tried.
Forgot to include the link mentioned in the first paragraph:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=136508
> Looks like we use egrep to build mozilla-config.h in configure:
> http://mxr.mozilla.org/mozilla-central/source/configure.in#9197
>
> and we build up a really big egrep pattern for some reason.

> I suspect the "egrep: Regular expression too big" error message is
> spurious

> maybe it's some kind of encoding issue

Some encodings take up more space than others.  So maybe it's both an
encoding issue and a real space issue.
This is a very puzzling bug.  But I can now say with some confidence
that it's ultimately caused by a bug in grep/egrep/fgrep 2.5.1 that
only surfaces when those programs are compiled are run in 64-bit mode.
As best I can tell, the bug has nothing to do with encodings.

A quick fix is to rename the original /usr/bin/egrep to something like
egrep-orig, then run the lipo command to create an egrep that's a
32-bit-mode thin binary:

$ sudo mv /usr/bin/egrep /usr/bin/egrep-orig
$ sudo lipo /usr/bin/egrep-orig -thin i386 -output /usr/bin/egrep

After this you can build the entire tree, without error.  (I'm not
able to reproduce the error I reported in comment #16.)
grep, egrep and fgrep are i386/x86_64 universal binaries on both OS X
10.6 and OS X 10.7.  But for some reason this only causes trouble on
10.7.  My guess is that only on 10.7 does our build process run
command-line utilities in 64-bit mode.  I'll try to write a simple
test utility to confirm or deny this.
> My guess is that only on 10.7 does our build process run
> command-line utilities in 64-bit mode.  I'll try to write a simple
> test utility to confirm or deny this.

My guess was wrong -- i386/x86_64 universal binaries run in 64-bit
mode when called from 'configure', on OS X 10.6, 10.6 and 10.7.  And
the i386/x86_64 universal egrep binary copied from 10.6 also works
fine.

So maybe the problem is with how the 64-bit part of the egrep binary
gets compiled on OS X 10.7.
i386/x86_64 universal binaries run in 64-bit mode when called from
'configure', on OS X *10.5*, 10.6 and 10.7.
> I can now say with some confidence that it's ultimately caused by a
> bug in grep/egrep/fgrep 2.5.1 that only surfaces when those programs
> are compiled are run in 64-bit mode.

I've now managed to track it down.  As with gettext and glib, OS X
10.7's version of gcc (gcc 4.2) seems picker than earlier versions (at
least than earlier Apple versions) -- it finds errors where previous
versions didn't, and gets confused by things that are actually errors,
but which it doesn't detect.  (Earlier versions also failed to detect
them, but weren't confused by them.)

In the case of Gnu grep 2.5.1, it's an undetected error:  There are a
lot of "#ifdef MBS_SUPPORT" blocks in src/dfa.c, src/search.c and
lib/regex.c.  On OS X, MBS_SUPPORT (== multi-byte character support)
is defined in dfa.c and search.c, but not in regex.c.  This last is a
bug, and means that different parts of grep (and of fgrep and egrep)
don't agree on whether or not multi-byte characters are supported.
This doesn't appear to cause any trouble when grep 2.5.1 is compiled
on OS X 10.5 or 10.6.  But it *does* cause trouble when grep 2.5.1 is
compiled on recent DPs of OS X 10.7.

When you fix this problem, the "egrep: Regular expression too big"
errors building the Mozilla tree stop happening.

By the way, the (internal) error number corresponding to this message
is REG_ESIZE.  In lib/posix/regex.h this is defined as:

"Compiled pattern bigger than 2^16 bytes."

But the "pattern" used in configure.in is, while large, only 593 bytes
long.  So the "egrep: Regular expression too big" messages *are*
spurious.
Assignee: nobody → smichaud
The problem with Gnu grep 2.5.1 (which Apple has bundled with OS X at
least since version 10.5) is something that Apple needs to take care
of.  I'll open an Apple bug to make sure they know about it.

For now I don't think we should do anything ourselves.

If this bug survives into OS X 10.7 release versions, we can get
around it by calling egrep with the 'arch -arch i386' command, to
ensure that we always use the 32-bit binary.  But of course we don't
want to have to do this :-)

In the meantime people can use my workaround from comment #21.
Assignee: smichaud → nobody
Component: XPCOM → Build Config
QA Contact: xpcom → build-config
Assignee: nobody → smichaud
Nomis101: Please confirm that my workaround from comment #21 works for you ... or let us know if it doesn't.
Side note:  Builds made on OS X 10.7 are almost certainly not usable
on OS X 10.5, which we still support (even on the trunk).  So it's
going to be a while before we build any "official" distros on 10.7 --
even nightlies.

But given the amount of weirdness I've already found in the short time
I've been building on 10.7, and given that not all of this causes
build failures, we need to start keeping an eye out for strange
behavior that may turn out to be caused by what I'm calling 10.7's
gcc's "pickiness".
(In reply to comment #27)
> Nomis101: Please confirm that my workaround from comment #21 works for you
> ... or let us know if it doesn't.

Yes, this also works for me. :-)
I've opened a bug with Apple.  Here's the text of my bug report:

Attempts to build the Mozilla tree on recent DPs of OS X Lion result
in strange errors.  I've seen these errors with build 11A444d plus
XCode Developer Preview 4, and with build 11A459e plus XCode Developer
Preview 5, but they may go back further.  This issue is being tracked
at:

https://bugzilla.mozilla.org/show_bug.cgi?id=655339

I've found that these errors are ultimately caused by a bug in the GNU
grep 2.5.1.  This has been bundled with OS X since at least version
10.5.  But (very strangely) the bug only manifests itself (as far as I
can tell) in grep distros compiled on OS X 10.7, and then only in
64-bit binaries.

Here's a description of the bug, mostly quoted from
bugzilla.mozilla.org's bug 655339:

There are a lot of "#ifdef MBS_SUPPORT" blocks in src/dfa.c,
src/search.c and lib/regex.c.  On OS X, MBS_SUPPORT (== multi-byte
character support) is defined in dfa.c and search.c, but not in
regex.c.  This last is a bug, and means that different parts of grep
(and of fgrep and egrep) don't agree on whether or not multi-byte
characters are supported.  This doesn't appear to cause any trouble
when grep 2.5.1 is compiled on OS X 10.5 or 10.6.  But it *does* cause
trouble when grep 2.5.1 is compiled to a 64-bit binary on recent DPs
of OS X 10.7.

I've created a patch that fixes this bug, and also fixes the problems
building the Mozilla tree on OS X 10.7.  I've attached it to bug
655339, and will also try to attach it here.

To reproduce the Mozilla build errors you need to know how to build
the Mozilla tree.  Basic instructions are available at
https://developer.mozilla.org/En/Simple_Firefox_build.  But the
official MacPorts release doesn't yet support OS X 10.7.  So instead
you'll need to install an unofficial release of MacPorts 1.9.2
available at
http://blog.touchspot.at/2011/04/28/macports-1-9-2-for-mac-os-x-10-7-lion/.
Whiteboard: rdar://9463556
Blocks: 659817
(In reply to comment #4)
> I haven't tried to build FF on 10.7 yet. Too lazy and too little reason to
> install the requirements like autoconf213, libidl, hg, and yasm manually
> since MacPorts doesn't work on 10.7 yet. I'll probably have to figure it out
> in the next couple of weeks.

Josh, since DP2 I am able to build all required MacPorts again. Not sure if this is because of changes in Lion or if they made changes to the ports.
No longer blocks: 659817
This bug has not been fixed in Apple's latest Lion+XCode combo -- Lion
DP4 (build 11A480b) plus XCode 4.1 DP 6.  Apple hasn't yet commented
on my bug report.

My workaround from comment #21 still works, though.
Maybe we should write a replacement in Perl, Python or C? Is there a scripting language that we depend on already?
If "we" is the build system, then Python.
(In reply to comment #33 and comment #34)

If Apple never fixes this bug.

But I think my workaround from comment #21 will suffice until we start
doing "official" builds on OS X 10.7 -- which can't happen until after
we drop support for OS X 10.5.
(In reply to comment #32)
> My workaround from comment #21 still works, though.

Yes, but you have to redo it everytime you install a new DP of Xcode 4.1. :-(
And note that Python still suffers from bug 659881 -- though recent
comments at http://bugs.python.org/issue9516 show the Python
developers are working on it.
>> My workaround from comment #21 still works, though.
>
> Yes, but you have to redo it everytime you install a new DP of Xcode
> 4.1. :-(

Yes, this is a pain.  But those who build on 10.7 are going to be on
the bleeding edge for a while, and not just because of this bug.
Note to Steven:  I still needed your work around in comment #21 to build on OS X Lion (official, the one that came out yesterday)
> Note to Steven: I still needed your work around in comment #21 to
> build on OS X Lion (official, the one that came out yesterday)

Sigh ... but not totally unexpected :-(

What's the build number for the official release?  The GM (Gold
Master) DP's build number was 11A511, and I'm wondering if the
official release is different.
I have a patch pending in bug 673209 to make configure fail fast when egrep fails so we don't see spurious GCC compilation failures. It may be appropriate to change this bug's summary once that is committed, as people will then see an obvious failure with egrep on 10.7.
> It may be appropriate to change this bug's summary once that is
> committed, as people will then see an obvious failure with egrep on
> 10.7.

It'd probably be best to add the new error (from your patch for bug
673209) to this bug's summary -- people will keep seeing the old error
for a while.
If we do something about this programmatically, I still think it's
best to substitute (on OS X) a call to "arch -arch i386 egrep" for
every call to "egrep", as I suggested in comment #26.
(In reply to comment #40)
> What's the build number for the official release?  The GM (Gold
> Master) DP's build number was 11A511, and I'm wondering if the
> official release is different.

The official release is the same build number than the GM, 11A511. Is egrep installed with Lion, I assumed this is from Xcode?
$ ls -lc /usr/bin/egrep 
-rwxr-xr-x  3 root  wheel  232064 20 Jul 16:53 /usr/bin/egrep

egrep comes with OS X. I just updated Xcode to the compatible Lion version.

Xcode 4.1
Build version 4B110
> The official release is the same build number than the GM, 11A511.

Thanks for the info.

> Is egrep installed with Lion, I assumed this is from Xcode?

It *is* installed with Lion (I just checked a partition where I'd installed the Lion GM but not XCode).  Though XCode might also install a copy, especially if the XCode distro has a more recent build (though all versions will be 2.5.1, presumably for licensing reasons).
The build of XCode that Apple currently has available for download (build 4B110) is more recent than the build that came out at the same time as the Lion Gold Master DP (build 4B95).

I'll download and install 4B110 and see if that makes any difference ... but I think that's unlikely.
No, this doesn't make any difference. Today I've tried to build with Lion 11A511 and Xcode 4B110. And I still see the error.
Attached patch Simple, hacky workaround (obsolete) — — Splinter Review 
(Following up comment #44)

> If we do something about this programmatically, I still think it's
> best to substitute (on OS X) a call to "arch -arch i386 egrep" for
> every call to "egrep", as I suggested in comment #26.

Turns out this is much harder than I realized.  autoconf always uses
egrep directly (like cat, cmp, cp and so forth), rather than sometimes
through the mediation of an environment variable (like ar ($AR), bison
($BISON), cc ($CC) and so forth.  So it's not at all clear how I'd go
about replacing all calls to "egrep" in the build system with calls to
"arch -arch i386 egrep".

But, fortunately, there are currently only two calls to egrep whose
PATTERN is long enough to trigger this bug.  And it's very easy to
just replace these two calls.

Which is what this patch does.

Yes, it's as ugly as sin.  But I can't think of a better way to work
around Apple's bug, while they take their sweet time fixing it.
Attachment #547774 - Flags: review?(ted.mielczarek)
Comment on attachment 547774 [details] [diff] [review]
Simple, hacky workaround

I haven't yet tried this patch on any other platforms than OS X 10.7.  But I'm currently an all-platform tryserver run.
Blocks: 673541
I'm going to be on vacation all next week, and mostly away from the Internet.

My tryserver builds haven't finished yet, but none has yet failed because of this patch.
Steven's patch fixes the build for me. If for some reason we decide not to use it, I can try to replace these uses of grep with python.
I got past this problem by installing grep 2.9 through MacPorts.
Comment on attachment 547774 [details] [diff] [review]
Simple, hacky workaround

Review of attachment 547774 [details] [diff] [review]:
-----------------------------------------------------------------

I worry that this will live on forever like most of the other hacks in our codebase, but there's not a great alternative here.

::: configure.in
@@ +9323,5 @@
> +# binary, even on 64-bit systems.  It should work on OS X 10.4.5 and up.  We
> +# (apparently) only need this hack when egrep's "pattern" is particularly
> +# long (as in the following code).  See bug 655339.
> +case "$host" in
> +*-darwin*)

Do you want to make this even more specific and say x86_64-apple-darwin*) ? (Probably doesn't matter all that much, as we only support i386 and x86-64 now.)
Attachment #547774 - Flags: review?(ted.mielczarek) → review+
> Do you want to make this even more specific and say
> x86_64-apple-darwin*)?

Sounds good to me.  I've done this in the copy of my patch for
landing.

I've also carried forward the r+.
Attachment #547774 - Attachment is obsolete: true
Attachment #549831 - Flags: review+
Whiteboard: rdar://9463556 → rdar://9463556 [inbound]
http://hg.mozilla.org/mozilla-central/rev/6f56ca77a9ba
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla8
Blocks: 675300
FYI, this fix broke my 10.7 build, so here's the details in case someone
else has the same problem.

My system had a working 'egrep' in /opt/local/bin/egrep which is x86-64 only.
Running it under "arch -arch i386" fails with:
arch: posix_spawnp: egrep: Bad CPU type in executable
configure: error: Error outputting config definitions

The solution is:
sudo mv  /opt/local/bin/egrep  /opt/local/bin/egrep-orig

so that /usr/bin/egrep is used instead.
(In reply to comment #59)

Someone else reported the same thing to me in email.  I'm told the
64-bit-only egrep in /opt/local/bin is installed by MacPorts.  So I
might have to change my patch for this bug (for example by using "arch
-arch i386 /usr/bin/egrep").  But first I want to find out whether
MacPorts does a 64-bit-only install by default, or whether that's just
an option.

(I don't use MacPorts, so I'm not very familiar with it.  But sometime
next week I'll try installing it on a spare partition.)
(In reply to Steven Michaud from comment #60)
> But first I want to find out whether
> MacPorts does a 64-bit-only install by default, or whether that's just
> an option.

On 10.6 and 10.7 MacPorts builds in x86_64 by default. You can see this in /opt/local/etc/macports/macports.conf 
Is says there:

# CPU architecture to compile for. Defaults to i386 or ppc on Mac OS X 10.5
# and earlier, depending on the CPU type detected at runtime. On Mac OS X 10.6
# the default is x86_64 if the CPU supports it, i386 otherwise.

If you want to build in i386 on 10.6 and later, you need to change the build_arch. Or build an UB with +universal.
Maybe it would be better to check for the specific problem version of egrep? I patched my file like this to get it working with my MacPorts egrep. I freely admit to a lack of scripting experience so probably there's a better way to do it.


diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -9324,7 +9324,11 @@ xpcom/xpcom-private.h
 # long (as in the following code).  See bug 655339.
 case "$host" in
 *-apple-darwin*)
-    FIXED_EGREP="arch -arch i386 egrep"
+    if [[ `egrep -V` == *2.5.1* ]]; then
+        FIXED_EGREP="arch -arch i386 egrep"
+    else
+        FIXED_EGREP="egrep"
+    fi
     ;;
 *)
     FIXED_EGREP="egrep"
(In reply to Steven Michaud from comment #50)
> Yes, it's as ugly as sin.  But I can't think of a better way to work
> around Apple's bug, while they take their sweet time fixing it.

Have you reported the bug?  bugs don't get fixed if they're not reported...
According to comment 26 and the status whiteboard, I'd think Steven reported this to apple, yes.
The text of my bug report to Apple is in comment #30.
Whiteboard: rdar://9463556 [inbound] → rdar://9463556
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: