Last Comment Bug 655339 - [10.7] error: expected initializer before ‘NS_ATTR_MALLOC’
: [10.7] error: expected initializer before ‘NS_ATTR_MALLOC’
Status: RESOLVED FIXED
rdar://9463556
:
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: x86 Mac OS X
: -- normal with 2 votes (vote)
: mozilla8
Assigned To: Steven Michaud [:smichaud] (Retired)
:
Mentors:
: 673237 (view as bug list)
Depends on:
Blocks: lion-compatibility 673541 675300
  Show dependency treegraph
 
Reported: 2011-05-06 12:33 PDT by Nomis101
Modified: 2013-11-23 01:05 PST (History)
23 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
affected
affected


Attachments
Patch for Gnu grep 2.5.1 (*not* a fix for this bug) (4.92 KB, patch)
2011-05-18 12:46 PDT, Steven Michaud [:smichaud] (Retired)
no flags Details | Diff | Splinter Review
Simple, hacky workaround (2.46 KB, patch)
2011-07-22 12:30 PDT, Steven Michaud [:smichaud] (Retired)
ted: review+
Details | Diff | Splinter Review
Follow Ted's suggestion (2.70 KB, patch)
2011-08-01 09:33 PDT, Steven Michaud [:smichaud] (Retired)
smichaud: review+
Details | Diff | Splinter Review

Description Nomis101 2011-05-06 12:33:55 PDT
If I build FF trunk or TB trunk on 10.7 it stops with an error. The error is the same, witch 10.7 SDK and with 10.6 SDK. 
OS X 10.7 11A444d, Xcode 4.1 4B47f

The error is:

In file included from /Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc_oom.cpp:45:
../../dist/include/mozilla/mozalloc_abort.h:61: error: expected initializer before ‘NS_NORETURN’
/Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc_oom.cpp: In function ‘void mozalloc_handle_oom()’:
/Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc_oom.cpp:54: error: ‘mozalloc_abort’ was not declared in this scope
In file included from /Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc_abort.cpp:56:
../../dist/include/mozilla/mozalloc_abort.h:61: error: expected initializer before ‘NS_NORETURN’
/Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc_abort.cpp: In function ‘void mozalloc_abort(const char*)’:
/Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc_abort.cpp:88: error: ‘_exit’ was not declared in this scope
make[6]: *** [mozalloc_abort.o] Error 1
make[6]: *** Waiting for unfinished jobs....
make[6]: *** [mozalloc_oom.o] Error 1
In file included from /Volumes/Macintosh_HD/Users/polysom/Documents/Developer/temp/src/mozilla/memory/mozalloc/mozalloc.cpp:67:
../../dist/include/mozilla/mozalloc.h:109: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:113: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:117: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:120: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:124: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:127: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:131: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h:134: error: expected initializer before ‘NS_ATTR_MALLOC’
../../dist/include/mozilla/mozalloc.h: In function ‘void* operator new(size_t)’:
../../dist/include/mozilla/mozalloc.h:229: error: ‘moz_xmalloc’ was not declared in this scope
../../dist/include/mozilla/mozalloc.h: In function ‘void* operator new(size_t, const std::nothrow_t&)’:
../../dist/include/mozilla/mozalloc.h:235: error: ‘moz_malloc’ was not declared in this scope
../../dist/include/mozilla/mozalloc.h: In function ‘void* operator new [](size_t)’:
../../dist/include/mozilla/mozalloc.h:241: error: ‘moz_xmalloc’ was not declared in this scope
../../dist/include/mozilla/mozalloc.h: In function ‘void* operator new [](size_t, const std::nothrow_t&)’:
../../dist/include/mozilla/mozalloc.h:247: error: ‘moz_malloc’ was not declared in this scope
../../dist/include/mozilla/mozalloc.h: In function ‘void* operator new(size_t, const mozilla::fallible_t&)’:
../../dist/include/mozilla/mozalloc.h:303: error: ‘moz_malloc’ was not declared in this scope
../../dist/include/mozilla/mozalloc.h: In function ‘void* operator new [](size_t, const mozilla::fallible_t&)’:
../../dist/include/mozilla/mozalloc.h:309: error: ‘moz_malloc’ was not declared in this scope
make[6]: *** [mozalloc.o] Error 1
make[5]: *** [libs] Error 2
make[4]: *** [libs_tier_base] Error 2
make[3]: *** [tier_base] Error 2
make[2]: *** [default] Error 2
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2
Comment 1 Chris Jones [:cjones] inactive; ni?/f?/r? if you need me 2011-05-11 15:44:15 PDT
Does your $objdir/dist/include/mozilla-config.h #define these macros?  Do you have other patches applied when you see these build errors?
Comment 2 Nomis101 2011-05-12 13:10:11 PDT
(In reply to comment #1)
> Does your $objdir/dist/include/mozilla-config.h #define these macros?
This file is empty, it only includes this three lines:

#ifndef _MOZILLA_CONFIG_H_
#define _MOZILLA_CONFIG_H_

#endif /* _MOZILLA_CONFIG_H_ */

> you have other patches applied when you see these build errors?
Today I've tried it with fresh code with no patches. Same result.
Comment 3 Chris Jones [:cjones] inactive; ni?/f?/r? if you need me 2011-05-12 15:34:40 PDT
(In reply to comment #2)
> (In reply to comment #1)
> > Does your $objdir/dist/include/mozilla-config.h #define these macros?
> This file is empty, it only includes this three lines:
> 
> #ifndef _MOZILLA_CONFIG_H_
> #define _MOZILLA_CONFIG_H_
> 
> #endif /* _MOZILLA_CONFIG_H_ */

That's bad.  Are you following these --- https://developer.mozilla.org/En/Simple_Firefox_build --- build instructions?

I'm pretty sure FF has been built on 10.7; maybe you need some patches that aren't in 10.7.  CC'ing folks who would know better.
Comment 4 Josh Aas 2011-05-12 15:49:46 PDT
I haven't tried to build FF on 10.7 yet. Too lazy and too little reason to install the requirements like autoconf213, libidl, hg, and yasm manually since MacPorts doesn't work on 10.7 yet. I'll probably have to figure it out in the next couple of weeks.

Also, you won't be able to make a proper release build on 10.7 since Xcode 4 doesn't support the 10.5 SDK and targeting the OS version isn't exactly the same (even if it should be in theory).
Comment 5 Steven Michaud [:smichaud] (Retired) 2011-05-12 15:58:38 PDT
I haven't yet tried to build on 10.7, either -- out of laziness, and having (I think) more urgent things to do.

But I don't use MacPorts.  So I may end up trying to build on 10.7 before the new MacPorts comes out.  If/when I do, I'll comment here.

You do realize, I hope, that without MacPorts you'll need to install all the build requirements manually.
Comment 6 Nomis101 2011-05-13 09:26:49 PDT
(In reply to comment #3)
> That's bad.  Are you following these ---
> https://developer.mozilla.org/En/Simple_Firefox_build --- build instructions?
Yes, I did it like it works on 10.6. And with a very simple and basic mozconfig.

(In reply to comment #4)
> I haven't tried to build FF on 10.7 yet. Too lazy and too little reason to
> install the requirements like autoconf213, libidl, hg, and yasm manually
> since MacPorts doesn't work on 10.7 yet. I'll probably have to figure it out
> in the next couple of weeks.
I've updated a copy of 10.6 to 10.7.  So everything of the requirements are still installed from 10.6.

But I remember that I was able to build FF and TB with the first DP of 10.7 (so I was able to fill Bug 643150). So the build must be broken by any of the updates.
Comment 7 Steven Michaud [:smichaud] (Retired) 2011-05-13 16:33:07 PDT
In a somewhat masochistic mood, I've started trying to build the
various requirements (and prerequirements) for the Mozilla tree by
hand on the latest Lion DP (build 11A444d).

The very first component I tried to build, gettext, failed to build
with an utterly inexplicable error.  I got the same error building
both an older gettext distro (gettext-0.17) and the current distro
(gettext-0.18.1.1).

I strongly suspect this is due to a bug in the gcc 4.2.1 that's
included with build 11A444d:

$ gcc --version
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.14.00)
Copyright (C) 2007 Free Software Foundation, Inc.

Please try building one of those gettext distros, and see if you get
the same error (or one that seems close).  If you do, I'd bet the
error you report here is also due to a bug in build 11A444d's gcc
4.2.1.

Here's the error I got building gettext-0.18.1.1:

stpncpy.c:34: error: expected declaration specifiers or ‘...’ before numeric constant
stpncpy.c:34: error: expected ‘)’ before ‘!=’ token
stpncpy.c:34: error: expected ‘)’ before ‘?’ token
stpncpy.c:34: error: expected declaration specifiers or ‘...’ before numeric constant
stpncpy.c:34: error: expected ‘)’ before ‘!=’ token
stpncpy.c:34: error: expected ‘)’ before ‘?’ token
Comment 8 Steven Michaud [:smichaud] (Retired) 2011-05-13 16:53:26 PDT
I get the same error building gettext-0.17 on the previous Lion DP
(build 11A430e):

gcc --version
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.11.00)
Copyright (C) 2007 Free Software Foundation, Inc.

I no longer have a copy of the first DP (build 11A390) ... though I
suppose I could reinstall it.
Comment 9 Steven Michaud [:smichaud] (Retired) 2011-05-13 18:07:33 PDT
Actually, my errors are no longer so inexplicable -- it looks like the offending file redefined stpncpy.  I wish the error message had *said* this.

Sigh.
Comment 10 Nomis101 2011-05-14 01:42:59 PDT
(In reply to comment #7)
> Please try building one of those gettext distros, and see if you get
> the same error (or one that seems close).  If you do, I'd bet the
> error you report here is also due to a bug in build 11A444d's gcc
> 4.2.1.

OK, I will try this after I have installed the new 10.7 and Xcode Update. Maybe this will fix some problems.
What I can say is, that I get the same error (from my initital post) with Apples gcc 4.2.1 and with Clang 2.9.
Comment 11 Steven Michaud [:smichaud] (Retired) 2011-05-14 09:32:03 PDT
>> Please try building one of those gettext distros, and see if you
>> get the same error (or one that seems close).  If you do, I'd bet
>> the error you report here is also due to a bug in build 11A444d's
>> gcc 4.2.1.
>
> OK, I will try this ...

You don't need to bother.  What I first thought was a spurious error
turns out to be a real one (though the error messages were misleading
and mostly wrong).  So this isn't a gcc 4.2.1 (or clang) bug (aside
from the uselessness of the error messages).

Do install the appropriate XCode update for your DP, though, and let
us know if this makes a difference.  As I assume you're aware, each DP
seed has it's own (seperate) XCode download.

At some point I'll finish installing the build prerequisites by hand,
and be able to try a build myself.
Comment 12 Nomis101 2011-05-14 09:39:05 PDT
Today I've also tried to build gettext 0.18.1.1 on 10.7 (11A459e) with Xcode 4.1 DP5. I get:

stpncpy.c:34: error: expected declaration specifiers or ‘...’ before numeric constant
stpncpy.c:34: error: expected ‘)’ before ‘!=’ token
stpncpy.c:34: error: expected ‘)’ before ‘?’ token
make[4]: *** [stpncpy.lo] Error 1
make[3]: *** [all] Error 2
make[2]: *** [all-recursive] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1


Than I've tried to build it with Clang 2.9 from http://llvm.org/releases/download.html#2.9 and I get a similar error:

stpncpy.c:34:1: error: expected parameter declarator
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:5: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
    ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
In file included from /usr/include/secure/_string.h:32:
/usr/include/secure/_common.h:38:63: note: instantiated from:
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                                                              ^
stpncpy.c:34:1: error: expected ')'
stpncpy.c:34:1: note: to match this '('
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:5: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
    ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
In file included from /usr/include/secure/_string.h:32:
/usr/include/secure/_common.h:38:54: note: instantiated from:
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                                                     ^
stpncpy.c:34:1: error: expected ')'
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:27: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
                          ^
stpncpy.c:34:1: note: to match this '('
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:4: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
   ^
stpncpy.c:34:1: error: expected ')'
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:111:4: note: instantiated from:
   ? __builtin___stpncpy_chk (dest, src, len, __darwin_obsz (dest))     \
   ^
stpncpy.c:34:1: note: to match this '('
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:3: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
  ^
stpncpy.c:34:1: error: conflicting types for '__builtin_object_size'
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:5: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
    ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
In file included from /usr/include/secure/_string.h:32:
/usr/include/secure/_common.h:38:32: note: instantiated from:
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                               ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:61:56: note: '__builtin_object_size' is a builtin
      with type 'unsigned long (void *, int)'
  return __builtin___memcpy_chk (__dest, __src, __len, __darwin_obsz0(__dest));
                                                       ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
In file included from /usr/include/secure/_string.h:32:
/usr/include/secure/_common.h:38:32: note: instantiated from:
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                               ^
stpncpy.c:34:1: error: definition of builtin function '__builtin_object_size'
__stpncpy (char *dest, const char *src, size_t n)
^
stpncpy.c:28:20: note: instantiated from:
# define __stpncpy stpncpy
                   ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
/usr/include/secure/_string.h:110:5: note: instantiated from:
  ((__darwin_obsz0 (dest) != (size_t) -1)                               \
    ^
In file included from stpncpy.c:25:
In file included from ./string.h:26:
In file included from /usr/include/string.h:190:
In file included from /usr/include/secure/_string.h:32:
/usr/include/secure/_common.h:38:32: note: instantiated from:
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                               ^
stpncpy.c:39:7: error: use of undeclared identifier 'n'
  if (n >= 4)
      ^
stpncpy.c:41:19: error: use of undeclared identifier 'n'
      size_t n4 = n >> 2;
                  ^
stpncpy.c:45:16: error: use of undeclared identifier 'src'
          c = *src++;
               ^
stpncpy.c:49:16: error: use of undeclared identifier 'src'
          c = *src++;
               ^
stpncpy.c:53:16: error: use of undeclared identifier 'src'
          c = *src++;
               ^
stpncpy.c:57:16: error: use of undeclared identifier 'src'
          c = *src++;
               ^
stpncpy.c:64:7: error: use of undeclared identifier 'n'
      n -= dest - s;
      ^
stpncpy.c:69:3: error: use of undeclared identifier 'n'
  n &= 3;
  ^
stpncpy.c:70:7: error: use of undeclared identifier 'n'
  if (n == 0)
      ^
stpncpy.c:75:12: error: use of undeclared identifier 'src'
      c = *src++;
           ^
stpncpy.c:76:9: error: use of undeclared identifier 'n'
      --n;
        ^
stpncpy.c:80:11: error: use of undeclared identifier 'n'
      if (n == 0)
          ^
stpncpy.c:85:10: error: use of undeclared identifier 'n'
  while (n-- > 0)
         ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make[4]: *** [stpncpy.lo] Error 1
make[3]: *** [all] Error 2
make[2]: *** [all-recursive] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1
Comment 13 Steven Michaud [:smichaud] (Retired) 2011-05-14 13:17:04 PDT
> At some point I'll finish installing the build prerequisites by
> hand, and be able to try a build myself.

I finished building all the prerequisites, then tried doing a
mozilla-central build ... and got exactly the same error that Nomis101
has reported.

I built on what's now the second-to-latest DP (build 11A444d), with
the appropriate XCode download also installed.  Later I'll try with
the current DP (build 11A459e), plus its own XCode download (I'm
downloading the bits now).  Assuming the error persists, next week
I'll start trying to figure out why it happens, and see what we can do
about it.

Installing the build prerequisites by hand is *not* straightforward.
But you can get a head start by downloading current Mac Ports source
distros, and checking 1) source changes they've made and 2) what
configure parameters they use.  At some point I'll document how I
build them ... but I may not have time for that for a while.

For the record, here are the prerequisites I installed, in the order I
installed them.  Except for autoconf, I always used the latest
versions.  (Autoconf 2.6.8 is needed for translating changes I make to
glib's configure.ac file.)

gettext
pkgconfig
autoconf 2.13
autoconf 2.68
glib
libIDL
docutils
mercurial
yasm
Comment 14 Steven Michaud [:smichaud] (Retired) 2011-05-14 13:25:29 PDT
> autoconf 2.13
> autoconf 2.68

Just so that nobody asks:  I installed Autoconf 2.13 with '--program-suffix=213' and Autoconf 2.68 with '--program-suffix=268'.  The two different versions shouldn't get mixed up, or interfere with each other.
Comment 15 Nomis101 2011-05-15 15:00:55 PDT
Today I've erased my Lion HD and installed everything new. Installing Macports and anything that is needed to build Mozilla on 10.7 seems easy.
First I've installed this unofficial version of MacPorts 1.9.2 http://blog.touchspot.at/2011/04/28/macports-1-9-2-for-mac-os-x-10-7-lion/ Than I've changed the macports.conf file, so that it builds x86_64 on 10.7 (don't know if this is necessary). After that I've installed everything in the way:
$ sudo port -v selfupdate
$ sudo port install libidl configure.cflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk" configure.cxxflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk"
$ sudo port install autoconf213 configure.cflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk" configure.cxxflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk"
$ sudo port install mercurial configure.cflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk" configure.cxxflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk"
$ sudo port install yasm configure.cflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk" configure.cxxflags="-O3 -isysroot /Developer/SDKs/MacOSX10.7.sdk"

The only problem I've had was db46 (needed for mercurial), you need first to install the Java Runtime for 10.7. I've not tried to build FF with this new installed files for now.
Comment 16 Steven Michaud [:smichaud] (Retired) 2011-05-16 16:25:27 PDT
This problem persists on the latest DP (build 11A459e).

I also have an empty mozilla-config.h file.

Looking through my build logs, I noticed two instances of the
following error around the time that mozilla-config.h gets built:

egrep: Regular expression too big

So for the hell of it I tried compiling and installing the current
(latest) version of GNU grep (which also includes fgrep and egrep) --
version 2.8.  And what do you know, my mozilla-config.h is no longer
empty, and the build gets a lot futher along before it errors out (on
something else that's probably unrelated)!

I haven't a clue why a newer version of egrep should make a
difference.  Boris and Ted, do you have any ideas?
Comment 17 Ted Mielczarek [:ted.mielczarek] 2011-05-16 17:27:53 PDT
Looks like we use egrep to build mozilla-config.h in configure:
http://mxr.mozilla.org/mozilla-central/source/configure.in#9197

and we build up a really big egrep pattern for some reason. (I didn't really analyze what it's doing there.)
Comment 18 Steven Michaud [:smichaud] (Retired) 2011-05-16 17:46:50 PDT
I suspect the "egrep: Regular expression too big" error message is
spurious: OS X 10.5, 10.6 and 10.7 all have the same version (2.51) of
Gnu grep. And at the following link someone talks about making the
error go away by changing his LC_CTYPE from "de_AT.UTF-8" to "de_AT".

So maybe it's some kind of encoding issue. But playing with the
LC_CTYPE, LC_ALL, LC_MESSAGES and LANG environment variables before
building didn't make any difference ... at least not the changes I
tried.
Comment 19 Steven Michaud [:smichaud] (Retired) 2011-05-16 17:53:29 PDT
Forgot to include the link mentioned in the first paragraph:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=136508
Comment 20 Steven Michaud [:smichaud] (Retired) 2011-05-16 18:07:24 PDT
> Looks like we use egrep to build mozilla-config.h in configure:
> http://mxr.mozilla.org/mozilla-central/source/configure.in#9197
>
> and we build up a really big egrep pattern for some reason.

> I suspect the "egrep: Regular expression too big" error message is
> spurious

> maybe it's some kind of encoding issue

Some encodings take up more space than others.  So maybe it's both an
encoding issue and a real space issue.
Comment 21 Steven Michaud [:smichaud] (Retired) 2011-05-17 14:45:09 PDT
This is a very puzzling bug.  But I can now say with some confidence
that it's ultimately caused by a bug in grep/egrep/fgrep 2.5.1 that
only surfaces when those programs are compiled are run in 64-bit mode.
As best I can tell, the bug has nothing to do with encodings.

A quick fix is to rename the original /usr/bin/egrep to something like
egrep-orig, then run the lipo command to create an egrep that's a
32-bit-mode thin binary:

$ sudo mv /usr/bin/egrep /usr/bin/egrep-orig
$ sudo lipo /usr/bin/egrep-orig -thin i386 -output /usr/bin/egrep

After this you can build the entire tree, without error.  (I'm not
able to reproduce the error I reported in comment #16.)
Comment 22 Steven Michaud [:smichaud] (Retired) 2011-05-17 14:53:19 PDT
grep, egrep and fgrep are i386/x86_64 universal binaries on both OS X
10.6 and OS X 10.7.  But for some reason this only causes trouble on
10.7.  My guess is that only on 10.7 does our build process run
command-line utilities in 64-bit mode.  I'll try to write a simple
test utility to confirm or deny this.
Comment 23 Steven Michaud [:smichaud] (Retired) 2011-05-17 16:11:27 PDT
> My guess is that only on 10.7 does our build process run
> command-line utilities in 64-bit mode.  I'll try to write a simple
> test utility to confirm or deny this.

My guess was wrong -- i386/x86_64 universal binaries run in 64-bit
mode when called from 'configure', on OS X 10.6, 10.6 and 10.7.  And
the i386/x86_64 universal egrep binary copied from 10.6 also works
fine.

So maybe the problem is with how the 64-bit part of the egrep binary
gets compiled on OS X 10.7.
Comment 24 Steven Michaud [:smichaud] (Retired) 2011-05-17 16:17:42 PDT
i386/x86_64 universal binaries run in 64-bit mode when called from
'configure', on OS X *10.5*, 10.6 and 10.7.
Comment 25 Steven Michaud [:smichaud] (Retired) 2011-05-18 12:46:39 PDT
Created attachment 533379 [details] [diff] [review]
Patch for Gnu grep 2.5.1 (*not* a fix for this bug)

> I can now say with some confidence that it's ultimately caused by a
> bug in grep/egrep/fgrep 2.5.1 that only surfaces when those programs
> are compiled are run in 64-bit mode.

I've now managed to track it down.  As with gettext and glib, OS X
10.7's version of gcc (gcc 4.2) seems picker than earlier versions (at
least than earlier Apple versions) -- it finds errors where previous
versions didn't, and gets confused by things that are actually errors,
but which it doesn't detect.  (Earlier versions also failed to detect
them, but weren't confused by them.)

In the case of Gnu grep 2.5.1, it's an undetected error:  There are a
lot of "#ifdef MBS_SUPPORT" blocks in src/dfa.c, src/search.c and
lib/regex.c.  On OS X, MBS_SUPPORT (== multi-byte character support)
is defined in dfa.c and search.c, but not in regex.c.  This last is a
bug, and means that different parts of grep (and of fgrep and egrep)
don't agree on whether or not multi-byte characters are supported.
This doesn't appear to cause any trouble when grep 2.5.1 is compiled
on OS X 10.5 or 10.6.  But it *does* cause trouble when grep 2.5.1 is
compiled on recent DPs of OS X 10.7.

When you fix this problem, the "egrep: Regular expression too big"
errors building the Mozilla tree stop happening.

By the way, the (internal) error number corresponding to this message
is REG_ESIZE.  In lib/posix/regex.h this is defined as:

"Compiled pattern bigger than 2^16 bytes."

But the "pattern" used in configure.in is, while large, only 593 bytes
long.  So the "egrep: Regular expression too big" messages *are*
spurious.
Comment 26 Steven Michaud [:smichaud] (Retired) 2011-05-18 12:53:35 PDT
The problem with Gnu grep 2.5.1 (which Apple has bundled with OS X at
least since version 10.5) is something that Apple needs to take care
of.  I'll open an Apple bug to make sure they know about it.

For now I don't think we should do anything ourselves.

If this bug survives into OS X 10.7 release versions, we can get
around it by calling egrep with the 'arch -arch i386' command, to
ensure that we always use the 32-bit binary.  But of course we don't
want to have to do this :-)

In the meantime people can use my workaround from comment #21.
Comment 27 Steven Michaud [:smichaud] (Retired) 2011-05-18 13:07:21 PDT
Nomis101: Please confirm that my workaround from comment #21 works for you ... or let us know if it doesn't.
Comment 28 Steven Michaud [:smichaud] (Retired) 2011-05-18 13:20:04 PDT
Side note:  Builds made on OS X 10.7 are almost certainly not usable
on OS X 10.5, which we still support (even on the trunk).  So it's
going to be a while before we build any "official" distros on 10.7 --
even nightlies.

But given the amount of weirdness I've already found in the short time
I've been building on 10.7, and given that not all of this causes
build failures, we need to start keeping an eye out for strange
behavior that may turn out to be caused by what I'm calling 10.7's
gcc's "pickiness".
Comment 29 Nomis101 2011-05-18 13:41:39 PDT
(In reply to comment #27)
> Nomis101: Please confirm that my workaround from comment #21 works for you
> ... or let us know if it doesn't.

Yes, this also works for me. :-)
Comment 30 Steven Michaud [:smichaud] (Retired) 2011-05-18 14:33:08 PDT
I've opened a bug with Apple.  Here's the text of my bug report:

Attempts to build the Mozilla tree on recent DPs of OS X Lion result
in strange errors.  I've seen these errors with build 11A444d plus
XCode Developer Preview 4, and with build 11A459e plus XCode Developer
Preview 5, but they may go back further.  This issue is being tracked
at:

https://bugzilla.mozilla.org/show_bug.cgi?id=655339

I've found that these errors are ultimately caused by a bug in the GNU
grep 2.5.1.  This has been bundled with OS X since at least version
10.5.  But (very strangely) the bug only manifests itself (as far as I
can tell) in grep distros compiled on OS X 10.7, and then only in
64-bit binaries.

Here's a description of the bug, mostly quoted from
bugzilla.mozilla.org's bug 655339:

There are a lot of "#ifdef MBS_SUPPORT" blocks in src/dfa.c,
src/search.c and lib/regex.c.  On OS X, MBS_SUPPORT (== multi-byte
character support) is defined in dfa.c and search.c, but not in
regex.c.  This last is a bug, and means that different parts of grep
(and of fgrep and egrep) don't agree on whether or not multi-byte
characters are supported.  This doesn't appear to cause any trouble
when grep 2.5.1 is compiled on OS X 10.5 or 10.6.  But it *does* cause
trouble when grep 2.5.1 is compiled to a 64-bit binary on recent DPs
of OS X 10.7.

I've created a patch that fixes this bug, and also fixes the problems
building the Mozilla tree on OS X 10.7.  I've attached it to bug
655339, and will also try to attach it here.

To reproduce the Mozilla build errors you need to know how to build
the Mozilla tree.  Basic instructions are available at
https://developer.mozilla.org/En/Simple_Firefox_build.  But the
official MacPorts release doesn't yet support OS X 10.7.  So instead
you'll need to install an unofficial release of MacPorts 1.9.2
available at
http://blog.touchspot.at/2011/04/28/macports-1-9-2-for-mac-os-x-10-7-lion/.
Comment 31 Stefan Arentz [:st3fan] 2011-05-30 17:59:10 PDT
(In reply to comment #4)
> I haven't tried to build FF on 10.7 yet. Too lazy and too little reason to
> install the requirements like autoconf213, libidl, hg, and yasm manually
> since MacPorts doesn't work on 10.7 yet. I'll probably have to figure it out
> in the next couple of weeks.

Josh, since DP2 I am able to build all required MacPorts again. Not sure if this is because of changes in Lion or if they made changes to the ports.
Comment 32 Steven Michaud [:smichaud] (Retired) 2011-06-08 13:03:07 PDT
This bug has not been fixed in Apple's latest Lion+XCode combo -- Lion
DP4 (build 11A480b) plus XCode 4.1 DP 6.  Apple hasn't yet commented
on my bug report.

My workaround from comment #21 still works, though.
Comment 33 Stefan Arentz [:st3fan] 2011-06-08 13:23:04 PDT
Maybe we should write a replacement in Perl, Python or C? Is there a scripting language that we depend on already?
Comment 34 Kyle Huey [:khuey] (khuey@mozilla.com) 2011-06-08 13:24:37 PDT
If "we" is the build system, then Python.
Comment 35 Steven Michaud [:smichaud] (Retired) 2011-06-08 13:36:45 PDT
(In reply to comment #33 and comment #34)

If Apple never fixes this bug.

But I think my workaround from comment #21 will suffice until we start
doing "official" builds on OS X 10.7 -- which can't happen until after
we drop support for OS X 10.5.
Comment 36 Nomis101 2011-06-08 13:37:36 PDT
(In reply to comment #32)
> My workaround from comment #21 still works, though.

Yes, but you have to redo it everytime you install a new DP of Xcode 4.1. :-(
Comment 37 Steven Michaud [:smichaud] (Retired) 2011-06-08 13:40:40 PDT
And note that Python still suffers from bug 659881 -- though recent
comments at http://bugs.python.org/issue9516 show the Python
developers are working on it.
Comment 38 Steven Michaud [:smichaud] (Retired) 2011-06-08 13:43:57 PDT
>> My workaround from comment #21 still works, though.
>
> Yes, but you have to redo it everytime you install a new DP of Xcode
> 4.1. :-(

Yes, this is a pain.  But those who build on 10.7 are going to be on
the bleeding edge for a while, and not just because of this bug.
Comment 39 Seth Spitzer 2011-07-21 13:40:54 PDT
Note to Steven:  I still needed your work around in comment #21 to build on OS X Lion (official, the one that came out yesterday)
Comment 40 Steven Michaud [:smichaud] (Retired) 2011-07-21 13:48:08 PDT
> Note to Steven: I still needed your work around in comment #21 to
> build on OS X Lion (official, the one that came out yesterday)

Sigh ... but not totally unexpected :-(

What's the build number for the official release?  The GM (Gold
Master) DP's build number was 11A511, and I'm wondering if the
official release is different.
Comment 41 Chenxia Liu [:liuche] 2011-07-21 14:24:25 PDT
*** Bug 673237 has been marked as a duplicate of this bug. ***
Comment 42 Gregory Szorc [:gps] 2011-07-21 14:39:07 PDT
I have a patch pending in bug 673209 to make configure fail fast when egrep fails so we don't see spurious GCC compilation failures. It may be appropriate to change this bug's summary once that is committed, as people will then see an obvious failure with egrep on 10.7.
Comment 43 Steven Michaud [:smichaud] (Retired) 2011-07-21 15:00:38 PDT
> It may be appropriate to change this bug's summary once that is
> committed, as people will then see an obvious failure with egrep on
> 10.7.

It'd probably be best to add the new error (from your patch for bug
673209) to this bug's summary -- people will keep seeing the old error
for a while.
Comment 44 Steven Michaud [:smichaud] (Retired) 2011-07-21 15:05:07 PDT
If we do something about this programmatically, I still think it's
best to substitute (on OS X) a call to "arch -arch i386 egrep" for
every call to "egrep", as I suggested in comment #26.
Comment 45 Nomis101 2011-07-21 16:37:00 PDT
(In reply to comment #40)
> What's the build number for the official release?  The GM (Gold
> Master) DP's build number was 11A511, and I'm wondering if the
> official release is different.

The official release is the same build number than the GM, 11A511. Is egrep installed with Lion, I assumed this is from Xcode?
Comment 46 Christoph Diehl [:posidron] 2011-07-21 17:06:46 PDT
$ ls -lc /usr/bin/egrep 
-rwxr-xr-x  3 root  wheel  232064 20 Jul 16:53 /usr/bin/egrep

egrep comes with OS X. I just updated Xcode to the compatible Lion version.

Xcode 4.1
Build version 4B110
Comment 47 Steven Michaud [:smichaud] (Retired) 2011-07-21 17:22:46 PDT
> The official release is the same build number than the GM, 11A511.

Thanks for the info.

> Is egrep installed with Lion, I assumed this is from Xcode?

It *is* installed with Lion (I just checked a partition where I'd installed the Lion GM but not XCode).  Though XCode might also install a copy, especially if the XCode distro has a more recent build (though all versions will be 2.5.1, presumably for licensing reasons).
Comment 48 Steven Michaud [:smichaud] (Retired) 2011-07-21 17:26:55 PDT
The build of XCode that Apple currently has available for download (build 4B110) is more recent than the build that came out at the same time as the Lion Gold Master DP (build 4B95).

I'll download and install 4B110 and see if that makes any difference ... but I think that's unlikely.
Comment 49 Nomis101 2011-07-21 17:36:09 PDT
No, this doesn't make any difference. Today I've tried to build with Lion 11A511 and Xcode 4B110. And I still see the error.
Comment 50 Steven Michaud [:smichaud] (Retired) 2011-07-22 12:30:18 PDT
Created attachment 547774 [details] [diff] [review]
Simple, hacky workaround

(Following up comment #44)

> If we do something about this programmatically, I still think it's
> best to substitute (on OS X) a call to "arch -arch i386 egrep" for
> every call to "egrep", as I suggested in comment #26.

Turns out this is much harder than I realized.  autoconf always uses
egrep directly (like cat, cmp, cp and so forth), rather than sometimes
through the mediation of an environment variable (like ar ($AR), bison
($BISON), cc ($CC) and so forth.  So it's not at all clear how I'd go
about replacing all calls to "egrep" in the build system with calls to
"arch -arch i386 egrep".

But, fortunately, there are currently only two calls to egrep whose
PATTERN is long enough to trigger this bug.  And it's very easy to
just replace these two calls.

Which is what this patch does.

Yes, it's as ugly as sin.  But I can't think of a better way to work
around Apple's bug, while they take their sweet time fixing it.
Comment 51 Steven Michaud [:smichaud] (Retired) 2011-07-22 12:42:15 PDT
Comment on attachment 547774 [details] [diff] [review]
Simple, hacky workaround

I haven't yet tried this patch on any other platforms than OS X 10.7.  But I'm currently an all-platform tryserver run.
Comment 52 Steven Michaud [:smichaud] (Retired) 2011-07-22 15:15:29 PDT
I'm going to be on vacation all next week, and mostly away from the Internet.

My tryserver builds haven't finished yet, but none has yet failed because of this patch.
Comment 53 Rafael Ávila de Espíndola (:espindola) (not reading bugmail) 2011-07-22 19:20:40 PDT
Steven's patch fixes the build for me. If for some reason we decide not to use it, I can try to replace these uses of grep with python.
Comment 54 Michael Newton 2011-07-24 10:57:39 PDT
I got past this problem by installing grep 2.9 through MacPorts.
Comment 55 Ted Mielczarek [:ted.mielczarek] 2011-07-25 07:43:18 PDT
Comment on attachment 547774 [details] [diff] [review]
Simple, hacky workaround

Review of attachment 547774 [details] [diff] [review]:
-----------------------------------------------------------------

I worry that this will live on forever like most of the other hacks in our codebase, but there's not a great alternative here.

::: configure.in
@@ +9323,5 @@
> +# binary, even on 64-bit systems.  It should work on OS X 10.4.5 and up.  We
> +# (apparently) only need this hack when egrep's "pattern" is particularly
> +# long (as in the following code).  See bug 655339.
> +case "$host" in
> +*-darwin*)

Do you want to make this even more specific and say x86_64-apple-darwin*) ? (Probably doesn't matter all that much, as we only support i386 and x86-64 now.)
Comment 56 Steven Michaud [:smichaud] (Retired) 2011-08-01 09:33:49 PDT
Created attachment 549831 [details] [diff] [review]
Follow Ted's suggestion

> Do you want to make this even more specific and say
> x86_64-apple-darwin*)?

Sounds good to me.  I've done this in the copy of my patch for
landing.

I've also carried forward the r+.
Comment 57 Steven Michaud [:smichaud] (Retired) 2011-08-03 13:08:39 PDT
Comment on attachment 549831 [details] [diff] [review]
Follow Ted's suggestion

Landed on mozilla-inbound:
http://hg.mozilla.org/integration/mozilla-inbound/rev/6f56ca77a9ba
Comment 58 Marco Bonardo [::mak] (Away 6-20 Aug) 2011-08-04 03:15:57 PDT
http://hg.mozilla.org/mozilla-central/rev/6f56ca77a9ba
Comment 59 Mats Palmgren (vacation) 2011-08-05 15:23:52 PDT
FYI, this fix broke my 10.7 build, so here's the details in case someone
else has the same problem.

My system had a working 'egrep' in /opt/local/bin/egrep which is x86-64 only.
Running it under "arch -arch i386" fails with:
arch: posix_spawnp: egrep: Bad CPU type in executable
configure: error: Error outputting config definitions

The solution is:
sudo mv  /opt/local/bin/egrep  /opt/local/bin/egrep-orig

so that /usr/bin/egrep is used instead.
Comment 60 Steven Michaud [:smichaud] (Retired) 2011-08-05 15:33:19 PDT
(In reply to comment #59)

Someone else reported the same thing to me in email.  I'm told the
64-bit-only egrep in /opt/local/bin is installed by MacPorts.  So I
might have to change my patch for this bug (for example by using "arch
-arch i386 /usr/bin/egrep").  But first I want to find out whether
MacPorts does a 64-bit-only install by default, or whether that's just
an option.

(I don't use MacPorts, so I'm not very familiar with it.  But sometime
next week I'll try installing it on a spare partition.)
Comment 61 Nomis101 2011-08-05 15:42:32 PDT
(In reply to Steven Michaud from comment #60)
> But first I want to find out whether
> MacPorts does a 64-bit-only install by default, or whether that's just
> an option.

On 10.6 and 10.7 MacPorts builds in x86_64 by default. You can see this in /opt/local/etc/macports/macports.conf 
Is says there:

# CPU architecture to compile for. Defaults to i386 or ppc on Mac OS X 10.5
# and earlier, depending on the CPU type detected at runtime. On Mac OS X 10.6
# the default is x86_64 if the CPU supports it, i386 otherwise.

If you want to build in i386 on 10.6 and later, you need to change the build_arch. Or build an UB with +universal.
Comment 62 Michael Newton 2011-08-17 16:40:39 PDT
Maybe it would be better to check for the specific problem version of egrep? I patched my file like this to get it working with my MacPorts egrep. I freely admit to a lack of scripting experience so probably there's a better way to do it.


diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -9324,7 +9324,11 @@ xpcom/xpcom-private.h
 # long (as in the following code).  See bug 655339.
 case "$host" in
 *-apple-darwin*)
-    FIXED_EGREP="arch -arch i386 egrep"
+    if [[ `egrep -V` == *2.5.1* ]]; then
+        FIXED_EGREP="arch -arch i386 egrep"
+    else
+        FIXED_EGREP="egrep"
+    fi
     ;;
 *)
     FIXED_EGREP="egrep"
Comment 63 Jeremy Huddleston 2011-08-20 21:00:35 PDT
(In reply to Steven Michaud from comment #50)
> Yes, it's as ugly as sin.  But I can't think of a better way to work
> around Apple's bug, while they take their sweet time fixing it.

Have you reported the bug?  bugs don't get fixed if they're not reported...
Comment 64 Boris Zbarsky [:bz] 2011-08-20 21:04:34 PDT
According to comment 26 and the status whiteboard, I'd think Steven reported this to apple, yes.
Comment 65 Steven Michaud [:smichaud] (Retired) 2011-08-20 21:20:46 PDT
The text of my bug report to Apple is in comment #30.

Note You need to log in before you can comment on or make changes to this bug.