1363774 - Broken awk results in an empty old-configure which breaks the build in a confusing way

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Victor Ng [:vng] [:crankycoder]]

I was trying to build Firefox in Artifact mode on OSX.  My gawk was out of date using 4.1.3 instead of 4.1.4.

You can see a failed link to libreadline in the build output below.  I'm not sure why this happened - it seems like a bug in homebrew, but I just forced gawk to upgrade with `brew upgrade gawk` and I'm getting a successful build now.

We should probably just check the minimum version of gawk on OSX though.

```
$ rm -rf old-configure configure
Victors-MacBook-Pro-2:mozilla-unified victorng$ ./mach build
 0:00.30 /usr/bin/make -f client.mk -s configure
 0:01.15 Generating /Users/victorng/dev/moz-central/mozilla-unified/configure
 0:01.16 cd /Users/victorng/dev/moz-central/mozilla-unified/obj-x86_64-apple-darwin16.5.0
 0:01.16 /Users/victorng/dev/moz-central/mozilla-unified/configure
 0:01.31 Reexecuting in the virtualenv
 0:01.47 Adding configure options from /Users/victorng/dev/moz-central/mozilla-unified/mozconfig
 0:01.47   --enable-artifact-builds
 0:01.63 checking for a shell... /bin/sh
 0:01.71 checking for host system type... x86_64-apple-darwin16.5.0
 0:01.71 checking for target system type... x86_64-apple-darwin16.5.0
 0:01.75 checking for a shell... /bin/sh
 0:01.83 checking for host system type... x86_64-apple-darwin16.5.0
 0:01.83 checking for target system type... x86_64-apple-darwin16.5.0
 0:01.87 checking for llvm-config... not found
 0:01.87 checking whether cross compiling... no
 0:01.89 checking for tar... /usr/local/bin/gtar
 0:01.89 checking for unzip... /usr/bin/unzip
 0:01.89 checking for zip... /usr/bin/zip
 0:01.89 checking for the Mozilla API key... no
 0:01.89 checking for the Google API key... no
 0:01.89 checking for the Bing API key... no
 0:01.89 checking for the Adjust SDK key... no
 0:01.89 checking for awk... /usr/local/bin/gawk
 0:01.89 checking for perl... /usr/local/bin/perl
 0:01.90 checking for minimum required perl version >= 5.006... 5.024000
 0:01.92 checking for full perl installation... yes
 0:01.92 checking for gmake... /Applications/Xcode.app/Contents/Developer/usr/bin/make
 0:01.92 checking for doxygen... not found
 0:01.92 checking for xargs... /usr/bin/xargs
 0:01.92 checking for dsymutil... /usr/bin/dsymutil
 0:01.92 checking for mkfshfs... /sbin/newfs_hfs
 0:01.92 checking for hfs_tool... not found
 0:01.92 checking for autoconf... /usr/local/bin/autoconf213
 0:01.92 Refreshing /Users/victorng/dev/moz-central/mozilla-unified/old-configure with /usr/local/bin/autoconf213
 0:02.08 dyld: Library not loaded: /usr/local/opt/readline/lib/libreadline.6.dylib
 0:02.08   Referenced from: /usr/local/bin/awk
 0:02.08   Reason: image not found
 0:02.10 Traceback (most recent call last):
 0:02.10   File "/Users/victorng/dev/moz-central/mozilla-unified/configure.py", line 124, in <module>
 0:02.10     sys.exit(main(sys.argv))
 0:02.10   File "/Users/victorng/dev/moz-central/mozilla-unified/configure.py", line 29, in main


```

Comment 1

[(not currently active) Ted Mielczarek]

So I don't think this is related to the *version* of awk at all, but simply that your awk didn't work and we didn't notice, but it broke generating old-configure. I finally tracked it down to autoconf 2.13 using awk to put line numbers into the generated configure:
http://git.savannah.gnu.org/gitweb/?p=autoconf.git;a=blob;f=bin/autoconf.in;h=e6b0819fab30b18f8c3084c76097da301445fb0d;hb=74b21b9b2ce75529e3d93df7367f4f4b0c6b6adf#l149

Nothing checks the return code of that pipeline so if it fails we just wind up with an empty old-configure. A simple thing to do here would be to check that `script` is not empty after we run autoconf:
https://dxr.mozilla.org/mozilla-central/rev/b21b974d60d3075ae24f6fb1bae75d0f122f28fc/build/moz.configure/old.configure#103

Comment 2

[(not currently active) Ted Mielczarek]

Attachment: bug 1363774 - error if generated old-configure is empty due to broken autoconf.

Review commit: https://reviewboard.mozilla.org/r/138162/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/138162/

Comment 3

[Mike Hommey [:glandium]]

Comment on attachment 8866543 [details]
bug 1363774 - error if generated old-configure is empty due to broken autoconf.

https://reviewboard.mozilla.org/r/138162/#review143374

::: build/moz.configure/old.configure:108
(Diff revision 1)
>          script = subprocess.check_output([
>              shell, autoconf,
>              '--localdir=%s' % os.path.dirname(old_configure),
>              old_configure + '.in'])
> +        if not script:
> +            die('Generated old-configure is empty! Check that your autoconf 2.13 and awk programs work!')

Not sure we should single out awk here. It could be awk, sed, and anything else that autoconf uses. Talking about autoconf seems enough.

Comment 4

[(not currently active) Ted Mielczarek]

That's fair. The error message in comment 0 does seem to make it pretty obvious. As we were remote-debugging this via IRC it was being obscured because we wrote out an empty old-configure file, and then re-running the build didn't regenerate it because it was up-to-date, so the build just failed in a more-cryptic way (config.data was missing).

Comment 5

[(not currently active) Ted Mielczarek]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c87511369fbbf3e6a7d632c4a9478b9478efa216
bug 1363774 - error if generated old-configure is empty due to broken autoconf. r=glandium

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/c87511369fbb