1680862 - js/src/aclocal.m4 includes files outside of topsrcdir, `./mach build` fails at configure

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P5)

Description

[Rob Wu [:robwu]]

I tried to run ./mach build, but it consistently failed, even after running ./mach bootstrap and clobber. It appears to be caused by the fact that the relative path in js/src/aclocal.m4 is resolved relative to the object directory, instead of js/src/. Because I have nested topsrcdir in another (older/unrelated) topsrcdir, an incompatible file is included from the unrelated topsrcdir, which results in an error in the configure step of the build.

I came to this conclusion as follows:

• The following directories are relevant:
  • /path/to/gecko/ <- some old checkout, from before bug 1641786
    • this is significant because of the old build/autoconf/compiler-opts.m4 file containing debug-label.
  • /path/to/gecko/worktree2/ <-- latest checkout from yesterday.
  • /path/to/gecko/worktree2/objdir-debug/ <-- from mozconfig: MOZ_OBJDIR=./objdir-debug
• The output of ./mach build (relevant parts below) refers to with-debug-label.
• Even after removing all traces of with-debug-label from repository, the same error returned, and with-debug-label magically appeared in js/src/old-configure.
• With strace -o/tmp/trace -fs 10000 ./mach build, I see that ../../build/autoconf/compiler-opts.m4 is read, but that it contained an old version of the file, not from the topsrcdir that I am using.
  • The command that was being run is: "/usr/bin/m4" "-I/path/to/gecko/worktree2/build/autoconf" "-I/path/to/gecko/worktree2/js/src" "-DAC_LOCALDIR=/path/to/gecko/worktree2/js/src" "autoconf.m4" "/path/to/gecko/worktree2/js/src/old-configure.in"
  • Called from /path/to/gecko/worktree2/build/autoconf/autoconf.sh --localdir="/path/to/gecko/worktree2/js/src" "/path/to/gecko/worktree2/js/src/old-configure.in"
• When I changed MOZ_OBJDIR=./objdir-debug to MOZ_OBJDIR=./objdir-prefix/objdir-debug, the bug went away.

To reproduce, recreate the directory structure above (the compiler-opts.m4 file + a checkout in worktree2) and run ./mach build with the following mozconfig file:

ac_add_options --enable-debug-symbols
ac_add_options --enable-debug
mk_add_options MOZ_OBJDIR=./objdir-debug

OS: ArchLinux,
m4 (GNU M4) 1.4.18

Relevant parts of ./mach build output:

 0:03.24 Adding configure options from /path/to/gecko/worktree2/.mozconfig
 0:03.24   --enable-debug-symbols
 0:03.24   --enable-debug
...
 0:21.76 creating ./config.data
 0:21.79 js/src> Refreshing /path/to/gecko/worktree2/js/src/old-configure with /path/to/gecko/worktree2/build/autoconf/autoconf.sh
...
 0:25.56 js/src> updating cache ./config.cache
 0:25.57 js/src> creating ./config.data
 0:25.60 js/src> ERROR: Missing option in `@old_configure_options` in /path/to/gecko/worktree2/js/sub.configure: --with-debug-label
...
Exception: Process executed with non-0 exit code 1: ['/path/to/.mozbuild/_virtualenvs/mach/bin/python', '/path/to/gecko/worktree2/configure.py']

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

I'm able to reproduce this as described when doing a ./mach build from the nested source directory.

Comment 2

[Rob Wu [:robwu]]

Linking this to mach-busted.

As reported, the work-around for developers is to change MOZ_OBJDIR to a deeper directory.
It's not pretty, but it works until this is fixed.

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Hey, this is a bit of an weird workflow. Why do you have two Firefox checkouts, one of them nested within the other?
Due to the resource constraints of the build team, I'm not sure that we'll be able to address this.

Comment 4

[Rob Wu [:robwu]]

I have many of them to be able to work on multiple features in parallel.

I still have one copy of the .git database due to the use of git worktree.

Do you have an guesses why this bug is happening?

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

It's not clear to me why this is happening just yet.
A more common workflow is to have your projects adjacent to each other, like so:

/path/to/dev/
    - gecko/
        - [all of the files of a checkout]
    - worktree2/
        - [all of the files of another checkout]

If you move worktree2 to the parent directory, does the issue go away?

Comment 6

[Rob Wu [:robwu]]

It's not clear to me why this is happening just yet.

That's unfortunate. My workflow had already been unblocked by the MOZ_OBJDIR= workaround. Directory traversel outside of topsrcdir looks like dangerous, and I was hoping to learn whether this bug could potentially result in bad consequences. I acknowledge that you may not have time to look into this.

If you move worktree2 to the parent directory, does the issue go away?

Of course, because the file lookup would fail. But it's just a work-around, like setting MOZ_OBJDIR.

As for why the directories are nested: By having a common parent working dir, I can simply add a new worktree with:

$ cd /path/to/gecko/
$ git worktree add worktree2 HEAD
$ cd worktree2

To create a worktree in an adjacent directories, the command would just need to change to:

$ cd /path/to/gecko/
$ git worktree add ../worktree2 HEAD
$ cd ../worktree2

or

$ cd /path/to/
$ git -C gecko worktree add ../worktree2 HEAD
$ cd worktree2

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

it's just a work-around, like setting MOZ_OBJDIR.

Eh, I'd argue that the more common workflow with work trees is to have them adjacent to each other, not nested.

Directory traversel outside of topsrcdir looks like dangerous, and I was hoping to learn whether this bug could potentially result in bad consequences.

You're right that something is wrong here, and that is worth investigating. I just can't quite justify working on this right now (but, of course, I'd be happy to take a look at a patch!)

Comment 8

[Rob Wu [:robwu]]

I know what's happening.

1. autoconf.sh is called (via https://searchfox.org/mozilla-central/rev/014fe72eaba26dcf6082fb9bbaf208f97a38594e/build/moz.configure/old.configure#201-219)

• The working dir is the object dir (from MOZ_OBJDIR).

2. m4 program is called at https://searchfox.org/mozilla-central/rev/014fe72eaba26dcf6082fb9bbaf208f97a38594e/build/autoconf/autoconf.sh#124-125

• The working dir is still MOZ_OBJDIR
• m4 is passed -I/path/to/gecko/worktree2/js/src as one of its parameters, which is supposedly used to look up inclusions from aclocal.m4. As a reminder, aclocal.m4 looks like this:

  ..
  builtin(include, ../../build/autoconf/compiler-opts.m4)dnl
  ..
  

• According to the documentation of m4, the working directory is read first - https://www.gnu.org/software/m4/manual/m4-1.4.15/html_node/Search-Path.html

  m4 allows included files to be found in other directories than the current working directory.

  If the --prepend-include or -B command-line option was provided (see Invoking m4), those directories are searched first, in reverse order that those options were listed on the command line. Then m4 looks in the current working directory. Next comes the directories specified with the --include or -I option, in the order found on the command line. Finally, if the M4PATH environment variable is set, it is expected to contain a colon-separated list of directories, which will be searched in order.

• In short, the order of lookup is --prepend-include/-B (not set), working directory (set to MOZ_OBJDIR), and only then --include/-I (set)

To fix this, one of the following options look:

• Somehow replace ../../ with the absolute path to topsrcdir of (or prepend the directory of js/src/aclocal.m4) in js/src/aclocal.m4
• Edit autoconf.sh to add --prepend-include or -B in the options.
• Change old.configure to change the working directory to the location of the directory containing old-configure

I don't know how to do the first one (heck before today I didn't even know how m4 worked...),
the second option doesn't sound great because autoconf.sh is a (vendored) copy from autoconf.
The third option doesn't look too bad, and seems to fix the issue. I'm going to attach a patch.

Comment 9

[Rob Wu [:robwu]]

Attachment: Bug 1680862 - Use fixed cwd in old.configure for m4 includes

js/src/aclocal.m4 contains includes starting with ../../.
As explained in https://bugzilla.mozilla.org/show_bug.cgi?id=1680862#c8,
m4 will first try to resolve this path relative to the working directory
and only if that path doesn't exists, fall back to the location set by
localdir (from -I).

The working directory is usually MOZ_OBJDIR, an immediate subdirectory
of topsrcdir, so ../../ resolves to a location outside of topsrcdir.
Usually, that path does not exist, and m4 falls back to localdir that
was passed via -I.

But if that path existed and is incompatible with the current Gecko
checkout, then the build will fail (see bug report). To prevent this
from happening, this patch fixes the working directory to localdir,
so that m4 will immediately find the expected file.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ef4d9653b1ae
Use fixed cwd in old.configure for m4 includes r=mhentges

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/ef4d9653b1ae