1322025 - ./configure --help does not work

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[terabit.funtoo]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36

Steps to reproduce:

r./configure --help 
#on a fresh clone and bootstrap 


Actual results:

checking for a shell... Traceback (most recent call last):
  File "./configure.py", line 107, in <module>
    sys.exit(main(sys.argv))
  File "./configure.py", line 26, in main
    sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 334, in run
    self.include_file(path)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 325, in include_file
    exec_(code, self)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 59, in exec_
    exec(object, globals, locals)
  File "/crypt/firefox/firefox/moz.configure", line 112, in <module>
    when='--enable-compile-environment')
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 630, in include_impl
    self.include_file(what)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 325, in include_file
    exec_(code, self)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 59, in exec_
    exec(object, globals, locals)
  File "/crypt/firefox/firefox/build/moz.configure/toolchain.configure", line 974, in <module>
    include('windows.configure', when=is_windows)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 630, in include_impl
    self.include_file(what)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 325, in include_file
    exec_(code, self)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 59, in exec_
    exec(object, globals, locals)
  File "/crypt/firefox/firefox/build/moz.configure/windows.configure", line 8, in <module>
    help='Windows SDK version to target. Win 8.1 (603) is currently'
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 562, in option_impl
    self._value_for(when, need_help_dependency=True)):
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 409, in _value_for
    return self._value_for_depends(obj, need_help_dependency)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 925, in method_call
    cache[args] = self.func(instance, *args)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 419, in _value_for_depends
    return obj.result
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 943, in __get__
    setattr(instance, name, self.func(instance))
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 134, in result
    resolved_args = [self.sandbox._value_for(d) for d in deps]
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 409, in _value_for
    return self._value_for_depends(obj, need_help_dependency)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 925, in method_call
    cache[args] = self.func(instance, *args)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 419, in _value_for_depends
    return obj.result
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 943, in __get__
    setattr(instance, name, self.func(instance))
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 89, in result
    resolved_args = [self.sandbox._value_for(d) for d in self.dependencies]
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 409, in _value_for
    return self._value_for_depends(obj, need_help_dependency)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 925, in method_call
    cache[args] = self.func(instance, *args)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 419, in _value_for_depends
    return obj.result
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 943, in __get__
    setattr(instance, name, self.func(instance))
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 89, in result
    resolved_args = [self.sandbox._value_for(d) for d in self.dependencies]
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 409, in _value_for
    return self._value_for_depends(obj, need_help_dependency)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 925, in method_call
    cache[args] = self.func(instance, *args)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 419, in _value_for_depends
    return obj.result
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 943, in __get__
    setattr(instance, name, self.func(instance))
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 89, in result
    resolved_args = [self.sandbox._value_for(d) for d in self.dependencies]
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 409, in _value_for
    return self._value_for_depends(obj, need_help_dependency)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 925, in method_call
    cache[args] = self.func(instance, *args)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 419, in _value_for_depends
    return obj.result
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 943, in __get__
    setattr(instance, name, self.func(instance))
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 90, in result
    return self.func(*resolved_args)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 938, in wrapped
    return new_func(*args, **kwargs)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 671, in wrapper
    ret = template(*args, **kwargs)
  File "/crypt/firefox/firefox/build/moz.configure/checks.configure", line 53, in wrapped
    ret = func(*args, **kwargs)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 938, in wrapped
    return new_func(*args, **kwargs)
  File "/crypt/firefox/firefox/build/moz.configure/init.configure", line 286, in shell
    return find_program(shell)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 936, in wrapped
    self._apply_imports(func, glob)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 736, in _apply_imports
    glob[what] = self._get_one_import('%s%s' % (_from, what))
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/configure/__init__.py", line 756, in _get_one_import
    exec_(import_line, {}, glob)
  File "/crypt/firefox/firefox/python/mozbuild/mozbuild/util.py", line 59, in exec_
    exec(object, globals, locals)
  File "<string>", line 1, in <module>
ImportError: No module named which


Expected results:

It should have listed configure options as documented and instructed in https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Configuring_Build_Options

Comment 1

[terabit.funtoo]

On that last link , it says "The full list of options can be obtained by running ./configure --help."

I need to know all the available build options so I can disable the problematic ones. all in all this should work but it doesn't.

Comment 2

[Mike Hommey [:glandium]]

Bisected to bug 1316957

Comment 4

[Mike Hommey [:glandium]]

Attachment: Bug 1322025 - Enforce --help requirement on indirect dependencies.

Bug 1313306 relaxed the --help dependency requirement in some cases, but
while doing so, the requirement was also removed in other, unexpected
cases. Specifically, the --help dependency ended up not being required
on indirect dependencies that should have had it, had the --help
dependency been explicit on the direct dependency.

Review commit: https://reviewboard.mozilla.org/r/107084/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107084/

Comment 5

[Mike Hommey [:glandium]]

Attachment: Bug 1322025 - Enforce --help requirement on option's when argument.

Options with a `when` argument (either directly, or inherited through
only_when() or an include) require --help per _value_for_option, but
that code path is not exercised during a lint pass.

With this change, along the previous one, we now correctly detect that
bug 1316957 was not supposed to work as is.

Review commit: https://reviewboard.mozilla.org/r/107086/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107086/

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1322025 - Make DependsFunction.func less public.

We're going to change the function signature for CombinedDependsFunction,
so make it visible in the API that the function member is not meant to
be used directly. The linter still does, though, because it needs to
look in their guts.

At the same time, avoid setting DependsFunction names via the function
name itself, because in upcoming changes, it will not be modifiable in
some cases.

Review commit: https://reviewboard.mozilla.org/r/107088/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107088/

Comment 7

[Mike Hommey [:glandium]]

Attachment: Bug 1322025 - Don't wrap the combination function in CombinedDependsFunction.

Several things were wrong with the wrapping:
- the equality test on functions was actually comparing the memoized
  functions, which have a type memoize, which inherits from dict. So
  they weren't comparing actual functions, but the dict used to store
  the cache of their invocation.
- each CombinedDependsFunction created for the same combination function
  used a different wrapped function, so even if the dict problem wasn't
  there, the equality test still wouldn't work, except if the function
  wrapping itself was memoized.
- the memoization was not particularly useful.

Also, for upcoming changes, we'd actually like the combination function to
take an iterable instead of a variable argument list, so that items of
the iterable can be skipped.

Review commit: https://reviewboard.mozilla.org/r/107090/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107090/

Comment 8

[Mike Hommey [:glandium]]

Attachment: Bug 1322025 - Don't automatically add --help dependencies to CombinedDependsFunctions.

Adding those dependencies, retrospectively, only worked around the poor
handling of --help requirements by the linter, that we fixed a few
commits ago. This is now not necessary anymore.

Review commit: https://reviewboard.mozilla.org/r/107092/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107092/

Comment 9

[Mike Hommey [:glandium]]

Attachment: Bug 1322025 - Allow to combine two DependsFunctions with "|".

Ideally, it would have been better if it were "or", but it's not
possible to override "or" in python ; __or__ is for "|".

This does feel magic, but it's also shorter than adding something like
@depends_any(), and while we're only adding "|" as of this change, we
can add other operations such as "&" in the future, or __getattr__ for
things like milestone.is_nightly.

An alternative form in moz.configure could require the @depends function
to be called, e.g. "a() | b()" instead of "a | b", but I'm not
particularly convinced that one is less magic than the other.

This feature is hooked up such that b is not resolved if a is true,
although in practice, it will still be resolved in Sandbox.run... but
not when --help is passed. In the long run, the forced resolution of
@depends functions will be removed from Sandbox.run.

Review commit: https://reviewboard.mozilla.org/r/107094/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107094/

Comment 10

[Mike Hommey [:glandium]]

Attachment: Bug 1322025 - Provide variants of host and target that depend on --help.

We want to avoid giving --help dependencies to host and target, so that
they we don't spawn config.guess and config.sub when running configure
--help, and don't need to reach out to the which module to find a
suitable shell to execute them.

So, when --help is given, we return a fake host/target namespace, and
avoid the config.guess/config.sub-invoking code being executed.

Then, by giving the --help option to the linter, it can properly find
that the config.guess/config.sub-invoking code doesn't need the
dependency on --help.

This effectively unbreaks configure --help after bug 1313306.

Review commit: https://reviewboard.mozilla.org/r/107096/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107096/

Comment 11

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8830193 [details]
Bug 1322025 - Enforce --help requirement on indirect dependencies.

https://reviewboard.mozilla.org/r/107084/#review108692

Comment 12

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8830194 [details]
Bug 1322025 - Enforce --help requirement on option's when argument.

https://reviewboard.mozilla.org/r/107086/#review108694

Comment 13

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8830195 [details]
Bug 1322025 - Make DependsFunction.func less public.

https://reviewboard.mozilla.org/r/107088/#review108700

Comment 14

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8830196 [details]
Bug 1322025 - Don't wrap the combination function in CombinedDependsFunction.

https://reviewboard.mozilla.org/r/107090/#review108740

::: python/mozbuild/mozbuild/test/configure/test_configure.py:875
(Diff revision 1)
> +        with self.moz_configure('''
> +            @depends(when=True)
> +            def always():
> +                return True
> +            @depends(when=True)
> +            def always2():
> +                return True
> +            with only_when(always2):
> +                option('--with-foo', help='foo', when=always)
> +                set_config('FOO', depends('--with-foo', when=always)(lambda x: x))
> +        '''):
> +            self.get_config()

Does this raise without the other changes in this commit? I don't see exactly how it's testing this issue.

Comment 15

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8830197 [details]
Bug 1322025 - Don't automatically add --help dependencies to CombinedDependsFunctions.

https://reviewboard.mozilla.org/r/107092/#review108742

Comment 16

[Mike Hommey [:glandium]]

Comment on attachment 8830196 [details]
Bug 1322025 - Don't wrap the combination function in CombinedDependsFunction.

https://reviewboard.mozilla.org/r/107090/#review108744

::: python/mozbuild/mozbuild/test/configure/test_configure.py:875
(Diff revision 1)
> +        with self.moz_configure('''
> +            @depends(when=True)
> +            def always():
> +                return True
> +            @depends(when=True)
> +            def always2():
> +                return True
> +            with only_when(always2):
> +                option('--with-foo', help='foo', when=always)
> +                set_config('FOO', depends('--with-foo', when=always)(lambda x: x))
> +        '''):
> +            self.get_config()

Yeah, it's not entirely testing it. That is, it passes by chance without the patch, while it does pass for a good reason with it. I'll try to make it fail without the patch.

Comment 17

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8830198 [details]
Bug 1322025 - Allow to combine two DependsFunctions with "|".

https://reviewboard.mozilla.org/r/107094/#review108750

::: python/mozbuild/mozbuild/configure/__init__.py:125
(Diff revision 1)
> +    def any(iterable):
> +        # Like the builtin any(), but returns the first element that is true,
> +        # instead of True. If none are true, returns the last element.
> +        for i in iterable:

This is fine, I guess `first` be a better name.

Comment 18

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8830199 [details]
Bug 1322025 - Provide variants of host and target that depend on --help.

https://reviewboard.mozilla.org/r/107096/#review108772

::: python/mozbuild/mozbuild/test/configure/lint.py:60
(Diff revision 1)
>      def lint(self, project):
>          sandbox = LintSandbox({
>              'OLD_CONFIGURE': os.path.join(topsrcdir, 'old-configure'),
>              'MOZCONFIG': os.path.join(os.path.dirname(test_path), 'data',
>                                        'empty_mozconfig'),
> -        }, ['--enable-project=%s' % project])
> +        }, ['--enable-project=%s' % project, '--help'])

Would it be interesting to run the lint with help as well as without it? It doesn't look like it, but it's not entirely obvious to me.

Comment 19

[Mike Hommey [:glandium]]

(In reply to Chris Manchester (:chmanchester) from comment #18)
> Comment on attachment 8830199 [details]
> Bug 1322025 - Provide variants of host and target that depend on --help.
> 
> https://reviewboard.mozilla.org/r/107096/#review108772
> 
> ::: python/mozbuild/mozbuild/test/configure/lint.py:60
> (Diff revision 1)
> >      def lint(self, project):
> >          sandbox = LintSandbox({
> >              'OLD_CONFIGURE': os.path.join(topsrcdir, 'old-configure'),
> >              'MOZCONFIG': os.path.join(os.path.dirname(test_path), 'data',
> >                                        'empty_mozconfig'),
> > -        }, ['--enable-project=%s' % project])
> > +        }, ['--enable-project=%s' % project, '--help'])
> 
> Would it be interesting to run the lint with help as well as without it? It
> doesn't look like it, but it's not entirely obvious to me.

Without --help, it would say there's a missing --help dependency on target.

Comment 20

[Mike Hommey [:glandium]]

Comment on attachment 8830193 [details]
Bug 1322025 - Enforce --help requirement on indirect dependencies.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/107084/diff/1-2/

Comment 21

[Mike Hommey [:glandium]]

Comment on attachment 8830194 [details]
Bug 1322025 - Enforce --help requirement on option's when argument.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/107086/diff/1-2/

Comment 22

[Mike Hommey [:glandium]]

Comment on attachment 8830195 [details]
Bug 1322025 - Make DependsFunction.func less public.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/107088/diff/1-2/

Comment 23

[Mike Hommey [:glandium]]

Comment on attachment 8830196 [details]
Bug 1322025 - Don't wrap the combination function in CombinedDependsFunction.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/107090/diff/1-2/

Comment 24

[Mike Hommey [:glandium]]

Comment on attachment 8830197 [details]
Bug 1322025 - Don't automatically add --help dependencies to CombinedDependsFunctions.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/107092/diff/1-2/

Comment 25

[Mike Hommey [:glandium]]

Comment on attachment 8830198 [details]
Bug 1322025 - Allow to combine two DependsFunctions with "|".

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/107094/diff/1-2/

Comment 26

[Mike Hommey [:glandium]]

Comment on attachment 8830199 [details]
Bug 1322025 - Provide variants of host and target that depend on --help.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/107096/diff/1-2/

Comment 27

[Mike Hommey [:glandium]]

Comment on attachment 8830198 [details]
Bug 1322025 - Allow to combine two DependsFunctions with "|".

https://reviewboard.mozilla.org/r/107094/#review108750

> This is fine, I guess `first` be a better name.

I made it `first_true`

Comment 28

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8830196 [details]
Bug 1322025 - Don't wrap the combination function in CombinedDependsFunction.

https://reviewboard.mozilla.org/r/107090/#review108802

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/97c019e39b8f
Enforce --help requirement on indirect dependencies. r=chmanchester
https://hg.mozilla.org/integration/autoland/rev/42f5b6502a64
Enforce --help requirement on option's when argument. r=chmanchester
https://hg.mozilla.org/integration/autoland/rev/175dabab373b
Make DependsFunction.func less public. r=chmanchester
https://hg.mozilla.org/integration/autoland/rev/0b17ca34cb2e
Don't wrap the combination function in CombinedDependsFunction. r=chmanchester
https://hg.mozilla.org/integration/autoland/rev/5e9a5b5829d1
Don't automatically add --help dependencies to CombinedDependsFunctions. r=chmanchester
https://hg.mozilla.org/integration/autoland/rev/9c02acca893d
Allow to combine two DependsFunctions with "|". r=chmanchester
https://hg.mozilla.org/integration/autoland/rev/c78a57144263
Provide variants of host and target that depend on --help. r=chmanchester

Comment 30

[Mike Hommey [:glandium]]

Fixed on aurora by backing out the root cause.
https://hg.mozilla.org/releases/mozilla-aurora/rev/cceb7f61ef17d0c10d42f4e4a9820d40cc46528d

Comment 31

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/97c019e39b8f
https://hg.mozilla.org/mozilla-central/rev/42f5b6502a64
https://hg.mozilla.org/mozilla-central/rev/175dabab373b
https://hg.mozilla.org/mozilla-central/rev/0b17ca34cb2e
https://hg.mozilla.org/mozilla-central/rev/5e9a5b5829d1
https://hg.mozilla.org/mozilla-central/rev/9c02acca893d
https://hg.mozilla.org/mozilla-central/rev/c78a57144263