949906 - GYP integration into config.status broke Sphinx docs

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

This is full of fail.

$ mach build-docs

Error running mach:

    ['build-docs']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

BuildReaderError: ==============================
ERROR PROCESSING MOZBUILD FILE
==============================

The error occurred while processing the following file:

    /Users/gps/src/firefox/media/mtransport/third_party/moz.build

The error appears to be part of the mozbuild.frontend.reader Python module itself! It is possible you have stumbled across a legitimate bug.

Traceback (most recent call last):
  File "/Users/gps/src/firefox/python/mozbuild/mozbuild/frontend/reader.py", line 679, in read_mozbuild
    metadata=metadata):
  File "/Users/gps/src/firefox/python/mozbuild/mozbuild/frontend/reader.py", line 764, in _read_mozbuild
    gyp_dir.variables))
  File "/Users/gps/src/firefox/python/mozbuild/mozbuild/frontend/gyp_reader.py", line 109, in read_from_gyp
    params=params)
  File "/Users/gps/src/firefox/media/webrtc/trunk/tools/gyp/pylib/gyp/__init__.py", line 132, in Load
    params['parallel'])
  File "/Users/gps/src/firefox/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 2560, in Load
    variables, includes, depth, check, True)
  File "/Users/gps/src/firefox/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 391, in LoadTargetBuildFile
    build_file_data, PHASE_EARLY, variables, build_file_path)
  File "/Users/gps/src/firefox/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 1142, in ProcessVariablesAndConditionsInDict
    variables, build_file, 'variables')
  File "/Users/gps/src/firefox/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 1149, in ProcessVariablesAndConditionsInDict
    expanded = ExpandVariables(value, phase, variables, build_file)
  File "/Users/gps/src/firefox/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 881, in ExpandVariables
    (contents, p.returncode))
GypError: Call to '/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python ./build/dir_exists.py ./third_party/platformsdk_win7/files/redist/x86' returned exit status 2. while trying to load /Users/gps/src/firefox/media/mtransport/third_party/nICEr/nicer.gyp


  File "/Users/gps/src/firefox/tools/docs/mach_commands.py", line 42, in build_docs
    for sandbox in reader.walk_topsrcdir():
  File "/Users/gps/src/firefox/python/mozbuild/mozbuild/frontend/reader.py", line 645, in walk_topsrcdir
    filesystem_absolute=True, read_tiers=True):
  File "/Users/gps/src/firefox/python/mozbuild/mozbuild/frontend/reader.py", line 703, in read_mozbuild
    sys.exc_info()[2], other_error=e)

Comment 1

[Gregory Szorc [:gps]]

Attachment: Ability to not process GYP projects

This wallpaper job should do it.

Comment 2

[Gregory Szorc [:gps]]

Attachment: Add a callback to modify evaluated moz.build sandboxes, fix Sphinx docs

More generic solution, as discussed IRL.

Comment 3

[Mike Hommey [:glandium]]

Comment on attachment 8347071 [details] [diff] [review]
Add a callback to modify evaluated moz.build sandboxes, fix Sphinx docs

Review of attachment 8347071 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/frontend/reader.py
@@ +602,5 @@
>          self._log = logging.getLogger(__name__)
>          self._read_files = set()
>          self._execution_stack = []
>  
> +    def read_topsrcdir(self, sandbox_post_eval_cb=None):

It would be simpler to pass it to __init__ and store it in the BuildReader itself.

@@ +678,5 @@
>          annotations into moz.build files that is independent of the sandbox's
>          execution context.
>  
> +        If process_gyp is False, GYP processing will not occur. The default is
> +        to process GYP projects referenced in GYP_DIRS variables.

This comment is now irrelevant ;)

Comment 4

[Gregory Szorc [:gps]]

Attachment: Add a callback to modify evaluated moz.build sandboxes, fix Sphinx docs;

Incorporated review feedback.

Comment 5

[Mike Hommey [:glandium]]

Comment on attachment 8347091 [details] [diff] [review]
Add a callback to modify evaluated moz.build sandboxes, fix Sphinx docs;

Review of attachment 8347091 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/frontend/reader.py
@@ +652,5 @@
>                  filesystem_absolute=True, read_tiers=True):
>                  yield s
>  
>      def read_mozbuild(self, path, read_tiers=False, filesystem_absolute=False,
> +            descend=True, metadata={}, sandbox_post_eval_cb=None):

Leftover

@@ +709,5 @@
>              raise BuildReaderError(list(self._execution_stack),
>                  sys.exc_info()[2], other_error=e)
>  
>      def _read_mozbuild(self, path, read_tiers, filesystem_absolute, descend,
> +            metadata, sandbox_post_eval_cb=None):

Leftover

@@ +727,5 @@
>          sandbox.exec_file(path, filesystem_absolute=filesystem_absolute)
>          sandbox.execution_time = time.time() - time_start
> +
> +        if self._sandbox_post_eval_cb:
> +            self._sandbox_post_eval_cb(sandbox)

For completeness, you should add the same further below for the gyp sandboxes.

Comment 6

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/24b43ecb4cad

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/24b43ecb4cad