1563797 - Use 'backports.shutil_which' instead of the 'which' module across the tree

Status: RESOLVED FIXED

(Firefox Build System :: General, task)

Description

[Andrew Halberstadt [:ahal]]

We currently have a vendored library called which that we use throughout the tree. The problem is that it only supports Python 2 and the last commit was more than 10 years ago. We should stop using this module and instead vendor backports.shutil_which which is a drop-in replacement for Python 3's shutil.which.

This module will let us write code like:

try:
    from shutil import which
except ImportError:
    from shutil_which import which

There are a few differences with the which module we'll need to fix. Namely the old module raised a WhichError if the binary wasn't found, whereas the new one simply returns None. The arguments are also slightly different.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1563797 - Vendor backports.shutil_which

I couldn't use 'mach vendor python' to get this module in-tree because that
puts a 'backports' module on the PYTHONPATH. This is a problem because many
(most?) compatibility backports also use a 'backports' library. So in the
likely event that a user happens to have one of these installed in their
system Python, we'll search that module before the vendored one and fail to
find it.

This gets around the issue by foregoing 'backports' and just putting
'shutil_which' on the path

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1563797 - Use 'backports.shutil_which' instead of 'which' across the tree

Depends on D36838

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1563797 - Remove third_party/python/which

This module has been replaced by 'shutil_which' and is no longer needed.

Depends on D37097

Comment 4

[Andrew Halberstadt [:ahal]]

There are a few subtle differences Callek discovered on Windows as well:

1. A program is only found if it has an extension in PATHEXT. E.g, calling which("autoconf2.13") won't find anything because there's no file called autoconf2.13.exe anywhere. This can be worked around by prepending "." to PATHEXT. There's an issue tracking this:
   https://bugs.python.org/issue31405

2. The current working directory will be searched before things on the PATH. This is intentional to match up with the behaviour of Windows' built-in functionality in the command prompt.

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1563797 - Use 'backports.shutil_which' instead of 'which' in the build system

Depends on D37097

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1563797 - Use 'backports.shutil_which' instead of 'which' in configure

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1563797 - [mozfile] Add a 'which' function to make 'shutil.which' consistent on Windows

Depends on D36838

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e29cc4ee9158
Vendor backports.shutil_which r=Callek
https://hg.mozilla.org/integration/autoland/rev/7e4c633a3566
[mozfile] Add a 'which' function to make 'shutil.which' consistent on Windows r=Callek
https://hg.mozilla.org/integration/autoland/rev/7f92f148ea54
Use 'backports.shutil_which' instead of 'which' across the tree r=Callek
https://hg.mozilla.org/integration/autoland/rev/423da1d755f5
Use 'backports.shutil_which' instead of 'which' in the build system r=firefox-build-system-reviewers,chmanchester

Comment 9

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/e29cc4ee9158
https://hg.mozilla.org/mozilla-central/rev/7e4c633a3566
https://hg.mozilla.org/mozilla-central/rev/7f92f148ea54
https://hg.mozilla.org/mozilla-central/rev/423da1d755f5

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9837631a9c71
Use 'backports.shutil_which' instead of 'which' in configure r=glandium

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c9e597d3ee12
Use 'backports.shutil_which' instead of 'which' in configure r=glandium

Comment 14

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/c9e597d3ee12

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/525b371e48a9
Remove third_party/python/which r=Callek

Comment 16

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/525b371e48a9

Comment 17

[Landry Breuil (:gaston)]

Fwiw, with 74.0b1 i fail to pass configure (our infra just runs the old-style ./configure from a separate builddir)

Traceback (most recent call last):                                                                                                         
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/configure.py", line 170, in <module>
    sys.exit(main(sys.argv))                                                                                                               
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/configure.py", line 46, in main
    sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))                                                                  
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 444, in run
    self.include_file(path)                                          
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 435, in include_file
    exec_(code, self)                                                                                                                      
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/util.py", line 52, in exec_
    exec(object, globals, locals)                                                                                                          
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/moz.configure", line 7, in <module>                                                  
    include('build/moz.configure/init.configure')                    
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 767, in include_impl
    self.include_file(what)                                                                                                                  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 435, in include_file               exec_(code, self)                                                                                                                        File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/util.py", line 52, in exec_
    exec(object, globals, locals)                                                                                                            File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/build/moz.configure/init.configure", line 254, in <module>                               @imports(_from='six', _import='ensure_text')                                                                                             File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 750, in decorator                  depends = DependsFunction(self, func, dependencies, when=when)                                                                         
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 129, in __init__
    sandbox._value_for(self)                                                                                                               
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 541, in _value_for             
    return self._value_for_depends(obj)                              
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/util.py", line 1017, in method_call
    cache[args] = self.func(instance, *args)
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 550, in _value_for_depends
    value = obj.result()
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/util.py", line 1017, in method_call
    cache[args] = self.func(instance, *args)
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 155, in result
    return self._func(*resolved_args)
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 1155, in wrapped
    return new_func(*args, **kwargs)
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/build/moz.configure/init.configure", line 309, in virtualenv_python2
    found_python = find_program(python)
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 1155, in wrapped
    return new_func(*args, **kwargs)
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/build/moz.configure/util.configure", line 185, in find_program
    path = which(os.path.basename(file), path=os.path.dirname(file), exts=exts)
  File "/usr/obj/ports/firefox-74.0beta1/firefox-74.0/testing/mozbase/mozfile/mozfile/mozfile.py", line 338, in which
    from shutil_which import  which as shutil_which 
ImportError: No module named shutil_which

no virtualenv at all is initialized afaict.

this is with PYTHON=/usr/local/bin/python2.7 in the env, and from my understanding for this version the which from third_party/python/backports/shutil_which should be used, but it doenst seem so? tried with PYTHON=/usr/local/bin/python3.7 as default and that also fails.

what in the env could cause such clusterfuck failure ? should we move to just use 3.7 everywhere ?

@ahal should i file a regression bug ?

Comment 18

[Andrew Halberstadt [:ahal]]

Sure, please file a bug and make sure to CC myself, :glandium and :rstewart.

I have zero knowledge around OpenBSD build processes, but previously configure depended on a vendored third_party/python/which module. Now it depends on third_party/python/shutil_which. When running configure via our mach command dispatcher, these modules will be placed on the sys.path. If you are invoking configure some other way, you must have had logic to find that old which module, so maybe that just needs to be updated to find shutil_which instead?

Tbh, configure (and the build system in general) is not my area of expertise. Though I'm fairly certain that running configure with Python 3 is expected to fail at this time.