1614994 - 74.0b1: ImportError: No module named shutil_which

Status: VERIFIED FIXED

(Firefox Build System :: General, defect)

Description

[Landry Breuil (:gaston)]

(from bug #1563797 comment 17)

74.0b1 fails to boostrap configure, from my understanding it fails to find shutil_which which is located in third_party/python/backports.

Our build process runs (and has done so since forever, and i guess other unix/linux downstreams do the same)

cd /usr/obj/ports/firefox-74.0beta1/build-amd64 && /usr/obj/ports/firefox-74.0beta1/firefox-74.0/configure (with many configure args)

by default python2 is used (PYTHON in the env points to /usr/local/bin/python2.7) and i fail to see what 'adds' backports to the path where python should look for modules.

should it be from backports.shutil_which import backport_which ? i think i tried that and it failed.

i also tried

cd /usr/obj/ports/firefox-74.0beta1/build-amd64 &&  /usr/obj/ports/firefox-74.0beta1/firefox-74.0/configure.py (with OLD_CONFIGURE set in env)

as that's more or less what the configure wrapper does but it also fails

Comment 1

[Andrew Halberstadt [:ahal]]

Looks like my comment in the other bug wasn't quite accurate and the shutil_which module was moved around a bit since last I checked. The path is now third_party/python/backports/shutil_which, so you'll need to make sure third_party/python/backports is on your PYTHONPATH:

$ export PYTHONPATH=$PYTHONPATH:<mozsrcdir>/third_party/python/backports

Though like I mentioned, configure.py has always depended on modules in the third_party/python dir, so your build process must have been accounting for that previously somehow. Or maybe there was some magic on our end that was making it work that we neglected to update. I'm not aware of anything though.

Comment 2

[Landry Breuil (:gaston)]

setting PYTHONPATH yields a different failure with python2.7:

  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/configure.py", line 170, in <module>
    sys.exit(main(sys.argv))
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/configure.py", line 46, in main
    sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 444, in run
    self.include_file(path)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 435, in include_file
    exec_(code, self)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 52, in exec_
    exec(object, globals, locals)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/moz.configure", line 7, in <module>
    include('build/moz.configure/init.configure')
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 767, in include_impl
    self.include_file(what)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 435, in include_file
    exec_(code, self)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 52, in exec_
    exec(object, globals, locals)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/build/moz.configure/init.configure", line 254, in <module>
    @imports(_from='six', _import='ensure_text')
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 750, in decorator
    depends = DependsFunction(self, func, dependencies, when=when)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 129, in __init__
    sandbox._value_for(self)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 541, in _value_for
    return self._value_for_depends(obj)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 1017, in method_call
    cache[args] = self.func(instance, *args)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 550, in _value_for_depends
    value = obj.result()
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 1017, in method_call
    cache[args] = self.func(instance, *args)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 155, in result
    return self._func(*resolved_args)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 1155, in wrapped
    return new_func(*args, **kwargs)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/build/moz.configure/init.configure", line 334, in virtualenv_python2
    if version < (2, 7, 0):
UnboundLocalError: local variable 'version' referenced before assignment

i cant believe how complicated this mozbuild contraption became...

Comment 3

[Landry Breuil (:gaston)]

that code in https://searchfox.org/mozilla-central/source/build/moz.configure/init.configure#324 looks wrong as version is tested for even if not set

Comment 4

[Landry Breuil (:gaston)]

that bit seems fallout of #1608880

Comment 5

[Landry Breuil (:gaston)]

if i comment out the version check configure does its thing with 2.7.

Comment 6

[Landry Breuil (:gaston)]

and fwiw i see nothing in the code that would add third_party/python/backports to PYTHONPATH.. but maybe i looked wrong.

Comment 8

[Ricky Stewart]

Attachment: Bug 1614994 - Fix "ImportError: No module named shutil_which" in configure

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0803b57c77a1
Fix "ImportError: No module named shutil_which" in configure r=froydnj

Comment 11

[Landry Breuil (:gaston)]

How was it tested ? with the commit backported on top of 74.0b2, configure still fails the same ?

1581622286===>  Configuring for firefox-74.0beta2                    
Traceback (most recent call last): 
File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/configure.py", line 170, in <module>                                                 
    sys.exit(main(sys.argv))                                                                                                               
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/configure.py", line 46, in main                                                      
    sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))                                                                  
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 444, in run
    self.include_file(path)                                          
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 435, in include_file
    exec_(code, self)                                                
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 52, in exec_
    exec(object, globals, locals)                                                                                                          
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/moz.configure", line 7, in <module>                                                  
    include('build/moz.configure/init.configure')                    
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 767, in include_impl
    self.include_file(what)                                          
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 435, in include_file
    exec_(code, self)                                                
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 52, in exec_
    exec(object, globals, locals)                                    
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/build/moz.configure/init.configure", line 254, in <module>
    @imports(_from='six', _import='ensure_text')                                                                                           
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 750, in decorator
    depends = DependsFunction(self, func, dependencies, when=when)                                                                         
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 129, in __init__
    sandbox._value_for(self)                                         
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 541, in _value_for
    return self._value_for_depends(obj)                              
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 1017, in method_call
    cache[args] = self.func(instance, *args)                                                                                               
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 550, in _value_for_depends
    value = obj.result()                                             
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 1017, in method_call
    cache[args] = self.func(instance, *args)                         
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 155, in result
    return self._func(*resolved_args)                                
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 1155, in wrapped
    return new_func(*args, **kwargs)                                 
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/build/moz.configure/init.configure", line 312, in virtualenv_python2
    found_python = find_program(python)                              
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 1155, in wrapped
    return new_func(*args, **kwargs)                                 
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/build/moz.configure/util.configure", line 185, in find_program
    path = which(os.path.basename(file), path=os.path.dirname(file), exts=exts)                                                            
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/testing/mozbase/mozfile/mozfile/mozfile.py", line 338, in which
    from shutil_which import which as shutil_which                   
ImportError: No module named shutil_which   

it works if i 'fix' the added line to be

sys.path.insert(0, os.path.join(topsrcdir, 'third_party', 'python', 'backports'))

without shutil_which at the end.

Comment 12

[tt_1]

I can confirm @gaston

Comment 13

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/0803b57c77a1

Comment 15

[Landry Breuil (:gaston)]

This is not fixed :)

Comment 16

[Landry Breuil (:gaston)]

[Tracking Requested - why for this release]:

ftbfs for many downstreams

Comment 17

[Ricky Stewart]

Attachment: Bug 1614994 - Adjust sys.path manipulation in configure to ensure we can pick up shutil_which

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2e6f2482bd44
Adjust sys.path manipulation in configure to ensure we can pick up shutil_which r=froydnj

Comment 19

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/2e6f2482bd44

Comment 20

[Ryan VanderMeulen [:RyanVM]]

Please nominate this for Beta uplift when you get a chance.

Comment 21

[Ricky Stewart]

Comment on attachment 9126463 [details]
Bug 1614994 - Fix "ImportError: No module named shutil_which" in configure

Beta/Release Uplift Approval Request

• User impact if declined: Some downstream consumers will fail to build from source
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Small change to prevent breakage in one particular branch of logic in configure
• String changes made/needed:

Comment 22

[Pascal Chevrel:pascalc]

Comment on attachment 9126463 [details]
Bug 1614994 - Fix "ImportError: No module named shutil_which" in configure

Uplift approved for 74.0b6, thanks.

Comment 23

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/releases/mozilla-beta/rev/1dd41a1fbf2c
https://hg.mozilla.org/releases/mozilla-beta/rev/9fb1d309475e

Comment 24

[Landry Breuil (:gaston)]

can confirm that 74.0b6 passes configure without addition patches :)