Closed Bug 1614994 Opened 4 years ago Closed 4 years ago

74.0b1: ImportError: No module named shutil_which

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Platform:
All
OpenBSD
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox-esr68 unaffected, firefox73 unaffected, firefox74+ fixed, firefox75+ fixed)

Status:
VERIFIED FIXED
Milestone:
mozilla75
Tracking Flags:
Tracking Status
firefox-esr68 --- unaffected
firefox73 --- unaffected
firefox74 + fixed
firefox75 + fixed

People

(Reporter: gaston, Assigned: rstewart)

References

(Regression)

Details

(Keywords: regression)

Attachments

(2 files)

Bug 1614994 - Fix "ImportError: No module named shutil_which" in configure
47 bytes, text/x-phabricator-request
pascalc
: approval-mozilla-beta+
Details | Review
Bug 1614994 - Adjust sys.path manipulation in configure to ensure we can pick up shutil_which
47 bytes, text/x-phabricator-request
pascalc
: approval-mozilla-beta+
Details | Review

(from bug #1563797 comment 17)

74.0b1 fails to boostrap configure, from my understanding it fails to find shutil_which which is located in third_party/python/backports.

Our build process runs (and has done so since forever, and i guess other unix/linux downstreams do the same)

cd /usr/obj/ports/firefox-74.0beta1/build-amd64 && /usr/obj/ports/firefox-74.0beta1/firefox-74.0/configure (with many configure args)

by default python2 is used (PYTHON in the env points to /usr/local/bin/python2.7) and i fail to see what 'adds' backports to the path where python should look for modules.

should it be from backports.shutil_which import backport_which ? i think i tried that and it failed.

i also tried

cd /usr/obj/ports/firefox-74.0beta1/build-amd64 &&  /usr/obj/ports/firefox-74.0beta1/firefox-74.0/configure.py (with OLD_CONFIGURE set in env)

as that's more or less what the configure wrapper does but it also fails

Looks like my comment in the other bug wasn't quite accurate and the shutil_which module was moved around a bit since last I checked. The path is now third_party/python/backports/shutil_which, so you'll need to make sure third_party/python/backports is on your PYTHONPATH:

$ export PYTHONPATH=$PYTHONPATH:<mozsrcdir>/third_party/python/backports

Though like I mentioned, configure.py has always depended on modules in the third_party/python dir, so your build process must have been accounting for that previously somehow. Or maybe there was some magic on our end that was making it work that we neglected to update. I'm not aware of anything though.

setting PYTHONPATH yields a different failure with python2.7:

  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/configure.py", line 170, in <module>
    sys.exit(main(sys.argv))
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/configure.py", line 46, in main
    sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 444, in run
    self.include_file(path)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 435, in include_file
    exec_(code, self)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 52, in exec_
    exec(object, globals, locals)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/moz.configure", line 7, in <module>
    include('build/moz.configure/init.configure')
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 767, in include_impl
    self.include_file(what)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 435, in include_file
    exec_(code, self)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 52, in exec_
    exec(object, globals, locals)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/build/moz.configure/init.configure", line 254, in <module>
    @imports(_from='six', _import='ensure_text')
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 750, in decorator
    depends = DependsFunction(self, func, dependencies, when=when)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 129, in __init__
    sandbox._value_for(self)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 541, in _value_for
    return self._value_for_depends(obj)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 1017, in method_call
    cache[args] = self.func(instance, *args)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 550, in _value_for_depends
    value = obj.result()
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 1017, in method_call
    cache[args] = self.func(instance, *args)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 155, in result
    return self._func(*resolved_args)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 1155, in wrapped
    return new_func(*args, **kwargs)
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/build/moz.configure/init.configure", line 334, in virtualenv_python2
    if version < (2, 7, 0):
UnboundLocalError: local variable 'version' referenced before assignment

i cant believe how complicated this mozbuild contraption became...

that code in https://searchfox.org/mozilla-central/source/build/moz.configure/init.configure#324 looks wrong as version is tested for even if not set

that bit seems fallout of #1608880

Regressed by: 1608880
Has Regression Range: --- → yes
Keywords: regression

if i comment out the version check configure does its thing with 2.7.

and fwiw i see nothing in the code that would add third_party/python/backports to PYTHONPATH.. but maybe i looked wrong.

Assignee: nobody → rstewart
Status: NEW → ASSIGNED
Pushed by rstewart@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0803b57c77a1
Fix "ImportError: No module named shutil_which" in configure r=froydnj

How was it tested ? with the commit backported on top of 74.0b2, configure still fails the same ?

1581622286===>  Configuring for firefox-74.0beta2                    
Traceback (most recent call last): 
File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/configure.py", line 170, in <module>                                                 
    sys.exit(main(sys.argv))                                                                                                               
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/configure.py", line 46, in main                                                      
    sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))                                                                  
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 444, in run
    self.include_file(path)                                          
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 435, in include_file
    exec_(code, self)                                                
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 52, in exec_
    exec(object, globals, locals)                                                                                                          
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/moz.configure", line 7, in <module>                                                  
    include('build/moz.configure/init.configure')                    
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 767, in include_impl
    self.include_file(what)                                          
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 435, in include_file
    exec_(code, self)                                                
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 52, in exec_
    exec(object, globals, locals)                                    
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/build/moz.configure/init.configure", line 254, in <module>
    @imports(_from='six', _import='ensure_text')                                                                                           
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 750, in decorator
    depends = DependsFunction(self, func, dependencies, when=when)                                                                         
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 129, in __init__
    sandbox._value_for(self)                                         
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 541, in _value_for
    return self._value_for_depends(obj)                              
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 1017, in method_call
    cache[args] = self.func(instance, *args)                                                                                               
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 550, in _value_for_depends
    value = obj.result()                                             
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/util.py", line 1017, in method_call
    cache[args] = self.func(instance, *args)                         
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 155, in result
    return self._func(*resolved_args)                                
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 1155, in wrapped
    return new_func(*args, **kwargs)                                 
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/build/moz.configure/init.configure", line 312, in virtualenv_python2
    found_python = find_program(python)                              
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/python/mozbuild/mozbuild/configure/__init__.py", line 1155, in wrapped
    return new_func(*args, **kwargs)                                 
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/build/moz.configure/util.configure", line 185, in find_program
    path = which(os.path.basename(file), path=os.path.dirname(file), exts=exts)                                                            
  File "/usr/obj/ports/firefox-74.0beta2/firefox-74.0/testing/mozbase/mozfile/mozfile/mozfile.py", line 338, in which
    from shutil_which import which as shutil_which                   
ImportError: No module named shutil_which

it works if i 'fix' the added line to be

sys.path.insert(0, os.path.join(topsrcdir, 'third_party', 'python', 'backports'))

without shutil_which at the end.

I can confirm @gaston

https://hg.mozilla.org/mozilla-central/rev/0803b57c77a1

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
status-firefox75: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla75

This is not fixed :)

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

[Tracking Requested - why for this release]:

ftbfs for many downstreams

status-firefox74: --- → affected
status-firefox75: fixed → affected
tracking-firefox74: --- → ?
Pushed by rstewart@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2e6f2482bd44
Adjust sys.path manipulation in configure to ensure we can pick up shutil_which r=froydnj

https://hg.mozilla.org/mozilla-central/rev/2e6f2482bd44

Status: REOPENED → RESOLVED
Closed: 4 years ago → 4 years ago
status-firefox75: affected → fixed
Resolution: --- → FIXED

Please nominate this for Beta uplift when you get a chance.

status-firefox73: --- → unaffected
status-firefox-esr68: --- → unaffected
tracking-firefox74: ? → +
tracking-firefox75: --- → +
Flags: needinfo?(rstewart)

Comment on attachment 9126463 [details]
Bug 1614994 - Fix "ImportError: No module named shutil_which" in configure

Beta/Release Uplift Approval Request

  • User impact if declined: Some downstream consumers will fail to build from source
  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Small change to prevent breakage in one particular branch of logic in configure
  • String changes made/needed:
Flags: needinfo?(rstewart)
Attachment #9126463 - Flags: approval-mozilla-beta?
Attachment #9126818 - Flags: approval-mozilla-beta?

Comment on attachment 9126463 [details]
Bug 1614994 - Fix "ImportError: No module named shutil_which" in configure

Uplift approved for 74.0b6, thanks.

Attachment #9126463 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #9126818 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

can confirm that 74.0b6 passes configure without addition patches :)

Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: