Problems ./mach bootstrap complains about sqlite3 module
Categories
(Firefox Build System :: Bootstrap Configuration, defect)
Tracking
(firefox-esr91 unaffected, firefox96 unaffected, firefox97 fixed, firefox98 fixed)
Tracking | Status | |
---|---|---|
firefox-esr91 | --- | unaffected |
firefox96 | --- | unaffected |
firefox97 | --- | fixed |
firefox98 | --- | fixed |
People
(Reporter: smurfd, Assigned: smurfd)
References
(Regression)
Details
(Keywords: regression)
Attachments
(4 files)
Yesterday it had been a while since i did a hg pull && hg update
and a ./mach clobber && ./mach bootstrap
I have done one this year atleast, but not sure when... probably first couple of days on the new year...
On openSUSE linux, have done the above procedure many many many times before.
See attached log for a more detailed output of mach bootstrap
It ends with an error :
File "/usr/lib64/python3.6/_import_failed/sqlite3.py", line 19, in <module>
to install it.""")
ImportError: Module 'sqlite3' is not installed.
I can however import sqlite3 using the python3 shell :
Python 3.6.15 (default, Sep 23 2021, 15:41:43) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>>
I have tried todo the following :
- remove ~/.mozbuild
- remove the obj dir
- clone a new mozilla central tree
and all times trying a new :./mach bootstrap
without any luck...
I also tried to use Python 3.9 but that made no difference... let me know if you want some more info, i can reproduce it :)
The workaround i found was applying this diff, Not sure what breakage that causes though.. then i can bootstrap and build atleast:
diff --git a/testing/mozbase/mozprofile/mozprofile/permissions.py b/testing/mozbase/mozprofile/mozprofile/permissions.py
--- a/testing/mozbase/mozprofile/mozprofile/permissions.py
+++ b/testing/mozbase/mozprofile/mozprofile/permissions.py
@@ -11,7 +11,6 @@ from __future__ import absolute_import
import codecs
import os
-import sqlite3
from six import string_types
from six.moves.urllib import parse
Comment 1•3 years ago
|
||
I'm guessing this is caused by this bug.
Can you provide some console output for me to help start diagnosing this?
python3 -c "import sqlite3; print(sqlite3)"
- In
mozprofile/permissions.py
, right before whereimport sqlite3
would be, instead add the lines:
import sys
print(sys.path)
from mach.site import MozSiteMetadata
print(vars(MozSiteMetadata.current))
Updated•3 years ago
|
Assignee | ||
Comment 2•3 years ago
|
||
Ofcourse, see attached for the output of python3 -m
and ./mach bootstrap
after adding the python code
Comment 3•3 years ago
|
||
Hmm, you're using SitePackagesSource.VENV
, which is good.
Also, your /usr/lib64/python3.6
(where sqlite3
is) is before /usr/lib64/python3.6/_import_failed
in the sys.path
, which is also good. I'm not sure why _import_failed/sqlite3
is being found first 🤔.
Next steps:
- If you re-add the
import sqlite3
below thoseprint()
statements, it still fails, yeah? - What if you create a
test.py
file with the following contents (putting$HOME
back), then doingpython3 test.py
?
import sys
sys.path = ['$HOME/.mozbuild/srcdirs/mc-4240be0810f8/_virtualenvs/mach/lib64/python3.6/site-packages', '$HOME/Code/Mozilla/mc/build', '$HOME/Code/Mozilla/mc/config', '$HOME/Code/Mozilla/mc/config/mozunit', '$HOME/Code/Mozilla/mc/dom/bindings', '$HOME/Code/Mozilla/mc/dom/bindings/parser', '$HOME/Code/Mozilla/mc/layout/tools/reftest', '$HOME/Code/Mozilla/mc/python/l10n', '$HOME/Code/Mozilla/mc/python/mach', '$HOME/Code/Mozilla/mc/python/mozboot', '$HOME/Code/Mozilla/mc/python/mozbuild', '$HOME/Code/Mozilla/mc/python/mozlint', '$HOME/Code/Mozilla/mc/python/mozperftest', '$HOME/Code/Mozilla/mc/python/mozrelease', '$HOME/Code/Mozilla/mc/python/mozterm', '$HOME/Code/Mozilla/mc/python/mozversioncontrol', '$HOME/Code/Mozilla/mc/security/manager/tools', '$HOME/Code/Mozilla/mc/taskcluster', '$HOME/Code/Mozilla/mc/testing', '$HOME/Code/Mozilla/mc/testing/awsy', '$HOME/Code/Mozilla/mc/testing/condprofile', '$HOME/Code/Mozilla/mc/testing/firefox-ui/harness', '$HOME/Code/Mozilla/mc/testing/marionette/client', '$HOME/Code/Mozilla/mc/testing/marionette/harness', '$HOME/Code/Mozilla/mc/testing/mozbase/manifestparser', '$HOME/Code/Mozilla/mc/testing/mozbase/mozcrash', '$HOME/Code/Mozilla/mc/testing/mozbase/mozdebug', '$HOME/Code/Mozilla/mc/testing/mozbase/mozdevice', '$HOME/Code/Mozilla/mc/testing/mozbase/mozfile', '$HOME/Code/Mozilla/mc/testing/mozbase/mozgeckoprofiler', '$HOME/Code/Mozilla/mc/testing/mozbase/mozhttpd', '$HOME/Code/Mozilla/mc/testing/mozbase/mozinfo', '$HOME/Code/Mozilla/mc/testing/mozbase/mozinstall', '$HOME/Code/Mozilla/mc/testing/mozbase/mozleak', '$HOME/Code/Mozilla/mc/testing/mozbase/mozlog', '$HOME/Code/Mozilla/mc/testing/mozbase/moznetwork', '$HOME/Code/Mozilla/mc/testing/mozbase/mozpower', '$HOME/Code/Mozilla/mc/testing/mozbase/mozprocess', '$HOME/Code/Mozilla/mc/testing/mozbase/mozprofile', '$HOME/Code/Mozilla/mc/testing/mozbase/mozproxy', '$HOME/Code/Mozilla/mc/testing/mozbase/mozrunner', '$HOME/Code/Mozilla/mc/testing/mozbase/mozscreenshot', '$HOME/Code/Mozilla/mc/testing/mozbase/mozsystemmonitor', '$HOME/Code/Mozilla/mc/testing/mozbase/moztest', '$HOME/Code/Mozilla/mc/testing/mozbase/mozversion', '$HOME/Code/Mozilla/mc/testing/raptor', '$HOME/Code/Mozilla/mc/testing/talos', '$HOME/Code/Mozilla/mc/testing/web-platform', '$HOME/Code/Mozilla/mc/testing/web-platform/tests/tools/third_party/funcsigs', '$HOME/Code/Mozilla/mc/testing/web-platform/tests/tools/third_party/h2', '$HOME/Code/Mozilla/mc/testing/web-platform/tests/tools/third_party/hpack', '$HOME/Code/Mozilla/mc/testing/web-platform/tests/tools/third_party/html5lib', '$HOME/Code/Mozilla/mc/testing/web-platform/tests/tools/third_party/hyperframe', '$HOME/Code/Mozilla/mc/testing/web-platform/tests/tools/third_party/pywebsocket3', '$HOME/Code/Mozilla/mc/testing/web-platform/tests/tools/third_party/webencodings', '$HOME/Code/Mozilla/mc/testing/web-platform/tests/tools/wptrunner', '$HOME/Code/Mozilla/mc/testing/web-platform/tests/tools/wptserve', '$HOME/Code/Mozilla/mc/testing/xpcshell', '$HOME/Code/Mozilla/mc/third_party/python/Jinja2', '$HOME/Code/Mozilla/mc/third_party/python/MarkupSafe/src', '$HOME/Code/Mozilla/mc/third_party/python/PyYAML/lib3', '$HOME/Code/Mozilla/mc/third_party/python/aiohttp', '$HOME/Code/Mozilla/mc/third_party/python/appdirs', '$HOME/Code/Mozilla/mc/third_party/python/async_timeout', '$HOME/Code/Mozilla/mc/third_party/python/attrs', '$HOME/Code/Mozilla/mc/third_party/python/blessings', '$HOME/Code/Mozilla/mc/third_party/python/cbor2', '$HOME/Code/Mozilla/mc/third_party/python/certifi', '$HOME/Code/Mozilla/mc/third_party/python/chardet', '$HOME/Code/Mozilla/mc/third_party/python/click', '$HOME/Code/Mozilla/mc/third_party/python/colorama', '$HOME/Code/Mozilla/mc/third_party/python/compare_locales', '$HOME/Code/Mozilla/mc/third_party/python/cookies', '$HOME/Code/Mozilla/mc/third_party/python/cram', '$HOME/Code/Mozilla/mc/third_party/python/diskcache', '$HOME/Code/Mozilla/mc/third_party/python/distro', '$HOME/Code/Mozilla/mc/third_party/python/dlmanager', '$HOME/Code/Mozilla/mc/third_party/python/ecdsa', '$HOME/Code/Mozilla/mc/third_party/python/esprima', '$HOME/Code/Mozilla/mc/third_party/python/fluent.migrate', '$HOME/Code/Mozilla/mc/third_party/python/fluent.syntax', '$HOME/Code/Mozilla/mc/third_party/python/gyp/pylib', '$HOME/Code/Mozilla/mc/third_party/python/idna', '$HOME/Code/Mozilla/mc/third_party/python/idna-ssl', '$HOME/Code/Mozilla/mc/third_party/python/importlib_metadata', '$HOME/Code/Mozilla/mc/third_party/python/iso8601', '$HOME/Code/Mozilla/mc/third_party/python/jsmin', '$HOME/Code/Mozilla/mc/third_party/python/json-e', '$HOME/Code/Mozilla/mc/third_party/python/jsonschema', '$HOME/Code/Mozilla/mc/third_party/python/mohawk', '$HOME/Code/Mozilla/mc/third_party/python/mozilla_version', '$HOME/Code/Mozilla/mc/third_party/python/multidict', '$HOME/Code/Mozilla/mc/third_party/python/packaging', '$HOME/Code/Mozilla/mc/third_party/python/pathspec', '$HOME/Code/Mozilla/mc/third_party/python/pip', '$HOME/Code/Mozilla/mc/third_party/python/pip_tools', '$HOME/Code/Mozilla/mc/third_party/python/ply', '$HOME/Code/Mozilla/mc/third_party/python/pyasn1', '$HOME/Code/Mozilla/mc/third_party/python/pyasn1_modules', '$HOME/Code/Mozilla/mc/third_party/python/pylru', '$HOME/Code/Mozilla/mc/third_party/python/pyparsing', '$HOME/Code/Mozilla/mc/third_party/python/pyrsistent', '$HOME/Code/Mozilla/mc/third_party/python/python-hglib', '$HOME/Code/Mozilla/mc/third_party/python/pytoml', '$HOME/Code/Mozilla/mc/third_party/python/redo', '$HOME/Code/Mozilla/mc/third_party/python/requests', '$HOME/Code/Mozilla/mc/third_party/python/requests_unixsocket', '$HOME/Code/Mozilla/mc/third_party/python/responses', '$HOME/Code/Mozilla/mc/third_party/python/rsa', '$HOME/Code/Mozilla/mc/third_party/python/sentry_sdk', '$HOME/Code/Mozilla/mc/third_party/python/setuptools', '$HOME/Code/Mozilla/mc/third_party/python/six', '$HOME/Code/Mozilla/mc/third_party/python/slugid', '$HOME/Code/Mozilla/mc/third_party/python/taskcluster', '$HOME/Code/Mozilla/mc/third_party/python/taskcluster_taskgraph', '$HOME/Code/Mozilla/mc/third_party/python/taskcluster_urls', '$HOME/Code/Mozilla/mc/third_party/python/tqdm', '$HOME/Code/Mozilla/mc/third_party/python/typing_extensions', '$HOME/Code/Mozilla/mc/third_party/python/urllib3', '$HOME/Code/Mozilla/mc/third_party/python/voluptuous', '$HOME/Code/Mozilla/mc/third_party/python/wheel', '$HOME/Code/Mozilla/mc/third_party/python/yamllint', '$HOME/Code/Mozilla/mc/third_party/python/yarl', '$HOME/Code/Mozilla/mc/third_party/python/zipp', '$HOME/Code/Mozilla/mc/toolkit/components/telemetry/tests/marionette/harness', '$HOME/Code/Mozilla/mc/tools', '$HOME/Code/Mozilla/mc/tools/moztreedocs', '$HOME/Code/Mozilla/mc/xpcom/ds/tools', '$HOME/Code/Mozilla/mc/xpcom/geckoprocesstypes_generator', '$HOME/Code/Mozilla/mc/xpcom/idl-parser', '$HOME/.mozbuild/srcdirs/mc-4240be0810f8/_virtualenvs/mach', '$HOME/.mozbuild/srcdirs/mc-4240be0810f8/_virtualenvs/mach/lib/python3.6/site-packages', '', '/usr/lib/python36.zip', '/usr/lib64/python3.6/lib-dynload', '/usr/lib64/python3.6', '/usr/lib64/python3.6/_import_failed']
import sqlite3
- If it fails, can you try bisecting it slowly until you find out which
sys.path
change is causing the failure? You'll want to bisect back to whatsys.path
is pre-modification.
Comment 4•3 years ago
|
||
Set release status flags based on info from the regressing bug 1730712
Assignee | ||
Comment 5•3 years ago
|
||
hmmmm i start by answering the questions.
- yeah it fails.
- that test.py works :) with full path and with $HOME
the only thing causing it to fail is if i comment away/usr/lib64/python3.6
or by adding/usr/lib64/python3.6/_import_failed
above the python3.6 folder. which i assume is as it should be.
I have however done some findings.
Tried in a debian VM, no issue.
Tried in a openSUSE VM issue appears. or rather, with a fresh clone, im able to run bootstrap once ... if i run it immediately after, it fails... if i jam it a couple of times 3-6 it will run :D
(the same goes for my non-VM openSUSE, after accidently updating the tree)
did a script like :
import sys
print(sys.path)
it would give the output :
['$HOME/Code/Mozilla/mc', '/usr/lib/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '$HOME/.local/lib/python3.6/site-packages', '/usr/lib64/python3.6/site-packages', '/usr/lib64/python3.6/_import_failed', '/usr/lib/python3.6/site-packages']
Im starting to suspect it being Python3.6 that is causing issues. It is a strange one though...
Maby ill try firing up a CentOS vm (they have python36 as default aswell) during the weekend and see if it reproduces.
Comment 6•3 years ago
|
||
Ooh, good findings, yeah!
I wonder if you make a script outside of the Firefox source that just does import sqlite3
, and try running it a bunch of times, if it will fail on its own sometimes.
If so, then it's an issue with how the distro is adding /usr/lib64/python3.6/_import_failed
to the sys.path
.
Updated•3 years ago
|
Assignee | ||
Comment 7•3 years ago
|
||
heh no no, cant be that easy :)
(ie import sqlite3
script running well outside of firefox source 100 times in a row)
Something is weird with openSUSE though...
Tried to reproduce it on centOS python 3.6, debian running python 3.7 and ubuntu bionic running python 3.6.
no luck.
Tried on previous stable openSUSE 15.2 release running python 3.6, it happens again...
Started printing out alot of stuff. Like this module name : https://searchfox.org/mozilla-central/rev/f7eeca4e34daea900fbedbc392c8c99f90923143/python/mach/mach/main.py#263 ie doing a print("------", module_name)
right after that row.
When it failed, it always stop with this one : https://searchfox.org/mozilla-central/rev/f7eeca4e34daea900fbedbc392c8c99f90923143/build/mach_initialize.py#63
Somehow the last part of the list didnt come with....
Reverted this : https://phabricator.services.mozilla.com/D134201 then it works flawlessly 10 times ./mach bootstrap
in a row, and building successfully
Its tough to compare distributions though, since the bootstrap scripts are different... but that should not really matter, right?
hmm might try their rolling release that has newer python version as default...
Comment 8•3 years ago
|
||
Its tough to compare distributions though, since the bootstrap scripts are different... but that should not really matter, right?
hmm might try their rolling release that has newer python version as default...
Another test you can try to force a failure, that isn't platform-specific (like ./mach bootstrap
), is to do ./mach python -c pass
. That's one of the simplest Mach commands you can run, since it just does a "no-op" and exits.
Reverted this : https://phabricator.services.mozilla.com/D134201 then it works flawlessly 10 times ./mach bootstrap in a row, and building successfully
Hmm, I'm not surprised that that is the cause indeed, but I'd definitely like to dig in further to see why it's failing (and inconsistently).
When it failed, it always stop with this one : https://searchfox.org/mozilla-central/rev/f7eeca4e34daea900fbedbc392c8c99f90923143/build/mach_initialize.py#63
Probably because it's a transitive dependency of remote/mach_commands.py
that imports sqlite3
:
File "$HOME/Code/Mozilla/mc/remote/mach_commands.py", line 36, in <module>
import mozprofile
File "$HOME/Code/Mozilla/mc/testing/mozbase/mozprofile/mozprofile/__init__.py", line 17, in <module>
from mozprofile.cli import *
File "$HOME/Code/Mozilla/mc/testing/mozbase/mozprofile/mozprofile/cli.py", line 18, in <module>
from .profile import FirefoxProfile
File "$HOME/Code/Mozilla/mc/testing/mozbase/mozprofile/mozprofile/profile.py", line 28, in <module>
from .permissions import Permissions
File "$HOME/Code/Mozilla/mc/testing/mozbase/mozprofile/mozprofile/permissions.py", line 14, in <module>
import sqlite3
File "/usr/lib64/python3.6/_import_failed/sqlite3.py", line 19, in <module>
to install it.""")
Can you try this test again (the printing of sys.path
), but show the output when that prints and the failure is encountered?
I'd like to see if "/usr/lib64/python3.6/_import_failed"
is higher in the list when that failure case is triggered.
Then, if you're able to see where it gets moved higher in the sys.path
, that would be immensely helpful :)
Thanks for your help so far!
Assignee | ||
Comment 9•3 years ago
|
||
found it! :)
attaching a file with the diff with all the print's and also the output of those.
https://searchfox.org/mozilla-central/rev/f7eeca4e34daea900fbedbc392c8c99f90923143/python/mach/mach/site.py#348
actually this is the thing that fails, and if i run this in a script, it fails intermittently... :( :)
https://searchfox.org/mozilla-central/rev/f7eeca4e34daea900fbedbc392c8c99f90923143/python/mach/mach/site.py#826-835
import sys
import site
print(set(sys.path)-set([site.getusersitepackages()] + site.getsitepackages()))
$ python3 pp.py
{'/usr/lib/python36.zip', '/usr/lib64/python3.6/_import_failed', '/usr/lib64/python3.6/lib-dynload', '/usr/lib64/python3.6', '$HOME/Code/Mozilla'}
$ python3 pp.py
{'/usr/lib/python36.zip', '$HOME/Code/Mozilla', '/usr/lib64/python3.6', '/usr/lib64/python3.6/_import_failed', '/usr/lib64/python3.6/lib-dynload'}
$ python3 pp.py
{'/usr/lib64/python3.6/lib-dynload', '$HOME/Code/Mozilla', '/usr/lib64/python3.6/_import_failed', '/usr/lib64/python3.6', '/usr/lib/python36.zip'}
$ python3 pp.py
{'/usr/lib/python36.zip', '/usr/lib64/python3.6/lib-dynload', '/usr/lib64/python3.6/_import_failed', '/usr/lib64/python3.6', '$HOME/Code/Mozilla'}
File bug for opensuse? or python? guess i pyenv v3.6.15 in another vm...
Assignee | ||
Comment 10•3 years ago
|
||
for completeness sake
Assignee | ||
Comment 11•3 years ago
•
|
||
issue is reproducable in a virtualenv with python 3.6.15 in debian vm. so guess its python that has a bug.
-- edit --
i can reproduce a randomizing thing with that script on my mac aswell :/ .. or atleast it randomizes the output...
Assignee | ||
Comment 12•3 years ago
•
|
||
fixed script
import sys
import site
print(sorted(set(sys.path)-set([site.getusersitepackages()] + site.getsitepackages())))
sorting on length seems to fix it.. since the path for python3.6 is shorter than import_failed :)
stdlib_paths = subprocess.check_output(
[
self.python_path,
"-c",
"import sys; import site; "
"print("
" sorted(set(sys.path)"
" - set([site.getusersitepackages()] + site.getsitepackages()), key=len)"
")",
],
-- edit --
its a known feature it seems
https://stackoverflow.com/questions/61414947/why-dont-python-sets-preserve-insertion-order
Comment 13•3 years ago
|
||
Wow, nice find Nicklas! That's exactly it, set()
was used which changed the order of the system sys.path
.
sorted(set(sys.path))
is better ('cause it's consistent), but it might still change the path from what the system would normally have it ordered as.
I think the correct way to remove dupes will be list(OrderedDict.fromkeys(site_packages + user_site_packages))
.
If you're feeling ambitious, I'll gladly review a patch if you want to give it a shot, though there's also enough information here for me to write and land the patch myself.
You've put in the leg work here though, so I'd rather that you get the patch credit :)
Assignee | ||
Comment 14•3 years ago
|
||
Not sure if this is exactly as you had thought, but its a good start
Updated•3 years ago
|
Assignee | ||
Comment 15•3 years ago
|
||
that works for me :)
Updated•3 years ago
|
Comment 16•3 years ago
|
||
Comment 17•3 years ago
|
||
bugherder |
Comment 18•3 years ago
|
||
Comment on attachment 9260616 [details]
Bug 1751028 - Fix randomized sys.path order when using system Python packages r=mhentges
Beta/Release Uplift Approval Request
- User impact if declined: Some Linux devs will encounter intermittent build failures
- Is this code covered by automated tests?: No
- Has the fix been verified in Nightly?: Yes
- Needs manual test from QE?: No
- If yes, steps to reproduce:
- List of other uplifts needed: None
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): Only affects build, it's a small change
- String changes made/needed:
Comment 19•3 years ago
|
||
Comment on attachment 9260616 [details]
Bug 1751028 - Fix randomized sys.path order when using system Python packages r=mhentges
Approved for 97.0b9.
Comment 20•3 years ago
|
||
bugherder uplift |
Description
•