1686279 - "Glean could not be found, so telemetry will not be reported."

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, defect)

Description

[Matthew Gaudet (he/him) [:mgaudet]]

I keep getting the following.

Glean could not be found, so telemetry will not be reported. You may need to run |mach bootstrap|

when using mach; however, running mach bootstrap has -not- rectified the issue.

By running rm -rf ~/.mozbuild/_virtualenvs/ then mach create-mach-environment I see the following error:

Collecting iso8601>=0.1.10
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    iso8601>=0.1.10 from https://files.pythonhosted.org/packages/98/ad/5b6342bec1ce40b5f4783fa552f10109e159b734909d11a71513b240d7dd/iso8601-0.1.13-py3-none-any.whl#sha256=6f02f01dd13320a7f280e58516dc8d1950dfaf77527cc365a398cd9de4d3c692 (from glean-parser==1.29.0->-r /home/matthew/unified/build/glean_requirements.txt (line 61))
Could not install glean_sdk, so telemetry will not be collected. Continuing.
Python 3 mach environment created.
virtualenv at /home/matthew/.mozbuild/_virtualenvs/mach_py2 is already up to date.
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Requirement already satisfied: cffi==1.14.4 in /home/matthew/.mozbuild/_virtualenvs/mach_py2/lib/python2.7/site-packages (from -r /home/matthew/unified/build/zstandard_requirements.txt (line 7)) (1.14.4)
Requirement already satisfied: pycparser==2.20 in /home/matthew/.mozbuild/_virtualenvs/mach_py2/lib/python2.7/site-packages (from -r /home/matthew/unified/build/zstandard_requirements.txt (line 45)) (2.20)
Requirement already satisfied: zstandard==0.13.0 in /home/matthew/.mozbuild/_virtualenvs/mach_py2/lib/python2.7/site-packages (from -r /home/matthew/unified/build/zstandard_requirements.txt (line 49)) (0.13.0)
WARNING: You are using pip version 20.3.1; however, version 20.3.3 is available.
You should consider upgrading via the '/home/matthew/.mozbuild/_virtualenvs/mach_py2/bin/python -m pip install --upgrade pip' command.
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Requirement already satisfied: psutil==5.7.0 in /home/matthew/.mozbuild/_virtualenvs/mach_py2/lib/python2.7/site-packages (from -r /home/matthew/unified/build/psutil_requirements.txt (line 7)) (5.7.0)
WARNING: You are using pip version 20.3.1; however, version 20.3.3 is available.
You should consider upgrading via the '/home/matthew/.mozbuild/_virtualenvs/mach_py2/bin/python -m pip install --upgrade pip' command.
Python 2 mach environment created.

Central Revision: 7643eaee62a3

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Hey can you do a ./mach create-mach-environment -f for me and paste the full logs?

Comment 2

[Matthew Gaudet (he/him) [:mgaudet]]

Attachment: create-env-log.txt

Attached!

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Hah, hmm, this is a bit of a thinker.
You're running into this issue because glean_parser-1.29.0 depends on iso8601 but only if Python <= 3.6.
Additionally, generating a requirements list with hashes only includes dependencies that appear on the machine doing the list-generation.
So, since we generated the list on Python 3.7+, iso8601 and its hashes aren't included, and pip correctly chokes.

A workaround is to use Python 3.7+, if possible.
I knew I'd have to dig into cross-python/platform hash-baking eventually, and it looks like this is the shove that's going to make that happen :)

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Hmm, as confirmed by pip-compile docs, there's no nice way for cross-environment generation of a single requirements.txt.

However, we have a small set of dependencies for mach, so I'll doctor our current requirements.txt for glean to work with Python 3.6 :)

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1686279: Support Python 3.6 with glean_parser

glean_parser depends on iso8601, but only on Python <= 3.6.
Add the iso8601 hash to our frozen dependency hash list accordingly.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5c40775b6d3c
Support Python 3.6 with glean_parser r=sheehan

Comment 7

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/5c40775b6d3c

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

Huh, little follow-up here:

• In my patch, I should've used PEP 508 syntax to denote that iso8601 is a "python <= 3.6 dependency". Since I didn't do that, iso8601 is installed for all machines, even those that are Python > 3.6.
• pip-tools should theoretically be able to embed "environment markers": specific issue, related issue.
• Even once pip-tools embeds environment markers, we still need to be wary of rogue packages having setup.py specify environment-specific dependencies programmatically ('transitive_dep' if sys.version == x else None).