1799434 - `mach vendor python` changes unrelated files and doesn't add hashes

Status: RESOLVED DUPLICATE of bug 1782785

(Firefox Build System :: General, defect)

Description

[Jan-Erik Rediger [:janerik]]

STR:

1. In third_party/python/requirements.in update glean_parser:

-glean_parser==6.1.2
+glean_parser==6.3.0

(Side-note: This file currently has Windows line endings \r\n).

2. Run mach vendor python

What I expected:

Updates in third_party/python/glean_parser/ and some related files in third_party, updated hashes in third_party/python/poetry.lock and third_party/python/requirements.txt

What I got:

Updates in unrelated files, such as third_party/python/taskcluster_taskgraph/taskgraph/decision.py or third_party/python/fluent.migrate/fluent/migrate/transforms.py.
No hashes in either requirements.txt or poetry.lock

Full status:

$ git status
## only-glean_parser-update...central
M  third_party/python/fluent.migrate/fluent/migrate/transforms.py
R  third_party/python/glean_parser/glean_parser-6.1.2.dist-info/AUTHORS.md -> third_party/python/glean_parser/glean_parser-6.3.0.dist-info/AUTHORS.md
R  third_party/python/glean_parser/glean_parser-6.1.2.dist-info/LICENSE -> third_party/python/glean_parser/glean_parser-6.3.0.dist-info/LICENSE
R  third_party/python/glean_parser/glean_parser-6.1.2.dist-info/METADATA -> third_party/python/glean_parser/glean_parser-6.3.0.dist-info/METADATA
R  third_party/python/glean_parser/glean_parser-6.1.2.dist-info/RECORD -> third_party/python/glean_parser/glean_parser-6.3.0.dist-info/RECORD
R  third_party/python/glean_parser/glean_parser-6.1.2.dist-info/WHEEL -> third_party/python/glean_parser/glean_parser-6.3.0.dist-info/WHEEL
R  third_party/python/glean_parser/glean_parser-6.1.2.dist-info/entry_points.txt -> third_party/python/glean_parser/glean_parser-6.3.0.dist-info/entry_points.txt
R  third_party/python/glean_parser/glean_parser-6.1.2.dist-info/top_level.txt -> third_party/python/glean_parser/glean_parser-6.3.0.dist-info/top_level.txt
M  third_party/python/glean_parser/glean_parser/__main__.py
M  third_party/python/glean_parser/glean_parser/data_review.py
M  third_party/python/glean_parser/glean_parser/javascript.py
M  third_party/python/glean_parser/glean_parser/parser.py
M  third_party/python/glean_parser/glean_parser/schemas/metrics.2-0-0.schema.yaml
A  third_party/python/glean_parser/glean_parser/templates/data_review.jinja2
M  third_party/python/glean_parser/glean_parser/templates/rust.jinja2
M  third_party/python/glean_parser/glean_parser/translation_options.py
M  third_party/python/poetry.lock
M  third_party/python/requirements.in
M  third_party/python/requirements.txt
M  third_party/python/taskcluster_taskgraph/taskgraph/decision.py

Rewritten third_party/python/requirements.txt:

$ head -5 third_party/python/requirements.txt
aiohttp==3.7.4.post0; python_version >= "3.6"
appdirs==1.4.4
async-timeout==3.0.1; python_full_version >= "3.5.3" and python_version >= "3.6"
attrs==19.2.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")
blessings==1.7; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")

My Python:

$ mach python --version
Python 3.10.7

Linux machine (Fedora). Same thing happens on my macbook.

Comment 1

[Johan Lorenzo [:jlorenzo]]

I confirm I reproduce the same behavior as :janerik's. I wanted to bump taskcluster-taskgraph for bug 1793219 and I couldn't get hashes anymore. For the record, I purged the poetry cache with ./obj-aarch64-apple-darwin21.6.0/_virtualenvs/vendor/bin/python -m poetry cache clear pypi --all

Comment 2

[Alex Hochheiden [:ahochheiden] [he/him]]

(In reply to Jan-Erik Rediger [:janerik] from comment #0)

(Side-note: This file currently has Windows line endings \r\n).

A change recently landed that strips the Carriage Return (\r), so that vendoring on Windows going forward will be identical to Linux/Mac. I don't think a vendor has landed with that change yet, so seeing the current diff losing the Carriage Returns is confusing, but intended. It should be one of the last times we ever see this difference.

2. Run mach vendor python
   Updates in unrelated files, such as third_party/python/taskcluster_taskgraph/taskgraph/decision.py or third_party/python/fluent.migrate/fluent/migrate/transforms.py.

The way ./mach vendor python works is by purging everything that's vendored in mozilla-central/third_party/python (Minus a few exceptions) and fetching everything specified in requirements.in (Plus their dependencies) fresh from PyPi. This means that even if you only change one requirement in requirements.in you can get many differences. In this case, I think what's vendored in-tree is wrong. I ran ./mach vendor python without changing any requirements and got the same modifications you did. I manually inspected the taskcluster-taskgraph wheel for version 3.2.1 and the code that's being changed is being changed to what's in that wheel, which is what we want. I assume the same is true for fluent.migrate, but I did not check.

No hashes in either requirements.txt or poetry.lock

I vendored glean-parser at 6.3.0 and definitely see changed hashes:

--- a/third_party/python/requirements.txt
+++ b/third_party/python/requirements.txt
@@ -92,9 +92,9 @@ fluent.syntax==0.18.1 \
 giturlparse==0.10.0; python_version >= "3.6" \
     --hash=sha256:04ba1a3a099c3093fa8d24a422913c6a9b2c2cd22bcffc939cf72e3e98f672d7 \
     --hash=sha256:2595ab291d30717cda8474b874c9fd509f1b9802ad7f6968c36a45e4b13eb337
-glean-parser==6.1.2 \
-    --hash=sha256:e801af6463b7e0ba79d97ddfc0a58d9d71121c93cea601417571e33fa8142270 \
-    --hash=sha256:12a0fecedc1144d77fa571e0422ff3fea4dbadc381d631bea800a6b2f58f4f7f
+glean-parser==6.3.0 \
+    --hash=sha256:b6cb6ab1247f3eb8e64427644dbdc0ef1fbf46af593225d66e24aa967dcaa23e \
+    --hash=sha256:946ef3d7fe24fbbe8101964791b7ad0e609d1ee6a21a87b33f805de8577e6f40
 idna-ssl==1.1.0; python_version < "3.7" and python_version >= "3.6" \

--- a/third_party/python/poetry.lock
+++ b/third_party/python/poetry.lock
@@ -204,7 +204,7 @@ python-versions = ">=3.6"

 [[package]]
 name = "glean-parser"
-version = "6.1.2"
+version = "6.3.0"
 description = "Parser tools for Mozilla's Glean telemetry"
 category = "main"
 optional = false
@@ -252,8 +252,8 @@ python-versions = "!=3.0.*,!=3.1.*,!=3.2
 zipp = ">=0.5"

 [package.extras]
-testing = ["importlib-resources (>=1.3)", "pep517", "packaging"]
-docs = ["rst.linker", "sphinx"]
+docs = ["sphinx", "rst.linker"]
+testing = ["packaging", "pep517", "importlib-resources (>=1.3)"]

 [[package]]
 name = "iso8601"
@@ -757,7 +757,7 @@ testing = ["pytest (>=4.6)", "pytest-che
 [metadata]
 lock-version = "1.1"
 python-versions = "^3.6"
-content-hash = "eb26337dadb86c9a4895059df8799e1d81eb31e85150dea8931c2207a0bef168"
+content-hash = "c46ae33bc287223a8858fb21ac1f6b7beba10667daea79aaefcc7e6e33ef0f43"

 [metadata.files]
 aiohttp = [
@@ -875,8 +875,8 @@ giturlparse = [
     {file = "giturlparse-0.10.0.tar.gz", hash = "sha256:2595ab291d30717cda8474b874c9fd509f1b9802ad7f6968c36a45e4b13eb337"},
 ]
 glean-parser = [
-    {file = "glean_parser-6.1.2-py3-none-any.whl", hash = "sha256:e801af6463b7e0ba79d97ddfc0a58d9d71121c93cea601417571e33fa8142270"},
-    {file = "glean_parser-6.1.2.tar.gz", hash = "sha256:12a0fecedc1144d77fa571e0422ff3fea4dbadc381d631bea800a6b2f58f4f7f"},
+    {file = "glean_parser-6.3.0-py3-none-any.whl", hash = "sha256:b6cb6ab1247f3eb8e64427644dbdc0ef1fbf46af593225d66e24aa967dcaa23e"},
+    {file = "glean_parser-6.3.0.tar.gz", hash = "sha256:946ef3d7fe24fbbe8101964791b7ad0e609d1ee6a21a87b33f805de8577e6f40"},
 ]

Note: I used Notepad++ to re-add the CR LF line endings to remove the diff noise to make it easier to see the hash changes.

I'm surprised you were able to vendor at all (due to bug 1782785). I'm assuming you modified poetry in the venv (obj\_virtualenvs\vendor\Lib\site-packages\poetry\repositories\pypi_repository.py#284)?

Other than ./mach vendor python not being usable without modifying the venv (which is bug 1782785) I don't think there's any problem here. The diffs you're seeing in fluent.migrate and taskcluster_taskgraph were probably introduced by the most recent vendor and weren't caught by CI due to the ./mach vendor python diff test being disabled.

Comment 3

[Alex Hochheiden [:ahochheiden] [he/him]]

(In reply to Alex Hochheiden [:ahochheiden] from comment #2)

Other than ./mach vendor python not being usable without modifying the venv (which is bug 1782785) I don't think there's any problem here. The diffs you're seeing in fluent.migrate and taskcluster_taskgraph were probably introduced by the most recent vendor and weren't caught by CI due to the ./mach vendor python diff test being disabled.

I see, the symptom of the problem in bug 1782785 changed. ./mach vendor python now finishes without error, it just outputs garbage without hashes. I'll mark this as a duplicate, since the "changes unrelated files" portion of this bug is expected behavior.