(In reply to Jan-Erik Rediger [:janerik] from comment #0) > (Side-note: This file currently has Windows line endings `\r\n`). > A change recently landed that strips the Carriage Return (`\r`), so that vendoring on Windows going forward will be identical to Linux/Mac. I don't think a vendor has landed with that change yet, so seeing the current diff losing the Carriage Returns is confusing, but intended. It should be one of the last times we ever see this difference. > 2. Run `mach vendor python` > Updates in unrelated files, such as `third_party/python/taskcluster_taskgraph/taskgraph/decision.py` or `third_party/python/fluent.migrate/fluent/migrate/transforms.py`. The way `./mach vendor python` works is by purging everything that's vendored in `mozilla-central/third_party/python` (Minus a few exceptions) and fetching everything specified in `requirements.in` (Plus their dependencies) fresh from PyPi. This means that even if you only change one requirement in `requirements.in` you can get many differences. In this case, I think what's vendored in-tree is wrong. I ran `./mach vendor python` without changing any requirements and got the same modifications you did. I manually inspected the `taskcluster-taskgraph` wheel for version `3.2.1` and the code that's being changed is being changed to what's in that wheel, which is what we want. I assume the same is true for `fluent.migrate`, but I did not check. > No hashes in either requirements.txt or poetry.lock I vendored `glean-parser` at `6.3.0` and definitely see changed hashes: ``` --- a/third_party/python/requirements.txt +++ b/third_party/python/requirements.txt @@ -92,9 +92,9 @@ fluent.syntax==0.18.1 \ giturlparse==0.10.0; python_version >= "3.6" \ --hash=sha256:04ba1a3a099c3093fa8d24a422913c6a9b2c2cd22bcffc939cf72e3e98f672d7 \ --hash=sha256:2595ab291d30717cda8474b874c9fd509f1b9802ad7f6968c36a45e4b13eb337 -glean-parser==6.1.2 \ - --hash=sha256:e801af6463b7e0ba79d97ddfc0a58d9d71121c93cea601417571e33fa8142270 \ - --hash=sha256:12a0fecedc1144d77fa571e0422ff3fea4dbadc381d631bea800a6b2f58f4f7f +glean-parser==6.3.0 \ + --hash=sha256:b6cb6ab1247f3eb8e64427644dbdc0ef1fbf46af593225d66e24aa967dcaa23e \ + --hash=sha256:946ef3d7fe24fbbe8101964791b7ad0e609d1ee6a21a87b33f805de8577e6f40 idna-ssl==1.1.0; python_version < "3.7" and python_version >= "3.6" \ ``` ``` --- a/third_party/python/poetry.lock +++ b/third_party/python/poetry.lock @@ -204,7 +204,7 @@ python-versions = ">=3.6" [[package]] name = "glean-parser" -version = "6.1.2" +version = "6.3.0" description = "Parser tools for Mozilla's Glean telemetry" category = "main" optional = false @@ -252,8 +252,8 @@ python-versions = "!=3.0.*,!=3.1.*,!=3.2 zipp = ">=0.5" [package.extras] -testing = ["importlib-resources (>=1.3)", "pep517", "packaging"] -docs = ["rst.linker", "sphinx"] +docs = ["sphinx", "rst.linker"] +testing = ["packaging", "pep517", "importlib-resources (>=1.3)"] [[package]] name = "iso8601" @@ -757,7 +757,7 @@ testing = ["pytest (>=4.6)", "pytest-che [metadata] lock-version = "1.1" python-versions = "^3.6" -content-hash = "eb26337dadb86c9a4895059df8799e1d81eb31e85150dea8931c2207a0bef168" +content-hash = "c46ae33bc287223a8858fb21ac1f6b7beba10667daea79aaefcc7e6e33ef0f43" [metadata.files] aiohttp = [ @@ -875,8 +875,8 @@ giturlparse = [ {file = "giturlparse-0.10.0.tar.gz", hash = "sha256:2595ab291d30717cda8474b874c9fd509f1b9802ad7f6968c36a45e4b13eb337"}, ] glean-parser = [ - {file = "glean_parser-6.1.2-py3-none-any.whl", hash = "sha256:e801af6463b7e0ba79d97ddfc0a58d9d71121c93cea601417571e33fa8142270"}, - {file = "glean_parser-6.1.2.tar.gz", hash = "sha256:12a0fecedc1144d77fa571e0422ff3fea4dbadc381d631bea800a6b2f58f4f7f"}, + {file = "glean_parser-6.3.0-py3-none-any.whl", hash = "sha256:b6cb6ab1247f3eb8e64427644dbdc0ef1fbf46af593225d66e24aa967dcaa23e"}, + {file = "glean_parser-6.3.0.tar.gz", hash = "sha256:946ef3d7fe24fbbe8101964791b7ad0e609d1ee6a21a87b33f805de8577e6f40"}, ] ``` Note: I used Notepad++ to re-add the `CR LF` line endings to remove the diff noise to make it easier to see the hash changes. I'm surprised you were able to vendor at all (due to bug 1765416). I'm assuming you modified poetry in the venv (`obj\_virtualenvs\vendor\Lib\site-packages\poetry\repositories\pypi_repository.py#284`). Other than `./mach vendor python` not being usable without modifying the venv (which is bug 1765416) I don't think there's any problem here. The diffs you're seeing in `fluent.migrate` and `taskcluster_taskgraph` were probably introduced by the most recent vendor and weren't caught by CI due to the `./mach vendor python` diff test being disabled.
Bug 1799434 Comment 2 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
(In reply to Jan-Erik Rediger [:janerik] from comment #0) > (Side-note: This file currently has Windows line endings `\r\n`). > A change recently landed that strips the Carriage Return (`\r`), so that vendoring on Windows going forward will be identical to Linux/Mac. I don't think a vendor has landed with that change yet, so seeing the current diff losing the Carriage Returns is confusing, but intended. It should be one of the last times we ever see this difference. > 2. Run `mach vendor python` > Updates in unrelated files, such as `third_party/python/taskcluster_taskgraph/taskgraph/decision.py` or `third_party/python/fluent.migrate/fluent/migrate/transforms.py`. The way `./mach vendor python` works is by purging everything that's vendored in `mozilla-central/third_party/python` (Minus a few exceptions) and fetching everything specified in `requirements.in` (Plus their dependencies) fresh from PyPi. This means that even if you only change one requirement in `requirements.in` you can get many differences. In this case, I think what's vendored in-tree is wrong. I ran `./mach vendor python` without changing any requirements and got the same modifications you did. I manually inspected the `taskcluster-taskgraph` wheel for version `3.2.1` and the code that's being changed is being changed to what's in that wheel, which is what we want. I assume the same is true for `fluent.migrate`, but I did not check. > No hashes in either requirements.txt or poetry.lock I vendored `glean-parser` at `6.3.0` and definitely see changed hashes: ``` --- a/third_party/python/requirements.txt +++ b/third_party/python/requirements.txt @@ -92,9 +92,9 @@ fluent.syntax==0.18.1 \ giturlparse==0.10.0; python_version >= "3.6" \ --hash=sha256:04ba1a3a099c3093fa8d24a422913c6a9b2c2cd22bcffc939cf72e3e98f672d7 \ --hash=sha256:2595ab291d30717cda8474b874c9fd509f1b9802ad7f6968c36a45e4b13eb337 -glean-parser==6.1.2 \ - --hash=sha256:e801af6463b7e0ba79d97ddfc0a58d9d71121c93cea601417571e33fa8142270 \ - --hash=sha256:12a0fecedc1144d77fa571e0422ff3fea4dbadc381d631bea800a6b2f58f4f7f +glean-parser==6.3.0 \ + --hash=sha256:b6cb6ab1247f3eb8e64427644dbdc0ef1fbf46af593225d66e24aa967dcaa23e \ + --hash=sha256:946ef3d7fe24fbbe8101964791b7ad0e609d1ee6a21a87b33f805de8577e6f40 idna-ssl==1.1.0; python_version < "3.7" and python_version >= "3.6" \ ``` ``` --- a/third_party/python/poetry.lock +++ b/third_party/python/poetry.lock @@ -204,7 +204,7 @@ python-versions = ">=3.6" [[package]] name = "glean-parser" -version = "6.1.2" +version = "6.3.0" description = "Parser tools for Mozilla's Glean telemetry" category = "main" optional = false @@ -252,8 +252,8 @@ python-versions = "!=3.0.*,!=3.1.*,!=3.2 zipp = ">=0.5" [package.extras] -testing = ["importlib-resources (>=1.3)", "pep517", "packaging"] -docs = ["rst.linker", "sphinx"] +docs = ["sphinx", "rst.linker"] +testing = ["packaging", "pep517", "importlib-resources (>=1.3)"] [[package]] name = "iso8601" @@ -757,7 +757,7 @@ testing = ["pytest (>=4.6)", "pytest-che [metadata] lock-version = "1.1" python-versions = "^3.6" -content-hash = "eb26337dadb86c9a4895059df8799e1d81eb31e85150dea8931c2207a0bef168" +content-hash = "c46ae33bc287223a8858fb21ac1f6b7beba10667daea79aaefcc7e6e33ef0f43" [metadata.files] aiohttp = [ @@ -875,8 +875,8 @@ giturlparse = [ {file = "giturlparse-0.10.0.tar.gz", hash = "sha256:2595ab291d30717cda8474b874c9fd509f1b9802ad7f6968c36a45e4b13eb337"}, ] glean-parser = [ - {file = "glean_parser-6.1.2-py3-none-any.whl", hash = "sha256:e801af6463b7e0ba79d97ddfc0a58d9d71121c93cea601417571e33fa8142270"}, - {file = "glean_parser-6.1.2.tar.gz", hash = "sha256:12a0fecedc1144d77fa571e0422ff3fea4dbadc381d631bea800a6b2f58f4f7f"}, + {file = "glean_parser-6.3.0-py3-none-any.whl", hash = "sha256:b6cb6ab1247f3eb8e64427644dbdc0ef1fbf46af593225d66e24aa967dcaa23e"}, + {file = "glean_parser-6.3.0.tar.gz", hash = "sha256:946ef3d7fe24fbbe8101964791b7ad0e609d1ee6a21a87b33f805de8577e6f40"}, ] ``` Note: I used Notepad++ to re-add the `CR LF` line endings to remove the diff noise to make it easier to see the hash changes. I'm surprised you were able to vendor at all (due to bug 1765416). I'm assuming you modified poetry in the venv (`obj\_virtualenvs\vendor\Lib\site-packages\poetry\repositories\pypi_repository.py#284`). Other than `./mach vendor python` not being usable without modifying the venv (which is bug 1765416) I don't think there's any problem here. The diffs you're seeing in `fluent.migrate` and `taskcluster_taskgraph` were probably introduced by the most recent vendor and weren't caught by CI due to the `./mach vendor python` diff test being disabled.
(In reply to Jan-Erik Rediger [:janerik] from comment #0) > (Side-note: This file currently has Windows line endings `\r\n`). > A change recently landed that strips the Carriage Return (`\r`), so that vendoring on Windows going forward will be identical to Linux/Mac. I don't think a vendor has landed with that change yet, so seeing the current diff losing the Carriage Returns is confusing, but intended. It should be one of the last times we ever see this difference. > 2. Run `mach vendor python` > Updates in unrelated files, such as `third_party/python/taskcluster_taskgraph/taskgraph/decision.py` or `third_party/python/fluent.migrate/fluent/migrate/transforms.py`. The way `./mach vendor python` works is by purging everything that's vendored in `mozilla-central/third_party/python` (Minus a few exceptions) and fetching everything specified in `requirements.in` (Plus their dependencies) fresh from PyPi. This means that even if you only change one requirement in `requirements.in` you can get many differences. In this case, I think what's vendored in-tree is wrong. I ran `./mach vendor python` without changing any requirements and got the same modifications you did. I manually inspected the `taskcluster-taskgraph` wheel for version `3.2.1` and the code that's being changed is being changed to what's in that wheel, which is what we want. I assume the same is true for `fluent.migrate`, but I did not check. > No hashes in either requirements.txt or poetry.lock I vendored `glean-parser` at `6.3.0` and definitely see changed hashes: ``` --- a/third_party/python/requirements.txt +++ b/third_party/python/requirements.txt @@ -92,9 +92,9 @@ fluent.syntax==0.18.1 \ giturlparse==0.10.0; python_version >= "3.6" \ --hash=sha256:04ba1a3a099c3093fa8d24a422913c6a9b2c2cd22bcffc939cf72e3e98f672d7 \ --hash=sha256:2595ab291d30717cda8474b874c9fd509f1b9802ad7f6968c36a45e4b13eb337 -glean-parser==6.1.2 \ - --hash=sha256:e801af6463b7e0ba79d97ddfc0a58d9d71121c93cea601417571e33fa8142270 \ - --hash=sha256:12a0fecedc1144d77fa571e0422ff3fea4dbadc381d631bea800a6b2f58f4f7f +glean-parser==6.3.0 \ + --hash=sha256:b6cb6ab1247f3eb8e64427644dbdc0ef1fbf46af593225d66e24aa967dcaa23e \ + --hash=sha256:946ef3d7fe24fbbe8101964791b7ad0e609d1ee6a21a87b33f805de8577e6f40 idna-ssl==1.1.0; python_version < "3.7" and python_version >= "3.6" \ ``` ``` --- a/third_party/python/poetry.lock +++ b/third_party/python/poetry.lock @@ -204,7 +204,7 @@ python-versions = ">=3.6" [[package]] name = "glean-parser" -version = "6.1.2" +version = "6.3.0" description = "Parser tools for Mozilla's Glean telemetry" category = "main" optional = false @@ -252,8 +252,8 @@ python-versions = "!=3.0.*,!=3.1.*,!=3.2 zipp = ">=0.5" [package.extras] -testing = ["importlib-resources (>=1.3)", "pep517", "packaging"] -docs = ["rst.linker", "sphinx"] +docs = ["sphinx", "rst.linker"] +testing = ["packaging", "pep517", "importlib-resources (>=1.3)"] [[package]] name = "iso8601" @@ -757,7 +757,7 @@ testing = ["pytest (>=4.6)", "pytest-che [metadata] lock-version = "1.1" python-versions = "^3.6" -content-hash = "eb26337dadb86c9a4895059df8799e1d81eb31e85150dea8931c2207a0bef168" +content-hash = "c46ae33bc287223a8858fb21ac1f6b7beba10667daea79aaefcc7e6e33ef0f43" [metadata.files] aiohttp = [ @@ -875,8 +875,8 @@ giturlparse = [ {file = "giturlparse-0.10.0.tar.gz", hash = "sha256:2595ab291d30717cda8474b874c9fd509f1b9802ad7f6968c36a45e4b13eb337"}, ] glean-parser = [ - {file = "glean_parser-6.1.2-py3-none-any.whl", hash = "sha256:e801af6463b7e0ba79d97ddfc0a58d9d71121c93cea601417571e33fa8142270"}, - {file = "glean_parser-6.1.2.tar.gz", hash = "sha256:12a0fecedc1144d77fa571e0422ff3fea4dbadc381d631bea800a6b2f58f4f7f"}, + {file = "glean_parser-6.3.0-py3-none-any.whl", hash = "sha256:b6cb6ab1247f3eb8e64427644dbdc0ef1fbf46af593225d66e24aa967dcaa23e"}, + {file = "glean_parser-6.3.0.tar.gz", hash = "sha256:946ef3d7fe24fbbe8101964791b7ad0e609d1ee6a21a87b33f805de8577e6f40"}, ] ``` Note: I used Notepad++ to re-add the `CR LF` line endings to remove the diff noise to make it easier to see the hash changes. I'm surprised you were able to vendor at all (due to bug 1782785). I'm assuming you modified poetry in the venv (`obj\_virtualenvs\vendor\Lib\site-packages\poetry\repositories\pypi_repository.py#284`)? Other than `./mach vendor python` not being usable without modifying the venv (which is bug 1782785) I don't think there's any problem here. The diffs you're seeing in `fluent.migrate` and `taskcluster_taskgraph` were probably introduced by the most recent vendor and weren't caught by CI due to the `./mach vendor python` diff test being disabled.