1627072 - The generated requirements.txt from `mach vendor python` has a bunch of goofy, apparently nondeterministic content

Status: RESOLVED FIXED

(Firefox Build System :: General, task)

Description

[Ricky Stewart]

Looking at requirements.txt, it contains some apparently meaningless, non-determinstic content (the via /tmp/... lines). The contents of those lines seem also seem to depend on your OS, so unfortunately whenever I vendor a new package it changes the contents of all those lines, e.g.:

 attrs==18.1.0 \
     --hash=sha256:4b90b09eeeb9b88c35bc642cbac057e45a5fd85367b985bd2809c62b7b939265 \
     --hash=sha256:e0d0eb91441a3b53dab4d9b743eafc1ac44476296a2053b6ca3af0b139faf87b \
-    # via -r /tmp/tmplu66or93, mozilla-version, pytest
+    # via -r /var/folders/5g/4jg8tpgn28g7dkynf55bgtj40000gn/T/tmpywm8w3m2, mozilla-version, pytest
 biplist==1.0.3 \
     --hash=sha256:4c0549764c5fe50b28042ec21aa2e14fe1a2224e239a1dae77d9e7f3932aa4c6 \
-    # via -r /tmp/tmplu66or93
+    # via -r /var/folders/5g/4jg8tpgn28g7dkynf55bgtj40000gn/T/tmpywm8w3m2
 blessings==1.7 \
     --hash=sha256:98e5854d805f50a5b58ac2333411b0482516a8210f23f43308baeb58d77c157d \
     --hash=sha256:b1fdd7e7a675295630f9ae71527a8ebc10bfefa236b3d6aa4932ee4462c17ba3 \
     --hash=sha256:caad5211e7ba5afe04367cdd4cfc68fa886e2e08f6f35e76b7387d2109ccea6e \
-    # via -r /tmp/tmplu66or93
+    # via -r /var/folders/5g/4jg8tpgn28g7dkynf55bgtj40000gn/T/tmpywm8w3m2
 certifi==2018.4.16 \

This isn't a functional problem since those lines are effectively comments that downstream consumers don't ever care about but it does make for a more confusing experience when reading over the generated changes to verify them.

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

This is caused by us creating a temporary directory and doctoring the requirements.in file before calling pip-compile.
Normally, those "via /tmp" lines would look like:

# via -r requirements.in

Which is a lot more valuable.
Perhaps pip-compile allows configuring the name, or we can find some other workaround.
Alternatively, perhaps we don't need to use this _update_packages() function? Will dig tomorrow.

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1627072 - Runs pip-compile in same directory as temporary requirements file r?rstewart

pip-compile can annotate each requirement with a reason why it is included (e.g.: transitive dependency, or depended-on directly).

When annotating direct dependencies, it notes it as "via -r <direct path to requirements.in file>".
Since we were using a temporary directory, the path of the directory was being included, making the output non-deterministic.

This change ensures that we run pip-compile in the same working directory as the temporary requirements file, enabling
the annotations to be deterministic: "via -r requirements-mach-vendor-python.in".

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5a93d460a035
Runs pip-compile in same directory as temporary requirements file r=rstewart

Comment 4

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/5a93d460a035