1804178 - Replace use of `pytoml` with `toml` (or similar)

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, task)

Description

[Ray Kraesig [:rkraesig]]

From the pytoml README.md:

Deprecated

The pytoml project is no longer being actively maintained. Consider using the toml package instead.

In particular, pytoml only supports up to TOML v0.4.0. However, Cargo.toml files are based on TOML v0.5.0, and often use single-quoted keys, which were not valid in TOML v0.4.0; pytoml chokes on these, which has necessitated some hacky workarounds in vendor_rust.py. On the other hand, the toml library recommended above can handle them just fine.

Comment 1

[Ray Kraesig [:rkraesig]]

For context: I encountered issues while attempting to vendor a patched dependency -- ./mach vendor rust choked on its Cargo.toml because it contained a [target.'cfg(windows)'.dependencies] section. That could be worked around, but not doing so seems like less hassle in the long run.

(I have a patchset currently in testing.)

Comment 2

[Ray Kraesig [:rkraesig]]

Attachment: Bug 1804178 - [1/6] Add `toml` to mach requirements r?#build

The pytoml library is abandoned, and does not support TOML v0.5.0,
which Cargo uses. This prevents mach from properly parsing many
Cargo.toml files.

Include the toml library as a slightly more modern and currently-
maintained alternative for use in mach.

Supply-chain notes:

• toml v0.10.2 is MIT-licensed, which is an approved MPL-compatible
  license.
• I have personally audited this version of toml and consider it
  "safe-to-run", as much as any Python code can be: other than opening
  and reading/writing a file when specifically requested via load/
  dump respectively, it performs only string manipulation and
  ordinary object creation.

Comment 3

[Ray Kraesig [:rkraesig]]

Attachment: Bug 1804178 - [2/6] Convert use of `pytoml` to `toml` in vendor_rust.py r?glandium

Replace pytoml with toml in vendor_rust.py.

This is a minimal substitution that preserves existing behavior (along
with a flake8 lint pass). Further simplification will follow in upcoming
commits.

Depends on D164150

Comment 4

[Ray Kraesig [:rkraesig]]

Attachment: Bug 1804178 - [3/6] Remove explicit sorting of TOML output r?glandium

Remove explicit sorting, as this is no longer needed to guarantee
determinism as of Python 3.6 (our minimum-supported version).

(This is a separate commit solely to make it clear that the reordering
of .cargo/config.in is indeed merely a reordering and contains no
hidden functional changes.)

Depends on D164151

Comment 5

[Ray Kraesig [:rkraesig]]

Attachment: Bug 1804178 - [4/6] Remove pytoml output postprocessing r?glandium

The postprocessing phase that cleaned up pytoml's output has little
effect on toml's output, which is already largely clean. Remove the
additional code.

Depends on D164152

Comment 6

[Ray Kraesig [:rkraesig]]

Attachment: Bug 1804178 - [5/6] Remove regex-based license-scraping hackery r?glandium

pytoml could not handle single-quoted keys in table names, as these
were added in TOML v0.5.0; but toml has no problem with them.

Remove regex-based scraping of Cargo.toml files to find a crate's
associated license, replacing it with toml.load.

(This has been manually tested to ensure that dependencies with
noncompliant licenses or unrecognized license files continue to be
rejected.)

Depends on D164153

Comment 7

[Ray Kraesig [:rkraesig]]

Attachment: Bug 1804178 - [6/6] Switch from pytoml to toml in (almost) all other contexts r?#build!,#releng!

Convert all other imports of pytoml within mozilla-central to imports
of toml.

As both libraries patterned their API on that of the Python standard-
library module json, this is mostly a straight replace, with two
caveats:

• pytoml's exception messages when loading a file provide the file
  name; toml's do not. Some additional machinery has been added or
  repurposed in a few places to ensure that the relevant filename is
  printed.

• In python_lockfile.py, the order of arguments to dump needed to
  be reversed. (toml follows json in this; pytoml differs.)

This patchset does not remove pytoml from requirements.in, as pytoml
is still used by compare-locales.

Depends on D164154

Comment 8

[Eemeli Aro [:eemeli]]

Is there a plan to follow this change with a (later) further migration to tomllib, which was added to the standard library in Python 3.11, or is the plan to stay with toml going forward?

Comment 9

[Ray Kraesig [:rkraesig]]

There are no plans to migrate to tomllib, for two reasons:

1. Our current minimum supported Python version is 3.6 or 3.7. I expect being able to rely on Python 3.11 is far enough out that it would be premature to make plans surrounding it, and there doesn't seem to be a backport up on PyPI.
2. tomllib lacks any TOML output capability, so we'd need another TOML library for writing anyway.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/903be63b308c
[1/6] Add `toml` to mach requirements  r=firefox-build-system-reviewers,glandium
https://hg.mozilla.org/integration/autoland/rev/d2acef30c803
[2/6] Convert use of `pytoml` to `toml` in vendor_rust.py  r=glandium
https://hg.mozilla.org/integration/autoland/rev/f8cf90fc83b8
[3/6] Remove explicit sorting of TOML output  r=glandium
https://hg.mozilla.org/integration/autoland/rev/4aa38981a711
[4/6] Remove pytoml output postprocessing  r=glandium
https://hg.mozilla.org/integration/autoland/rev/56746370d08e
[5/6] Remove regex-based license-scraping hackery  r=glandium
https://hg.mozilla.org/integration/autoland/rev/7ce002299574
[6/6] Switch from pytoml to toml in (almost) all other contexts  r=firefox-build-system-reviewers,releng-reviewers,glandium,ahal
https://hg.mozilla.org/integration/autoland/rev/9bfbfd1eca71
1804178, 1804178, 1804178, 1804178, 1804178: apply code formatting via Lando

Comment 11

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/903be63b308c
https://hg.mozilla.org/mozilla-central/rev/d2acef30c803
https://hg.mozilla.org/mozilla-central/rev/f8cf90fc83b8
https://hg.mozilla.org/mozilla-central/rev/4aa38981a711
https://hg.mozilla.org/mozilla-central/rev/56746370d08e
https://hg.mozilla.org/mozilla-central/rev/7ce002299574
https://hg.mozilla.org/mozilla-central/rev/9bfbfd1eca71