1741261 - Rewrite intl/update-icu4x.sh to python

Status: NEW

(Core :: Internationalization, task, P3)

Description

[Greg Tatum [:gregtatum]]

I prototyped this script out using bash as it was quick and easy for me to do, but it would better and more maintainable to have it as a python script. Existing intl python scripts have been one-off python scripts, but for bonus points it would be nice to make it an actual mach command with nice things like proper CLI documentation and maybe even tests.

Comment 1

[lukas.bernhard]

I started on implementing this as a mach command. The main issue is that both my mach command as well as the in-tree update-icu4x.sh fail to run with a variety of backtraces depending on the icu4x tag, CLDR version and rustc version. Is there a particular combination of these that is known-good?

Comment 2

[Greg Tatum [:gregtatum]]

The example I wrote in it still works for me. If you are having issues, then perhaps it's a cross-OS issue?

./update-icu4x.sh https://github.com/unicode-org/icu4x.git icu@0.3.0 39.0.0

I wrote it on macOS.

ICU4X still hasn't stabilized so some assumptions may need to be updated for the latest version.

Comment 3

[Greg Tatum [:gregtatum]]

If it still doesn't work please provide the output of the errors, and I can look at it.

Comment 4

[lukas.bernhard]

I tried to run invoke the original update-icu4x.sh script with a couple of different release tags and CLDR versions on ubuntu22.04 with the following result:

{0.3,0.4} 39.0
Build failure with:
warning: build/expando.c:4:24: error: pasting "RUST_VERSION_OPENSSL_" and "(" does not give a valid preprocessing token

0.5 39.0
Runtime failure:
thread 'main' panicked at 'called Result::unwrap() on an Err value: InvalidLanguage', provider/cldr/src/reader.rs:38:88

0.5 41.0
Runtime failure:
thread 'main' panicked at 'Please call close before dropping FilesystemExporter', provider/blob/src/export/blob_exporter.rs:33:13

For building 0.6 one needs to add --features=bin to the cargo command line.
0.6 39.0
Runtime failure:
Error: Extracting archive failed (InvalidArchive("Invalid file path"))
Location:
provider/datagen/src/bin/datagen.rs:245:13

0.6 41.0
Runtime failure:
Error: ICU4X data error (key: list/and@1): data for CodePointTrie of Script
Location:
/rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/convert/mod.rs:546:9

Comment 5

[lukas.bernhard]

Attachment: mach-update-icu4x.diff

FWIW, with the icu4x 1.0 release the data generation works for me. Attached is a patch adding a mach command for data generation.
./mach update-icu4x https://github.com/unicode-org/icu4x.git icu@1.0.0 41.0.0 release-71-1 generates the icu4x.postcard file.