1881272 - update desktop_l10n.py to support l10n target string repo in github

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: L10N, enhancement)

Description

[[e-mail] (:bhearsum)]

We will be migrating away from the current l10n-central mercurial repositories, and into a single target repository in GitHub. As part of this, we'll need to update our l10n repack scripts (desktop_l10n.py and its dependencies) to support fetching localized strings from there instead of mercurial. It looks like this is largely isolated in locales.py.

We need to support both methods of cloning at the same time to ensure that it's possible to rollback if needed. This most likely means some additional config or command line flag that we can switch off of.

One thing to watch out for is that we need to ensure that the directory layout is the same before we pass off to the build system - otherwise that is likely to fail, or produce incorrect builds.

Comment 1

[[e-mail] (:bhearsum)]

https://github.com/mozilla-l10n/firefox-l10n/tree/cthulhu is set up in the same way the final target repo will be. We can use it to test against while working on this bug.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: Bug 1881272: remove support for unused 'l10n_repos', 'tag_override', and 'gecko_locale_revisions' in mozharness r?#releng-reviewers!

I didn't do archaeology on these, but I checked the current tree for usages of the underscored and hyphenated versions of these. There are a handful of places where they get passed around, but I was unable to find anywhere that actually sets them.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: Bug 1881272: Fix urllib imports in gittool.py r?#releng-reviewers!

Ideally we should drop these obfuscated embedded modules, but I would prefer not to block on that for the moment. So in the meantime, here is the raw diff of the decoded version:
--- old.git.py 2024-05-27 15:06:58.352627967 -0400
+++ git.py 2024-05-27 13:58:50.365958418 -0400
@@ -1,7 +1,7 @@
"""Functions for interacting with hg"""
import os
import subprocess
-import urllib
+import urllib.parse
import re

from util.commands import run_cmd, remove_path, run_quiet_cmd
@@ -28,8 +28,8 @@

def get_repo_name(repo):
bits = urllib.parse.urlsplit(repo)

• host = urllib.quote(bits.netloc, "")
• path = urllib.quote(bits.path.lstrip("/"), "")

• host = urllib.parse.quote(bits.netloc, "")
• path = urllib.parse.quote(bits.path.lstrip("/"), "")
  return os.path.join(host, path)

Comment 4

[[e-mail] (:bhearsum)]

Attachment: Bug 1881272: ignore rare, recoverable error condition during l10n repack r?eemeli!

Comment 5

[[e-mail] (:bhearsum)]

Attachment: Bug 1881272: add support for pulling l10n strings from git r?#releng-reviewers!,eemeli!

This turned out to be quite straightforward and boring. Aside from the plumbing required to allow the git repo in the config and make gittool.py available, the actual changes to the code are a simple fork that clone the git repository if it's there, and fall back to the old repo (and logic) if it's not.

Because the git repository is laid out the same way as the directory we clone the current Mercurial repositories into, everything else Just Works (tm).

I did two simple tests of this on try:

• https://treeherder.mozilla.org/jobs?repo=try&revision=ccbf29629b30347aa8acf166ee53d8cd4670d9e0 is without git enabled
• https://treeherder.mozilla.org/jobs?repo=try&revision=f71f6b91f8e6aea0566d42fea15c19b8a681e881 is with git enabled.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ccfc9b7a1f39
remove support for unused 'l10n_repos', 'tag_override', and 'gecko_locale_revisions' in mozharness r=releng-reviewers,ahal,jcristau
https://hg.mozilla.org/integration/autoland/rev/f8cd7bfc3c98
Fix urllib imports in gittool.py r=releng-reviewers,jcristau
https://hg.mozilla.org/integration/autoland/rev/4c893612d034
ignore rare, recoverable error condition during l10n repack r=eemeli
https://hg.mozilla.org/integration/autoland/rev/7bace6d14823
add support for pulling l10n strings from git r=eemeli,releng-reviewers,jcristau

Comment 7

[Norisz Fay [:noriszfay]]

Backed out for causing l10n related Android bustages

Backout link

Push with failures

Failure log

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9a3443e0ae69
remove support for unused 'l10n_repos', 'tag_override', and 'gecko_locale_revisions' in mozharness r=releng-reviewers,ahal,jcristau
https://hg.mozilla.org/integration/autoland/rev/4c60ec51029b
Fix urllib imports in gittool.py r=releng-reviewers,jcristau
https://hg.mozilla.org/integration/autoland/rev/c6a2f04dd616
ignore rare, recoverable error condition during l10n repack r=eemeli
https://hg.mozilla.org/integration/autoland/rev/1c676eb3a477
add support for pulling l10n strings from git r=eemeli,releng-reviewers,jcristau

Comment 9

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/9a3443e0ae69
https://hg.mozilla.org/mozilla-central/rev/4c60ec51029b
https://hg.mozilla.org/mozilla-central/rev/c6a2f04dd616
https://hg.mozilla.org/mozilla-central/rev/1c676eb3a477

Comment 10

[[e-mail] (:bhearsum)]

Attachment: Bug 1881272: remove support for unused 'l10n_repos', 'tag_override', and 'gecko_locale_revisions' in mozharness

Comment 11

[[e-mail] (:bhearsum)]

Attachment: Bug 1881272: Fix urllib imports in gittool.py

Comment 12

[[e-mail] (:bhearsum)]

Attachment: Bug 1881272: ignore rare, recoverable error condition during l10n repack

Comment 13

[[e-mail] (:bhearsum)]

Attachment: Bug 1881272: add support for pulling l10n strings from git

Comment 14

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/202b57c0c242
https://hg.mozilla.org/releases/mozilla-esr115/rev/2d9eac2bada7
https://hg.mozilla.org/releases/mozilla-esr115/rev/71d7839978d6
https://hg.mozilla.org/releases/mozilla-esr115/rev/e87730e72627