1791284 - Searchfox l10n / l10n-central submodule ("subrepo") problem tracker

Status: NEW

(Webtools :: Searchfox, task)

Description

[Andrew Sutherland [:asuth] (he/him)]

Got 2 indexing warnings today which seem like they may be transient in nature as both crh and da seem to exist but we should keep an eye on this and report any problems here:

+ popd
/mnt/index-scratch/l10n/git ~
+ for submod in *
+ pushd ckb
/mnt/index-scratch/l10n/git/ckb /mnt/index-scratch/l10n/git ~
+ git pull
Already up to date.
+ popd
/mnt/index-scratch/l10n/git ~
+ for submod in *
+ pushd crh
/mnt/index-scratch/l10n/git/crh /mnt/index-scratch/l10n/git ~
+ git pull

ERROR [Errno 110] Connection timed out
Run the command again with `git -c cinnabar.check=traceback <command>` to see the full traceback.
+ echo 'WARNING: The crh subrepo inside l10n-central seems to be missing!'
WARNING: The crh subrepo inside l10n-central seems to be missing!
--
+ popd
/mnt/index-scratch/l10n/git ~
+ for submod in *
+ pushd cy
/mnt/index-scratch/l10n/git/cy /mnt/index-scratch/l10n/git ~
+ git pull
Already up to date.
+ popd
/mnt/index-scratch/l10n/git ~
+ for submod in *
+ pushd da
/mnt/index-scratch/l10n/git/da /mnt/index-scratch/l10n/git ~
+ git pull

ERROR [Errno 110] Connection timed out
Run the command again with `git -c cinnabar.check=traceback <command>` to see the full traceback.
+ echo 'WARNING: The da subrepo inside l10n-central seems to be missing!'
WARNING: The da subrepo inside l10n-central seems to be missing!

I did log into the machine and saw no indications that there were additional failures that were not included due to line limits.

Bug 1669735 initially landed support for l10n-central with some custom sub-tree iteration due to cinnabar interaction complications, although I think :glandium's response in https://github.com/glandium/git-cinnabar/issues/261 implies that we could actually use the protocol.allow mechanism to configure out git for that repo or our root config so that maybe it's okay with the hg protocol? (or "cinnabar"?)

cc'ing :flod mainly since it sounds like maybe we left it that manual action would need to be taken to add new l10n repositories, but I'm not sure I see any bugs that suggest this ever happened from bugs or git log .gitmodules from the repo (beyond a removal of x-testing).

I think we could automatically update the set of repositories without too much trouble; something like the following successfully scrapes the set of locales for me AFAICT (if we start provisioning ripgrep, which we absolutely should do anyways; the alternative would be provisioning pcregrep):

curl -s https://hg.mozilla.org/l10n-central/ | rg '^.*?href="/l10n-central/([^/]+)/".*?$' -r '$1'

Output looks like:

ace
ach
af
ak
an
...
xh
zam
zh-CN
zh-TW
zu

Piping that into wc gets me a count of 155 whereas on the searchfox listing of https://searchfox.org/l10n/source/ there seems to be a tbody childElementCount of 154 less one row for .gitmodules (and ignoring the thead) gives us 153, or 2 missing languages (unless some have been removed too? but then I'd expect we would have been seeing error messages on a daily basis already?). This needs to be addressed but is less bad than I was expecting.

Comment 1

[Francesco Lodolo [:flod]]

Both crh and da repositories are old, especially da. crh was added in 2018, da was migrated from CVS in 2008. We also never remove repositories.

The two locales missing from searchfox are af (must be a bug in the initial import, since it's another CVS locale), and tg, which is the latest we've added (bug 1684867).

Unfortunately I forgot the process is manual, so having it automated would be great.

I guess you could also use the json output with jq to extract them, instead of grepping the HTML?

curl -s "https://hg.mozilla.org/l10n-central/?style=json" | jq -r ".entries[].name"

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Ah, thanks, yes! I tried using my wgetjson alias that sets an accept header for JSON, but didn't get JSON out; I forgot that the mercurial server has that style mode and should have tried clicking around more. Passing JSON through jq is indeed much preferred, many thanks for providing that!

Needinfo on myself to address that.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Actually, while I'm poking buttons on this repo, are there any links that it would be useful to have appear in the navigation side panel, like "Edit this file in pontoon" or something like that where we can potentially apply a simple transform from the relative source path to an absolute URL?

Comment 4

[Francesco Lodolo [:flod]]

No, there is no easy way to do it (files in these repositories map to 4 different projects in Pontoon, for some locales files might be there but the project is not set up in Pontoon, some locales are completely missing from Pontoon, etc.)

I also don't think there is much overlap between folks using searchfox for l10n and Pontoon users, so not worth the additional effort.