1918304 - Incomplete localization with local build (locale es-MX)

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[cangas.oz]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:132.0) Gecko/20100101 Firefox/132.0

Steps to reproduce:

./mach configure
./mach build
./mach package
./mach build installers-es-MX
132.0a1 linux x64

Actual results:

build was successful but very patchy translations of menus items: very few items were translated into spanish

Expected results:

almost all menu items should be in spanish, eg. settings -> ajustes but no

Comment 1

[Francesco Lodolo [:flod]]

Can you attach a screenshot with specific strings untranslated?

While es-MX is not at 100%, as far as I can see all Settings strings are translated, which points to a build problem.

P.S. I'm sure that team would appreciate all the help they can get https://pontoon.mozilla.org/es-MX/firefox/

Comment 3

[Francesco Lodolo [:flod]]

mach build installers-* seems to fail for me

./mach build installers-es-MX
 0:00.52 W Clobber not needed.
  Parallelism determined by memory: using 10 jobs for 10 cores based on 64.0 GiB RAM and estimated job size of 1.0 GiB
 0:00.90 /usr/bin/make -j10 -s installers-es-MX
 0:01.02 /bin/sh: clone: command not found
 0:01.02 make[3]: *** [merge-es-MX] Error 127
 0:01.02 make[2]: *** [l10n-es-MX] Error 2
 0:01.02 make[1]: *** [installers-es-MX] Error 2
 0:01.02 make: *** [installers-es-MX] Error 2
 0:01.23 W 990 compiler warnings present.

EDIT: it works with an explicit ac_add_options --with-l10n-base=/make/this/a/absolute/path. ~/.mozbuild/l10n-central is re-created but it's empty.

Comment 4

[Francesco Lodolo [:flod]]

The build works for me, I don't see untranslated content, but I have a feeling something funny is going on here.

@reporter

Can you check the content of ~/.mozbuild/l10n-central/? Is it empty, a Mercurial repository (.hg folder present) or Git (.git folder present)?

This should work around the issue for now:

1. Manually clone https://github.com/mozilla-l10n/firefox-l10n on your system
2. Add an option to mozconfig ac_add_options --with-l10n-base=/make/this/a/absolute/path that points to the folder
3. Run ./mach build installers-es-MX again

Comment 5

[Francesco Lodolo [:flod]]

@Ben
Is that something that you could check (at least locally)?

From the error in the log, it looks like it can't find git on my system.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Francesco Lodolo [:flod] from comment #5)

@Ben
Is that something that you could check (at least locally)?

From the error in the log, it looks like it can't find git on my system.

Seems plausible. I assume this error is coming from https://searchfox.org/mozilla-central/rev/4a8bd8efdfaa43dd14a16d3cb15bf86796fd1def/toolkit/locales/l10n.mk#165, and that $(GIT) is empty (which is allowed by the build system.

I suspect that the best thing to do here is add a check for this being set in l10n.mk and throw a better error if it's not. (It would be ideal if we had a way to make this required at the build system level when doing a repack, but I'm not sure we have that level of configurability for it.)

Comment 7

[cangas.oz]

I did do this:

Manually clone https://github.com/mozilla-l10n/firefox-l10n on your system
Add an option to mozconfig ac_add_options --with-l10n-base=/make/this/a/absolute/path that points to the folder
Run ./mach build installers-es-MX again

and it did work. But the instructions page on how to package this does say it will download the translation files itself to .mozbuild/l10-central(In reply to Francesco Lodolo [:flod] from comment #1)

Can you attach a screenshot with specific strings untranslated?

While es-MX is not at 100%, as far as I can see all Settings strings are translated, which points to a build problem.

P.S. I'm sure that team would appreciate all the help they can get https://pontoon.mozilla.org/es-MX/firefox/

I'm not sure about how to provide the untranslated strings, merely looking at the running build is my symptom...

Comment 8

[cangas.oz]

But I would prefer not to have to download the whole repository from git. I tried downloading the es-MX translations only from https://hg.mozilla.org/l10n-central/ and then sett the ac_add_options but that produced the same result: only very few strings were translated.

Comment 9

[cangas.oz]

One further datum: .mozbuild/l10-central/es-MX did get populated during the build but the translations did not work.

Comment 10

[Francesco Lodolo [:flod]]

(In reply to cangas.oz from comment #8)

But I would prefer not to have to download the whole repository from git. I tried downloading the es-MX translations only from https://hg.mozilla.org/l10n-central/ and then sett the ac_add_options but that produced the same result: only very few strings were translated.

That repository is obsolete, that's the reason why you're getting partial translations in the first place.

But I would prefer not to have to download the whole repository from git.

That's the only way, but automation should do it for you.

One other thing to try is to manually remove .mozbuild/l10-central/ to see if that downloads the git repo on your system.

With that said, it's clear what the problem is at this point, downloading the git repository separately and adding the option is a workaround to get you unblocked.

Comment 11

[Francesco Lodolo [:flod]]

Attachment: Bug 1918304 - Fix l10n git repository cloning step in ./mach build installers-$AB_CD, r=bhearsum!

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1904586

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8cbe1aad4c5c
Fix l10n git repository cloning step in ./mach build installers-$AB_CD, r=bhearsum

Comment 14

[Norisz Fay [:noriszfay]]

Backed out for causing l10n and android bustages

Backout link

Push with failures - android bustage
Push with failures - l10n bustage

Failure log - android bustage
Failure log - l10n bustage

Comment 15

[Francesco Lodolo [:flod]]

@ben
Can you help me understand what the problem could be here? I will probably unassign myself from the bug, because it's starting to feel out of my depth.

Since it was completely broken locally, I assumed nothing was really using that part of the makefile, so it's surprising to see failures. Also, locally it works for me with or without the ~/mozbuild/l10n-central folder.

1. Why wasn't it failing before? Something was putting content in ~/mozbuild/l10n-central?
2. The logs are complaining about git pull failing because the clone is not a branch. No clue how you'd get into that state.

I'm guessing I could remove the git pull part, and that might get rid of the failures, but the whole thing seems very fuzzy.

Comment 16

[Francesco Lodolo [:flod]]

Try build without the git pull part (not sure it selected all the right tasks though)
https://treeherder.mozilla.org/jobs?repo=try&revision=13f7186ba1fd558e1c06f239a606e0e4e5504f66

Comment 17

[[e-mail] (:bhearsum)]

(In reply to Francesco Lodolo [:flod] from comment #15)

@ben
Can you help me understand what the problem could be here? I will probably unassign myself from the bug, because it's starting to feel out of my depth.

Since it was completely broken locally, I assumed nothing was really using that part of the makefile, so it's surprising to see failures. Also, locally it works for me with or without the ~/mozbuild/l10n-central folder.

1. Why wasn't it failing before? Something was putting content in ~/mozbuild/l10n-central?

In automation, mozharness clones the l10n repo: https://searchfox.org/mozilla-central/rev/181e5bb2645236a617d42e3740420098097f7a0f/testing/mozharness/mozharness/mozilla/l10n/locales.py#163-167:

[task 2024-09-13T16:30:36.860Z] 16:30:36     INFO -  16:30:36     INFO - Using env: {'GIT_SHARE_BASE_DIR': '/builds/hg-shared',
[task 2024-09-13T16:30:36.860Z] 16:30:36     INFO -  16:30:36     INFO -  'PATH': '/usr/local/bin:/bin:/usr/bin'}
[task 2024-09-13T16:30:36.874Z] 16:30:36     INFO -  16:30:36     INFO -  /builds/worker/checkouts/gecko/testing/mozharness/external_tools/gittool.py:30: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
[task 2024-09-13T16:30:36.874Z] 16:30:36     INFO -  16:30:36     INFO -    import sys, imp, base64, zlib
[task 2024-09-13T16:30:36.900Z] 16:30:36     INFO -  16:30:36     INFO -  2024-09-13 16:30:36,899 creating bare repo /builds/hg-shared/github.com/mozilla-l10n%2Ffirefox-l10n
[task 2024-09-13T16:30:36.900Z] 16:30:36     INFO -  16:30:36     INFO -  2024-09-13 16:30:36,900 removing /builds/hg-shared/github.com/mozilla-l10n%2Ffirefox-l10n
[task 2024-09-13T16:30:36.900Z] 16:30:36     INFO -  16:30:36     INFO -  2024-09-13 16:30:36,900 git init --bare -q /builds/hg-shared/github.com/mozilla-l10n%2Ffirefox-l10n
[task 2024-09-13T16:30:36.902Z] 16:30:36     INFO -  16:30:36     INFO -  2024-09-13 16:30:36,902 Checking dest /builds/worker/checkouts/l10n-central
[task 2024-09-13T16:30:36.903Z] 16:30:36     INFO -  16:30:36     INFO -  fatal: not a git repository (or any parent up to mount point /builds/worker)
[task 2024-09-13T16:30:36.903Z] 16:30:36     INFO -  16:30:36     INFO -  Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[task 2024-09-13T16:30:36.903Z] 16:30:36     INFO -  16:30:36     INFO -  2024-09-13 16:30:36,903 /builds/worker/checkouts/l10n-central doesn't appear to be a valid git directory; clobbering
[task 2024-09-13T16:30:36.903Z] 16:30:36     INFO -  16:30:36     INFO -  2024-09-13 16:30:36,903 removing /builds/worker/checkouts/l10n-central
[task 2024-09-13T16:30:36.903Z] 16:30:36     INFO -  16:30:36     INFO -  2024-09-13 16:30:36,903 git init -q /builds/worker/checkouts/l10n-central
[task 2024-09-13T16:30:36.917Z] 16:30:36     INFO -  16:30:36     INFO -  2024-09-13 16:30:36,917 command: START
[task 2024-09-13T16:30:36.917Z] 16:30:36     INFO -  16:30:36     INFO -  2024-09-13 16:30:36,917 command: git fetch -q https://github.com/mozilla-l10n/firefox-l10n +refs/heads/*:refs/remotes/origin/*
[task 2024-09-13T16:30:36.917Z] 16:30:36     INFO -  16:30:36     INFO -  2024-09-13 16:30:36,917 command: cwd: /builds/hg-shared/github.com/mozilla-l10n%2Ffirefox-l10n
[task 2024-09-13T16:30:36.917Z] 16:30:36     INFO -  16:30:36     INFO -  2024-09-13 16:30:36,917 command: output:
[task 2024-09-13T16:31:34.368Z] 16:31:34     INFO -  16:31:34     INFO -  2024-09-13 16:31:34,367 command: END (57.45s elapsed)

(and much more on this in the log).

2. The logs are complaining about git pull failing because the clone is not a branch. No clue how you'd get into that state.

It looks like the pull is being done on the bare repository, which has no checkout to update (pull is essential fetch + merge by default):

[task 2024-09-13T17:06:54.214Z] 17:06:54     INFO -  if  test -d /builds/worker/checkouts/l10n-central/.git ; then \
[task 2024-09-13T17:06:54.214Z] 17:06:54     INFO -  	git -C /builds/worker/checkouts/l10n-central pull --quiet ; \
[task 2024-09-13T17:06:54.214Z] 17:06:54     INFO -  else \
[task 2024-09-13T17:06:54.214Z] 17:06:54     INFO -  	echo 'Error: folder is not a git repository /builds/worker/checkouts/l10n-central' ; \
[task 2024-09-13T17:06:54.214Z] 17:06:54     INFO -  	exit 1 ; \

I'm guessing I could remove the git pull part, and that might get rid of the failures, but the whole thing seems very fuzzy.

You could also considering moving it behind an ifndef MOZ_AUTOMATION to avoid hitting that new code path for automation altogether.

Comment 18

[Francesco Lodolo [:flod]]

Thanks for the explanation.

The try build without git pull seems green enough to me. I will give it a try with ifndef MOZ_AUTOMATION, since I think there's value in pulling the repository, and see how that goes.

Comment 19

[cangas.oz]

I have tried again. Manually cloning all translations and pointing to that download works, but attempting the automatic updating of l10 provides the following:
hg pull -u
hg update --clean
./mach configure
./mach build
./mach package
./mach build installers-es-MX ===>
0:00.87 W Clobber not needed.
Parallelism determined by memory: using 12 jobs for 12 cores based on 15.5 GiB RAM and estimated job size of 1.0 GiB
0:11.90 /usr/bin/gmake -j12 -s installers-es-MX
0:12.17 /bin/sh: 3: clone: not found

Comment 20

[Francesco Lodolo [:flod]]

This bug is not fixed yet. Once that happens, you will need to manually remove .mozbuild/l10n-central before trying again, as that folder needs to be used for a Git repository.

Comment 21

[cangas.oz]

OK, thanks. (I did remove the content of l10-central before I tried though)

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/598fb32492b0
Fix l10n git repository cloning step in ./mach build installers-$AB_CD, r=bhearsum

Comment 23

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/598fb32492b0

Comment 24

[cangas.oz]

thank you

Comment 25

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:flod, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox131 to wontfix.

For more information, please visit BugBot documentation.

Comment 26

[Francesco Lodolo [:flod]]

The code that I fixed is behind a NIGHTLY_BUILD flag, I don't think uplifting has any practical effect.