1310221 - Remove broken locales for Thunderbird builds [meta]

Status: RESOLVED WORKSFORME

(Thunderbird :: Build Config, defect)

Description

[aleth [:aleth]]

Making langpack /builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/objdir-tb/dist/linux-x86_64/xpi/thunderbird-52.0a1.ta-LK.langpack.xpi
/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/objdir-tb/_virtualenv/bin/python /builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/config/nsinstall.py -D ../../dist/linux-x86_64/xpi/
/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/objdir-tb/_virtualenv/bin/python -m mozbuild.action.preprocessor -DNDEBUG=1 -DTRIMMED=1 -DMOZ_APP_MAXVERSION=52.0a1 -DDLL_PREFIX=lib -DDLL_SUFFIX=.so -DBIN_SUFFIX= -DDIR_MACOS= -DDIR_RESOURCES= -DA11Y_LOG=1 -DACCESSIBILITY=1 -DATK_MAJOR_VERSION=2 -DATK_MINOR_VERSION=2 -DATK_REV_VERSION=0 -DBUILD_CTYPES=1 -DCROSS_COMPILE='' -DD_INO=d_ino -DE10S_TESTING_ONLY=1 -DEARLY_BETA_OR_EARLIER=1 -DENABLE_INTL_API=1 -DENABLE_MARIONETTE=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DENABLE_TESTS=1 -DEXPOSE_INTL_API=1 -DFIREFOX_VERSION=52.0a1 -DFORCE_PR_LOG=1 -DGL_PROVIDER_GLX=1 -DGTEST_HAS_RTTI=0 -DHAVE_64BIT_BUILD=1 -DHAVE_FONTCONFIG_FCFREETYPE_H=1 -DHAVE_FORCEINLINE=1 -DHAVE_LOCALECONV=1 -DHAVE_MOVEMAIL=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DMALLOC_H='<malloc.h>' -DMOZILLA_OFFICIAL=1 -DMOZILLA_UAVERSION='"52.0"' -DMOZILLA_VERSION='"52.0a1"' -DMOZILLA_VERSION_U=52.0a1 -DMOZ_ACCESSIBILITY_ATK=1 -DMOZ_APP_UA_NAME='""' -DMOZ_APP_UA_VERSION='"52.0a1"' -DMOZ_B2G_OS_NAME='""' -DMOZ_B2G_VERSION='"1.0.0"' -DMOZ_BINARY_EXTENSIONS=1 -DMOZ_BUILD_APP=../mail -DMOZ_CONTENT_SANDBOX=1 -DMOZ_CRASHREPORTER=1 -DMOZ_CRASHREPORTER_ENABLE_PERCENT=100 -DMOZ_DATA_REPORTING=1 -DMOZ_DISTRIBUTION_ID='"org.mozilla"' -DMOZ_DLL_SUFFIX='".so"' -DMOZ_ENABLE_DBUS=1 -DMOZ_ENABLE_GCONF=1 -DMOZ_ENABLE_GIO=1 -DMOZ_ENABLE_MASK_AS_SHORTHAND=1 -DMOZ_ENABLE_PROFILER_SPS=1 -DMOZ_ENABLE_SKIA=1 -DMOZ_ENABLE_XREMOTE=1 -DMOZ_FEEDS=1 -DMOZ_FFMPEG=1 -DMOZ_FFVPX=1 -DMOZ_FMP4=1 -DMOZ_GAMEPAD=1 -DMOZ_GLUE_IN_PROGRAM=1 -DMOZ_GMP_SANDBOX=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_JSDOWNLOADS=1 -DMOZ_LOGGING=1 -DMOZ_MACBUNDLE_ID=org.mozilla.daily -DMOZ_MORK=1 -DMOZ_PEERCONNECTION=1 -DMOZ_PERMISSIONS=1 -DMOZ_PLACES=1 -DMOZ_PULSEAUDIO=1 -DMOZ_RAW=1 -DMOZ_SAMPLE_TYPE_FLOAT32=1 -DMOZ_SANDBOX=1 -DMOZ_SCTP=1 -DMOZ_SECUREELEMENT=1 -DMOZ_SEPARATE_MANIFEST_FOR_THEME_OVERRIDES=1 -DMOZ_SRTP=1 -DMOZ_STACKWALKING=1 -DMOZ_STATIC_JS=1 -DMOZ_THUNDERBIRD=1 -DMOZ_TREE_CAIRO=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_UPDATER=1 -DMOZ_UPDATE_CHANNEL=nightly -DMOZ_USER_DIR='".mozilla"' -DMOZ_VORBIS=1 -DMOZ_VPX_ERROR_CONCEALMENT=1 -DMOZ_VPX_NO_MEM_REPORTING=1 -DMOZ_WEBM_ENCODER=1 -DMOZ_WEBRTC=1 -DMOZ_WEBRTC_ASSERT_ALWAYS=1 -DMOZ_WEBRTC_SIGNALING=1 -DMOZ_WEBSPEECH=1 -DMOZ_WEBSPEECH_TEST_BACKEND=1 -DMOZ_WIDGET_GTK=3 -DMOZ_X11=1 -DMOZ_XUL=1 -DNIGHTLY_BUILD=1 -DNO_NSPR_10_SUPPORT=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DSPIDERMONKEY_PROMISE=1 -DSTATIC_JS_API=1 -DUSE_SKIA=1 -DUSE_SKIA_GPU=1 -DU_STATIC_IMPLEMENTATION=1 -DU_USING_ICU_NAMESPACE=0 -DVPX_X86_ASM=1 -DXP_LINUX=1 -DXP_UNIX=1 -DAB_CD=ta-LK -DAB_CD=ta-LK -DMOZ_LANGPACK_EID=[e-mail] -DMOZ_APP_ID='{3550f703-e582-4d05-9a08-453d09bdfdc6}' -DMOZ_APP_VERSION=52.0a1 -DMOZ_APP_MAXVERSION=52.0a1 -DLOCALE_SRCDIR=/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/l10n/ta-LK/mail -DPKG_BASENAME='thunderbird-52.0a1.ta-LK.linux-x86_64' -DPKG_INST_BASENAME='thunderbird-52.0a1.ta-LK.linux-x86_64.installer'  -DTK_DEFINES=/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/l10n/ta-LK/toolkit/defines.inc -DAPP_DEFINES=/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/l10n/ta-LK/mail/defines.inc /builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/toolkit/locales/generic/install.rdf -o ../../dist/xpi-stage/locale-ta-LK/install.rdf
Traceback (most recent call last):
  File "/tools/python27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/tools/python27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/python/mozbuild/mozbuild/action/preprocessor.py", line 18, in <module>
    main(sys.argv[1:])
  File "/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/python/mozbuild/mozbuild/action/preprocessor.py", line 14, in main
    pp.handleCommandLine(args, True)
  File "/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/python/mozbuild/mozbuild/preprocessor.py", line 481, in handleCommandLine
    self.processFile(input=input, output=out)
  File "/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/python/mozbuild/mozbuild/preprocessor.py", line 387, in processFile
    self.do_include(input, False)
  File "/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/python/mozbuild/mozbuild/preprocessor.py", line 776, in do_include
    self.handleLine(l)
  File "/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/python/mozbuild/mozbuild/preprocessor.py", line 554, in handleLine
    self.write(aLine)
  File "/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/python/mozbuild/mozbuild/preprocessor.py", line 435, in write
    filteredLine = self.applyFilters(aLine)
  File "/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/python/mozbuild/mozbuild/preprocessor.py", line 410, in applyFilters
    aLine = f[1](aLine)
  File "/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/python/mozbuild/mozbuild/preprocessor.py", line 732, in filter_substitution
    return self.varsubst.sub(repl, aLine)
  File "/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/python/mozbuild/mozbuild/preprocessor.py", line 730, in repl
    raise Preprocessor.Error(self, 'UNDEFINED_VAR', varname)
mozbuild.preprocessor.Error: ('/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/toolkit/locales/generic/install.rdf', 16, 'UNDEFINED_VAR', 'MOZ_LANG_TITLE')
make: *** [langpack-ta-LK] Error 1

Comment 1

[aleth [:aleth]]

https://archive.mozilla.org/pub/thunderbird/nightly/2016/10/2016-10-14-12-24-44-comm-central-l10n/comm-central-linux64-l10n-nightly-ta-LK-bm71-build1-build409.txt.gz

Comment 2

[aleth [:aleth]]

Looks like the undefined MOZ_LANG_TITLE should be set in the localized version of

https://dxr.mozilla.org/comm-central/source/mozilla/toolkit/locales/en-US/defines.inc#1

so this isn't an in-tree issue?

Comment 3

[Francesco Lodolo [:flod]]

As I wrote in another bug, you really should triage the list of locales you're building.

Current status for ta-LK
https://l10n.mozilla.org/teams/ta-LK

Last sign-off: Thunderbird 9

File is empty
https://hg.mozilla.org/releases/l10n/mozilla-aurora/ta-LK/file/default/toolkit/defines.inc

Having said that, I assume this should fallback to en-US, so it sounds like bug 1307247 comment 9.

Comment 4

[Axel Hecht [:Pike]]

I fixed this once in 2013 in bug 913470, but it regresssed again in 2014 with https://hg.mozilla.org/releases/l10n/mozilla-aurora/ta-LK/diff/4a4b1221a19f/toolkit/defines.inc.

Existing but empty .inc files are a definite build breaker, see http://hg.mozilla.org/mozilla-central/file/default/toolkit/locales/l10n.mk#l158. Also part of the tests for repacks on firefox, https://hg.mozilla.org/mozilla-central/file/default/browser/locales/Makefile.in#l188.

This is a bug about ta-LK, I'm with flod, the best fix is to remove the build, across all channels.

Comment 5

[aleth [:aleth]]

(In reply to Francesco Lodolo [:flod] from comment #3)
> As I wrote in another bug, you really should triage the list of locales
> you're building.

Thanks! I'm slowly working my way through the big stack of c-c/c-a l10n bustage, hopefully we can get there soon.

Comment 6

[Philipp Kewisch [:Fallen][📱📆]]

I agree we should remove some locales, I have been wanting to do this for a while now but there was never enough pressure.

I suggest to remove the following locales due to their last sign-off:

Locale   Missing    Last Sign-off
bn-BD    2237       tb6
ta-LK    6498       tb9
pa-IN    739        tb9
vi       2693       tb9
el       2892       tb12
si       1477       tb17
gl       472 	    tb19 
be       1393       tb31

The following locale have nothing in the status field, I'm assuming this basically means the sign-off is too long ago. I suspect they should also be removed. Pike, can you confirm what the empty status field means?

af       3176
ka       882

The following locales have a high number of missing strings, but a more recent sign-off. Maybe we should at least ask them about their level of participation:

he      1989        tb45
id      1306        tb49

The data was taken from tb_beta, the numbers on tb_aurora are even higher. Removing said locales would boost our minimum translated percent to above 80 for all locales.

Comment 7

[Francesco Lodolo [:flod]]

The empty value for a locale in beta means that there's no sign-off, that's expected given that they're only in aurora and not shipping in beta
http://hg.mozilla.org/releases/comm-beta/file/default/mail/locales/shipped-locales

I would suggest to open bugs for each of these localizations, saying that you plan to drop their localization from Thunderbird, and see what they have to ask. ta-LK and be can be removed directly, since there's no community.

One other point that you need to consider is the number of active users you have in these locales, and what you want to do with them, i.e. to which other language you'd move them.

Comment 8

[Axel Hecht [:Pike]]

Additional comment on the numbers, you should check how much of Thunderbird would be devtools/shared at this point, and include that in to your thinking of how many missing strings are OK or not. As devtools is considered to be good in English for many locales on the Firefox side.

Re old sign-offs, you may want to consider following what we did on the sign-off process, announced at https://groups.google.com/forum/#!msg/mozilla.dev.l10n/t4IolW4moMY/BBcxbFxAAQAJ;context-place=forum/mozilla.dev.l10n. We've stopped asking localizers for sign-offs, and do the technical review unprompted on our own schedule, taking whatever's good to take.

We talk about removing builds when the total goes bad, but we don't use sign-offs as a signal no more.

Comment 9

[Philipp Kewisch [:Fallen][📱📆]]

Jasmin will be contacting localizers to help move this bug forward!

Comment 10

[Francesco Lodolo [:flod]]

(In reply to Philipp Kewisch [:Fallen] from comment #9)
> Jasmin will be contacting localizers to help move this bug forward!

We need to get his permissions upgraded, otherwise bugs will be filed as unconfirmed and remain invisible to most (I'm following the entire product and didn't get a notification for the bn-BD bug)
https://wiki.mozilla.org/BMO/UserGuide#How_to_apply_for_upgraded_permissions

Comment 11

[Philipp Kewisch [:Fallen][📱📆]]

I'll take care of the permissions.

Comment 12

[Wayne Mery (:wsmwk)]

Thanks Jasmin.

Comment 13

[Wayne Mery (:wsmwk)]

I suggest we don't drop a locale until it falls below our target, in case someone else picks up the locale. For example ...
bd-BN at 82%
pa-IN at 84%

But keep the bug report open for tracking. Does that make sense?

Comment 14

[Francesco Lodolo [:flod]]

(In reply to Wayne Mery (:wsmwk) from comment #13)
> I suggest we don't drop a locale until it falls below our target, in case
> someone else picks up the locale.

The content won't be removed from the repository, so picking up the language would only be a matter of re-enabling it in localization tools.

bn-BD Is missing 2600 strings, 82% seems unlikely. What's the source?
https://l10n.mozilla.org/shipping/dashboard?tree=tb_central

Also note that, sadly, the completion percentage alone is misleading: pa-IN is missing over 1000 strings, but also has 2700 identical message, which means it's basically in English…

Comment 15

[Wayne Mery (:wsmwk)]

My bad. I was looking at the % at the top of https://pontoon.mozilla.org/bn-BD/ which is just an average.  Thunderbird isn't even listed there. likewise https://pontoon.mozilla.org/pa-IN/

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/a6386623e224
Remove bn-BD, pa-IN and ta-LK locales. r=me. DONTBUILD

Comment 17

[Philipp Kewisch [:Fallen][📱📆]]

I've removed bn-BD, pa-IN and ta-LK now. gl and vi are waiting on a reply.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/5d088abc2f19
Remove bn-BD, pa-IN and ta-LK locales. r=me. DONTBUILD

Comment 19

[Axel Hecht [:Pike]]

Flagging myself for removing these locales from the dashboard once comm-central merges to comm-beta. Did that for tb_central already, though.

Comment 20

[Wayne Mery (:wsmwk)]

(In reply to Axel Hecht [:Pike] from comment #19)
> Flagging myself for removing these locales from the dashboard once
> comm-central merges to comm-beta. Did that for tb_central already, though.

Axel, comm-central v60 is now beta so you can proceed?

Comment 21

[Wayne Mery (:wsmwk)]

(In reply to Philipp Kewisch [:Fallen]  from comment #17)
> I've removed bn-BD, pa-IN and ta-LK now. gl and vi are waiting on a reply.

these locales are still appearing in ship-it and I have been manually removing them. Plus af, cak, fa, ka, and uz because iirc they are not found in shipped-locales.

Can we sort that out this week?  (My manual edit of ship-it locales can be error prone)

Comment 22

[Axel Hecht [:Pike]]

I've cleared up the locale list of builds. I didn't do the sign-off part, I'm hoping that Philipp gives me an idea why the data we have is the data we have.

PS: I think it'd be good to have the maintained list of shipped-locales in comm-central and let that ride the trains rather than just having 'en-US' in there.

Comment 23

[Wayne Mery (:wsmwk)]

Can you think of anything left to do here?

Comment 24

[Rob Lemley [:rjl]]

Nothing on my end.

Comment 25

[Wayne Mery (:wsmwk)]

Good to close then. Thanks.