997652 - [Tarako][Email] Tarako build missing locale data

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Vendcom, defect)

Description

[Bingqing Li]

Attachment: screenshot-1

The last letter of each word in Email app is in upper case.
Pls see the attachment.


Environment:
-----------------------------------------------------
Gaia      5a5690d6691d3bc23e83d4433b458192f4197a8f
Gecko     https://hg.mozilla.org/releases/mozilla-b2g28_v1_3t/rev/69214e49e489
BuildID   20140416164002
Version   28.1
ro.build.version.incremental=eng.cltbld.20140416.203531
ro.build.date=Wed Apr 16 20:35:38 EDT 2014

Comment 1

[Bingqing Li]

Attachment: screenshot-2

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

This indicates locales are not being loaded.  This is likely a problem in the build you are using or (guessing) the system is configured to use a locale the e-mail app does not have a locale for.

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

There might also be some other problem that would be visible in the logcat, although little has changed in email on the Tarako branch recently, so it's more likely a build problem.  If the problem still reproduces on a more recent build and there's no known general system regression, please be sure to capture a logcat while reproducing and provide it here.  Instructions are available at https://wiki.mozilla.org/Gaia/Email/RequiredBugInfo.  For future reference, note that we always want a logcat for email bugs.

Comment 4

[Bingqing Li]

It looks fine on build 20140417164004. Let's keep this open for a few more builds, is that OK?


Build Info
------------------------------------------------------
Gaia      f0872318d46781bb59d0a13a2e1fffb115b8ca2b
Gecko     https://hg.mozilla.org/releases/mozilla-b2g28_v1_3t/rev/9ff031daa73c
BuildID   20140417164004
Version   28.1
ro.build.version.incremental=eng.cltbld.20140417.201933
ro.build.date=Thu Apr 17 20:19:40 EDT 2014

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

Sure; if the builds are flakey it is a real problem.  And there's no point immediately resolving this just to open a similar one a day later.

Comment 6

[Bingqing Li]

Andrew,

I hadn't encountered this issue since build 20140417164004, so I think this bug can be closed.

Thanks for your help.

Comment 7

[Andrew Sutherland [:asuth] (he/him)]

Good to hear; thanks for making the effort to check back in on this!

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

It happened again, build log at https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-b2g28_v1_3t-tarako/2014/05/2014-05-19-16-40-02/logs/ suggests that there is a missing Makefile dependency that is resulting in the optimize task being run concurrently with per-app Makefiles.

I'm going to skim the Makefile git log for a bit to see if we already have a bug/fix we can uplift before investigating that too much more.

Comment 12

[Jason Smith [:jsmith]]

Naoki - Can you find out if this is a test blocker or not? There seems to be a pile of bugs coming in around this with the fact that we can't test localizations on tarako.

Comment 13

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

As far as I understand we're not responsible for l10n; the vendor is over all.  I think we are responsible for l10n for the marketplace though.

I don't think switching language is a blocker for us for email...  If we can't switch the Marketplace to the correct language, then that might be a blocker for us.

Sent email and waiting to hear back from Steve or Joe.

Comment 14

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Oops, I missed reading an email.  What I was referring to in the previous comment was bug 1013327 

Looking at the languages missing it's hi-IN, and ta.

I think this is more of a build issue; we should probably file a separate bug to pull the language translations from our vendor and make our builds.  This is not an email bug, but a build issue.

Comment 15

[Andrew Sutherland [:asuth] (he/him)]

Attachment: preliminary investigation notes

I've been looking into this using the definitely known bad build from this most recent time and the definitely known good build from last time.

In the bad build our locales-obj/en-US.json file contains only the shared locale contents but none of the email app's own strings.

The build logs in general show that the multilocale target (and its multilocale-clean recursive byproduct) do not have their dependencies correctly declared and effectively race and interleave with the email app's own build process.  Unfortunately, I'm not actually sure that the race is causing the problem in question.  I've been digging into the somewhat under-documented locale build process a bit more to try and understand which consistent ordering is desired and whether the race actually explains the failure.  (The multilocale step mainly just messes with ini files, it's the subsequent optimize stage that builds the locales-obj...)

Note that there's potentially a huge amount of variability in the races since kernel builds are occurring concurrently with the gaia stuff that's happening in parallel.  Additionally, the jobs don't put out a lot of output so what's actually happening under the hood can be somewhat opaque (hence the quick strace investigation in the log.)  This makes local reproduction somewhat difficult.

The en-US failure is especially odd and suggests there could be a different race/problem going on since en-US should potentially be a stable locale despite what multilocale/multilocale-clean do (unless files transiently don't exist) and appears to be ignored by some of the logic.  Unfortunately the app zips don't include the locale subtrees so specific intermediary states are somewhat guesswork.

Other note: This stuff doesn't/won't happen on trunk because a whole bunch of changes have landed for the build system since 1.3.  Bug 922463, bug 968666, bug 897352, somewhat bug 968661.

I'm hoping to get to this later "today", but I have some non-build blocker-related stuff that will pre-empt finishing this up.

Comment 16

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: email_lang.log

I checked with the OEM version, which has language translations... except for the email app.  I'm not 100 % sure if it's because they haven't gotten to it in the 5/12 build or if it wasn't switched over.

What I did notice is that it didn't have the capitalization mismatch associated with a missing language.

Comment 17

[Andrew Sutherland [:asuth] (he/him)]

:nhirata, is it possible to provide a copy of the email application.zip from that build so I can look at the build byproducts?  (Or point me at steps of where to fetch the build so I can extract the zip or flash-then-extract it?)

Comment 18

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: email.gaiamobile.org.zip

Attached is the email.gaiamobile.org contents

Comment 19

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Other people are noticing the l10n issue with email.  Nominating for 1.3T.

FYI, bn-BD will be used not bn-IN.

Comment 20

[James Zhang (Spreadtrum)]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #14)
> Oops, I missed reading an email.  What I was referring to in the previous
> comment was bug 1013327 
> 
> Looking at the languages missing it's hi-IN, and ta.
> 
> I think this is more of a build issue; we should probably file a separate
> bug to pull the language translations from our vendor and make our builds. 
> This is not an email bug, but a build issue.

Yang, please comment here. hi-IN and ta are git repos on my side.

Comment 21

[yang.zhao]

Hi,all!
   We init the 'hi-IN' and 'ta' git repos on our side,not fetch it from mozillaorg .So you should pull the two language translations from our side.
   I found the the l10n in email app couldn't change when the language change.Maybe it's something wrong in email's l10n translation.

Comment 22

[Andrew Sutherland [:asuth] (he/him)]

(In reply to yang.zhao from comment #21)
>    I found the the l10n in email app couldn't change when the language
> change.Maybe it's something wrong in email's l10n translation.

The email app needs to be restarted after a locale change in order for all the strings to update.  Is that what you're talking about?

In terms of build issues, if the build is done as "-j1" instead of "-j8" or whatever, the races should be avoided and eliminate that potential set of problems.  I am a little backlogged on travel prep but will try and provide a candidate Makefile patch shortly to better express the app makefile dependency.

Comment 23

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Thanks James.  I made a pull of the languages, and am working on building it.  I have a feeling that the issue is on our end.  I filed bug 1025390.  Once I finish building, I should be able to verify if this is on our end or not.  I will unnom this bug if the issue is our builds and dup this bug to bug 1025390, unless you think otherwise?

Andrew, I believe that the switching of languages is discussed in bug 1008859?  Or is that not the same issue?  If it is the same issue, maybe we can continue the discussion in that bug?

Comment 24

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

This is our build issue.  I found after pulling the locale and then adding LOCALE_BASEDIR=gaia-l10n/ to my build.sh line, I was able to show that email is translated correctly.

There's a few strings that doesn't seem to be translated in my build; I think that's a separate issue.

Comment 25

[James Zhang (Spreadtrum)]

Kaizhen, can you help add spreadtrum gaia-l10n git repo to tarako v1.3t manifest? Thank you.

<remote name="sprd-b2g" fetch="http://sprdsource.spreadtrum.com:8085/b2g" />
<project name="l10n/hi-IN/gaia" path="gaia-l10n/hi-IN" remote="sprd-b2g" revision="v1.3t" />
<project name="l10n/ta/gaia" path="gaia-l10n/ta"  remote="sprd-b2g" revision="v1.3t"  />

Comment 26

[Kai-Zhen Li [:seinlin][:kli]]

(In reply to James Zhang (Spreadtrum) from comment #25)
> Kaizhen, can you help add spreadtrum gaia-l10n git repo to tarako v1.3t
> manifest? Thank you.
> 
> <remote name="sprd-b2g" fetch="http://sprdsource.spreadtrum.com:8085/b2g" />
> <project name="l10n/hi-IN/gaia" path="gaia-l10n/hi-IN" remote="sprd-b2g"
> revision="v1.3t" />
> <project name="l10n/ta/gaia" path="gaia-l10n/ta"  remote="sprd-b2g"
> revision="v1.3t"  />

Hwine, v1.3 branch of these two l10n repos are contributed by partner, and their corresponding git url are:
https://git.mozilla.org/releases/l10n/ta/gaia
https://git.mozilla.org/releases/l10n/hi-IN/gaia

Could you suggest me how can we get them on git.m.o?  Thanks!

Comment 27

[hwine]

(In reply to Kai-Zhen Li from comment #26)
> Hwine, v1.3 branch of these two l10n repos are contributed by partner, and
> their corresponding git url are:
> https://git.mozilla.org/releases/l10n/ta/gaia
> https://git.mozilla.org/releases/l10n/hi-IN/gaia
> 
> Could you suggest me how can we get them on git.m.o?  Thanks!

No, this is a new process you are proposing. Currently, all l10n work is done directly in Mercurial.

How these partner repositories get integrated is up to the Mozilla l10n team. Axel?

Once the new process is decided upon, we'll help implement it.

Comment 28

[Axel Hecht [:Pike]]

I mentioned this in bug 1025390, we shouldn't load the SPRD l10n repos into ours.

Notably, this only affects email, and one potential culprit is that email is one of one or two apps using a build dir on 1.3, so it could be that this is something slightly off-beat in SPRD's build generation.

Comment 29

[Axel Hecht [:Pike]]

PS: this is only our bug if this can be reproduces on a 1.3 non-Tarako build, suggested locale to test would be French.

Comment 30

[Aki Sasaki (not active)]

The current setup:
* releng builds from community localized repos
* partners can add other locales, but then are on the hook for the localization, building, and testing of those locales.

As discussed in today's meeting, we're going to push back here.  Moz QA should either remove these locales from their test list for 1.3t or use partner builds when testing.

Comment 31

[James Burke [:jrburke]]

I tested a 1.3 non-Tarako build (Hamachi) via the pvt builds, with French, and it was fine. 

So current theory is how the build is run from the partner. As asuth mentioned in comment 22, a simple solution may be to use "-j1" with the `make` command that is used for Gaia. Will ni James Zhang to see if we can get confirmation from partner side to see if they try that, if that solves the issue.

Comment 32

[James Zhang (Spreadtrum)]

Loop Ying.
1.Check 1.3t manifest
2.Check build.sh -j1 on our hudson

Comment 33

[ying.xu]

I didn't see this bug with latest build.
but I prefer to keep this bug open

Comment 34

[Joe Cheng [:jcheng] (please needinfo)]

let's make sure this gets fixed for release. 1.3T+

Comment 35

[ying.xu]

we don't meet this bug ever before.
Our customer in India did not too.

Can you reproduce this bug again?
If not , I'm going to close it.

Comment 36

[James Zhang (Spreadtrum)]

Is it the same as bug 1008859 ?

Comment 37

[Peipei Cheng (needinfo if you need my action)]

(In reply to James Zhang (Spreadtrum) from comment #36)
> Is it the same as bug 1008859 ?

No, this is font problem. I think it's already fixed. we should close it.

Comment 38

[Peipei Cheng (needinfo if you need my action)]

Close as worksforme.