985363 - [tarako]customize multilocales

Status: RESOLVED FIXED

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[yang.zhao]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.19 (KHTML, like Gecko) Ubuntu/12.04 Chromium/18.0.1025.151 Chrome/18.0.1025.151 Safari/535.19

Steps to reproduce:

build different languages as the customers need

Comment 1

[yang.zhao]

I want to config the multilocales in a makefile,so I need to konw the following mappings.
Could anyone tell me the following mappings?Or where can I get these documents?
language -> short name
language -> Gaia short name
language -> Gecko short name
language -> keyboard layout
language -> fonts

If I forgot some mappings that is needed ,please let me konw.Thank you!

Comment 2

[James Zhang (Spreadtrum)]

Tomorrow our customer will give the detail language list, then we can configure font/l10n/keybord by PRODUCT_LOCALES.

For example, "PRODUCT_LOCALES := zh_CN en_US" means that we only need copy English and Chinese font/l10n to system image.
Add lianxiang and yang here to reduce ROM size by PRODUCT_LOCALES. We'll modify external/moztt/font.mk and gaia/local.mk first.

Comment 3

[[:fabrice] Fabrice Desré]

James, see also https://developer.mozilla.org/en-US/Firefox_OS/Building#Building_multilocale

Comment 4

[Tzu-Lin Huang [:dwi2][:tzhuang]]

Hi James, Yang,

As Fabrice stated, you can follow https://developer.mozilla.org/en-US/Firefox_OS/Building#Building_multilocale for multilocales customization.
And I don't think we have font customization or "language -> fonts" mapping in Gaia side.
Also, keyboard layouts (for v1.3t) are at https://github.com/mozilla-b2g/gaia/tree/v1.3t/keyboard/layouts

Comment 5

[yang.zhao]

hi,Evelyn
 I'd like to know the mappings between language ，keyboard and fonts.Such as
     language        language name    keyborad layout                       font
   Chinese(中文）      zh-CN,zh-TW   zh-Hant-Zhuyin,zh-Hans-Pinyin     DroidSansFallback.ttf

But the links in above comment doesn't contain the mapping relations,especially the fonts.So could anyone tell me about this?Thank you!

Comment 6

[James Zhang (Spreadtrum)]

(In reply to Fabrice Desré [:fabrice] from comment #3)
> James, see also
> https://developer.mozilla.org/en-US/Firefox_OS/Building#Building_multilocale

We can add all of this language by https://developer.mozilla.org/en-US/Firefox_OS/Building#Building_multilocale.
But we need cut some language by device/product configure "PRODUCT_LOCALES", for memory shrink and customization.

Comment 7

[Steven Yang [:styang]]

Hi Thomas, are we able to control it by build script?

Comment 8

[Evelyn Hung]

(In reply to James Zhang from comment #6)
> (In reply to Fabrice Desré [:fabrice] from comment #3)
> > James, see also
> > https://developer.mozilla.org/en-US/Firefox_OS/Building#Building_multilocale
> 
> We can add all of this language by
> https://developer.mozilla.org/en-US/Firefox_OS/Building#Building_multilocale.
> But we need cut some language by device/product configure "PRODUCT_LOCALES",
> for memory shrink and customization.

Our customize framework doesn't cover fonts, because font files is not included in Gaia, it belongs to system. You have to write a script by yourself or modify your build script to copy corresponding font files to device.

Regarding the mapping of fonts and language, it really depends which font style you want to built-in in your system. There isn't a rule to say which language is mapping to which font style.

@Tim, can you comment more on fonts supported in FxOS?

Comment 9

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Modify B2G/external/moztt/font.mk to set the font files to include.

The upstream repo is at https://github.com/mozilla-b2g/moztt

Comment 10

[yang.zhao]

Attachment: font_zhCN.patch

Assume the  DroidSansFallback.ttf is only related zh_CN,we can add the "if" condition sentence to avoid the DroidSansFallback.ttf to be built into system if there is no zh_CN in PRODUCT_LOCALES.

Comment 11

[James Zhang (Spreadtrum)]

This patch is just a example.

Comment 12

[bhavana bajaj [:bajaj]]

James, looks like the info/documentation folks have provided could be used to solve the issue you raised. Please feel free to NI Steven or any other developers on this bug in case you need help. 

At this time I don't see a actionable bug here to make a blocking case hence clearing the nomination.

Comment 13

[James Zhang (Spreadtrum)]

(In reply to bhavana bajaj [:bajaj] from comment #12)
> James, looks like the info/documentation folks have provided could be used
> to solve the issue you raised. Please feel free to NI Steven or any other
> developers on this bug in case you need help. 
> 
> At this time I don't see a actionable bug here to make a blocking case hence
> clearing the nomination.

Before we reduce ROM size for moving system apps to system image, I think we should keep v1.3?, we still can't reset phone because system apps is in userdata image now, see bug 984623.

Comment 14

[James Zhang (Spreadtrum)]

I think mozilla should provide the detail information of moztt font, then we can reduce ROM size by customer's configure.

Comment 15

[James Zhang (Spreadtrum)]

for example,
in external/moztt/fonts.mk 

DroidSansFallback.ttf -- for zh_CN zh_TW
external/moztt/Lohit/lohit-bengali-ttf-2.5.3/Lohit-Bengali.ttf:system/fonts/Lohit-Bengali.ttf -- for bn-BD

We need this list, thanks.

Comment 16

[Tim Guan-tin Chien [:timdream] (please needinfo)]

I don't have such list. I believe people can work out that list by using a font metadata inspector or even simply Google the font names in question.

Comment 17

[thomas tsai]

Gonk team is working on this.

Comment 18

[thomas tsai]

Attachment: Defining a product ENV variable to de-select the default fonts and a product provides its fonts.

The product can deselect the default fonts set by "MOZ_PRODUCT_HAS_FONTS := true". So the product can define its font sets in board.mk.

Comment 19

[Joe Cheng [:jcheng] (please needinfo)]

triage; this is needed to save some ROM storage sizes. Assign to Thomas since he has been working on this

Comment 20

[yang.zhao]

(In reply to thomas tsai from comment #18)
> Created attachment 8394598 [details] [diff] [review]
> Defining a product ENV variable to de-select the default fonts and a product
> provides its fonts.
> 
> The product can deselect the default fonts set by "MOZ_PRODUCT_HAS_FONTS :=
> true". So the product can define its font sets in board.mk.

We want to classify the fonts by language,then the fonts can be selected by PRODUCT_LOCALES.

Comment 21

[Michael Wu [:mwu]]

Comment on attachment 8394598 [details] [diff] [review]
Defining a product ENV variable to de-select the default fonts and a product provides its fonts.

Disabling all fonts is not an acceptable option. What would be ok is switching to a minimal required set of fonts. There are certain core fonts which we always want to be available, especially since they're hardcoded into our gecko font preferences.

Comment 22

[Michael Wu [:mwu]]

Identifying necessary fonts through build locale settings is also a good option. That shouldn't be the default except when we're low on space, however.

Comment 23

[yang.zhao]

hi,all
 The size of CharisSILCompact font style is 7.2M.Can't we remove it?
 There are several font styles,can we just keep only one type?

Comment 24

[James Zhang (Spreadtrum)]

Attachment: fontmk.patch

Add Yang's new patch.
We can't open all language value by PRODUCT_LOCALES because the system image size is 140MB, over 35MB than tarako system image max size. We can only choose some of them.

PRODUCT_LOCALES := ca hr cs nl en_US de el hu it pl pt_BR ro ru sr_Cyrl sk es tr bn_BD bg fr mk sr_Latn zh_CN zh_TW

Comment 25

[thomas tsai]

Attachment: device/sprd/sp6821a_gonk/prod_sp6821a.mk

Comment 26

[thomas tsai]

If the minimal font set is defined in font.mk and export some VAR like "MOZ_PRODUCT_HAS_FONTS" in board makefile to select the minimal font set. Then the project can customize its own additional fonts in board makefile.
The following is what the minimal font set is?

Comment 27

[thomas tsai]

Considering the font dependency in gecko and based on my user build, system.img takes about 103MB without Chinese fonts: MTLmr3m.ttf  MTLc3m.ttf? Is it acceptable to remove MTLmr3m.ttf  MTLc3m.ttf in 1.3t?

Comment 28

[Michael Wu [:mwu]]

(In reply to yang.zhao from comment #23)
> hi,all
>  The size of CharisSILCompact font style is 7.2M.Can't we remove it?

No. Charis SIL is our serif font.

>  There are several font styles,can we just keep only one type?

Generally no. In some cases we may be able to remove some weights, but I think gaia expects to use all the weights.

Comment 29

[Michael Wu [:mwu]]

(In reply to thomas tsai from comment #27)
> Considering the font dependency in gecko and based on my user build,
> system.img takes about 103MB without Chinese fonts: MTLmr3m.ttf  MTLc3m.ttf?
> Is it acceptable to remove MTLmr3m.ttf  MTLc3m.ttf in 1.3t?

MTLmr3m and MTLc3m are Japanese fonts (motoya). We can make it a locale dependent font.

Comment 30

[Michael Wu [:mwu]]

Comment on attachment 8395210 [details] [diff] [review]
fontmk.patch

Review of attachment 8395210 [details] [diff] [review]:
-----------------------------------------------------------------

This needs to be modified to turn on *only* when we are configured to use a minimal font set.

Adding Jonathan as a reviewer since he knows the coverage of the different fonts better than I do.

::: fonts.mk
@@ +15,5 @@
>  # Warning: this is actually a product definition, to be inherited from
>  
>  # Android fonts
> +
> +ifneq ($(findstring zh_CN, $PRODUCT_LOCALES),) #zh-CN,Chinese

nit: add a space after #. Don't put the locale name in the comment since it's already in the line. Same for all other comments.

@@ +92,2 @@
>      external/moztt/Padauk-2.80/Padauk.ttf:system/fonts/Padauk.ttf \
>      external/moztt/Padauk-2.80/Padauk-bold.ttf:system/fonts/Padauk-bold.ttf \

FYI - Paduak is used for Myanmar support.

Comment 31

[yang.zhao]

(In reply to Michael Wu [:mwu] from comment #30)
> Comment on attachment 8395210 [details] [diff] [review]
> fontmk.patch
> 
> Review of attachment 8395210 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> This needs to be modified to turn on *only* when we are configured to use a
> minimal font set.
> 
> Adding Jonathan as a reviewer since he knows the coverage of the different
> fonts better than I do.
> 
> ::: fonts.mk
> @@ +15,5 @@
> >  # Warning: this is actually a product definition, to be inherited from
> >  
> >  # Android fonts
> > +
> > +ifneq ($(findstring zh_CN, $PRODUCT_LOCALES),) #zh-CN,Chinese
> 
> nit: add a space after #. Don't put the locale name in the comment since
> it's already in the line. Same for all other comments.
> 
> @@ +92,2 @@
> >      external/moztt/Padauk-2.80/Padauk.ttf:system/fonts/Padauk.ttf \
> >      external/moztt/Padauk-2.80/Padauk-bold.ttf:system/fonts/Padauk-bold.ttf \
> 
> FYI - Paduak is used for Myanmar support.
Thank you very much!There are some mistakes in my patch,I'm fixing it now.

Comment 32

[Jonathan Kew [:jfkthame]]

Comment on attachment 8395210 [details] [diff] [review]
fontmk.patch

Review of attachment 8395210 [details] [diff] [review]:
-----------------------------------------------------------------

Some comments below. In general, this needs to be carefully weighed in terms of its impact on product capabilities. The fonts here are not just there to support the specific locale(s) we're targeting for Gaia localizations, product marketing, etc.; they're also needed for rendering of all web content the user might view.

So for example, if the Chinese (DroidSansFallback) and Japanese fonts are omitted from a build, because it's being localized and sold into a non-CJK locale, this means that any user buying that device will see boxes instead of readable content if they visit a Chinese or Japanese web page. That's a pretty severe limitation.

Is the ROM size the biggest concern here? If we could ship with the fonts in a compressed form, so they'd only take up (say) 50% of the current space, would that allow us to keep them all (or at least a larger "core" set of them) in the product?

::: fonts.mk
@@ +17,5 @@
>  # Android fonts
> +
> +ifneq ($(findstring zh_CN, $PRODUCT_LOCALES),) #zh-CN,Chinese
> +PRODUCT_PACKAGES += \
> +    DroidSansFallback.ttf

DroidSansFallback looks like it could be used for Korean as well as Chinese; or do we have a separate font for Korean?

It also covers a lot of miscellaneous stuff (technical symbols, dingbats, etc); offhand, I don't know how many of these are also covered by other fonts. So omitting it might lead to some unexpected gaps in coverage.

@@ +34,5 @@
>      frameworks/base/data/fonts/DroidSansThai.ttf:system/fonts/DroidSansThai.ttf \
> +    $(NULL)
> +endif
> +
> +ifneq ($(findstring ar, $PRODUCT_LOCALES),) #ar,Arabic

The DroidNaskh font would be needed for any locale that uses Arabic *script* (e.g. Persian, Urdu, etc), not only for the Arabic *language*.

@@ +40,5 @@
> +    frameworks/base/data/fonts/DroidNaskh-Regular.ttf:system/fonts/DroidNaskh-Regular.ttf \
> +    $(NULL)
> +endif
> +
> +ifneq ($(findstring bn_BD, $PRODUCT_LOCALES),) #bn-BD,Bengali

Also needed for bn_IN.

@@ +42,5 @@
> +endif
> +
> +ifneq ($(findstring bn_BD, $PRODUCT_LOCALES),) #bn-BD,Bengali
> +PRODUCT_COPY_FILES += \
> +    external/moztt/Lohit/lohit-bengali-ttf-2.5.3/Lohit-Bengali.ttf:system/fonts/Lohit-Bengali.ttf \

This patch must be based on an older branch, as we have replaced Lohit Bengali with Noto Bengali in the current moztt.

(We've also added new fonts, e.g. Khmer and Tibetan, that you might want to make optional.)

@@ +46,5 @@
> +    external/moztt/Lohit/lohit-bengali-ttf-2.5.3/Lohit-Bengali.ttf:system/fonts/Lohit-Bengali.ttf \
> +    $(NULL)
> +endif
> +
> +ifneq ($(findstring hi_IN, $PRODUCT_LOCALES),) #hi-IN,Indian

hi_IN is specifically Hindi, which would require the Devanagari font; is the idea here that it will pull in support for all the Indian languages? Or should they each be handled individually?

It'd probably be more logical to either
(a) handle each of these individually, and then let builders select which of the Indian languages to include, or
(b) do "findstring _IN, ..." here (omitting the language part of the locale) so that *all* the Indian languages will be supported if *any* Indian locale is requested.

In the latter case, Bengali should also be included; it is the language and script of the Indian state of West Bengal (presumably that'd be locale bn_IN) as well as the separate country of Bangladesh (bn_BD).

@@ +64,2 @@
>      frameworks/base/data/fonts/DroidSansArmenian.ttf:system/fonts/DroidSansArmenian.ttf \
>      frameworks/base/data/fonts/DroidSansGeorgian.ttf:system/fonts/DroidSansGeorgian.ttf \

Armenian and Georgian both seem like good candidates to exclude from a minimal-fonts build.

Comment 33

[yang.zhao]

(In reply to Jonathan Kew (:jfkthame) from comment #32)
 hi,Jonathan
    Could you do me a favor,please?Do you have any specs for the mapping relation between language and fonts?Thank you!

Comment 34

[yang.zhao]

hi,James
   Do we have to do this like now?Please see the comment #32.
   They do have risks if we ommit some fonts.

Comment 35

[yang.zhao]

(In reply to Jonathan Kew (:jfkthame) from comment #32)
> Comment on attachment 8395210 [details] [diff] [review]
> fontmk.patch
> 
> Review of attachment 8395210 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Some comments below. In general, this needs to be carefully weighed in terms
> of its impact on product capabilities. The fonts here are not just there to
> support the specific locale(s) we're targeting for Gaia localizations,
> product marketing, etc.; they're also needed for rendering of all web
> content the user might view.
> 
> So for example, if the Chinese (DroidSansFallback) and Japanese fonts are
> omitted from a build, because it's being localized and sold into a non-CJK
> locale, this means that any user buying that device will see boxes instead
> of readable content if they visit a Chinese or Japanese web page. That's a
> pretty severe limitation.
Yes,it's a pretty severe limitation,but this is only for v1.3t,since the ROM size is so limited.
> 
> Is the ROM size the biggest concern here? If we could ship with the fonts in
> a compressed form, so they'd only take up (say) 50% of the current space,
> would that allow us to keep them all (or at least a larger "core" set of
> them) in the product?
Yes,the ROM size is the biggest concern.So could you provide the fonts in a compressed form before the end of March?Because we will do some tests then.
> 
> ::: fonts.mk
> @@ +17,5 @@
> >  # Android fonts
> > +
> > +ifneq ($(findstring zh_CN, $PRODUCT_LOCALES),) #zh-CN,Chinese
> > +PRODUCT_PACKAGES += \
> > +    DroidSansFallback.ttf
> 
> DroidSansFallback looks like it could be used for Korean as well as Chinese;
> or do we have a separate font for Korean?
> 
> It also covers a lot of miscellaneous stuff (technical symbols, dingbats,
> etc); offhand, I don't know how many of these are also covered by other
> fonts. So omitting it might lead to some unexpected gaps in coverage.
> 
> @@ +34,5 @@
> >      frameworks/base/data/fonts/DroidSansThai.ttf:system/fonts/DroidSansThai.ttf \
> > +    $(NULL)
> > +endif
> > +
> > +ifneq ($(findstring ar, $PRODUCT_LOCALES),) #ar,Arabic
> 
> The DroidNaskh font would be needed for any locale that uses Arabic *script*
> (e.g. Persian, Urdu, etc), not only for the Arabic *language*.
> 
Now there are 24 kinds of languages(ca hr cs nl en_US de el hu it pl pt_BR ro ru sr_Cyrl sk es tr bn_BD bg fr mk sr_Latn zh_CN zh_TW), 'ar' is not in the languages,so we just write it like this in v1.3t.
> @@ +40,5 @@
> > +    frameworks/base/data/fonts/DroidNaskh-Regular.ttf:system/fonts/DroidNaskh-Regular.ttf \
> > +    $(NULL)
> > +endif
> > +
> > +ifneq ($(findstring bn_BD, $PRODUCT_LOCALES),) #bn-BD,Bengali
> 
> Also needed for bn_IN.
> 
'bn_IN' is not one of the 24 kinds of languages either.
> @@ +42,5 @@
> > +endif
> > +
> > +ifneq ($(findstring bn_BD, $PRODUCT_LOCALES),) #bn-BD,Bengali
> > +PRODUCT_COPY_FILES += \
> > +    external/moztt/Lohit/lohit-bengali-ttf-2.5.3/Lohit-Bengali.ttf:system/fonts/Lohit-Bengali.ttf \
> 
> This patch must be based on an older branch, as we have replaced Lohit
> Bengali with Noto Bengali in the current moztt.
> 
> (We've also added new fonts, e.g. Khmer and Tibetan, that you might want to
> make optional.)
> 
I haven't seen the modification in v1.3t.
> @@ +46,5 @@
> > +    external/moztt/Lohit/lohit-bengali-ttf-2.5.3/Lohit-Bengali.ttf:system/fonts/Lohit-Bengali.ttf \
> > +    $(NULL)
> > +endif
> > +
> > +ifneq ($(findstring hi_IN, $PRODUCT_LOCALES),) #hi-IN,Indian
> 
> hi_IN is specifically Hindi, which would require the Devanagari font; is the
> idea here that it will pull in support for all the Indian languages? Or
> should they each be handled individually?
> 
"hi-IN" is not in the 24 kinds of language.
> It'd probably be more logical to either
> (a) handle each of these individually, and then let builders select which of
> the Indian languages to include, or
> (b) do "findstring _IN, ..." here (omitting the language part of the locale)
> so that *all* the Indian languages will be supported if *any* Indian locale
> is requested.
> 
> In the latter case, Bengali should also be included; it is the language and
> script of the Indian state of West Bengal (presumably that'd be locale
> bn_IN) as well as the separate country of Bangladesh (bn_BD).
> 
> @@ +64,2 @@
> >      frameworks/base/data/fonts/DroidSansArmenian.ttf:system/fonts/DroidSansArmenian.ttf \
> >      frameworks/base/data/fonts/DroidSansGeorgian.ttf:system/fonts/DroidSansGeorgian.ttf \
> 
> Armenian and Georgian both seem like good candidates to exclude from a
> minimal-fonts build.
The less ROM size the better.

Comment 36

[Jonathan Kew [:jfkthame]]

(In reply to yang.zhao from comment #33)
> (In reply to Jonathan Kew (:jfkthame) from comment #32)
>  hi,Jonathan
>     Could you do me a favor,please?Do you have any specs for the mapping
> relation between language and fonts?Thank you!

That's not a simple question. Fonts are associated with scripts rather than languages - the same characters are used for more than one language.

In some cases, there's clearly one "major" language associated with a font, but in many cases the list of languages (and therefore possible locales) that might require a particular font is quite long. E.g. the same Devanagari font serves for both Hindi and Marathi in India, and for Nepali in Nepal - and for a large number of minority languages in both countries, although it's less likely we'll specifically localize for most of those.

One source for much of this information would be http://scriptsource.org/. Here you can look up individual languages to find out what writing system(s) they use - note that some languages may use more than one - and you can look up scripts to find out what languages use them. The lists are often long!

Because of all this, I think choosing fonts to include based on language or locale codes may not be the best way forward here - it seems simple at first, but as the number of locales increases, it's going to become a huge and hard-to-maintain mass of data.

It would probably be better to classify the fonts we have available according to the *script(s)* that each supports; and then any given product would need to explicitly choose which *scripts* are to be supported or excluded. This will be a much more manageable data set - and it will more accurately reflect the choices that are being made. E.g. when we choose whether or not to include the DroidNaskh font, we're making a choice about Arabic *script* support, not about any one language.

Comment 37

[Jonathan Kew [:jfkthame]]

(In reply to yang.zhao from comment #35)


> > Is the ROM size the biggest concern here? If we could ship with the fonts in
> > a compressed form, so they'd only take up (say) 50% of the current space,
> > would that allow us to keep them all (or at least a larger "core" set of
> > them) in the product?
> Yes,the ROM size is the biggest concern.So could you provide the fonts in a
> compressed form before the end of March?Because we will do some tests then.

This will require a (minor) Gecko patch, I believe. I'll try to get something up for testing this week.

Comment 38

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

We're having issue with the l10n; bug 980681

Comment 39

[Jonathan Kew [:jfkthame]]

Comment on attachment 8395210 [details] [diff] [review]
fontmk.patch

Clearing r? here for now. I've filed bug 987357 and bug 987582 re compressing many of the fonts. If that works well, we might prefer uplifting that work to Tarako rather than omitting many of the fonts altogether.

Comment 40

[thomas tsai]

Tarako not only has limited rom size(256MB) but also limited ram size(128MB) and slow cpu computing.
Need to verify the impact to ram and cpu loading.

Comment 41

[yang.zhao]

(In reply to Jonathan Kew (:jfkthame) from comment #39)
> Comment on attachment 8395210 [details] [diff] [review]
> fontmk.patch
> 
> Clearing r? here for now. I've filed bug 987357 and bug 987582 re
> compressing many of the fonts. If that works well, we might prefer uplifting
> that work to Tarako rather than omitting many of the fonts altogether.

hi,Jonathan
  1,Because of the limited ram size and cpu computing,so the compressed fonts shouldn't impact the performance too much.
  2,Set the minimal required set in fonts.mk and add locale dependent fonts in prod_sp6821a.mk is a good idea,I think.Does anyone work on this?
   Thank you~

Comment 42

[James Zhang (Spreadtrum)]

Agree, fonts controlled by prod_sp6821a.mk is better solution because the schedule is too urgent.

Comment 43

[yang.zhao]

Attachment: fontsmk.patch

fonts.mk

Comment 44

[yang.zhao]

Attachment: localmk.patch

local.mk

Comment 45

[yang.zhao]

Attachment: prod_sp6821a_font_localemk.patch

prod_sp6821.mk

Comment 46

[yang.zhao]

hi,thomas
    Could you review the three patches in last 3 comments?
    I adopted this method:set the minimal required set in fonts.mk and add locale dependent fonts in prod_sp6821a.mk .

Comment 47

[Kai-Zhen Li [:seinlin][:kli]]

Attachment: Tarako_MULTILOCALE.tgz

This is a reference how to build ffos which support bn-BD and en-US.

Comment 48

[Kai-Zhen Li [:seinlin][:kli]]

Attachment: Copy dependent fonts only

Michael, Can you help me review this patch? Thanks!

In this path when MOZ_CHROME_MULTILOCALE is defined, only default fonts and dependent fonts will be copied.

Comment 49

[yang.zhao]

(In reply to Kai-Zhen Li from comment #48)
> Created attachment 8397574 [details] [review]
> Copy dependent fonts only
> 
> Michael, Can you help me review this patch? Thanks!
> 
> In this path when MOZ_CHROME_MULTILOCALE is defined, only default fonts and
> dependent fonts will be copied.

hi,kaizhen
     I'd like to know where the MOZ_CHROME_MULTILOCALE is defined?Thank you!

Comment 50

[yang.zhao]

(In reply to yang.zhao from comment #49)
> (In reply to Kai-Zhen Li from comment #48)
> > Created attachment 8397574 [details] [review]
> > Copy dependent fonts only
> > 
> > Michael, Can you help me review this patch? Thanks!
> > 
> > In this path when MOZ_CHROME_MULTILOCALE is defined, only default fonts and
> > dependent fonts will be copied.
> 
> hi,kaizhen
>      I'd like to know where the MOZ_CHROME_MULTILOCALE is defined?Thank you!

Please ignore the last comment.I've seen it defined in .userconfig .

Comment 51

[yang.zhao]

(In reply to Kai-Zhen Li from comment #47)
> Created attachment 8397572 [details]
> Tarako_MULTILOCALE.tgz
> 
> This is a reference how to build ffos which support bn-BD and en-US.

I'v sended a pull request for gaia:
https://github.com/mozilla-b2g/gaia/pull/17742

Since we couldn't modify the gecko,so could you help merge the remainings?Thank you very much!

Comment 52

[yang.zhao]

(In reply to yang.zhao from comment #51)
> (In reply to Kai-Zhen Li from comment #47)
> > Created attachment 8397572 [details]
> > Tarako_MULTILOCALE.tgz
> > 
> > This is a reference how to build ffos which support bn-BD and en-US.
> 
> I'v sended a pull request for gaia:
> https://github.com/mozilla-b2g/gaia/pull/17742
> 
> Since we couldn't modify the gecko,so could you help merge the
> remainings?Thank you very much!
Sorry!Please ignore this again!

Comment 53

[Kai-Zhen Li [:seinlin][:kli]]

Yang, I'm also thinking how to make multilocales build easier, although current method is not complicated.

Comment 54

[Michael Wu [:mwu]]

Attachment: Use compressed fonts

This is a patch against device/sprd which turns on compressed fonts. Font compression saves us about 11mb on the rom.

Comment 55

[Michael Wu [:mwu]]

Hi James,

Can you merge this patch for device/sprd? Kai-Zhen tested the overhead of compressed fonts and it appears to be minimal. This saves us about 11mb without having to think about what fonts are necessary for what locales. The main UI fonts remain uncompressed.

Comment 56

[James Zhang (Spreadtrum)]

(In reply to Michael Wu [:mwu] from comment #55)
> Hi James,
> 
> Can you merge this patch for device/sprd? Kai-Zhen tested the overhead of
> compressed fonts and it appears to be minimal. This saves us about 11mb
> without having to think about what fonts are necessary for what locales. The
> main UI fonts remain uncompressed.

Is Any risk and performance issue?

Comment 57

[James Zhang (Spreadtrum)]

Comment on attachment 8398787 [details] [diff] [review]
Use compressed fonts

>diff --git a/sp6821a_gonk/prod_sp6821a.mk b/sp6821a_gonk/prod_sp6821a.mk
>index f2b0efd..30884e0 100644
>--- a/sp6821a_gonk/prod_sp6821a.mk
>+++ b/sp6821a_gonk/prod_sp6821a.mk
>@@ -58,6 +58,8 @@ $(call inherit-product, device/sprd/common/res/boot/boot_res.mk)
> $(call inherit-product, $(SRC_TARGET_DIR)/product/full_base.mk)
> $(call inherit-product, $(SRC_TARGET_DIR)/product/telephony.mk)
> 
>+MOZ_PRODUCT_COMPRESS_FONTS := true
>+
> # Overrides
> PRODUCT_NAME := sp6821a_gonk
> PRODUCT_DEVICE := $(TARGET_BOARD)

Please use export MOZ_PRODUCT_COMPRESS_FONTS := true
Like Fabrice add "export MOZILLA_MLS_KEY_FILE=$(ANDROID_BUILD_TOP)/$(BOARDDIR)/mls.key"

Comment 58

[James Zhang (Spreadtrum)]

(In reply to Kai-Zhen Li from comment #53)
> Yang, I'm also thinking how to make multilocales build easier, although
> current method is not complicated.

Yes, we use you method now, but mozilla MUST provide bn-BD official release git repo for us, not github temp url, thanks you.

Like "git clone https://git.mozilla.org/releases/l10n/es-ES/gecko", it's ok.
But "git clone https://git.mozilla.org/releases/l10n/bn-BD/gecko", it's not ready.

Comment 59

[Michael Wu [:mwu]]

(In reply to James Zhang from comment #57)
> Please use export MOZ_PRODUCT_COMPRESS_FONTS := true
> Like Fabrice add "export
> MOZILLA_MLS_KEY_FILE=$(ANDROID_BUILD_TOP)/$(BOARDDIR)/mls.key"

export is used for exporting variables to subprocesses. This variable is only used in the internal Android build system, so we don't need to export it. In the case of MOZILLA_MLS_KEY_FILE, it needs to be exported since gonk-misc/default-gecko-config needs it, and that is imported from a subprocess.

Comment 60

[Michael Wu [:mwu]]

(In reply to James Zhang from comment #56)
> Is Any risk and performance issue?

Kai-Zhen checked the memory usage, which appeared to be ok. The other thing that can be at risk is startup time - Kai-Zhen, can you see if that's ok?

Comment 61

[James Zhang (Spreadtrum)]

(In reply to Michael Wu [:mwu] from comment #59)
> (In reply to James Zhang from comment #57)
> > Please use export MOZ_PRODUCT_COMPRESS_FONTS := true
> > Like Fabrice add "export
> > MOZILLA_MLS_KEY_FILE=$(ANDROID_BUILD_TOP)/$(BOARDDIR)/mls.key"
> 
> export is used for exporting variables to subprocesses. This variable is
> only used in the internal Android build system, so we don't need to export
> it. In the case of MOZILLA_MLS_KEY_FILE, it needs to be exported since
> gonk-misc/default-gecko-config needs it, and that is imported from a
> subprocess.

Which git repo is controlled by 'MOZ_PRODUCT_COMPRESS_FONTS', I can't find this value.

Comment 62

[Michael Wu [:mwu]]

(In reply to James Zhang from comment #61)
> Which git repo is controlled by 'MOZ_PRODUCT_COMPRESS_FONTS', I can't find
> this value.

https://github.com/mozilla-b2g/moztt/blob/v1.3t/fonts.mk#L20

Comment 63

[James Zhang (Spreadtrum)]

(In reply to Michael Wu [:mwu] from comment #62)
> (In reply to James Zhang from comment #61)
> > Which git repo is controlled by 'MOZ_PRODUCT_COMPRESS_FONTS', I can't find
> > this value.
> 
> https://github.com/mozilla-b2g/moztt/blob/v1.3t/fonts.mk#L20

add MOZ_PRODUCT_COMPRESS_FONTS on my side.


commit 70ddde6596402977f14833acd380ee873f37a092
Author: james.zhang <[e-mail]>
Date:   Sat Mar 29 11:57:58 2014 +0800

    Bug#269156 merge patch from mozilla, Bug 985363 - [tarako]customize multilocales
    
    [bug number  ]
    [root cause  ]
    [changes     ]
    [side effects]
    [self test   ] mozilla test
    [reviewers   ]
    
    Change-Id: Idadfef7fc2ce10c5feadb7e6fc9ca0899ac8d865

Comment 64

[Kai-Zhen Li [:seinlin][:kli]]

(In reply to Michael Wu [:mwu] from comment #62)
> (In reply to James Zhang from comment #61)
> > Which git repo is controlled by 'MOZ_PRODUCT_COMPRESS_FONTS', I can't find
> > this value.
> 
> https://github.com/mozilla-b2g/moztt/blob/v1.3t/fonts.mk#L20

To use woff fonts in v1.3t, I think we also need to pick this commit.

Bug 985806 https://hg.mozilla.org/mozilla-central/rev/2378f03042a2

Comment 65

[Kai-Zhen Li [:seinlin][:kli]]

(In reply to Michael Wu [:mwu] from comment #60)
> (In reply to James Zhang from comment #56)
> > Is Any risk and performance issue?
> 
> Kai-Zhen checked the memory usage, which appeared to be ok. The other thing
> that can be at risk is startup time - Kai-Zhen, can you see if that's ok?

I think it's ok and we will also need to pick the the patch from Bug 985806 to v1.3t.

Comment 66

[Kai-Zhen Li [:seinlin][:kli]]

My test result about ttf vs woff on tarako: http://goo.gl/XYZMbV

Comment 67

[Michael Wu [:mwu]]

(In reply to Kai-Zhen Li from comment #64)
> To use woff fonts in v1.3t, I think we also need to pick this commit.
> 
> Bug 985806 https://hg.mozilla.org/mozilla-central/rev/2378f03042a2

That's for the freetype 2.5.3 update since they changed the way their configure script works.

Comment 68

[Kai-Zhen Li [:seinlin][:kli]]

(In reply to Michael Wu [:mwu] from comment #67)
> 
> That's for the freetype 2.5.3 update since they changed the way their
> configure script works.

Michael, Thanks! In my local, I backport to freetype 2.5.3, that's why I need it. Actually, freetype 2.5.2 is enough for woff.

Comment 69

[Kai-Zhen Li [:seinlin][:kli]]

Per comment 67 and comment 68.

Comment 70

[Kai-Zhen Li [:seinlin][:kli]]

Comment on attachment 8397574 [details] [review]
Copy dependent fonts only

I think we doesn't require it any more.

Comment 71

[Kai-Zhen Li [:seinlin][:kli]]

Attachment: Add l10n for tarako

Hi Michael, Can you help me to review this patch for v1.3t branch? I think it will be better to have l10n in our manifest, and this can make our daily build much simple.

Comment 72

[Michael Wu [:mwu]]

(In reply to Kai-Zhen Li from comment #71)
> Created attachment 8399748 [details] [review]
> Add l10n for tarako
> 
> Hi Michael, Can you help me to review this patch for v1.3t branch? I think
> it will be better to have l10n in our manifest, and this can make our daily
> build much simple.

Is this to fix the build bustage? The tarako build bustage is in gecko l10n, not gaia l10n. This is triggered by MOZ_CHROME_MULTILOCALE, but that was backed out in the latest update to device/sprd. I'm not sure if we actually need gecko l10n anymore.

Comment 73

[Kai-Zhen Li [:seinlin][:kli]]

(In reply to Michael Wu [:mwu] from comment #72)
> 
> Is this to fix the build bustage? The tarako build bustage is in gecko l10n,
> not gaia l10n. This is triggered by MOZ_CHROME_MULTILOCALE, but that was
> backed out in the latest update to device/sprd. I'm not sure if we actually
> need gecko l10n anymore.

We do not require gecko l10n anymore, partner will update device config to fix gecko l10n build break; We only require gaia l10n to get the correct build according to device config.

Comment 74

[cmtalbert]

Folks, this is blocking us from getting the ROM size of our PVT tarako builds into a flashable state. I've read the entire back and forth and must confess to still being a little lost. Let me see if I understand:

We need to land the compressed font changes to 1.3t which will save us space. 
Then we need to land the gaia l10n patch that Kai-Zhen Li has made so that we have l10n in gaia once more.

After that, we will have both a l10n-enabled build that can also be flashed? 

What is the ETA to getting this resolved? I have people ready to start testing these builds but they can't do much if they cannot flash them.

Comment 75

[Michael Wu [:mwu]]

(In reply to Kai-Zhen Li from comment #73)
> We do not require gecko l10n anymore, partner will update device config to
> fix gecko l10n build break; We only require gaia l10n to get the correct
> build according to device config.

Why are the gaia l10n repos necessary for this device when we haven't included them in any other device manifest?

Comment 76

[Michael Wu [:mwu]]

(In reply to Clint Talbert ( :ctalbert ) from comment #74)
> We need to land the compressed font changes to 1.3t which will save us
> space. 

That landed last week.

Comment 77

[James Zhang (Spreadtrum)]

(In reply to Michael Wu [:mwu] from comment #75)
> (In reply to Kai-Zhen Li from comment #73)
> > We do not require gecko l10n anymore, partner will update device config to
> > fix gecko l10n build break; We only require gaia l10n to get the correct
> > build according to device config.
> 
> Why are the gaia l10n repos necessary for this device when we haven't
> included them in any other device manifest?

1. We need gaia l10n repo to build English/India language for field test and multilocales test.
2. I think mozilla build should be same as spreadtrum build, then we can fix multilocales bug quickly.
3. We need English/Hindi/Tamil/Bengali for India open market, there are Hind and Tamil translation and keybord development task on spreadtrum and mozilla side, and I think we will have good work efficiency if we have the same build.

Comment 78

[Michael Wu [:mwu]]

Comment on attachment 8399748 [details] [review]
Add l10n for tarako

Our build infrastructure already pulls these repos into the same paths, so I don't think it's a good idea to simply replicate this in the manifests. It'll probably burn the tree. I don't mind putting the locales into the manifests, but you'll need to coordinate with releng if you want to move these repos into the manifest.

Comment 79

[Kai-Zhen Li [:seinlin][:kli]]

Chris, Partner'd like to maintain locales things into manifests. But it could not be a good idea for our build infrastructure. Do you have some suggestion for this? Thanks!

Comment 80

[James Zhang (Spreadtrum)]

Wait Hindi and Tamil translation and input method.

Comment 81

[Aki Sasaki (not active)]

(In reply to Kai-Zhen Li from comment #79)
> Chris, Partner'd like to maintain locales things into manifests. But it
> could not be a good idea for our build infrastructure. Do you have some
> suggestion for this? Thanks!

Aren't manifests able to source other manifests?
All non-locale repos could go in the current manifest.  A new manifest could source the current manifest and include locale repos, and the partner could use that one.  Would that work?

Comment 82

[Kai-Zhen Li [:seinlin][:kli]]

(In reply to Aki Sasaki [:aki] from comment #81)
> 
> Aren't manifests able to source other manifests?
> All non-locale repos could go in the current manifest.  A new manifest could
> source the current manifest and include locale repos, and the partner could
> use that one.  Would that work?

Aki, Thanks! This reasonable to me, as manifests can include other manifests file.
But we'll need to maintain two combinations of manifest for 1 device. for example:
./config.sh tarako       - tarako base only
./config.sh tarako-l10n  - tarako base + l10n

Michael, do you think if it is reasonable to maintain two combinations of manifest for 1 device?

James, how do you think too?

Comment 83

[James Zhang (Spreadtrum)]

(In reply to Kai-Zhen Li from comment #82)
> (In reply to Aki Sasaki [:aki] from comment #81)
> > 
> > Aren't manifests able to source other manifests?
> > All non-locale repos could go in the current manifest.  A new manifest could
> > source the current manifest and include locale repos, and the partner could
> > use that one.  Would that work?
> 
> Aki, Thanks! This reasonable to me, as manifests can include other manifests
> file.
> But we'll need to maintain two combinations of manifest for 1 device. for
> example:
> ./config.sh tarako       - tarako base only
> ./config.sh tarako-l10n  - tarako base + l10n
> 
> Michael, do you think if it is reasonable to maintain two combinations of
> manifest for 1 device?
> 
> James, how do you think too?

OK, I will take tarako-l10n.

Comment 84

[Joe Cheng [:jcheng] (please needinfo)]

Hi James, can this bug now be closed? Thanks

Comment 85

[James Zhang (Spreadtrum)]

(In reply to Joe Cheng [:jcheng] from comment #84)
> Hi James, can this bug now be closed? Thanks

OK.
But I need not compress font and Chinese font on my side.

Comment 86

[Joe Cheng [:jcheng] (please needinfo)]

Hi James, do you mean you only take Chinese fonts out from your builds? and others fonts are not compressed? thanks

Comment 87

[James Zhang (Spreadtrum)]

Yes, we have v1.3t permission, we will remove chinse font.

Comment 88

[James Zhang (Spreadtrum)]

Yes, we have v1.3t permission, we will remove chinese font.

Comment 89

[Kai-Zhen Li [:seinlin][:kli]]

James, would you plan to use the patch in bug 992150 which allows to install dependent fonts only?

Comment 90

[James Zhang (Spreadtrum)]

We land this WIP patch.

commit 62b610caf3ef7bb3f0fee5c379c4e1b0beb6f306
Author: yang.zhao <[e-mail]>
Date:   Tue May 6 17:12:36 2014 +0800

    merge moztt.patch from sprd_patch/gonk4.0/moztt.patch

Comment 91

[Kai-Zhen Li [:seinlin][:kli]]

Per comment 90, I think no more info is needed.