1215247 - Enable ECMAScript Internationalization API for Firefox on Android

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[[:fabrice] Fabrice Desré]

Bug 864843 only enables icu on b2gdroid.

Comment 1

[:Margaret Leibovic]

We've held off on shipping this API on Android because we're worried about the APK size increase it will bring. We'll need to address that concern before we can enable this by default.

Comment 2

[Jeff Walden [:Waldo]]

Copying over the almost-six-months-old bug 864843 comment 85, nearly unaltered, for easier reference:

"""
Enabling ICU enables the Intl API in JavaScript, exposed to webpages.  This lets people create efficiently reusable date/time and number formatters and text collator instances with very particular behaviors.  Old-school JS's support for any of this is practically non-existent, to the point where sites use *roundtrips to servers* to get what they want.  (I need not point out that this *particularly* hurts on mobile.)  More on that in https://hacks.mozilla.org/2014/12/introducing-the-javascript-internationalization-api/ that I wrote last year.  So that's a gain.

And along with that gain.  The best argument for taking this is going to be rather blunt.

In very short order every other browser engine is going to ship this.  All desktop does, now.  If mobile doesn't now, it will extraordinarily soon.  Where the browser ships with the OS, they don't care about size.  Where the browser's a separate download (Chrome, I think), they've eaten the cost and ship anyway.  We will be odd man out extraordinarily soon (if we aren't already).  Sites will begin using this, and sites will depend on it, almost certainly no matter what Firefox for <mobile platform> does.  The download size hit is IMO surpassed by the reality that people are going to expect this, and it simply won't be acceptable much longer to not ship this.

And yes, ideally we'd not add 3MB to the download size.  But I don't think we have that choice any more.
"""

Bug 1217790 has since arisen as an instance of people expecting mobile to work like desktop.

I'm not opposed to trying to slim down our embedded ICU.  But we've already removed the unused bits of ICU.  It seems very, very unwise to decide some locales are more equal than others and selectively remove data for the rest.  And while ICU does let us remove some existing Gecko code, my sense is that code removal will never come close to 3MB.

In the end, I expect our hand will be forced by bugs like bug 1217790, and we'll end up having to eat most of that 3MB increase.  But I'm perfectly willing to let time make that argument for me, without having to engage in a long, drawn-out, knockdown argument to get there faster.

Comment 3

[Makoto Kato [:m_kato]]

APK limit size was 50MB, but now, Google changes the limit to 100MB (http://android-developers.blogspot.ca/2015/09/support-for-100mb-apks-on-google-play.html).

Also, Chrome Android already supports Intl API and they reduce ICU data size by http://bugs.icu-project.org/trac/ticket/11537.

Comment 4

[Jeff Walden [:Waldo]]

Ah, right, that change.  Any day now I'm hoping to update our ICU to stay current, and at the same time to take advantage of that -- bug 1210165

Even still, we're probably still talking a honking lot of data even if it'll reduce that 3MB number.  The bug suggests a 7.2% size reduction in ICU data size.   Given it's reducing data that probably compresses reasonably (and we ship it compressed), and given that reduction may apply to CLDR data we've already removed, I'd guess we're talking a smaller-than-7.2% reduction on that 3MB, from our point of view.

Comment 5

[:Margaret Leibovic]

(In reply to Makoto Kato [:m_kato] from comment #3)
> APK limit size was 50MB, but now, Google changes the limit to 100MB
> (http://android-developers.blogspot.ca/2015/09/support-for-100mb-apks-on-
> google-play.html).

The limit that Google proposes is not the problem. People do not want to download large APKs, and they do not want them taking up space on their devices, so we have a (large and difficult) ongoing effort to reduce our APK size (see bug 942609).

3MB is a non-trivial increase, and I'm not saying we'll never be able to do that, but we need a strong reason to take that. It might be better to have a broader discussion on mobile-firefox-dev.

I wonder if this could be something that is downloaded on demand, in a similar vein to bug 1194338.

Comment 6

[Jeff Walden [:Waldo]]

(In reply to :Margaret Leibovic from comment #5)
> I wonder if this could be something that is downloaded on demand, in a
> similar vein to bug 1194338.

Downloadable fonts are naturally asynchronous.  Excepting FOUC-like concerns, there's no problem extending the asynchrony -- an already-async operation just happens later.

The Intl API, on the other hand, is fully synchronous.  There's no good way to delay the downloading/loading of Intl data until it's first used, not without freezing the entire app to download those megabytes of data.  (And if bug 1211255 is anything to go by, UI's likely going to use it -- or toLocaleString that delegates to the Intl API -- soon enough anyway.)

More generally, we've covered the gamut of {im,}possible tactics on m.d.platform in the past -- downloading Intl support lazily, removing unused ICU functionality, removing excess CLDR data, removing "unused" locales (noting that bundling the user's locale plus the most popular ones probably picks up size-heavy Chinese), using the platform's possibly-ancient ICU or (more generally) having an entirely different backend for one platform or another (that doesn't require a size increase), and so on.  I don't have thread links handy.

But as I said, I'm not looking to really argue, only do a context-dump for future reference and re-plant the mind worm, so I'll put off digging up a link til later.  :-)

Comment 7

[Henri Sivonen (:hsivonen)]

To pile on recent reasons why not having ICU on Android is sad regardless of the JS i18n API:

 * We have an ICU backend for nsICollation for Mac. We could use that code for all platforms and get rid of platform-specific nsCollations, expect we can't, because Android.

 * We have an IDN rewrite that uses ICU. Except we can't do that on Android.

This 3 megabyte saving on Android keep accumulating engineering cost due to us not being able to write code with the assumption that Gecko has ICU available everywhere Gecko runs.

(And no, we can't do a lazy download of ICU itself for these use cases.)

Comment 8

[Makoto Kato [:m_kato]]

When updating to 55.1, ICU file size is more growth.

default (w/o icu) -> 38034081 (libxul.so ... 18943190)
w/55.1 (current)  -> 42133177 (libxul.so ... 23042306)
w/56.1            -> 42071425 (libxul.so ... 22980553)

Comment 9

[Makoto Kato [:m_kato]]

(In reply to Henri Sivonen (:hsivonen) from comment #7)
> To pile on recent reasons why not having ICU on Android is sad regardless of
> the JS i18n API:
> 
>  * We have an ICU backend for nsICollation for Mac. We could use that code
> for all platforms and get rid of platform-specific nsCollations, expect we
> can't, because Android.
> 
>  * We have an IDN rewrite that uses ICU. Except we can't do that on Android.
> 
> This 3 megabyte saving on Android keep accumulating engineering cost due to
> us not being able to write code with the assumption that Gecko has ICU
> available everywhere Gecko runs.
> 
> (And no, we can't do a lazy download of ICU itself for these use cases.)

And,

- IndexedDB already uses ICU for bug 871846.  So Firefox android won't return same result by compare.
- <input type="number"> uses ICU by bug 844744.

Comment 10

[Masatoshi Kimura [:emk]]

ICU size is unlikely to be dramatically reduced. Also it is impossible to download it dynamically. If it cannot be added due to 3MB bloat, it will never be added forever. Why don't you close this bug as WONTFIX?

Comment 11

[Makoto Kato [:m_kato]]

(In reply to Masatoshi Kimura [:emk] from comment #10)
> ICU size is unlikely to be dramatically reduced. Also it is impossible to
> download it dynamically. If it cannot be added due to 3MB bloat, it will
> never be added forever. Why don't you close this bug as WONTFIX?

Since we will reduce more size, I am investigating this.  Because chrome for android removes a lot of data that is unused for browser.

Comment 12

[Richard Newman [:rnewman]]

(In reply to Masatoshi Kimura [:emk] from comment #10)
> ICU size is unlikely to be dramatically reduced. Also it is impossible to
> download it dynamically. If it cannot be added due to 3MB bloat, it will
> never be added forever. Why don't you close this bug as WONTFIX?

My hope -- just like Jeff's -- is that effort and time will force our hand; I don't _want_ to WONTFIX it.

Some combination of:

* Increasing (or more compelling) user benefit
* Increasing developer cost
* Decreasing the proposed size impact directly
* Identifying a progressively larger pile of related code and data that could be removed in 'exchange', thus lowering the 3MB impact
* Removing other unrelated features, leading to size opportunity
* Decreased market pressure on APK size

would move this towards landing.

It's not that we won't take _this_ 3MB _in particular_, just that bumping up to 41MB from 38MB (looking at 43.0b4) isn't something we feel is reasonable right now.

If we manage to ship downloadable fonts (Bug 1194338) and a few other smaller space reductions (Bug 1095719?), those alone might be enough to take this.

If someone spends the time to figure out Bug 945123, that would also buy us enough space (5MB!) that we'd take this outright.

This boils down to: keep an eye on Perfherder.

If we can trend that APK size down for a while, and figure out a good way to stop it creeping back up, then we can take this. But it'll be a very hard sell if we're looking at a 45-50MB APK in 2016.

The more effort y'all expend on removing code and resources from Gecko in general, the more likely it is that we can mark this as FIXED.

Comment 13

[Makoto Kato [:m_kato]]

(In reply to Richard Newman [:rnewman] from comment #12)
> If we manage to ship downloadable fonts (Bug 1194338) and a few other
> smaller space reductions (Bug 1095719?), those alone might be enough to take
> this.

Actually, gfx and harfbuzz code to render text already use ICU if ENABLE_INTL_API=1. So even if ICU data becomes downloadable, it will download it when rendering text.  It means that all users need download ICU data file to render text.

And locale support of bionic is buggy, so collation and date time format are still broken on Android backend.  (ex. bug 864753).

Comment 14

[Richard Newman [:rnewman]]

(In reply to Makoto Kato [:m_kato] from comment #13)
> (In reply to Richard Newman [:rnewman] from comment #12)
> > If we manage to ship downloadable fonts (Bug 1194338)…
> 
> … So even if ICU data becomes downloadable…

To be clear, the features I'm talking about are nothing to do with ICU; I mention them only to emphasize that we care about _total_ size.

If we can get big wins by removing other chunks of code and data from Fennec -- for example, the TrueType fonts that we ship, or the Gecko localized strings that we bake into omni.ja -- then that frees up space that we can fill with ICU.

(That doesn't mean you shouldn't be working on making the ICU code and data that we ship as small as possible, and ripping out obsolete compat code once it does ship, of course. Every little helps.)

Comment 15

[Simon Montagu :smontagu]

m_kato, do you have up-to-date patches to enable ICU on Android that you can attach here? After the repurposing of bug 864843 I'm not sure what the status is here in terms of code changes.

Comment 16

[Zibi Braniecki [:zbraniecki][:gandalf]]

Would it be possible for us to enable Intl API and ship it with just one locale for now to unbreak apps using Intl API? That's what node.js does by default now and it's certainly better than what Firefox for Android does.

Comment 17

[Jeff Walden [:Waldo]]

It doesn't seem very useful to ship people English strings when they read Chinese or Arabic only -- in fact it's *anti*-useful.  And it would make existing feature-detection of Intl mostly meaningless, by giving a far inferior version where people will reasonably expect Intl implies support for a decent number of locales.

Comment 18

[Makoto Kato [:m_kato]]

Attachment: Turn on ICU and Intl API for Android

Comment 19

[Henri Sivonen (:hsivonen)]

(In reply to Richard Newman [:rnewman] from comment #12)
> * Increasing developer cost

What kind of developer cost would need to be shown for the developer cost to allow this to be fixed?

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #16)
> Would it be possible for us to enable Intl API and ship it with just one
> locale for now to unbreak apps using Intl API?

I agree with Waldo that this would be anti-useful.

Comment 20

[Morgan Phillips [:mrrrgn]]

Because of the way we've been handling identifier validation we're facing a significant amount of work in order to become ECMA6 compliant (see bug 917436). Enabling the Internationalization API on FF Android would decrease our workload on that front, and decrease the amount of time we spend maintaining our own lookup table of valid unicode identifiers (see bug 1230490).

Comment 21

[Richard Newman [:rnewman]]

(In reply to Henri Sivonen (:hsivonen) from comment #19)
> (In reply to Richard Newman [:rnewman] from comment #12)
> > * Increasing developer cost
> 
> What kind of developer cost would need to be shown for the developer cost to
> allow this to be fixed?

Developer cost _alone_? Pretty big, sadly.

We can afford some amount of increased developer time, and shitty workarounds, even if we don't like it. We cannot afford -- quite literally, from a CPI perspective, but also in terms of adoption trends -- the impact on our retention numbers that increasing APK size will cause.

(There's some evidence that installer size is a problem for desktop, too, but those conversations have been slightly less urgent.)

But there's a more positive way to look at this. The obvious question is where to spend the developer time -- reimplementation of stuff in ICU, or in making Gecko smaller. I'm sure we'd all prefer the latter.

If you could say "we have nine months of work to do, but landing this would give it to us for free", then that would likely sway us -- but in exchange I'd ask for 2 or 3 months to be spent _right now_ to offset the space impact. That'd make everyone happy.

mfinkle and others have put in some time lately to get perfherder moving in the right direction:

https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=[fx-team,4eb0cde5431ee9aeb5eb14512ddb3da6d4702cf0,1]&series=[mozilla-inbound,4eb0cde5431ee9aeb5eb14512ddb3da6d4702cf0,1]

but we're still hovering around 42MB on fx-team, which is a Bad Thing. What can you take out to make ICU fit?

Comment 22

[Ameen Ross]

Looking at treeherder, I noticed that it recently dropped to around 39MB. That's 3MB lower than it used to be, equal to the size increase caused by ICU. Any chance this can be reconsidered?

Comment 23

[:Margaret Leibovic]

(In reply to Ameen Ross from comment #22)
> Looking at treeherder, I noticed that it recently dropped to around 39MB.
> That's 3MB lower than it used to be, equal to the size increase caused by
> ICU. Any chance this can be reconsidered?

I'll add this item for discussion at our Fennec roadmap funnel meeting on Monday.

We don't have an explicit APK size target goal, it's basically always about a cost/benefit analysis for individual features. I understand this bug is blocking other work, so I want to help find a way forward for landing this.

Comment 24

[Barbara Bermes [:barbara] - NI please!]

added it to funnel items

Comment 25

[:Margaret Leibovic]

(In reply to Barbara Bermes [:barbara] from comment #24)
> added it to funnel items

To clarify for people following along, we discussed this bug at our team's funnel meeting a few weeks ago, and decided that we're still blocked on the APK size increase.

I recognize that there is an increasing list of dependencies here, but we'll need someone to really champion this to explain why this current set of bugs would be worth a 3MB APK size increase.

Comment 26

[robert.rcampbell]

6.77 million people in Laos can not use the Firefox browser for their native language due to words not being tokenized. Lacking this support negatively impacts line breaking which impacts the rendering of non-ZWSP delimited text on websites. Side scrolling and broken responsive pages end up being the result (or lots of hacks and/or developer work to make matters right). 

ZWSP text only automatically happens on MS Office using a third-party extension, and basically nothing else. Otherwise, Lao people have to enter those invisible spaces - and they don't, because they don't think that way, and never will. 

Text entry can be a problem too for input boxes and etc. 

Firefox does not include Burmese support either. Nor does it include the additional benefits to Khmer and Thai, using ICU's code as opposed to the systems it currently uses. At least it does mean that Firefox has  some support Thai, and Khmer (I think?). We are talking about up to approximately 150 million people.

That's quite a few people who could be affected by this (though I am sure not all of them have a smartphone yet, just lots and lots of them - and more every day).

It goes without saying, most of these people use Chrome, Opera, or Safari instead. I think Microsoft's latest browser even may support it. Chrome / Safari have a newer version of ICU. Of course that means lots of bloat for those who speak English - but for those here it's essential. 3MB to people here doesn't mean much if they can't use it. Long gone are the days where you have to know English to use the net. If you want the emerging markets - Firefox has to support their languages.

As a web developer creating sites for these Southeast Asian languages, support is non-optional - it simply has to be there.

If you want a non-ICU fix to this issue, I am more than willing to help provide my knowledge on the language and the data I have worked on to help ICU have that support. I am not a seasoned programmer, but I am sure with some help, we could whip up some solution. But, that's a lot of work to add word detection - plus a boat load of data (since the most effective method of word tokenization is dictionary based). ICU supports this out of the box. The only alternative is a pattern based solution - and god luck with that one! Lao has so many exceptions to standard spelling conventions (unlike Thai) that a non-dictionary based word boundary detection system is quite difficult to do.

Just my two ₭ on the matter.

I'd really like to see Firefox shine in these emerging markets - so, please do consider adding ICU support.

Comment 27

[Morgan Phillips [:mrrrgn]]

(In reply to robert.rcampbell from comment #26)
> 6.77 million people in Laos can not use the Firefox browser for their native
> language due to words not being tokenized. Lacking this support negatively
> impacts line breaking which impacts the rendering of non-ZWSP delimited text
> on websites. Side scrolling and broken responsive pages end up being the
> result (or lots of hacks and/or developer work to make matters right). 
> 
> ZWSP text only automatically happens on MS Office using a third-party
> extension, and basically nothing else. Otherwise, Lao people have to enter
> those invisible spaces - and they don't, because they don't think that way,
> and never will. 
> 
> Text entry can be a problem too for input boxes and etc. 
> 
> Firefox does not include Burmese support either. Nor does it include the
> additional benefits to Khmer and Thai, using ICU's code as opposed to the
> systems it currently uses. At least it does mean that Firefox has  some
> support Thai, and Khmer (I think?). We are talking about up to approximately
> 150 million people.
> 
> That's quite a few people who could be affected by this (though I am sure
> not all of them have a smartphone yet, just lots and lots of them - and more
> every day).
> 
> It goes without saying, most of these people use Chrome, Opera, or Safari
> instead. I think Microsoft's latest browser even may support it. Chrome /
> Safari have a newer version of ICU. Of course that means lots of bloat for
> those who speak English - but for those here it's essential. 3MB to people
> here doesn't mean much if they can't use it. Long gone are the days where
> you have to know English to use the net. If you want the emerging markets -
> Firefox has to support their languages.
> 
> As a web developer creating sites for these Southeast Asian languages,
> support is non-optional - it simply has to be there.
> 
> If you want a non-ICU fix to this issue, I am more than willing to help
> provide my knowledge on the language and the data I have worked on to help
> ICU have that support. I am not a seasoned programmer, but I am sure with
> some help, we could whip up some solution. But, that's a lot of work to add
> word detection - plus a boat load of data (since the most effective method
> of word tokenization is dictionary based). ICU supports this out of the box.
> The only alternative is a pattern based solution - and god luck with that
> one! Lao has so many exceptions to standard spelling conventions (unlike
> Thai) that a non-dictionary based word boundary detection system is quite
> difficult to do.
> 
> Just my two ₭ on the matter.
> 
> I'd really like to see Firefox shine in these emerging markets - so, please
> do consider adding ICU support.

I wish we could upvote comments on bugzilla. +100 Bandwidth and storage will bcome less of a barrier each year as hardware gets cheaper. People's native languages aren't changing anytime soon.

Comment 28

[Morgan Phillips [:mrrrgn]]

(In reply to robert.rcampbell from comment #26)
> 6.77 million people in Laos can not use the Firefox browser for their native
> language due to words not being tokenized. Lacking this support negatively
> impacts line breaking which impacts the rendering of non-ZWSP delimited text
> on websites. Side scrolling and broken responsive pages end up being the
> result (or lots of hacks and/or developer work to make matters right). 
> 
> ZWSP text only automatically happens on MS Office using a third-party
> extension, and basically nothing else. Otherwise, Lao people have to enter
> those invisible spaces - and they don't, because they don't think that way,
> and never will. 
> 
> Text entry can be a problem too for input boxes and etc. 
> 
> Firefox does not include Burmese support either. Nor does it include the
> additional benefits to Khmer and Thai, using ICU's code as opposed to the
> systems it currently uses. At least it does mean that Firefox has  some
> support Thai, and Khmer (I think?). We are talking about up to approximately
> 150 million people.
> 
> That's quite a few people who could be affected by this (though I am sure
> not all of them have a smartphone yet, just lots and lots of them - and more
> every day).
> 
> It goes without saying, most of these people use Chrome, Opera, or Safari
> instead. I think Microsoft's latest browser even may support it. Chrome /
> Safari have a newer version of ICU. Of course that means lots of bloat for
> those who speak English - but for those here it's essential. 3MB to people
> here doesn't mean much if they can't use it. Long gone are the days where
> you have to know English to use the net. If you want the emerging markets -
> Firefox has to support their languages.
> 
> As a web developer creating sites for these Southeast Asian languages,
> support is non-optional - it simply has to be there.
> 
> If you want a non-ICU fix to this issue, I am more than willing to help
> provide my knowledge on the language and the data I have worked on to help
> ICU have that support. I am not a seasoned programmer, but I am sure with
> some help, we could whip up some solution. But, that's a lot of work to add
> word detection - plus a boat load of data (since the most effective method
> of word tokenization is dictionary based). ICU supports this out of the box.
> The only alternative is a pattern based solution - and god luck with that
> one! Lao has so many exceptions to standard spelling conventions (unlike
> Thai) that a non-dictionary based word boundary detection system is quite
> difficult to do.
> 
> Just my two ₭ on the matter.
> 
> I'd really like to see Firefox shine in these emerging markets - so, please
> do consider adding ICU support.

I wish we could upvote comments on bugzilla. +100 Bandwidth and storage will become less of a barrier each year as hardware gets cheaper. People's native languages aren't changing anytime soon.

Comment 29

[:Margaret Leibovic]

Thank you for the detailed comment, this does make a compelling case for shipping this feature.

I agree I want Firefox to be a global product, but unfortunately, in these emerging markets, Firefox has such a huge APK that people don't download it. As much as bandwidth and storage should theoretically become less of a barrier, they're still a huge barrier right now.

NI to Barbara, our product manager, to consider this more. I think she has the most context to figure out how to make the right tradeoffs here.

Comment 30

[Brad Lassey [:blassey] (use needinfo?)]

Shipping this feature doesn't magically fix web content being localized. Do we have telemetry for this APIs usage/uptake in the wild?

Comment 31

[Jeff Walden [:Waldo]]

(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #30)
> Shipping this feature doesn't magically fix web content being localized. Do
> we have telemetry for this APIs usage/uptake in the wild?

I'll register partial dissent.

The obvious APIs -- <global>.Intl.{Collator,DateTimeFormat,NumberFormat} -- are probably feature-test-used in various libraries now.  But *any* code that uses Number.prototype.toLocaleString and Date.prototype.toLocale{,Date,Time}String is improved by addition of the Intl API, because part of the Intl API specifies precise behavior for the old locale-sensitive functions that have existed since time immemorial, in terms of Intl's behavior.  This has two effects.

The major effect is that you can perform locale-sensitive, customized operations for other locales *at all*.  SpiderMonkey's non-Intl-supporting builds entirely ignore the arguments passed to functions like Date.prototype.toLocaleString (instead using the user's system locale, however, say, Fennec might specify it, and default formatting options).  Thus the two function calls in this example would return the exact same string.

js> var d = new Date(2016, 4, 1)
js> d.toLocaleString()
"5/1/2016, 12:00:00 AM"
js> d.toLocaleString("de-DE")
"1.5.2016, 00:00:00"

And if you wanted to do one-off customized formatting, you can, again, do it at all (without the Intl API's extra overhead of an explicit object to cache behavior, for purely one-off cases):

js> d.toLocaleString("de-DE", { month: "long" })
"Mai"

The less-major effect is that even operations that only need to be performed with respect to the user's locale, have improved data.  I don't have examples here, because it'll depend on exactly what the user's libc does when we ask it to do a formatting operation.  But almost certainly ICU's going to do a better job of this than, say, bionic is.

So, yes -- shipping Intl *does* magically fix some web content.  Given the existing locale-sensitive APIs didn't offer much control before, tho, sites that actually cared about this stuff probably never used them.  So the major impact is on a long tail of smaller sites that used locale-sensitive things to be nice, without demanding or expecting high-quality behavior from them.

Comment 32

[Nathan Froyd [:froydnj]]

(In reply to Richard Newman [:rnewman] from comment #21)
> mfinkle and others have put in some time lately to get perfherder moving in
> the right direction:
> 
> https://treeherder.mozilla.org/perf.html#/
> graphs?timerange=2592000&series=[fx-team,
> 4eb0cde5431ee9aeb5eb14512ddb3da6d4702cf0,1]&series=[mozilla-inbound,
> 4eb0cde5431ee9aeb5eb14512ddb3da6d4702cf0,1]
> 
> but we're still hovering around 42MB on fx-team, which is a Bad Thing. What
> can you take out to make ICU fit?

Looks like we've cut out about 3MB in the last three months, particularly sometime in February:

https://treeherder.mozilla.org/perf.html#/graphs?timerange=7776000&series=%5Bfx-team,4eb0cde5431ee9aeb5eb14512ddb3da6d4702cf0,1%5D&series=%5Bmozilla-inbound,4eb0cde5431ee9aeb5eb14512ddb3da6d4702cf0,1%5D

thanks to bug 123379.  Can we enable ICU now?

Comment 33

[Brad Lassey [:blassey] (use needinfo?)]

Waldo, telemetry on that would be very useful. If there is an in between step that doesn't involve shipping an extra 3MB, I'd be all for it.

Comment 34

[Nathan Froyd [:froydnj]]

(In reply to Nathan Froyd [:froydnj] from comment #32)
> (In reply to Richard Newman [:rnewman] from comment #21)
> > mfinkle and others have put in some time lately to get perfherder moving in
> > the right direction:
> > 
> > https://treeherder.mozilla.org/perf.html#/
> > graphs?timerange=2592000&series=[fx-team,
> > 4eb0cde5431ee9aeb5eb14512ddb3da6d4702cf0,1]&series=[mozilla-inbound,
> > 4eb0cde5431ee9aeb5eb14512ddb3da6d4702cf0,1]
> > 
> > but we're still hovering around 42MB on fx-team, which is a Bad Thing. What
> > can you take out to make ICU fit?
> 
> Looks like we've cut out about 3MB in the last three months, particularly
> sometime in February:
> 
> https://treeherder.mozilla.org/perf.html#/
> graphs?timerange=7776000&series=%5Bfx-team,
> 4eb0cde5431ee9aeb5eb14512ddb3da6d4702cf0,1%5D&series=%5Bmozilla-inbound,
> 4eb0cde5431ee9aeb5eb14512ddb3da6d4702cf0,1%5D
> 
> thanks to bug 123379.  Can we enable ICU now?

Er, that's bug 1233799, of course.

Comment 35

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Nathan Froyd [:froydnj] from comment #32)
> thanks to bug 123379.  Can we enable ICU now?
No, our APK size is too big and we're making an effort to reduce it such that Fennec can be viable in locales with low bandwidth and low storage phones. If anything, the fact that we have to expend so much effort to reduce the APK size is an argument against taking this feature.

Comment 36

[Richard Newman [:rnewman]]

To follow up on Brad's comment, here are some historical APK sizes, so that everyone has the same context for what's good and bad as the Android team does:


Firefox 14, June 2012:     17MB.
Firefox 21, May 2013:      23MB.
Firefox 26, December 2013: 25MB.
Firefox 33, November 2014: 34MB.
Firefox 36, February 2015: 36MB.
Firefox 45, now:           40MB.


For the sake of illustration, libxul alone is now bigger than the entirety of Firefox 14's APK, and libxul+omni.ja are now bigger than Firefox 26's entire APK.

Current Firefox 45, extracted:

libxul.so:   17MB
omni.ja:     12MB
classes.dex:  6MB


Even Firefox 14 was/is too big for many regions -- in some markets we're competing against Opera Mini and other such browsers that are under 5MB.

Just because we've shaved off 3MB by not shipping fonts doesn't mean we have a luxurious amount of free space -- it means we're now _only 15MB heavier_ than we'd like to be, and _only 35MB heavier_ than some markets expect.

As an org we need to put significant effort into removing platform code, assets, strings, etc. -- it's not OK to just aim for the status quo.

Comment 37

[Mike Hommey [:glandium]]

(In reply to Nathan Froyd [:froydnj] from comment #32)
> thanks to bug 123379.  Can we enable ICU now?

Disregarding the policy debate about whether or not to do it, because of bug 1239083, it's not a switch-flip anymore. See bug 1262102.

Comment 38

[Nathan Froyd [:froydnj]]

(In reply to Richard Newman [:rnewman] from comment #36)
> To follow up on Brad's comment, here are some historical APK sizes, so that
> everyone has the same context for what's good and bad as the Android team
> does:

Thanks for this information, that's really helpful.

> Current Firefox 45, extracted:
> 
> libxul.so:   17MB
> omni.ja:     12MB
> classes.dex:  6MB
> 
> 
> Even Firefox 14 was/is too big for many regions -- in some markets we're
> competing against Opera Mini and other such browsers that are under 5MB.

In the same vein of having that shared context, are those tiny browsers that Firefox is competing against also the dominant browsers in those regions?

My (admittedly uninformed, Western-centric) reading of comment 26 is that Firefox isn't even viable in some regions if ICU-esque features aren't available.  And if we're so focused on "the browser must be this small to get used" but missing the "the browser isn't practically usable in any form", that doesn't seem like a good tradeoff.

> Just because we've shaved off 3MB by not shipping fonts doesn't mean we have
> a luxurious amount of free space -- it means we're now _only 15MB heavier_
> than we'd like to be, and _only 35MB heavier_ than some markets expect.
>
> As an org we need to put significant effort into removing platform code,
> assets, strings, etc. -- it's not OK to just aim for the status quo.

Speaking as someone who has put a significant amount of effort into removing code, shrinking data structures, and the like...there's not very much left to remove without starting to gut platform features.

When I read comment 21 or comment 35, what I hear is "if folks not on the Fennec team want this feature, then those folks are going to have to put in the effort to remove things or slim existing things down; any reductions that the Fennec team itself implements aren't going to be considered as freeing up space for this feature".  Is that an accurate reading?  What would it take as evidence to show that folks not on the Fennec team have removed, say, 1MB of "stuff" from libxul?

Comment 39

[Zibi Braniecki [:zbraniecki][:gandalf]]

On top of what Nathan said:

It seems to me that we still maintain a pretty-substantial bulk of code *just* for Fennec that deals with all i18n that ICU and ECMA402 APIs could replace.

I know it's an old argument, but the 3mb that ICU adds would be then reduced by old code removal.

Also, as one of the ECMA402 editors, I'd like to point out that we are actively working on increasing the number of Intl APIs for the Web[0].

I expect that within a year, most of those proposals will be part of JavaScript. There is going to be more pressure on Fennec team and I don't see a way out of the gridlock.

Would it be possible for Fennec team to work with Android team to get access to the system ICU that Android provides?

[0] https://github.com/tc39/ecma402/#current-proposals

Comment 40

[Masatoshi Kimura [:emk]]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #39)
> Would it be possible for Fennec team to work with Android team to get access
> to the system ICU that Android provides?

If we rely on system ICU, why don't we rely on system WebView like Firefox for iOS? It will dramatically reduce the binary size and it will compete with Opera Mini.

Comment 41

[Zibi Braniecki [:zbraniecki][:gandalf]]

(In reply to Masatoshi Kimura [:emk] from comment #40)
> (In reply to Zibi Braniecki [:gandalf][:zibi] from comment #39)
> > Would it be possible for Fennec team to work with Android team to get access
> > to the system ICU that Android provides?
> 
> If we rely on system ICU, why don't we rely on system WebView like Firefox
> for iOS? It will dramatically reduce the binary size and it will compete
> with Opera Mini.

I'm not sure if it's supposed to be a technical question or a provocation, so I'll give it a benefit of doubt:

Because Mozilla is not competing in the internationalization libraries space with ICU. It's not aiming at creating a better custom Intl library to ICU and provide diversity in this field.

We leverage ICU everywhere, and unless we decided to change that (which I can see reasons for!), we'll use ICU to provide Intl APIs for Fennec. The only question is if we ship with our custom one, or use the system one (it may not work, but wanted to ask).

Comment 42

[Makoto Kato [:m_kato]]

(In reply to Masatoshi Kimura [:emk] from comment #40)
> (In reply to Zibi Braniecki [:gandalf][:zibi] from comment #39)
> > Would it be possible for Fennec team to work with Android team to get access
> > to the system ICU that Android provides?
> 
> If we rely on system ICU, why don't we rely on system WebView like Firefox
> for iOS? It will dramatically reduce the binary size and it will compete
> with Opera Mini.

Cannot.  Since we still support ICS, ICS doesn't use version 50+.

Also, ICU version is different each OS version.  And all ICU API has version suffix such as udat_open_56.

Comment 43

[Richard Newman [:rnewman]]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #39)

> I know it's an old argument, but the 3mb that ICU adds would be then reduced
> by old code removal.

Yeah, and if that's a substantial offset, it'll help a lot (if it's 3MB, then let's do this!). I recall from earlier discussions that it's not even close, but I'd love for that estimate to be wrong.


(In reply to Nathan Froyd [:froydnj] from comment #38)

> In the same vein of having that shared context, are those tiny browsers that
> Firefox is competing against also the dominant browsers in those regions?

As I understand it, yes. Our Indian contributors told me that 15MB is the largest feasible size for a browser in those regions, and 5MB is a much more acceptable size. Even our single-locale builds are over 30MB. Size isn't the only competitive element, but right now we aren't even in the running for most potential users.


> My (admittedly uninformed, Western-centric) reading of comment 26 is that
> Firefox isn't even viable in some regions if ICU-esque features aren't
> available.  And if we're so focused on "the browser must be this small to
> get used" but missing the "the browser isn't practically usable in any
> form", that doesn't seem like a good tradeoff.

I think it's more that: if our browser grows to 43MB, we will immediately lose some users in all markets (low-end Android users simply don't have that much space on their devices; some proportion literally won't be able to install updates!), and we will further exclude size-sensitive markets like India.

Our feeling is that the set of users impacted by huge APKs is significantly larger than the set of folks who have $900 multi-GB modern Android phones, really want to use Firefox, but exclusively use locales that work poorly in Fennec today.

(Again, no numbers to back this up, but I haven't heard any evidence to the contrary, either. I haven't heard many complaints from users about internationalization features, but I've heard *lots* of complaints, particularly from non-en users, about size. Lots and lots and lots.)
 

> Speaking as someone who has put a significant amount of effort into removing
> code, shrinking data structures, and the like...there's not very much left
> to remove without starting to gut platform features.

To an extent, I'm proposing gutting platform features. (Or, more moderately, splitting the monolithic Gecko to allow us to ship much less.)

That is to say: if it doesn't contribute to acquiring or retaining Firefox users, I would like it to be removed from the Gecko that we ship, with the ultimate goal of getting libxul and omni.ja down to 2013/2014 levels. Gecko in 2013/2014 was a lot smaller, and -- speaking as a product engineer -- met our needs just as well as today's Gecko.

Not everyone shares that position, but that's mine.

> any reductions
> that the Fennec team itself implements aren't going to be considered as
> freeing up space for this feature".  Is that an accurate reading?

Not really. We're all in this together, and I'm sorry if my comments seemed adversarial.

That there's a long way to go. Gecko itself has engorged itself to twice its size in the past two years, and it's by far the largest part of our application. And folks outside the Fennec frontend team are much better equipped to know what to cut and how to cut it than we are. Those three things mean that we need help. But we care about the total number, not about who gets us there.

(We apply the same high bar to asset, code, and dependency additions in front-end code, for the record.)


> would it take as evidence to show that folks not on the Fennec team have
> removed, say, 1MB of "stuff" from libxul?

Watching perfherder numbers nosedive would be enough for me. But this isn't a point-scoring thing; yes, it's really appreciated if you cut a meg off libxul, but if at the same time someone else slips 800KB of bloated assets into omni.ja, we're not really in a better place. This is a holistic problem, and it needs holistic solutions. Fennec needs to be getting smaller over time, not growing by a hundred kilobytes every month.

Comment 44

[Nathan Froyd [:froydnj]]

(In reply to Richard Newman [:rnewman] from comment #43)
> (In reply to Zibi Braniecki [:gandalf][:zibi] from comment #39)
> 
> > I know it's an old argument, but the 3mb that ICU adds would be then reduced
> > by old code removal.
> 
> Yeah, and if that's a substantial offset, it'll help a lot (if it's 3MB,
> then let's do this!). I recall from earlier discussions that it's not even
> close, but I'd love for that estimate to be wrong.

Just eyeballing the sizes of the data tables that turn up in the top 100 biggest objects, we'd probably jettison ~0.7MB of data (assuming we can turn whatever uses those tables into equivalent ICU calls).  Running size(1) on all the objects in intl/ says if we could remove all of intl/, we'd save a little under 1MB.  So significant, but not a huge win.

> I think it's more that: if our browser grows to 43MB, we will immediately
> lose some users in all markets (low-end Android users simply don't have that
> much space on their devices; some proportion literally won't be able to
> install updates!), and we will further exclude size-sensitive markets like
> India.
> 
> Our feeling is that the set of users impacted by huge APKs is significantly
> larger than the set of folks who have $900 multi-GB modern Android phones,
> really want to use Firefox, but exclusively use locales that work poorly in
> Fennec today.

This perspective is helpful, thank you for the explanation.

Comment 45

[robert.rcampbell]

> Our feeling is that the set of users impacted by huge APKs is significantly
> larger than the set of folks who have $900 multi-GB modern Android phones,
> really want to use Firefox, but exclusively use locales that work poorly in
> Fennec today.

Just a quick note on the $900 phone part...

My phone is an iMobile IQII AndroidOne bought in Laos for $115 USD, 16GB internal storage, Android 6, and Lao and Khmer work great on it... The only phone I've owned that was more expensive was my Samsung Galaxy Ace 3 - about $250. I've tried to intentionally restrict my device to a reasonable phone, because the web apps I make must run on middle and low end devices - because I develop local language resources.

I found a source in Phnom Penh selling the same phon I have now for $150, which we bought for the principal at our school. They are selling well in Thailand, because they are reasonably good phones, without the bloat and lock down of the Samsung devices that dominate there (and everywhere in SE Asia).

India also has an AndroidOne phone being sold there. I would assume it's doing well too.

This whole 'my phone has 2GB/4GB of internal storage' won't last much longer... Not with AndroidOne initiative pushing for better competition. Samsung will have to start competing or it will start dying, either of which aren't bad things.

In Vientiane, Laos, it wasn't uncommon to see locals with iPhones and expensive Androids. Here in Mondulkiri, Cambodia, there's a big market for cheaper phones (around $100 on the street, including some used iPhones). And this region is quite underdeveloped.

Of course, $100 is still not always in reach for your lower middle class and below, but I've seen some of them with smartphones driving tractor pulled wooden carts. Image is quite important here, and some people invest more in their phones than they pay for their transportation (motorbike).

Anyways, my perspective...

Comment 46

[Mark Finkle (:mfinkle) (use needinfo?)]

It's certainly true that Android One is pushing the physical devices into larger capacities. That's half (or maybe less) of the battle though. Downloading a large APK is probably a more of a blocker, given bandwidth speeds and costs.

Including better support for JavaScript Intl is an inevitability, but for downloadable mobile browsers, there are trade-offs. I'm glad to see some non-Mobile team people taking part in these discussions, and looking to find ways to help.

The 5MB browsers based on WebViews have some big advantages over Firefox. We appreciate the ideas and help.

Comment 48

[robert.rcampbell]

(In reply to Mark Finkle (:mfinkle) (use needinfo?) from comment #46)
> It's certainly true that Android One is pushing the physical devices into
> larger capacities. That's half (or maybe less) of the battle though.
> Downloading a large APK is probably a more of a blocker, given bandwidth
> speeds and costs.
> 
> Including better support for JavaScript Intl is an inevitability, but for
> downloadable mobile browsers, there are trade-offs. I'm glad to see some
> non-Mobile team people taking part in these discussions, and looking to find
> ways to help.
> 
> The 5MB browsers based on WebViews have some big advantages over Firefox. We
> appreciate the ideas and help.

Just curious. If you take out the ICU dictionaries for word tokenization, how much would that reduce on adoption size for ICU?

The reason I ask is that the word lists that make up that are large, and talking with some liguists recently, there may be a way to not need such large word lists for Thai, Lao, Khmer, Burmese, and other SE Asian languages... (This would require help from expert programmers who could code the methods into algorithms efficiently). I would be willing to help with the liguistics side of things... Any takers?

Comment 49

[Makoto Kato [:m_kato]]

(In reply to robert.rcampbell from comment #48)
> (In reply to Mark Finkle (:mfinkle) (use needinfo?) from comment #46)
> > It's certainly true that Android One is pushing the physical devices into
> > larger capacities. That's half (or maybe less) of the battle though.
> > Downloading a large APK is probably a more of a blocker, given bandwidth
> > speeds and costs.
> > 
> > Including better support for JavaScript Intl is an inevitability, but for
> > downloadable mobile browsers, there are trade-offs. I'm glad to see some
> > non-Mobile team people taking part in these discussions, and looking to find
> > ways to help.
> > 
> > The 5MB browsers based on WebViews have some big advantages over Firefox. We
> > appreciate the ideas and help.
> 
> Just curious. If you take out the ICU dictionaries for word tokenization,
> how much would that reduce on adoption size for ICU?

We don't include word / line breaker dictionary even if desktop's ICU.  Our dictionary is already small except to city timezone data.  But cities data may require for next ECMA-402.  Also, Chrome removes some localization data for date time and number that isn't important language too.

Comment 50

[robert.rcampbell]

(In reply to Makoto Kato [:m_kato] (PTO 6/20-21, 6/24) from comment #49)
> (In reply to robert.rcampbell from comment #48)
> > (In reply to Mark Finkle (:mfinkle) (use needinfo?) from comment #46)
> > > It's certainly true that Android One is pushing the physical devices into
> > > larger capacities. That's half (or maybe less) of the battle though.
> > > Downloading a large APK is probably a more of a blocker, given bandwidth
> > > speeds and costs.
> > > 
> > > Including better support for JavaScript Intl is an inevitability, but for
> > > downloadable mobile browsers, there are trade-offs. I'm glad to see some
> > > non-Mobile team people taking part in these discussions, and looking to find
> > > ways to help.
> > > 
> > > The 5MB browsers based on WebViews have some big advantages over Firefox. We
> > > appreciate the ideas and help.
> > 
> > Just curious. If you take out the ICU dictionaries for word tokenization,
> > how much would that reduce on adoption size for ICU?
> 
> We don't include word / line breaker dictionary even if desktop's ICU.  Our
> dictionary is already small except to city timezone data.  But cities data
> may require for next ECMA-402.  Also, Chrome removes some localization data
> for date time and number that isn't important language too.

Sorry, my question was vague. 

Basically, what I am asking is that if the current ICU dictionaries in the break iterator code (within ICU itself) were emptied of content, after being compiled, about how much would that remove from the total size of taking on ICU? (Sorry, don't have access to a machine to check it myself.) 

Is it possible for someone to compile an APK that has empty version of these files and see how big they'd end up being? (I'm not familiar with Japanese or Chinese, but cjdict.txt file also takes up quite a bit of room. Maybe there is a way to reduce this file as well?)

Of course an algorithm with a fallback for exceptions would still take up some space - in some cases, like for Lao possibly not being a large enough difference to be worth the effort, but it could knock down the size for all if not some of them. And then when you take out what ICU will replace in terms of functionality, I think that could come much closer to reducing the burden of taking on ICU in the big picture.

I see that the text files themselves are >3MB themselves for Thai, Lao, Khmer, Burmese... 

http://bugs.icu-project.org/trac/browser/icu/trunk/source/data/brkitr/dictionaries

If it is enough to legitimize Mozilla taking on ICU, I think it might be worth my time seeing if the ICU team would be willing to take on an effort to swap out the large dictionaries with algorithms approaches that use much smaller exception lists for words that don't quite fit an algorithm well. I could help with that since I have experience in 2 (Khmer and Lao) of the 4 largest languages in the region, and some limited experience in Thai. Plus, I know the person who did some of the work for Burmese (the 4th one), as I used to work with him in the same organization.

Also, I believe there may be some currently ongoing effort to do this change for one of the languages listed - but I don't know how soon that will happen.

Thoughts?

Comment 51

[Makoto Kato [:m_kato]]

Robert, we already remove all word breaker and line breaker data file on our tree and we built ICU with UCONFIG_NO_BREAK_ITERATION.  ICU data file is in https://dxr.mozilla.org/mozilla-central/source/config/external/icu/data, it is built by https://dxr.mozilla.org/mozilla-central/source/intl/icu_sources_data.py.

To built without UCONFIG_NO_BREAK_ITERATION, it isn't goal of this bug.  It is bug 820261 or bug 933631.

Comment 52

[Xidorn Quan [:xidorn] UTC+10]

I wonder whether there is a chance to have this fixed soon.

I think we really want to use ICU's bidi engine rather than maintaining an implementation ourselves (bug 924851). Actually, in general we just backport code from ICU to our implementation, but since there are certain differences, doing so isn't always a trivial work. We find bugs in our impl, but not in ICU, every now and then, which takes time to debug and fix...

Comment 54

[Masatoshi Kimura [:emk]]

Unless we migrate to the system WebView, it is simply impossible to trimming binary size down to 20MB even if it might have been possible as of Firefox 26. Waldo made an excellent post about this on dev-platform:
https://lists.mozilla.org/pipermail/dev-platform/2016-May/014750.html

This bug should be closed as WONTFIX to reflect the reality and prevent people from having a false expectation.

(In reply to Xidorn Quan [:xidorn] (UTC+10) from comment #52)
> I wonder whether there is a chance to have this fixed soon.

No. Go ahead with simulated ubidi until Fennec drops Gecko.

Comment 55

[:Margaret Leibovic]

Barbara has been revisiting our APK size requirements. She can prioritize this accordingly. We do want to work towards a solution here, so this is not a WONTFIX.

Comment 56

[Axel Hecht [:Pike]]

I pitched OBB, https://developer.android.com/google/play/expansion-files.html in bug 1262102 comment 6. Really just pitched, I hardly red that doc.

Comment 57

[Axel Hecht [:Pike]]

(In reply to Richard Newman [:rnewman] from comment #12)
> 
> If someone spends the time to figure out Bug 945123, that would also buy us
> enough space (5MB!) that we'd take this outright.

To fork out l10n files from build-time gecko, we'll need to rewrite the l10n code and the code that uses it. This is something we're starting now. This will be based on future-proof APIs, including code paths that depend on ICU, notably: Number/Date/List formatters, and some interactions with locale code matching. We'll also focus the existing l10n files on use case, so some size wins might come just by refacting strings into "commonly used on the web" and "error console stuff".

We'll not be able to provide all of that in one cycle, and we'll need to move forward on desktop Firefox.

Much of the conversation was had in a different context of engineering capacity on Android and product roadmap.

With the build problems hopefully being resolved soon, my suggestion is that we're enabling ICU on Android once we land l20n in gecko and convert Firefox/Toolkit. I suggest take we take the hit in apk size, and recover over a couple of releases.

The impact on Android is tracked in bug 1280688, the complete work is tracked in bug 1279002. The exact timeline is still TBD, London and summer vacations make planning of timelines hard. But soon (TM).

Comment 58

[Makoto Kato [:m_kato]]

Alex, although I have pointed out it at MozLondon's L20N meetup, L20N team doesn't seems to be high priority for android version.  But I will fix bug 1262102 this month or next month... (at least, Q3)...

Comment 59

[Makoto Kato [:m_kato]]

libxul.so size
before this ... 26287379 bytes
after this  ... 30692696 bytes

Comment 60

[Makoto Kato [:m_kato]]

Attachment: Bug 1215247 - Turn on ICU and Intl API for Android.

Intl API (ECMA-402) already supports most web browsers such as Edge (including Windows 10 Mobile), Chrome (including Android), and upcoming Safari 10(https://developer.apple.com/library/prerelease/content/releasenotes/General/WhatsNewInSafari/Articles/Safari_10_0.html).

Also IDN2008 support, and number control by <input type="number"> require ICU.  And, although gfx has own unicode table, we want to remove this if ICU is supported on all platform.

Also, new L20N framework (aka L20N) wants to use Intl API to localize UI.

So I would like to use ICU as default on Fennec Android.

Negative issue is file size.  ICU has big table, file size of libxul.so will be following.  This support requires additional 4.4MB.

before this ... 26287379 bytes
after this  ... 30692696 bytes

Although android OS already has ICU, ICU version is different per OS version.  And our ICU requirement is 50.1+ (Android 4.0 uses ICU 48) and exported fucntion name of ICU has ICU version suffix.  So we cannot use it.

Review commit: https://reviewboard.mozilla.org/r/68456/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/68456/

Comment 61

[Richard Newman [:rnewman]]

For the record, Android 4.3 (API 18) has ICU50, and 4.4 (API 19) has 51, per [1].

I wonder if it's more palatable to drop support for earlier Android versions in a future Firefox than to take a 4.4MB hit?

[1] <https://developer.android.com/reference/java/util/Locale.html>

Comment 62

[Xidorn Quan [:xidorn] UTC+10]

They don't expose ICU as part of their stable NDK API, so even if we can drop support for earlier Android versions, I don't think we can use them.

Since Android N, they started providing ICU4J, which we may be able to use in the future, I guess. [1]

[1] https://developer.android.com/preview/features/icu4j-framework.html

Comment 63

[Makoto Kato [:m_kato]]

(In reply to Richard Newman [:rnewman] from comment #61)
> For the record, Android 4.3 (API 18) has ICU50, and 4.4 (API 19) has 51, per
> [1].
> 
> I wonder if it's more palatable to drop support for earlier Android versions
> in a future Firefox than to take a 4.4MB hit?
> 
> [1] <https://developer.android.com/reference/java/util/Locale.html>

Now gfx uses Unicode 8.0 data table (ex. ICU 52 uses Unicode 6.2...) and we would like to replace it with ICU as possible.  If we use mixed ICU version on one Android version, font rendering etc will be different per ICU version.

So we should use in-tree data to avoid compatibility issue of Unicode and vendor customization.  Even if Chrome for Android, they  use owns in-tree code/data too instead of device's library/table.

Comment 64

[Jeff Walden [:Waldo]]

Comment on attachment 8776799 [details]
Bug 1215247 - Turn on ICU and Intl API for Android.

https://reviewboard.mozilla.org/r/68456/#review65790

Patch looks fine.  Policy decision isn't mine to make (but I happen to agree with it).

We'll want to rip out the ability to not build with ICU eventually, but that patching can/should wait til this has landed, and perhaps a cycle past that to be safe.

::: js/xpconnect/tests/unit/xpcshell.ini
(Diff revision 1)
>  head = head_ongc.js
>  [test_onGarbageCollection-05.js]
>  head = head_ongc.js
>  [test_reflect_parse.js]
>  [test_localeCompare.js]
> -# Bug 676965: test fails consistently on Android

This removal appears unrelated to this patch/bug?

Comment 65

[Xidorn Quan [:xidorn] UTC+10]

(In reply to Jeff Walden [:Waldo] (remove +bmo to email) from comment #64)
> Comment on attachment 8776799 [details]
> Bug 1215247 - Turn on ICU and Intl API for Android.
> 
> https://reviewboard.mozilla.org/r/68456/#review65790
> 
> Patch looks fine.  Policy decision isn't mine to make (but I happen to agree
> with it).
> 
> We'll want to rip out the ability to not build with ICU eventually, but that
> patching can/should wait til this has landed, and perhaps a cycle past that
> to be safe.

We would need to run build on infra to ensure that still works, otherwise it may be broken at any point. e.g. I'm planning replacing our bidi engine with the ICU one, and I'm waiting for this so that the change can be as simple as possible, without overhead to support both code path. But that would make the build depend on ICU, which means if I were not aware of the policy here, I would break ICU-disabled build.

Comment 66

[Sebastian Kaspari (:sebastian; :pocmo)]

Comment on attachment 8776799 [details]
Bug 1215247 - Turn on ICU and Intl API for Android.

https://reviewboard.mozilla.org/r/68456/#review65912

I can't really r+ a patch that adds 4+ MB to the APK size. We are currently going a long way to reduce the APK size by 1-2 MB (bug 1095719) and have made similar improvements in the past. Growing by 4+ MB basically reverts all those improvements.

Anyways, as this is more a platform-y thing, I'll forward this review request to snorp.

Comment 67

[James Willcox (:snorp) ([e-mail]) (he/him)]

Comment on attachment 8776799 [details]
Bug 1215247 - Turn on ICU and Intl API for Android.

https://reviewboard.mozilla.org/r/68456/#review65988

The story around the APK size has not changed, and as such we cannot accept this patch.

Comment 68

[Xidorn Quan [:xidorn] UTC+10]

The size comparison in comment 59 and comment 60 is about libxul.so. Do we have the comparison around actual APK file? IIRC there is an additional compression applied to libxul.so, so it's worth measuring how much it that could save.

Comment 69

[Masatoshi Kimura [:emk]]

(In reply to :Margaret Leibovic from comment #55)
> Barbara has been revisiting our APK size requirements. She can prioritize
> this accordingly. We do want to work towards a solution here, so this is not
> a WONTFIX.

I don't understand how comment #66 and #67 does not mean WONTFIX.

Comment 70

[Xidorn Quan [:xidorn] UTC+10]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #67)
> Comment on attachment 8776799 [details]
> Bug 1215247 - Turn on ICU and Intl API for Android.
> 
> https://reviewboard.mozilla.org/r/68456/#review65988
> 
> The story around the APK size has not changed, and as such we cannot accept
> this patch.

How can we make this patch acceptable? Not increasing the size is basically infeasible, but people have tried hard to mitigate the hit. Also the APK size has been shrinked for quite a bit these versions, wouldn't that give us some room to fit the ICU in?

Not only the I18n API, there is an increasing burden on platform team to maintain code which could have been replaced by ICU, so I really hope this can get fixed as soon as possible. Please advise if there is a possible way to make it.

Comment 71

[Xidorn Quan [:xidorn] UTC+10]

(Probably we can have ICU included in some other bug but keep I18n API disabled... that should reduce the hit significantly while unblock other parts to replace our code with the ICU path.)

Comment 72

[Richard Newman [:rnewman]]

(In reply to Xidorn Quan [:xidorn] (UTC+10) from comment #71)
> ... that should reduce the hit significantly…

Can you quantify this?

Comment 73

[Richard Newman [:rnewman]]

Oh, and an update on sizes:

Firefox 14, June 2012:     17MB.
Firefox 21, May 2013:      23MB.
Firefox 26, December 2013: 25MB.
Firefox 33, November 2014: 34MB.
Firefox 36, February 2015: 36MB.
Firefox 45:                40MB.
Firefox 47, June 2016:     38MB.
Firefox 48, now:           36MB.


We're back to the size we were when we said no to shipping ICU in Bug 864843 in 2015 (which is good or bad depending on how you look at it!).

If this trend continues, and if Nathan is right in Comment 44 about there being ~700KB of space to recover, then I think we might be close to taking this patch at the beginning of a cycle, if there's a firm commitment to ripping out duplication to continue shrinking. But that's just my opinion; Sebastian, snorp, Brad, and Barbara carry more weight.

Comment 74

[Makoto Kato [:m_kato]]

Sebastian, I have already reduced APK size around 1MB by various bugs such as removing XULs and etc for this issue.

What the limit of incremented size you accept?   Although you aren't interesting in localization issue, L20N team requires this feature (see comment #57).

When I remove all localization data such as date time format, number format, zone and currency and I also remove some collation data that is none-shipped languages (such as Vietnamese) on Fennec, I can reduce that incremented size is around 2.4MB (original was 4.4MB)

Also if data has only collation, number format and date time format by our shipped languages, and currency and zone by en-US only, its size is around 2.7MB.  (L20N will work with this option).

Again.  By turned on ICU, the following issue will be fixed / improved.  In other word, Fennec lacks these features at least.
- IDN2008 support
- IndexedDB sort order

According to rnewman's comment #73, we will be able to keep Firefox 45 size (40MB) with ICU support by my option (increment size will be 2.4MB or 2.7MB).

Comment 75

[Richard Newman [:rnewman]]

snorp, how much APK size might Bug 1291424 save us?

Comment 76

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Richard Newman [:rnewman] from comment #75)
> snorp, how much APK size might Bug 1291424 save us?

It should save us exactly 5MB.

Comment 77

[Sebastian Kaspari (:sebastian; :pocmo)]

(In reply to Makoto Kato [:m_kato] from comment #74)
> Sebastian, I have already reduced APK size around 1MB by various bugs such
> as removing XULs and etc for this issue.

Thank you for your help! It is most appreciated.

Every now and then I look at the perfherder graphs and just see a steady growth. We had a couple of drops but they required a significant amount of work and so I'm a bit worried if one code drop "reverts" all the effort.

> What the limit of incremented size you accept?   Although you aren't
> interesting in localization issue, L20N team requires this feature (see
> comment #57).

Oh, I didn't intent to make a statement on the usefulness of the API in general. That's why I forwarded the review request to snorp. I'm not the right person to make platform decisions.


> When I remove all localization data such as date time format, number format,
> zone and currency and I also remove some collation data that is none-shipped
> languages (such as Vietnamese) on Fennec, I can reduce that incremented size
> is around 2.4MB (original was 4.4MB)
> 
> Also if data has only collation, number format and date time format by our
> shipped languages, and currency and zone by en-US only, its size is around
> 2.7MB.  (L20N will work with this option).

2.x MB definitely sounds much better than 4.x MB. On Fennec we have a background service that is able to download assets at runtime (Currently used for fonts): Could we let the service download this data and could the code use/load this data if it exists locally? Or does it *need* to be in the APK?


> According to rnewman's comment #73, we will be able to keep Firefox 45 size
> (40MB) with ICU support by my option (increment size will be 2.4MB or 2.7MB).

Yes, especially with the additional 5 MB saved (and we have another 1.x MB safe in the pipeline for downloading hyphenation dictionaries at runtime) this seems much more doable!

Comment 78

[Zibi Braniecki [:zbraniecki][:gandalf]]

(In reply to Sebastian Kaspari (:sebastian) from comment #77)
> Yes, especially with the additional 5 MB saved (and we have another 1.x MB
> safe in the pipeline for downloading hyphenation dictionaries at runtime)
> this seems much more doable!

What prevents us from switching to use ICU for hyphenation?

Comment 79

[Jonathan Kew [:jfkthame]]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #78)
> What prevents us from switching to use ICU for hyphenation?

Does ICU provide hyphenation services? (Not to my knowledge, but that may be out-of-date...)

Comment 80

[Zibi Braniecki [:zbraniecki][:gandalf]]

Isn't that what we're looking for - http://icu-project.org/apiref/icu4c/classicu_1_1BreakIterator.html ?

At TC39 there are other vendors interested in exposing a generic JS Intl API for hyphenation and line-breaking and I thought they're talking about basing it on this.

Comment 81

[Henri Sivonen (:hsivonen)]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #67)
> The story around the APK size has not changed, and as such we cannot accept
> this patch.

In order to deliver on our Mission, we are in the business of delivering an implementation of the Web Platform instead of just providing a UI and sync client with *some* engine. A full implementation of the Web Platform takes space and the space it takes will increase over time. In this light, it's not reasonable to set APK size to some number and block Gecko development that would take the size above that number. It's especially not reasonable to set that number by comparing our APK size with that of browser UIs that wrap the system WebView. Yes, increasing the APK size will mean Firefox won't fit on some phones with low specs, but we've already had a failure (with Firefox OS) when trying to aim for the lowest of the low end. We can't lead by refusing to bundle ICU when Chrome for Android (our main competitor, which also ships a full implementation of the Web Platform instead of wrapping system WebView) ships it.

Furthermore, when attributing our size problem, it makes sense to look at the .so size of Firefox for Android vs. the .so size of Chrome for Android. On my phone, Fennec nightly has 11 MB less of decompressed .so size than release channel Chrome.

I suggest we land ICU and unblock everything that's currently blocked by this bug and if we want to make APK smaller, look for non-.so savings.

Comment 82

[Makoto Kato [:m_kato]]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #80)
> Isn't that what we're looking for -
> http://icu-project.org/apiref/icu4c/classicu_1_1BreakIterator.html ?
> 
> At TC39 there are other vendors interested in exposing a generic JS Intl API
> for hyphenation and line-breaking and I thought they're talking about basing
> it on this.

breaker dictionary on ICU is too big and we don't still turn on this feature on all platform.  But we also have another implementations for both.  And I am consider whether we can replace line breaker with rust by xi-unicode (bug 1290022).

Comment 83

[Makoto Kato [:m_kato]]

(when we turn on ICU's breaker, libxul's size will be incremented around 2-3MB)

Comment 84

[Jet Villegas (inactive)]

(In reply to Henri Sivonen (:hsivonen) from comment #81)
> I suggest we land ICU and unblock everything that's currently blocked by
> this bug and if we want to make APK smaller, look for non-.so savings.

We've got agreement on a plan to get ICU landed with the L20N changes that enable single-locale builds. The size savings from single-locale are expected to offset the ICU size hit. This work will happen in a branch and merge to m-c when ready.

Comment 85

[Nathan Froyd [:froydnj]]

(In reply to Jet Villegas (:jet) from comment #84)
> We've got agreement on a plan to get ICU landed with the L20N changes that
> enable single-locale builds. The size savings from single-locale are
> expected to offset the ICU size hit. This work will happen in a branch and
> merge to m-c when ready.

That's great!  Is there a metabug for all this work so interested parties can follow along?

Comment 86

[Jonathan Kew [:jfkthame]]

(In reply to Nathan Froyd [:froydnj] from comment #85)
> Is there a metabug for all this work so interested parties
> can follow along?

The team is currently using:
https://github.com/mozilla-l10n/roadmap/projects/1
https://github.com/mozilla-l10n/roadmap/projects/2

Comment 87

[Staś Małolepszy :stas]

https://hg.mozilla.org/projects/larch/rev/836e720e70ce173cf1be7602be8fda8e0faeccf5
Bug 1215247 - Turn on ICU and Intl API for Android.

Comment 88

[Staś Małolepszy :stas]

Gah!  Sorry, my bad, I wanted to reference this bug in bug 1307164 when I landed this patch on larch.  We need it for our L20n work there.

Comment 89

[Gregory Moore [:gmoore]]

Hi, I am working on bug 1301640 (that uses ICU), which is almost finished but is blocked by this bug. There is a workaround we could implement until this bug is fixed, but we wanted to check if there was an estimated timeline for that to happen, before we spend too long working on it. Please let me know, thanks.

Comment 90

[Richard Newman [:rnewman]]

Summing up discussions elsewhere: I no longer consider myself a blocker to this landing, so long as we continue in the same vein of space savings noted in Comment 84 and Comment 77.

Some excellent recent work has already knocked our APK size down by multiple MB. If switching to this API will unblock downloadable localization files, then we'll eventually end up saving another ~7MB of APK size, as well as making a bunch of developers happier. That's good enough for me.

Comment 91

[Makoto Kato [:m_kato]]

(In reply to Gregory Moore from comment #89)
> Hi, I am working on bug 1301640 (that uses ICU), which is almost finished
> but is blocked by this bug. There is a workaround we could implement until
> this bug is fixed, but we wanted to check if there was an estimated timeline
> for that to happen, before we spend too long working on it. Please let me
> know, thanks.

We will change the package to single locale build to land this.  So, until bug 945123 is resolved, we won't turn on ICU on Fennec.

Comment 92

[Masatoshi Kimura [:emk]]

(In reply to Makoto Kato [:m_kato] from comment #91)
> We will change the package to single locale build to land this.  So, until
> bug 945123 is resolved, we won't turn on ICU on Fennec.

And bug 945123 depends on Intl per the bug comments. How will the cyclic dependence be resolved?

Comment 93

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Pinging Joe for a product decision also.

Comment 94

[David Bolter [:davidb] (NeedInfo me for attention)]

Updating whiteboard based on recent caniuse data for intl.

Comment 95

[Joe Cheng [:jcheng] (please needinfo)]

had a conversation with :jet about this. let's land this and keep the fennec apk size under 30MB. Thanks

Comment 96

[James Willcox (:snorp) ([e-mail]) (he/him)]

What exactly is the plan for keeping the APK size low if the downloadable l10n stuff is not complete?

Comment 97

[Jet Villegas (inactive)]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #96)
> What exactly is the plan for keeping the APK size low if the downloadable
> l10n stuff is not complete?

The plan is to continue working on l10n separately. Landing that will happen after Quantum. Our most current Fennec adoption and partner stats is what got us to this agreement to land this bug at <= 30MB.

Comment 98

[Julien Wajsberg [:julienw]]

React-Intl uses this API with no polyfill by default. I suspect we'll start to see websites/webapps using React broken on Firefox/Android soon if this doesn't land quickly :/

Comment 99

[Jet Villegas (inactive)]

(In reply to Joe Cheng [:jcheng] (please needinfo) from comment #95)
> had a conversation with :jet about this. let's land this and keep the fennec
> apk size under 30MB. Thanks

Our latest tests show that Fennec 53 will already exceed the requested 30MB limit _without_ ICU, and we certainly won't get below that by adding code:

Firefox 53 = 30.6 MB (32,125,459 bytes)
Firefox 53 + ICU = 33.3 MB (34,809,542 bytes)

Joe: please confirm that this is OK to land (in Firefox 54) if we keep the +2.8 MB increase shown in our latest tests.

Comment 100

[Joe Cheng [:jcheng] (please needinfo)]

Jet: yes it is OK to get ICU landed and unblock as long as we keep the size in mind.

Comment 101

[Makoto Kato [:m_kato]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=27001defeb3a45fc002834ecfd7dc92210a1b31c

Comment 102

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4f25dbd725c8
Turn on ICU and Intl API for Android. r=Waldo

Comment 103

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/4f25dbd725c8

Comment 104

[Tooru Fujisawa [:arai]]

Updated following documents:
  https://developer.mozilla.org/en-US/Firefox/Releases/54

  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/normalize
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/localeCompare

  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl

  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Collator
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Collator/prototype
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Collator/supportedLocalesOf
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Collator/compare
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Collator/resolvedOptions

  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DateTimeFormat
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DateTimeFormat/prototype
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DateTimeFormat/supportedLocalesOf
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DateTimeFormat/formatToParts
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DateTimeFormat/format
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DateTimeFormat/resolvedOptions

  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/NumberFormat
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/NumberFormat/prototype
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/NumberFormat/supportedLocalesOf
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/NumberFormat/format
  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/NumberFormat/resolvedOptions

  https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/getCanonicalLocales

Comment 105

[William Lachance (:wlach)]

As predicted, this resulted in a large size regression on Android:

== Change summary for alert #5042 (as of February 08 2017 04:01 UTC) ==

Regressions:

  8%  installer size summary android-4-0-armv7-api15 opt nightly       32341918 -> 35049357.33
  8%  installer size summary android-4-2-x86 opt nightly               34533206.25 -> 37353992.83
  8%  installer size summary android-4-0-armv7-api15 debug nightly     33391792.75 -> 36061057.67

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=5042

Comment 106

[Makoto Kato [:m_kato]]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #105)
> As predicted, this resulted in a large size regression on Android:
> 
> == Change summary for alert #5042 (as of February 08 2017 04:01 UTC) ==
> 
> Regressions:
> 
>   8%  installer size summary android-4-0-armv7-api15 opt nightly      
> 32341918 -> 35049357.33
>   8%  installer size summary android-4-2-x86 opt nightly              
> 34533206.25 -> 37353992.83
>   8%  installer size summary android-4-0-armv7-api15 debug nightly    
> 33391792.75 -> 36061057.67
> 
> For up to date results, see:
> https://treeherder.mozilla.org/perf.html#/alerts?id=5042

Product team already accepts this by comment #100.

Comment 107

[Zibi Braniecki [:zbraniecki][:gandalf]]

Mind you, we do believe that we'll be able to get some of that space back through things like:
 - removing custom collations
 - shifting l10n to use l20n
 - using ICU uconv

Comment 108

[Henri Sivonen (:hsivonen)]

I'm super-happy that this has landed. Thank you to everyone who made it happen.

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #107)
>  - using ICU uconv

Having ICU everywhere does enable a bunch of code removals, sure, but that's not one of them.

uconv is on track to being replaced but not with ICU but with encoding_rs. I expect this to make the APK smaller though. (We already compile ICU without encoding converters to the extent allowed by ICU.)

Comment 109

[Zibi Braniecki [:zbraniecki][:gandalf]]

> uconv is on track to being replaced but not with ICU but with encoding_rs.

You're right, I knew we're planning a switch, just messed up the destination :)

> (We already compile ICU without encoding converters to the extent allowed by ICU.)

Can we pinpoint a subset that ICU does not allow us to exclude but we'd like to and file an issue in their trac?

Comment 110

[Henri Sivonen (:hsivonen)]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #109)
> Can we pinpoint a subset that ICU does not allow us to exclude but we'd like
> to and file an issue in their trac?

http://bugs.icu-project.org/trac/ticket/10868 (closed as workingasdesigned)

I guess we should have another look at first building ICU without -DUCONFIG_NO_CONVERSION=1, then using that copy of ICU to build the data files that we need and then re-building ICU with -DUCONFIG_NO_CONVERSION=1 using data files generated in the previous build step.

Comment 111

[Zibi Braniecki [:zbraniecki][:gandalf]]

:m_kato, is that something you'd be willing to try?

Comment 112

[Makoto Kato [:m_kato]]

We already removed several data that we didn't use.  Example, encoding table and line breaker rules.  Various unicode properties into gfx already use ICU API instead of owns tables and we remove it by if-def.

I can remove some L10N data by new DateTimeFormat API.  I will handle it soon.

Comment 114

[Florian Scholz (Open Web Docs)]

This is Nightly only for the moment. :arai updated the MDN docs accordingly, see https://bugzilla.mozilla.org/show_bug.cgi?id=1343725#c16 (thank you!)

The new tracking bug seems to be bug 1344625.

Comment 115

[Kohei Yoshino]

https://pontoon.mozilla.org/ja/ is also broken due to the lack of the Intl API support :/