1012365 - [Fennec] [mr] Improper UI text rendering for leading character ॲ (U+0972) on Samsung Galaxy S4 device

Status: VERIFIED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Chandrakant Dhutadmal]

User Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0 (Beta/Release)
Build ID: 20140421221237

Steps to reproduce:

1. Download Localized version of firefox (Marathi language) from
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-l10n/fennec-31.0a2.mr.android-arm.apk on mobile device (Samsung Galaxy S4) .

2. Go to the location of this .apk file in the device and click on the file to launch/ start installation.

3. Launch the fennec application.

4. Go to tools and see the options listed out.




Actual results:

Two options ( Add-Ons and Apps) are not displayed correctly. First character in both these words is "ॲ" which has unicode value "U+0972" of devnagari code page. Instead of this character a square box is appearing in the UI text.



Expected results:

"ॲ" should have been displayed properly in both these options. i.e ॲड-ऑन्स and ॲप्स should have been displayed corectly.

Comment 1

[sandeep shedmake]

@Chandrakant,

Please attach a screenshot of inappropriate rendering of U+0972 @Samsung_Galaxy_S4.


Thanks.

Comment 2

[Chandrakant Dhutadmal]

Attachment: Screenshot of Marathi Fennec incorrect text display

Screenshot attached. Problematic text is circled with red colour.

Comment 3

[sandeep shedmake]

@Chandrakant,

Visibility of empty square box instead of "U+0972;ॲ;DEVANAGARI LETTER CHANDRA A" implies missing glyph/shape in the font file.


Towards effective resolution of this bug, you need to install custom font (*.ttf having glyph U+0972) on your 'Samsung Galaxy S4' device.


This can be done two different methods:

1) Installation of a font app for _non-rooted_ Samsung Android phone

OR

2) Download and copy font/*.ttf (from PC) to your device (_rooted_ Samsung Android phone)



Unfortunately, this bug cannot be classified under 'localization' category. 

I am sure Samsung Android phones can render U+0972 too provided the font file contains the required glyph to be rendered.


Thereby, closing this bug as NOT_A_BUG / INVALID.

Comment 4

[Parag Nemade]

Attachment: shot_2014-05-19_12-34-28.png

This is what I got on Android 4.4.2 version. Looks like rendering is good.

Comment 5

[Parag Nemade]

Attachment: shot_2014-05-19_12-40-53.png

Previous attachment having different screenshot. Attaching the one that shows rendering happening on Andorid 4.4.2 with the given fennec apk installed.

Comment 6

[sandeep shedmake]

From Comment 5,


Hey Parag! Thanks for the screenshot.


Chandrakant, please have a look at screenshot attached in Comment 5 for correct rendering of [U+0972]. 

Meanwhile, what is the Android version @Samsung_Galaxy_S4 you used while filing this bug ?

Comment 7

[sandeep shedmake]

(In reply to sandeep shedmake from comment #6)
> From Comment 5,
> 
> 
> Hey Parag! Thanks for the screenshot.
> 
> 
> Chandrakant, please have a look at screenshot attached in Comment 5 for
> correct rendering of [U+0972]. 
> 


@Chandrakant:

Also, if it is not the case; add appropriate commment (with screenshots and references - if possible) in this bugzilla and i would re-open the bug towards FIXING it.



Thanks.
Sandeep

> Meanwhile, what is the Android version @Samsung_Galaxy_S4 you used while
> filing this bug ?

Comment 8

[Chandrakant Dhutadmal]

Attachment: MarathiFennecWebPage.png

@Sandeep.

1. Please refer to the fonts kept as an attachment (named SamsungDevanagariFile.zip) with the bug 

https://bugzilla.mozilla.org/show_bug.cgi?id=992100. This font is having the shape of required character "U+0972;ॲ;DEVANAGARI LETTER CHANDRA A". 

2. In the same device for which i have reported this bug, i can see the character properly displayed in the webpage text. Refer snapshot attached with this bug (named MarathiFennecWebPage.png. 

3. Device details- Samsung Galaxy S4, Model Number-GT-I9500, Android version 4.3, Kernel- 3.4.5-1984169.

If someone else with the same device can cross check what i have observed, that would lead us to find the exact cause of the bug.

Comment 9

[sandeep shedmake]

Test with the build from https://bugzilla.mozilla.org/show_bug.cgi?id=992100#c46 and observe if the leading/first character: ॲ  as in ॲड-ऑन्स and ॲप्स are displayed correctly.

Comment 10

[Chandrakant Dhutadmal]

Attachment: Fennec jkew .png

@Sandeep. Tested on the http://people.mozilla.org/~jkew/fennec-bug-992100.apk build. The ॲ  as in ॲड-ऑन्स and ॲप्स are not displayed correctly there also. Attached is the screenshot (Fenenc jkew.png) of the same.

Comment 11

[sandeep shedmake]

From Comment 10,

CCing Jonathan Kew for feedback.


As per https://bugzilla.mozilla.org/show_bug.cgi?id=992100#c61:
'rooting the phone and replacing the preinstalled font with a better one' should be an alternative here.

Comment 12

[Jonathan Kew [:jfkthame]]

(In reply to Chandrakant Dhutadmal from comment #8)
> Created attachment 8424703 [details]
> MarathiFennecWebPage.png
> 
> @Sandeep.
> 
> 1. Please refer to the fonts kept as an attachment (named
> SamsungDevanagariFile.zip) with the bug 
> 
> https://bugzilla.mozilla.org/show_bug.cgi?id=992100. This font is having the
> shape of required character "U+0972;ॲ;DEVANAGARI LETTER CHANDRA A". 

Actually, that font does *not* have U+0972. I'll attach a fontforge screenshot showing the Devanagari character repertoire it supports...

> 
> 2. In the same device for which i have reported this bug, i can see the
> character properly displayed in the webpage text. Refer snapshot attached
> with this bug (named MarathiFennecWebPage.png. 
> 

AFAIK, this is not webpage text, it is a Firefox UI page that is rendered by the Java front-end code.

There must be another font on the device that has U+0972, but is not usable in actual web content (like the Google search results shown in attachment 8426047 [details]), probably because it lacks the OpenType tables for proper Devanagari rendering.

To be more confident of what's happening, I'd need to examine the collection of fonts from /system/fonts on the problem device.

Comment 13

[Jonathan Kew [:jfkthame]]

Attachment: Devanagari characters supported in the SamsungDevanagari font

Comment 14

[Chandrakant Dhutadmal]

@ Jonathan- 

Strange!. My device has no other fonts installed apart from the ones available by default. And for Devnagari, there is only one font SamsungDevanagari.ttf version 1.00.

@ Parag- Could you let us know specification of your device and the font name ( and version of the font) installed on that device. Also can you check if "U+0972;ॲ;DEVANAGARI LETTER CHANDRA A" is in the font your system is using ?

Comment 15

[Jonathan Kew [:jfkthame]]

I looked at all the fonts in the zip file Jeff Beatty provided (bug 992100 comment 38), and AFACT *none* of them support the character U+0972.

On that device, therefore, I don't see how U+0972 could render properly in any context, unless a webpage provides its own Devanagari font via @font-face, or an additional font is installed in the user profile (e.g. using a font add-on).

Chandrakant, does the collection of fonts on your device (which displays U+0972 in attachment 8424703 [details], right?) match Jeff's from bug 992100, or are there any differences?

Comment 16

[Chandrakant Dhutadmal]

@ Jonathan- 

Yes. The set of 64 fonts is what i have in my device as well. I have compared with the set given to you by Jeff.

Comment 17

[sandeep shedmake]

As per https://bugzilla.mozilla.org/show_bug.cgi?id=1012622#c6 ,

The fact that it renders correctly within web pages but not in the Firefox UI shows that it's a problem with the Java side of the app, and not with Gecko itself.

Comment 18

[Jonathan Kew [:jfkthame]]

(In reply to sandeep shedmake from comment #17)
> As per https://bugzilla.mozilla.org/show_bug.cgi?id=1012622#c6 ,
> 
> The fact that it renders correctly within web pages but not in the Firefox
> UI shows that it's a problem with the Java side of the app, and not with
> Gecko itself.

In this case, it seems to be the opposite situation: the character renders within the UI but not in web content. See comment 12.

(In reply to Chandrakant Dhutadmal from comment #16)
> @ Jonathan- 
> 
> Yes. The set of 64 fonts is what i have in my device as well. I have
> compared with the set given to you by Jeff.

Do the file sizes exactly match (not just the list of files), or could any of them be different versions?

Comment 19

[sandeep shedmake]

(In reply to Jonathan Kew (:jfkthame) from comment #18)
> (In reply to sandeep shedmake from comment #17)
> > As per https://bugzilla.mozilla.org/show_bug.cgi?id=1012622#c6 ,
> > 
> > The fact that it renders correctly within web pages but not in the Firefox
> > UI shows that it's a problem with the Java side of the app, and not with
> > Gecko itself.
> 


Actually, there are two use-cases of occurance of ॲ (U+0972) mentioned in this bugzilla.


Use case 1: when ॲ (U+0972) is part of MenuItem and is a leading glyph

Screenshot of Improper rendering of ॲ (U+0972) --> https://bugzilla.mozilla.org/attachment.cgi?id=8424596



> In this case, it seems to be the opposite situation: the character renders
> within the UI but not in web content. See comment 12.
> 


Use case 2: when ॲ (U+0972) is part of Web Content

Screenshot of correct rendering of ॲ (U+0972) --> https://bugzilla.mozilla.org/attachment.cgi?id=8424703

Comment 20

[Chandrakant Dhutadmal]

@ Sandeep, Jonathan-

Yes. As Sandeep has mentioned, we are concerned about Use case 1: when ॲ (U+0972) is part of MenuItem and is a leading glyph.


@ Jonathan-

Yes. The version and file sizes of all the fonts in the set and in my device is same (I extracted the set of fonts once again, before putting this comment ! :) ). 

BTW, to be doubly sure, is there a tool which can compare two font files and its versions ? I would like to use the same as well.

Comment 21

[Jonathan Kew [:jfkthame]]

(In reply to Chandrakant Dhutadmal from comment #20)

> BTW, to be doubly sure, is there a tool which can compare two font files and
> its versions ? I would like to use the same as well.

Simplest would be to do a binary compare of the files (e.g. with "diff -b" on Linux or OS X; on Windows, try "fc /b"). They should normally be identical, if the exact same fonts are being shipped on the devices.

(If the files are -not- binary-identical, then it becomes a bit harder to find out whether the differences are actually significant. You can't rely on version numbers - they don't always get updated appropriately. Really obvious differences may show up if you view the fonts with FontForge, but it's hard to spot minor things that way. You can use TTX to dump the fonts to a textual "source" format and then use diff to find the differences between them at that level.)

Comment 22

[Jonathan Kew [:jfkthame]]

Attachment: 0972-test.html

Chandrakant: could you let me know whether this simplified testcase renders the ॲ character successfully for you? Thanks.

Comment 23

[Jonathan Kew [:jfkthame]]

Attachment: 0972-test.html

Just fixed the testcase to be slightly more robust - sorry for the noise.

Comment 24

[Chandrakant Dhutadmal]

Attachment: Simplfied test result

(In reply to Jonathan Kew (:jfkthame) from comment #22 and comment #23)

>> Chandrakant: could you let me know whether this simplified testcase renders >> the ॲ character successfully for you? Thanks.


@ Jonathan - No. The test did not produce the correct results as expected. I am attaching the snapshot of the result of the test case with this comment.

Comment 25

[Chandrakant Dhutadmal]

Attachment: File Comparison for fonts

(In reply to Jonathan Kew (:jfkthame) from comment #21)

Jonathan- There is no difference between the fonts on Binary level. I am attaching the result screenshot with this comment. I have used fc /b command on Windows to find the result.

Comment 26

[Jonathan Kew [:jfkthame]]

(In reply to Chandrakant Dhutadmal from comment #24)
> Created attachment 8428755 [details]
> Simplfied test result
> 
> (In reply to Jonathan Kew (:jfkthame) from comment #22 and comment #23)
> 
> >> Chandrakant: could you let me know whether this simplified testcase renders >> the ॲ character successfully for you? Thanks.
> 
> 
> @ Jonathan - No. The test did not produce the correct results as expected. I
> am attaching the snapshot of the result of the test case with this comment.

OK. How about if you now open a new tab (or touch the location bar) and bring up the History list - does the page title show the character correctly (it should read "Test for ॲ (U+0972)"), or do you get a box there as well?

Comment 27

[Jonathan Kew [:jfkthame]]

(In reply to Chandrakant Dhutadmal from comment #25)
> Created attachment 8428762 [details]
> File Comparison for fonts
> 
> (In reply to Jonathan Kew (:jfkthame) from comment #21)
> 
> Jonathan- There is no difference between the fonts on Binary level. I am
> attaching the result screenshot with this comment. I have used fc /b command
> on Windows to find the result.

You checked that there are "No differences found" for all 64 files? (The screenshot doesn't include them all, as lots of them would have scrolled away already.) I'm especially concerned to check SamsungDevangari.ttf and DroidSansFallback.ttf, though -any- difference could be interesting.

Comment 28

[Chandrakant Dhutadmal]

(In reply to Jonathan Kew (:jfkthame) from comment #26)
> (In reply to Chandrakant Dhutadmal from comment #24)
> > Created attachment 8428755 [details]
> > Simplfied test result
> > 
> > (In reply to Jonathan Kew (:jfkthame) from comment #22 and comment #23)
> > 
> > >> Chandrakant: could you let me know whether this simplified testcase renders >> the ॲ character successfully for you? Thanks.
> > 
> > 
> > @ Jonathan - No. The test did not produce the correct results as expected. I
> > am attaching the snapshot of the result of the test case with this comment.
> 
> OK. How about if you now open a new tab (or touch the location bar) and
> bring up the History list - does the page title show the character correctly
> (it should read "Test for ॲ (U+0972)"), or do you get a box there as well?

No. It is showing the box there as well.

Comment 29

[Chandrakant Dhutadmal]

(In reply to Jonathan Kew (:jfkthame) from comment #27)
> (In reply to Chandrakant Dhutadmal from comment #25)
> > Created attachment 8428762 [details]
> > File Comparison for fonts
> > 
> > (In reply to Jonathan Kew (:jfkthame) from comment #21)
> > 
> > Jonathan- There is no difference between the fonts on Binary level. I am
> > attaching the result screenshot with this comment. I have used fc /b command
> > on Windows to find the result.
> 
> You checked that there are "No differences found" for all 64 files? (The
> screenshot doesn't include them all, as lots of them would have scrolled
> away already.) I'm especially concerned to check SamsungDevangari.ttf and
> DroidSansFallback.ttf, though -any- difference could be interesting.

Yes. I checked for all 64 files. There are no differences between any of them. Screenshot was just to showcase sample results. It is same for all files.

Comment 30

[Jonathan Kew [:jfkthame]]

(In reply to Chandrakant Dhutadmal from comment #29)
> Yes. I checked for all 64 files. There are no differences between any of
> them. Screenshot was just to showcase sample results. It is same for all
> files.

OK, thanks for confirming.

So what I'm still unable to understand is where the glyph shown in your attachment 8424703 [details] comes from, as it doesn't appear to exist in any of those fonts.

Comment 31

[Chandrakant Dhutadmal]

Attachment: FileComparisonDEV-DROID.png

@Jonathan- Check the results for SamsungDevnagari.ttf and DroidSansFallback.ttf attached.

Comment 32

[Jonathan Kew [:jfkthame]]

Attachment: 0972-test.html

Aha - I just realized something that might have been going on here.

Here's a further testcase; how does this one look? And how does its title show up in the History?

Comment 33

[Chandrakant Dhutadmal]

(In reply to Jonathan Kew (:jfkthame) from comment #32)
> Created attachment 8428830 [details]
> 0972-test.html
> 
> Aha - I just realized something that might have been going on here.
> 
> Here's a further testcase; how does this one look? And how does its title
> show up in the History?

Yes. Now the test is successful. I see the character properly in second (fake) case. Even the title of the page, when seen in the history list is properly seen. What did u do?

Comment 34

[Jonathan Kew [:jfkthame]]

(In reply to Chandrakant Dhutadmal from comment #33)
> Yes. Now the test is successful. I see the character properly in second
> (fake) case. Even the title of the page, when seen in the history list is
> properly seen. What did u do?

OK, I suspect that's also how it worked in attachment 8424703 [details].

The secret is that it is NOT really U+0972; instead, I used the sequence U+0905 U+0945 (i.e. the Devanagari letter A, अ, plus the vowel sign CANDRA E, ॅ). This is really an incorrect spelling - it is not defined as a canonically-equivalent sequence in Unicode - but is more likely to be supported by fonts, as U+0972 is a more recent addition to the standard (added in Unicode 5.1).

So this really is a Samsung font problem: their SamsungDevanagari font lacks the newer Devanagari characters such as U+0972.

Comment 35

[Chandrakant Dhutadmal]

Does this mean all the users using Samsung devices will be deprived of having a clean GUI of Mozilla firefox?. Can Mozilla not think of any other mechanism whereby maximum number of users are satisfied with the experience?. Recently mozilla has implemented its own mechanism of changing the interface language instead of depending on the OS mechanism.

Comment 36

[sandeep shedmake]

(In reply to Jonathan Kew (:jfkthame) from comment #34)
> (In reply to Chandrakant Dhutadmal from comment #33)
> > Yes. Now the test is successful. I see the character properly in second
> > (fake) case. Even the title of the page, when seen in the history list is
> > properly seen. What did u do?
> 
> OK, I suspect that's also how it worked in attachment 8424703 [details].
> 
> The secret is that it is NOT really U+0972; instead, I used the sequence
> U+0905 U+0945 (i.e. the Devanagari letter A, अ, plus the vowel sign CANDRA
> E, ॅ). This is really an incorrect spelling - it is not defined as a
> canonically-equivalent sequence in Unicode - but is more likely to be
> supported by fonts, as U+0972 is a more recent addition to the standard
> (added in Unicode 5.1).
> 

Agreed. 

Fixed [1] 17 instances of incorrect spelling in 'mr' Fennec localizations at http://mozilla.locamotion.org/mr/mobile/mobile/ as ॲ = अ + ॅ are not canonically-equivalent sequence defined in Unicode.

[1] http://pastebin.mozilla.org/5265963

> So this really is a Samsung font problem: their SamsungDevanagari font lacks
> the newer Devanagari characters such as U+0972.

True.

Comment 37

[sandeep shedmake]

(In reply to Chandrakant Dhutadmal from comment #35)
> Does this mean all the users using Samsung devices will be deprived of
> having a clean GUI of Mozilla firefox?. 

Samsung Galaxy S4 device should incorporate the missing glyphs here. 


Using latest Unicode version compliant font is one other solution.

Just came across https://bugzilla.mozilla.org/show_bug.cgi?id=648548 [provide a UI to allow users to download fonts that are missing from their system]

Comment 38

[Chandrakant Dhutadmal]

(In reply to sandeep shedmake from comment #37)
> (In reply to Chandrakant Dhutadmal from comment #35)
> > Does this mean all the users using Samsung devices will be deprived of
> > having a clean GUI of Mozilla firefox?. 
> 
> Samsung Galaxy S4 device should incorporate the missing glyphs here. 
> 
> 
> Using latest Unicode version compliant font is one other solution.
> 
> Just came across https://bugzilla.mozilla.org/show_bug.cgi?id=648548
> [provide a UI to allow users to download fonts that are missing from their
> system]

Unfortunately not many users are aware of how to use another font containing proper glyphs. All stuff including rooting the device, renaming and replacing the font to original font name, etc are too technical for an end user. Even developers dont wish to do the same unless otherwise it is a testing device. 

Is there a way to bring this issue to the notice of samsung and request them formally to get it fixed

Comment 39

[Jeff Beatty [:gueroJeff]]

(In reply to Chandrakant Dhutadmal from comment #38)
> (In reply to sandeep shedmake from comment #37)
> > (In reply to Chandrakant Dhutadmal from comment #35)
> > > Does this mean all the users using Samsung devices will be deprived of
> > > having a clean GUI of Mozilla firefox?. 
> > 
> > Samsung Galaxy S4 device should incorporate the missing glyphs here. 
> > 
> > 
> > Using latest Unicode version compliant font is one other solution.
> > 
> > Just came across https://bugzilla.mozilla.org/show_bug.cgi?id=648548
> > [provide a UI to allow users to download fonts that are missing from their
> > system]
> 
> Unfortunately not many users are aware of how to use another font containing
> proper glyphs. All stuff including rooting the device, renaming and
> replacing the font to original font name, etc are too technical for an end
> user. Even developers dont wish to do the same unless otherwise it is a
> testing device. 
> 
> Is there a way to bring this issue to the notice of samsung and request them
> formally to get it fixed

Samsung claims support for Marathi in both http://bit.ly/1wjIKpv and http://bit.ly/1kmJGEW . If the character set/font issue exists in these devices, I'm certain that it will be resolved shortly. Unfortunately, we are still investigating the impact of delivering fonts with Firefox for Android. It's a difficult desicion to make as well; backwards compatibility in this area may not be worth the effort because of how rapidly development for Android devices moves (in other words, the fix could be implemented across the majority of Samsung devices before we come up with one).

My recommendation would be to continue to monitor this impact within Samsung devices. At this time, it seems there's not much more that can be done beyond what Jonathan has provided (Jonathan, you're welcome to correct me).

Comment 40

[sandeep shedmake]

(In reply to sandeep shedmake from comment #36)
> (In reply to Jonathan Kew (:jfkthame) from comment #34)
> > (In reply to Chandrakant Dhutadmal from comment #33)
> > > Yes. Now the test is successful. I see the character properly in second
> > > (fake) case. Even the title of the page, when seen in the history list is
> > > properly seen. What did u do?
> > 
> > OK, I suspect that's also how it worked in attachment 8424703 [details].
> > 
> > The secret is that it is NOT really U+0972; instead, I used the sequence
> > U+0905 U+0945 (i.e. the Devanagari letter A, अ, plus the vowel sign CANDRA
> > E, ॅ). This is really an incorrect spelling - it is not defined as a
> > canonically-equivalent sequence in Unicode - but is more likely to be
> > supported by fonts, as U+0972 is a more recent addition to the standard
> > (added in Unicode 5.1).
> > 
> 
> Agreed. 
> 
> Fixed [1] 17 instances of incorrect spelling in 'mr' Fennec localizations at
> http://mozilla.locamotion.org/mr/mobile/mobile/ as ॲ = अ + ॅ are not
> canonically-equivalent sequence defined in Unicode.
> 

[1] http://pastebin.mozilla.org/5278649  [REVISED]

Please, disregard the previous URL: http://pastebin.mozilla.org/5265963 .



From Comment 19,


@Chandrakant, please confirm the following:

With latest nightly http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-l10n/fennec-31.0a2.mr.android-arm.apk (wherein ॲ = अ + ॅ incorrect spelling are removed);

'Use case 2: when ॲ (U+0972) is part of Web Content' should now FAIL on your Samsung Galaxy S4 device.

Comment 41

[sandeep shedmake]

From Comment 19 and Comment 40,


Chandrakant,

With latest [Modified on 29-May-2014] nightly fennec-31.0a2.mr.android-arm.apk, attachment 8424703 [details] should now reveal empty square box as well.

Please test and verify it.


Thanks.

Comment 42

[Chandrakant Dhutadmal]

Attachment: CorrectWord.png

@ Sandeep.

Please check the attachment. The word is appearing correctly on web page. This is captured on firefox 31.0a2 (201-05-29).

Comment 43

[Brad Lassey [:blassey] (use needinfo?)]

So is this a problem with the localized string? or the font rendering? Who should the assignee be?

Comment 44

[Jonathan Kew [:jfkthame]]

(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #43)
> So is this a problem with the localized string? or the font rendering? Who
> should the assignee be?

This is a problem with the SamsungDevanagari font on some Galaxy devices (at least); it shows up with some of the localized strings in the Marathi localization, because Marathi uses the particular letter that is missing from the Samsung font.

The correct fix, then, is for Samsung to ship a non-broken Devanagari font, either by fixing theirs or by removing it and using Noto or Lohit or something. Do we have contacts at Samsung who can make this happen?

Failing that, it would be possible (but ugly) to work around the issue in the Marathi localization by deliberately mis-spelling words that should contain U+0972, replacing it with <U+0905, U+0945>. However, this would not fix the problem for general Web content in Marathi that might still use the (correct) U+0972 character. (It's also possible that the <U+0905, U+0945> combination might render *worse* with some fonts/devices.)

(I notice that there is existing content that uses this same workaround, e.g. on mr.wikipedia.org; this probably began before U+0972 was encoded (Unicode 5.1), or at least was widely supported in software. The practice has been questioned there, too: http://mr.wikipedia.org/wiki/%E0%A4%B5%E0%A4%BF%E0%A4%95%E0%A4%BF%E0%A4%AA%E0%A5%80%E0%A4%A1%E0%A4%BF%E0%A4%AF%E0%A4%BE_%E0%A4%9A%E0%A4%B0%E0%A5%8D%E0%A4%9A%E0%A4%BE:%E0%A4%A8%E0%A4%BF%E0%A4%B0%E0%A5%8D%E0%A4%B5%E0%A4%BE%E0%A4%B9/%E0%A4%AE%E0%A4%BF%E0%A4%A1%E0%A4%BF%E0%A4%AF%E0%A4%BE%E0%A4%B5%E0%A4%BF%E0%A4%95%E0%A4%BF_%E0%A4%A8%E0%A4%BE%E0%A4%AE%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%8D%E0%A4%B5#wikipedia_don.27t_follow_Marathi_Standard_.3F.)

On the Gecko or Firefox side, a possible workaround would be to ship a better Devanagari font ourselves, and blacklist the Samsung font - which seems like rather a large sledgehammer when the problem is a single letter, not the entire script.

Or we could possibly add a hack to the text code at some level so that we'd detect this particular case and render unsupported U+0972 using the U+0905,0945 sequence instead. I find that idea pretty distasteful - an ugly hack in Gecko code to work around a specific bad font from a specific vendor - but perhaps we should give it a try. I suspect there are a lot of these devices out there, and even if an update becomes available for them, it'll take a long time to filter out to all the affected users.

Not sure that directly answers your question, but I hope it explains the mess somewhat!

Comment 45

[Chandrakant Dhutadmal]

(In reply to Jonathan Kew (:jfkthame) from comment #44)
> (In reply to Brad Lassey [:blassey] (use needinfo?) from comment #43)
> > So is this a problem with the localized string? or the font rendering? Who
> > should the assignee be?
> 
> This is a problem with the SamsungDevanagari font on some Galaxy devices (at
> least); it shows up with some of the localized strings in the Marathi
> localization, because Marathi uses the particular letter that is missing
> from the Samsung font.
> 
> The correct fix, then, is for Samsung to ship a non-broken Devanagari font,
> either by fixing theirs or by removing it and using Noto or Lohit or
> something. Do we have contacts at Samsung who can make this happen?
> 
> Failing that, it would be possible (but ugly) to work around the issue in
> the Marathi localization by deliberately mis-spelling words that should
> contain U+0972, replacing it with <U+0905, U+0945>. However, this would not
> fix the problem for general Web content in Marathi that might still use the
> (correct) U+0972 character. (It's also possible that the <U+0905, U+0945>
> combination might render *worse* with some fonts/devices.)
> 
> (I notice that there is existing content that uses this same workaround,
> e.g. on mr.wikipedia.org; this probably began before U+0972 was encoded
> (Unicode 5.1), or at least was widely supported in software. The practice
> has been questioned there, too:
> http://mr.wikipedia.org/wiki/
> %E0%A4%B5%E0%A4%BF%E0%A4%95%E0%A4%BF%E0%A4%AA%E0%A5%80%E0%A4%A1%E0%A4%BF%E0%A
> 4%AF%E0%A4%BE_%E0%A4%9A%E0%A4%B0%E0%A5%8D%E0%A4%9A%E0%A4%BE:
> %E0%A4%A8%E0%A4%BF%E0%A4%B0%E0%A5%8D%E0%A4%B5%E0%A4%BE%E0%A4%B9/
> %E0%A4%AE%E0%A4%BF%E0%A4%A1%E0%A4%BF%E0%A4%AF%E0%A4%BE%E0%A4%B5%E0%A4%BF%E0%A
> 4%95%E0%A4%BF_%E0%A4%A8%E0%A4%BE%E0%A4%AE%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%8D
> %E0%A4%B5#wikipedia_don.27t_follow_Marathi_Standard_.3F.)
> 
> On the Gecko or Firefox side, a possible workaround would be to ship a
> better Devanagari font ourselves, and blacklist the Samsung font - which
> seems like rather a large sledgehammer when the problem is a single letter,
> not the entire script.
> 
> Or we could possibly add a hack to the text code at some level so that we'd
> detect this particular case and render unsupported U+0972 using the
> U+0905,0945 sequence instead. I find that idea pretty distasteful - an ugly
> hack in Gecko code to work around a specific bad font from a specific vendor
> - but perhaps we should give it a try. I suspect there are a lot of these
> devices out there, and even if an update becomes available for them, it'll
> take a long time to filter out to all the affected users.
> 
> Not sure that directly answers your question, but I hope it explains the
> mess somewhat!

I was testing fennec on various other devices (tablets) which uses Android OS. I noticed that the problem there is more grave than Samsung device(s). The fonts therein are totally messed up and the GUI and other text was looking horrible. So your idea of shipping the best Devanagari font with the browser itself seems more reasonable as it will not have dependency on any device's fonts.

Comment 46

[Jonathan Kew [:jfkthame]]

(In reply to Chandrakant Dhutadmal from comment #45)
> I was testing fennec on various other devices (tablets) which uses Android
> OS. I noticed that the problem there is more grave than Samsung device(s).
> The fonts therein are totally messed up and the GUI and other text was
> looking horrible. So your idea of shipping the best Devanagari font with the
> browser itself seems more reasonable as it will not have dependency on any
> device's fonts.

Please file a separate report about this, with details of the devices, Android versions involved, and screenshots to show what's happening. Let's keep this bug focused on the specific issue with the U+0972 character.

Comment 47

[Jonathan Kew [:jfkthame]]

Attachment: workaround for Devanagari fonts that lack support for U+0972 on Android.

This is an ugly hack, but AFAICT it solves the problem for Marathi on these Samsung devices, and should be harmless elsewhere - and it's no worse than various other font- or platform-specific hacks we've done in the past. In principle, we could target it more narrowly (e.g. checking the font name), but I don't think it's worth the extra code or runtime tests involved.

Comment 48

[Jonathan Kew [:jfkthame]]

Tryserver job: https://tbpl.mozilla.org/?tree=Try&rev=c0022fa906d3

Chandrakant: if you could test this build (install Fennec from http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-c0022fa906d3/try-android/fennec-32.0a1.en-US.android-arm.apk) and confirm whether it solves the problem with ॲ on your Samsung device, that would be really helpful - thanks. I think it should work even with the testcase in attachment 8428737 [details] here.

Comment 49

[sandeep shedmake]

Attachment: suggestions with [U+0972] - without decomposition

Comment 50

[sandeep shedmake]

Attachment: suggestions with [U+0905][U+0945] - with decomposition

Comment 51

[sandeep shedmake]

Decomposition of [U+0972] as [U+0905][U+0945] would impact:

(1) Localized search (see attachment 8431295 [details] and attachment 8431297 [details])

(2) Backspacing of U+0972 (in decompose format) won't happen in single go


Thereby, (1) and (2) could possibly lead to data loss.

Comment 52

[John Daggett (:jtd)]

Probably a good idea to log this or somehow report this to Samsung so that the underlying problem gets fixed.

Comment 53

[John Daggett (:jtd)]

Comment on attachment 8431148 [details] [diff] [review]
workaround for Devanagari fonts that lack support for U+0972 on Android.

Yeah, ick, but can't be helped I guess.

Comment 54

[Jonathan Kew [:jfkthame]]

Chandrakant: could you please try the build from comment 48 and confirm whether it makes pages with U+0972 display properly (such as attachment 8428737 [details], or the title and comment at https://github.com/anubhav-chattoraj/indic-tools/issues/9) on your Samsung device?

Thanks!

Comment 55

[Chandrakant Dhutadmal]

Attachment: TestCase1 .png

(In reply to Jonathan Kew (:jfkthame) from comment #54)
> Chandrakant: could you please try the build from comment 48 and confirm
> whether it makes pages with U+0972 display properly (such as attachment
> 8428737 [details], or the title and comment at
> https://github.com/anubhav-chattoraj/indic-tools/issues/9) on your Samsung
> device?
> 
> Thanks!

I am extremely sorry for not being able to respond quickly on this. Was busy in group Annual meet for last 3 days. :) 

I tried both these tests on (http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-c0022fa906d3/try-android/fennec-32.0a1.en-US.android-arm.apk) build. In both the test cases, the address bar is showing Square box, whereas the webpage displays it correctly. I am attaching the screenshots for both these test cases (TestCase1.png and TestCase2.png).

Comment 56

[Chandrakant Dhutadmal]

Attachment: TestCase2.png

(In reply to Jonathan Kew (:jfkthame) from comment #54)
> Chandrakant: could you please try the build from comment 48 and confirm
> whether it makes pages with U+0972 display properly (such as attachment
> 8428737 [details], or the title and comment at
> https://github.com/anubhav-chattoraj/indic-tools/issues/9) on your Samsung
> device?
> 
> Thanks!

I didn't know how to add multiple screenshots in the same response, hence attaching second test case screenshot seperately.(TestCase2.png).

Comment 57

[Jonathan Kew [:jfkthame]]

(In reply to Chandrakant Dhutadmal from comment #55)
> I am extremely sorry for not being able to respond quickly on this. Was busy
> in group Annual meet for last 3 days. :) 

No problem - thanks for all your patience and testing here!

> I tried both these tests on
> (http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-
> c0022fa906d3/try-android/fennec-32.0a1.en-US.android-arm.apk) build. In both
> the test cases, the address bar is showing Square box, whereas the webpage
> displays it correctly.

Yes, this is expected: the address bar is not rendered by Gecko, but by the Java front-end code, so the workaround for Gecko to display the character won't help there. Sorry, but that really needs to be fixed by a Samsung update; it's out of our hands.

(In reply to John Daggett (:jtd) from comment #52)
> Probably a good idea to log this or somehow report this to Samsung so that
> the underlying problem gets fixed.

Indeed. :blassey, do you know if we have any kind of channel to Samsung for this?

Comment 58

[Jonathan Kew [:jfkthame]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/52e25aa9d032

Comment 59

[Daniel Glazman (:glazou) (not active in Mozilla any more)]

I have pinged my peers at Samsung HQ in Korea about this bug. Stay tuned.

Comment 60

[Parag Nemade]

Harcoding characters in sourcecode just to match rendering for one single font is not a good idea. It would have been better if someone could catch Samsung guys and tried to get font updated to fix the reported issue in this bug.

Comment 61

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/52e25aa9d032

Comment 62

[John Daggett (:jtd)]

Font bug details passed along to Samsung via Daniel Glazman.

Comment 63

[Jonathan Kew [:jfkthame]]

(In reply to Parag Nemade from comment #60)
> Harcoding characters in sourcecode just to match rendering for one single
> font is not a good idea. It would have been better if someone could catch
> Samsung guys and tried to get font updated to fix the reported issue in this
> bug.

Definitely. I hope that we'll see a fix from Samsung, and then be able to revert this hack at some point. But for now, this provides a solution that works with the existing devices in users' hands.

Comment 64

[Jonathan Kew [:jfkthame]]

Comment on attachment 8431148 [details] [diff] [review]
workaround for Devanagari fonts that lack support for U+0972 on Android.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bad Devanagari font on Samsung devices
User impact if declined: missing character in Marathi-language text
Testing completed (on m-c, etc.): tested with try builds, landed on m-c
Risk to taking this patch (and alternatives if risky): very low - android-only patch, narrowly targeted to affect only a single Devanagari letter
String or IDL/UUID changes made by this patch: none

Comment 65

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/e54d37bf5c5f

Comment 66

[Catalin Suciu [:csuciu]]

Verified as fixed