42893 - Add UTF-16 (BE & LE) to composer charset menu

Status: RESOLVED FIXED

(Core :: Internationalization, defect, P4)

Description

[bobj]

We should allow users to create UTF-16 documents.  The converters
are already there, so do we just need to hook up the menu?

Comment 1

[Frank Tang]

reassign to cata

Comment 2

[bobj]

This should also be in the File|Save As Charset... menu.

Comment 3

[Frank Tang]

Mark as NEED MORE INFO.  Adding a menu is very simple, but there is risk that
the general result may not be correct.

Comment 4

[bobj]

I don't know what info is being requested.  Reassign back to ftang.


Date: Thu, 10 Aug 2000 17:23:59 -0700
From: Bob Jung <[e-mail]>
To: Frank Tang <[e-mail]>
Subject: Re: Your Bugzilla buglist needs attention.


Seems clear to me.  The suggestion is to provide a mechanism to save UTF16BE
and/or UTF16LE. 
What info do you want? 

-bob 

Comment 5

[Katsuhiko Momoi]

Can you provide some scenarios for what kind of users or situations 
UTF-16 would be useful? I drew a blank when I thought this issue except
that some Windows applications like Notepad can save into UCS-2.

Comment 6

[bobj]

That's the scenario I was thinking about.

Even if you do not create new utf-16 .txt files, we will need these menu items
if we want to edit these type of .txt files in Composer.

Window apps (e.g., notepad on W2K) can create utf-16 .txt files.

Comment 7

[hirata masakazu]

IMHO, there is a problem with current support of UTF-16 in Mozilla.
It maybe minor, yet I would rather see it fixed before Mozilla encourage
creating documents in UTF-16.
See bug 55258, bug 56626 and bug 56630.

Comment 8

[Teruko Kobayashi]

Changed QA contact to [e-mail].

Comment 9

[Frank Tang]

reassign this to yokoyama since this is a composer issue.
I am not sure how important is this.
clear the previous status and keywords
Mark it as P4 for now.

Comment 10

[Roy Yokoyama]

Updating the target milestone.

Comment 11

[nhottanscp]

Reassign to jbetak.

Comment 12

[[e-mail] (away - not reading bugmail)]

I temporarily enabled UTF-16BE and UTF-16LE in my build and verified that all 
the charset menus are still drawing from the same RDF source (rdf:charset-menu).

Depending on the exact location of these two items, we might need to do some 
additional work to separate the composer menu from other menus. Alternatively, 
would it be acceptable, if both items appeared on all charset menus (browser, 
mail, editor)?

I'm tentatively using "Unicode (UTF-16LE)" and "Unicode (UTF-16BE)". If these 
are my working titles, what should be the official menu item names?

Comment 13

[bobj]

I thought the res\charsetData.properties file was supposed to provide some
control over where the charsets were used.

In charsetData.properties there are
        *.notForBrowser = true
entries which I assume will prevent them from appearing the the browser.
Take a look at EditorSaveAsCharset.js

Comment 14

[bobj]

"Unicode (UTF-16LE)" and "Unicode (UTF-16BE)" is consistent with the
exising naming scheme, so sounds good to me.

Comment 15

[[e-mail] (away - not reading bugmail)]

Bob,

thanks so much for your feedback. I'll keep the titles then. I also gave the 
menus a second look and it continues to be my opinion that we are currently not 
to discriminate between the different charset menus. Adding the two new items 
is a matter of deleting their "notForBrowser" property from 
charsetData.properties and assigning them a title in charsetTitles.properties. 

However, the "notForBrowser" property is currently really a "notForUI" 
attribute. The menus are all constructed separately, but draw from the same RDF 
source. Same applies to the "Customize" dialog and "Save as charset". They all 
depend on the "notForBrowser" property, which then applies to all - the 
browser, mail and composer.

To sum it up, the most finely grained control we have right now is the first 
menu tier, where we could manipulate entries in XUL. The string bundles and 
enumerators available from JavaScript do not differentiate between the 
callers / different application modules.

Comment 16

[Katsuhiko Momoi]

I am not sure if it is such a bad thing not to be able to
customize the menu items for Composer, Mail, or 
Browser. That would be too much of a detail to ask from
most users. Maybe te current status is a blessing in disguise.

Comment 17

[[e-mail] (away - not reading bugmail)]

Bob, Kat,

we just discussed this with Naoki and it appears that in case that these two 
new entries are not acceptable across the whole product, the most consistent 
and expedient way of solving this issue might be adding them to the "Save as 
charset" dialog only. 

We could set a new property in the pertinent string bundles and parse it out in 
EditorSaveAsCharset.js. An addition to the "Customize..." dialog would be 
conceivable, but would lead to the same dilemma as described above. Adding any 
item, would affect all the menus.

Comment 18

[bobj]

I agree that ideally users should not need these entries when opening a 
document because we should be able to determine the encoding from the BOM.

But how do we provide user feedback when editing a UTF-16 document?  Normally
we have a bullet/check next the the current document encoding when you
pull down the menu.  What will we do in the UTF-16 case, if these entries
don't exist?  Actually this is probably a flaw in our original *notForBrowser
feature...

Comment 19

[Katsuhiko Momoi]

Doesn't the non-Browser item appear anyway? I thought controlling the menu appearance and 
having an item appear as a cached item are 2 different things. In fact, we see names
like US-ASCII cached even though it is not in the menu.

Comment 20

[[e-mail] (away - not reading bugmail)]

>Doesn't the non-Browser item appear anyway?

could this qualify as a bug? I believe the ".notFor..." properties were 
supposed to suppress such items from appearing in the corresponding application 
UI altogether. They get registered, enumerated and so on,however they should be 
ignored by the UI-generating code.

As I already mentioned to Naoki, the converter test bed contains a reference to 
four properties: ".notForBrowser", ".notForComposer", ".notForMailView" 
and ".notForMailEdit". Only ".notForBrowser" is used in other places, the 
others have only this singular reference. Would this indicated that the 
original design might have intended more finely grained control over the UI? 

http://lxr.mozilla.org/seamonkey/source/intl/uconv/tests/TestUConv.cpp#415

Comment 21

[bobj]

> Would this indicated that the original design might have intended
> more finely grained control over the UI? 

Yes, that is my recollection.

Comment 22

[[e-mail] (away - not reading bugmail)]

should we push this out to 0.9.1? Changing the granularity of control make be 
too aggressive right now. I'd have an idea about this or that change to the 
nsCharsetMenu.cpp to allow for better control through the string bundles, I'd 
need more time and scrutiny though.

Bob might be right about the requirement to provide visual feedback through the 
charset menu. However, we will be not able to add these decoders in composer 
without affecting other parts of the product. If that should be an acceptable 
alternative for 0.9, I would do it. We could then file a new bug calling for 
more granular menu content control in 0.9.1...

Comment 23

[[e-mail] (away - not reading bugmail)]

moving to 0.9.1, I feel this will provide us with more time to address the UI 
granularity issues

Comment 24

[[e-mail] (away - not reading bugmail)]

marking dependency on bug 39780, we really need to get this thing going. I'm 
postponing the fix one last time...

Comment 25

[[e-mail] (away - not reading bugmail)]

ccing roy. Roy, I could take bug 39780 and fix it in conjunction with this one 
as briefly discussed a week or two ago.

Comment 26

[[e-mail] (away - not reading bugmail)]

UI granularity issues are also still outstanding. Currently, I'm leaning 
towards a solution, where both "UTF-16BE" and "UTF-16LE" are visible in the UI 
(.notForBrowser==false or .notForBrowser==null), but only in the dynamic part 
of the menu and not in the static section. These decoders would appear there 
either automatically by opening a document with the corresponding encoding 
(most likely in composer as noted before) or via customization. If desired, I 
could come up with a kludge to seal off the customization route.

Comment 27

[Frank Tang]

move to moz0.9.3 since it is not moz0.9.2 critical. 

Comment 28

[[e-mail] (away - not reading bugmail)]

(ftang using jbetak account):
move it to future. This is not critical for moz0.9.2

Comment 29

[Jungshik Shin]

REF: bug 97054: UTF-16 character coding support.

Recently I hit upon a UTF-16LE encoded web page which MS IE
can handle but Mozilla can't. This may add strength to the case
for enabling UTF-16LE/BE (now?).

Comment 30

[bobj]

This bug is for composer only.
For browsing, the BOM should be enough to determine if the page is BE or LE.
If we are not detecting the BOM, that is a different bug.  Please file a
bug with the URL of the page that fails.

Comment 31

[Frank Tang]

jbetak's contract is up. Bulk move bugs to ftang

Comment 32

[Jungshik Shin]

A couple of improvements have been made regarding UTF-16/UTF-32 (detection) over
the last two years. Can we remove '.notForBrowser' for UTF-16? Considering that
some broken web sites (Sony Europe?) declare that they're in UTF-8 via http
header but actually are in UTF-16,  it's not a bad idea to expose (at least in
Customize menu) UTF-16/UTF-32 for browser as well as for composer, is it? 

BTW, having forgotten comment #21, I proposed '.notForOutgoing' to hide some
MIME charsets for which we don't have encoders (bug 133615). 'notForOutgoing' is
for both mailedit and 'save as charset' in composer. 

Comment 33

[Jungshik Shin]

Taking. It should have been fixed a long time ago. 
Fix coming up.

Comment 34

[Jungshik Shin]

Attachment: patch (needs more work)

I just added another more menu for UTF's, but that's probably not a good idea.
Instead, we need a new 'static more' menu for UTF's (like 'Autodetect').
Besides, we have to split 'notForOutgoing' into 'notForMailEdit' and
'notForComposer' as originally planned because UTF-16/32 shouldn't be allowed
for mail but should be allowed for composer.

Comment 35

[Jungshik Shin]

adding Neil to ask for help.  

Comment 36

[[e-mail]]

Doesn't Mail Compose have its own unique character coding menu anyway?

Comment 37

[Jungshik Shin]

bug 55300 was fixed. I'm closing this as well. For a more robust blocking of
UTF-16*/32* in mail-composer (with new notForMailEdit and notForMailView), I'm
goona file a new bug.