369424 - Incorrect characters is entered from the keyboard when system code page is CP1125 or CP1131

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[Yuri Proniakin]

User-Agent:       Mozilla/5.0 (OS/2; U; Warp 4.5; ru-RU; rv:1.7.12) Gecko/20050922
Build Identifier: Mozilla/5.0 (OS/2; U; Warp 4.5; ru-RU; rv:1.7.12) Gecko/20050922

If system code page is CP1125 (Ukraine) or CP1131 (Belarus) then attempt to enter any national character from the keyboard to any input area (URL location bar, input fields in a HTML page, mail message composer, etc.) results in entering wrong letters (latin instead of cyrillics).
Also any national character in the window titlebar gets replaced with '?'.

This bug affects all versions of all applications.

Reproducible: Always

Steps to Reproduce:
1. Edit CONFIG.SYS lines to 'COUNTRY=380,C:\OS2\SYSTEM\COUNTRY.SYS', 'CODEPAGE=1125,850', and 'DEVINFO=KBD,UA,C:\OS2\KEYBOARD.DCP' (this is quite safe if you use English version of OS/2).
2. Reboot.
3. Run (for example) Seamonkey.
4. Place cursor into URL location bar.
5. Press Left Alt-Shift combination to activate Cyrillic keyboard layout.
6. Try to type any letters.
Actual Results:  
Extended latin letters are entered (from the CP850 code page)

Expected Results:  
Cyrillic letters crom the CP1125 code page should be entered

Comment 1

[Mike Kaply [:mkaply]]

For references:

http://www.borgendale.com/codepage/cp1125.gif
http://www.borgendale.com/codepage/cp1131.gif

We would need unicode mapping tables for these two codepages. I'm checking into it.

Comment 2

[Mike Kaply [:mkaply]]

Attachment: Beginning of a patch

This is the start of a patch.

Converters are done. Now need to do the OS/2 hookup.

Comment 3

[Mike Kaply [:mkaply]]

Wow. this actually worked. Patch with the last few changes coming.

Comment 4

[Mike Kaply [:mkaply]]

Attachment: Add OS/2 only encoders/decoders for 1125 and 1131

Comment 5

[Simon Montagu :smontagu]

Comment on attachment 254309 [details] [diff] [review]
Add OS/2 only encoders/decoders for 1125 and 1131

>Index: intl/uconv/src/charsetData.properties

>@@ -112,6 +116,8 @@
> ibm864.LangGroup                   = ar
> ibm866.LangGroup                   = x-cyrillic
> ibm869.LangGroup                   = el
>+ibm1125.LangGroup                  = x-central-euro
>+ibm1131.LangGroup                  = x-central-euro

These should be x-cyrillic. The rest looks good.

Comment 6

[Mike Kaply [:mkaply]]

Fix checked in to trunk.

Comment 7

[Dave Yeo]

Hi, attempting a trunk build of Seamonkey last night (early morning PST Feb 20) I now get this error
g++ -o nsCP1125ToUnicode.o -c  -DMOZILLA_INTERNAL_API -DOSTYPE=\"OS22.45\" -DOSARCH=\"OS2\" -DBUILD_ID=2007022003 -II:/mozilla/intl/uconv/ucvibm/../util -I../../../dist/include/xpcom -I../../../dist/include/uconv -I../../../dist/include   -I../../../dist/include/ucvibm -I../../../dist/include/nspr  -DMOZ_PNG_READ -DPNG_NO_MMX_CODE -DMOZ_PNG_WRITE  -I../../../dist/sdk/include       -fno-rtti -fno-exceptions -Wall -Wconversion -Wpointer-arith -Wcast-align -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wno-long-long -pedantic -Zomf -pipe  -DNDEBUG -DTRIMMED -O2 -s   -DMOZILLA_CLIENT -include ../../../mozilla-config.h -Uunix -U__unix -U__unix__ -Wp,-MD,.deps/nsCP1125ToUnicode.pp I:/mozilla/intl/uconv/ucvibm/nsCP1125ToUnicode.cpp
I:/mozilla/intl/uconv/ucvibm/nsCP1125ToUnicode.cpp:50: error: `ShiftCell' was 
   not declared in this scope
I:/mozilla/intl/uconv/ucvibm/nsCP1125ToUnicode.cpp: In function `nsresult 
   nsCP1125ToUnicodeConstructor(nsISupports*, const nsIID&, void**)':
I:/mozilla/intl/uconv/ucvibm/nsCP1125ToUnicode.cpp:60: error: `uShiftTable' 
   undeclared (first use this function)
I:/mozilla/intl/uconv/ucvibm/nsCP1125ToUnicode.cpp:60: error: (Each undeclared 
   identifier is reported only once for each function it appears in.)
I:/mozilla/intl/uconv/ucvibm/nsCP1125ToUnicode.cpp:60: error: parse error 
   before `)' token
make.exe[6]: *** [nsCP1125ToUnicode.o] Error 1
make.exe[6]: Leaving directory `I:/mozilla/obj-i386-pc-os2-emx/intl/uconv/ucvibm'
make.exe[5]: *** [libs] Error 2

Similar error in Firefox and Thunderbird.
Is this something messed up in my tree or a bad patch?

Comment 8

[Mike Kaply [:mkaply]]

My bad. That patch needed a little change for trunk.

Comment 9

[Walter Meinl]

Mike, after your check-in I see
D:/usr/src/mozilla/intl/uconv/ucvibm/nsCP1125ToUnicode.cpp
D:/usr/src/mozilla/intl/uconv/ucvibm/nsCP1125ToUnicode.cpp: In function `
   nsresult nsCP1125ToUnicodeConstructor(nsISupports*, const nsIID&, void**)':
D:/usr/src/mozilla/intl/uconv/ucvibm/nsCP1125ToUnicode.cpp:57: error: cannot
   convert `uScanClassID' to `const uMappingTable*' for argument `1' to `
   nsresult CreateOneByteDecoder(const uMappingTable*, nsISupports*, const
   nsIID&, void**)'
make.exe[6]: *** [nsCP1125ToUnicode.o] Error 1
make.exe[6]: Leaving directory `D:/mozbuild/intl/uconv/ucvibm'
make.exe[5]: *** [libs] Error 2
make.exe[5]: Leaving directory `D:/mozbuild/intl/uconv'
make.exe[4]: *** [libs] Error 2
make.exe[4]: Leaving directory `D:/mozbuild/intl'
make.exe[3]: *** [libs_tier_necko] Error 2
make.exe[3]: Leaving directory `D:/mozbuild'
make.exe[2]: *** [tier_necko] Error 2
make.exe[2]: Leaving directory `D:/mozbuild'
make.exe[1]: *** [default] Error 2
make.exe[1]: Leaving directory `D:/mozbuild'
make.exe: *** [build] Error 2

Comment 10

[Mike Kaply [:mkaply]]

Sorry. I'm doing this blind since my Os/2 machine is at work.

Try now.

Comment 11

[Walter Meinl]

Yep, it works now, I come across the intl subdir
(however since the checkins for bug #177805 the --enable-default-toolkit=os2 build breaks later during gfx compile, as non-thebes builds are not longer supported. But that's another story)

Comment 12

[Yuri Proniakin]

It seems, now all work as perfect as it is possible. But I have a comment (or report about omission):

Menu "View" -> "Character Encoding" -> "More Encodings" -> "East Europian" among other encodings currently contains "Cyrillic (KOI-8R)" (which, in fact, is strictly "Cyrillic/Russian"), "Cyrillic/Ukrainian (KOI-8U)", and "Cyrillic/Russian (CP866)", but no "Cyrillic/Ukrainian (CP1125)", and no "Cyrillic/Belarussian (CP1131)".

Comment 13

[Simon Montagu :smontagu]

This is by design. Unless there are webpages that use these encodings, we only need to support them internally, and adding them to the menu would just be cluttering up the UI.

Comment 14

[Yuri Proniakin]

I suppose this, but presence of CP866 is confusing - AFAIK, it is also not used in the web pages. OTOH, plain text documents frequently are available through links too, and they are either in CP866, or CP1125, because both are defined as a state standards.

Comment 15

[Mike Kaply [:mkaply]]

In my research, 1125 and 131 seemed to be OS/2 specific codepages. Where are you seeing this content on the web? Can you provide links?

Comment 16

[Yuri Proniakin]

Ukrainian codepage (known here as CP1125) was defined in the national standard "RST 2018-91" (valid since Jan 1, 1992). Belarussian codepage (CP1131) was defined in the national standard "STB 956-94" (valid since Jul 1, 1994).

Can't speak about Belarus, but in the Ukraine using of standard was mandatory for any organs of government, banks, and many other institutions. As a result, when nowadays I am searching for some info (technical, legal, chronological, etc.) quite often search engines point me to documents, which are in this codepage. Of course, they are not HTML pages but .TXT files; nevertheless, they are available in the Internet (as there: http://uib.cherkassy.net/zoo/DOC/STRU_PER.TXT).

On the other hand, I never saw web pages in CP866 as well, but CP866 presents in the Mozilla menu (I assume - because of a lot of text files on HTTP and FTP servers). If text files is not an important inducement, then you can safely remove CP866 from the menu (and thus reduce cluttering :-).