104934 - Convert Mozilla to Unicode app

Status: RESOLVED FIXED

(Core Graveyard :: Tracking, defect)

Description

[Frank Tang]

Comment 1

[Frank Tang]

yokoyama, can you do some research on this one ?

Comment 2

[Frank Tang]

any link to reference materila ?

Comment 3

[Roy Yokoyama]

I did some investigation back then (see 58866)
http://bugzilla.mozilla.org/attachment.cgi?id=55992&action=view
The above patch is to use ::GetOpenFileNameW() instead of
::GetOpenFileNameA()version in NT.  However, the patch doesn't work.  
Eventual call to TranslateMessage() from ::GetOpenFileNameW() changes 
all the unsupported locale chars to '?'.

I am changing the summary to  "Convert Mozilla to Unicode app"

Comment 4

[Frank Tang]

push it off to future

Comment 5

[Roy Yokoyama]

I would suggest to have a tinderbox column for "Win32-Unicode" for
Unicode enabled mozilla binary.

Comment 6

[Roy Yokoyama]

Attachment: Unicode text support in windows title and File Open/Save

Add build environment variable MOZ_UNICODE to enable this.
   (ex.  SET MOZ_UNICODE=1)
except I needed to hard code 
+#define MOZ_UNICODE
in nsprpub/pr/src/md/windows/w95io.c

Comment 7

[Roy Yokoyama]

cc [e-mail]

Comment 8

[Roy Yokoyama]

Attachment: nsWindowAPI.h

Copy this file to mozilla\widget\src\windows\

Comment 9

[Roy Yokoyama]

Attachment: Revised. Patch to include NSPR, XPCOM and Widget

With this patch, we can
- diplay non-locale text in the windows title
- open/save non-locale supported filenames

Comment 10

[Roy Yokoyama]

Attachment: latest backup

Comment 11

[Roy Yokoyama]

Attachment: test

Comment 12

[Roy Yokoyama]

Startup test results [tested in Windows XP, PIII, 0.5GB RAM]
a) Without Unicode (current build)
   - 1st run : Average startup time :   3.6927 secs (3 trials- 3.672 3.75 3.656)
   - 2nd run : Average startup time :   3.7133 secs (3 trials- 3.656 3.734 3.75)
   - 3rd run : Average startup time :   3.6930 secs (3 trials- 3.641 3.766 3.672)

b) With Unicode (with MOZ_UNICODE flag ON)
   - 1st run : Average startup time :   3.6720 secs (3 trials- 3.672 3.703 3.641)
   - 2nd run : Average startup time :   3.6357 secs (3 trials- 3.61 3.64 3.657)
   - 3rd run : Average startup time :   3.6460 secs (3 trials- 3.641 3.672 3.625)

PageLoad Test results [tested in Windows XP, PIII, 0.5GB RAM]
a) Without Unicode (current build)
   Test id: 3D875FC36B
   Avg. Median : 1269 msec		Minimum     : 312 msec
   Average     : 1278 msec		Maximum     : 3873 msec

b) With Unicode (with MOZ_UNICODE flag ON)
   Test id: 3D8759E3DA
   Avg. Median : 1262 msec		Minimum     : 319 msec
   Average     : 1269 msec		Maximum     : 3984 msec

Comment 13

[Roy Yokoyama]

Startup test results [tested in Windows 98, PIII, 0.5GB RAM]
a) Without Unicode (current build)
   - 1st run : Average startup time :   2.7833 secs (3 trials- 2.75 2.75 2.85)
   - 2nd run : Average startup time :   2.7700 secs (3 trials- 2.75 2.81 2.75)
   - 3rd run : Average startup time :   2.8000 secs (3 trials- 2.8 2.8 2.8)

b) With Unicode (with MOZ_UNICODE flag ON)
   - 1st run : Average startup time :   2.7633 secs (3 trials- 2.8 2.75 2.74)
   - 2nd run : Average startup time :   2.7833 secs (3 trials- 2.8 2.8 2.75)
   - 3rd run : Average startup time :   2.7500 secs (3 trials- 2.75 2.75 2.75)

PageLoad Test results [tested in Windows 98, PIII, 0.5GB RAM]
a) Without Unicode (current build)
   Test id: 3D87758625
   Avg. Median : 668 msec		Minimum     : 260 msec
   Average     : 693 msec		Maximum     : 2102 msec

b) With Unicode (with MOZ_UNICODE flag ON)
   Test id: 3D87AA0318
   Avg. Median : 666 msec		Minimum     : 260 msec
   Average     : 668 msec		Maximum     : 2039 msec

Comment 14

[Roy Yokoyama]

Attachment: turning MOZ_UNICODE flag on by default in widget module

shanjian: please review.
kin: please super review.

Comment 15

[Shanjian Li]

Comment on attachment 99885 [details] [diff] [review]
turning MOZ_UNICODE flag on by default in widget module

r=shanjian, now this is near real, :-)

Comment 16

[kinmoz]

Comment on attachment 99885 [details] [diff] [review]
turning MOZ_UNICODE flag on by default in widget module

If MOZ_UNICODE is only used in mozilla/widget/src/windows why does it have to
be defined in mozilla/widget/src/build/Makefile.in?

Comment 17

[Roy Yokoyama]

>If MOZ_UNICODE is only used in mozilla/widget/src/windows why does it have to
>be defined in mozilla/widget/src/build/Makefile.in?
I had the same question to myself before I post the patch.:)

It appears that MOZ_UNICODE is required in the /build/Makefile.in as well because 
NS_GENERIC_FACTORY_CONSTRUCTOR(nsWindow) in /build/nsWinWidgetFactory.cpp
and nsWindow class in /src/windows/nsWindow.h get out of sync and weird thing
happens if we don't define it in both places.

Does this make sense to you, kin?

Comment 18

[kinmoz]

Comment on attachment 99885 [details] [diff] [review]
turning MOZ_UNICODE flag on by default in widget module

sr=[e-mail]

Comment 19

[Roy Yokoyama]

checked into the trunk

Comment 20

[Brian Ryner (not reading)]

This patch caused smoketest blocker bug 170184.  I've backed out the checkin.

Comment 21

[Brian Ryner (not reading)]

Does widget/src/windows/nsScrollbar.cpp need to override WindowClassW() from
nsWindow as well as WindowClass() ?

Comment 22

[Roy Yokoyama]

Attachment: Need to override WindowClassW()

bryner: Thanks.  It was the problem.  The form scroll-bar now shows up.
shanjian: can you review?
kin: can you super-review?

Comment 23

[Shanjian Li]

Comment on attachment 100246 [details] [diff] [review]
Need to override WindowClassW()

r=shanjian

Comment 24

[kinmoz]

Comment on attachment 100246 [details] [diff] [review]
Need to override WindowClassW()

sr=[e-mail]

Comment 25

[Roy Yokoyama]

checked into the trunk

Comment 26

[Roy Yokoyama]

marking as fixed

Comment 27

[[e-mail]]

Reopening because of bug 170969.

Comment 28

[Kevin McCluskey (gone)]

The checkin for this bug broke dragging of elements on www.dhtmlcentral.com. see
bug 171228

Comment 29

[Jacek Piskozub]

Aren't bug 171208 and bug 171261 also caused by this one? Both concern
non-Western Latin characters (Central Europen and Cyryllic).

Comment 30

[kinmoz]

Turning on MOZ_UNICODE by default also caused bug 170855, which makes it hard to
use viewer for layout debugging.

I'm wondering if we should turn this back off until the kinks get worked out for
all these regression bugs.

Comment 31

[Kevin McCluskey (gone)]

Here are the cvs commands to backout the change locally:

cvs update -j3.7 -j3.6 mozilla/widget/src/windows/Makefile.in
cvs update -j1.41 -j1.40 mozilla/widget/src/build/Makefile.in

I can checkin the changes to the trunk if you like.

Comment 32

[Roy Yokoyama]

Sure. Please turn it off for now.  I'll work on those bugs and
once everything is ironed out, then I'm going to turn it back on for the
third time. :(

Comment 33

[Kevin McCluskey (gone)]

I backed out the change using:

cvs update -j3.7 -j3.6 mozilla/widget/src/windows/Makefile.in
cvs update -j1.41 -j1.40 mozilla/widget/src/build/Makefile.in

sr=[e-mail] : Given through AIM message.

Comment 34

[Frank Tang]

roy, before you try the 3rd time to turn it on. Do the following thing
A. build a release build with it turn ON
B. run the smoke test on both mozilla, viewer and MFCEmbed for the following
platforms:

1. Win95 English
2. Win95 Ja
3. Win98 English
4. Win98 Ja
5. WinNT4 English
6. WinME JA
7. Win2K JA
8. WinXP English


If you have problem to find those configuration, please ask IQA to install the
os on the iqa lab for you. 

Comment 35

[Roy Yokoyama]

I'm not sure if you are aware; but I have run the test
few times on XP and ME before I turn the switch ON 
for the first time.
The following _days_ of smoketest also passed!!

These regression can't be found by the smoketest alone.
170855: Layout Viewer Test App has no URL bar
171208: Can't type SOME Polish characters
171228: [DHTML] Dragging is broken
170969: Mozilla doesn't start on Windows 95

I am going to fix the above bugs and test on suggested apps/platforms
before I turn the switch on.  However, you have to understand that 
I can't test all the features that mozilla support.

Comment 36

[Roy Yokoyama]

Attachment: Changing default NS_foo to A functions instead of W functions

While debugging webshell/tests/viewer app, I realized that
the testapp doesn't instanciate nsToolkit object. Thus 
::RegisterClassW() never get called; but nsWindow continued
to act as if moz is an unicode app.

shanjian: can you review?

Comment 37

[[e-mail]]

Comment on attachment 102354 [details] [diff] [review]
Changing default NS_foo to A functions instead of W functions

>@@ -527,6 +525,20 @@
> 
>     nsToolkit::mIsNT = (osversion.dwPlatformId == VER_PLATFORM_WIN32_NT);
>     if (nsToolkit::mIsNT)  {
>+#ifdef MOZ_UNICODE 
>+      nsToolkit::mDefWindowProc = &DefWindowProcW;
>+      nsToolkit::mCallWindowProc = &CallWindowProcW;
>+      nsToolkit::mSetWindowLong = &SetWindowLongW;
>+      nsToolkit::mSendMessage = &SendMessageW;
>+      nsToolkit::mDispatchMessage = &DispatchMessageW;
>+      nsToolkit::mGetMessage = &GetMessageW;
>+      nsToolkit::mPeekMessage = &PeekMessageW;
>+      nsToolkit::mGetOpenFileName = &GetOpenFileNameW;
>+      nsToolkit::mGetSaveFileName = &GetSaveFileNameW;
>+      nsToolkit::mGetClassName = &GetClassNameW;
>+      nsToolkit::mCreateWindowEx = &CreateWindowExW;
>+      nsToolkit::mRegisterClass = &RegisterClassW; 
>+#endif /* MOZ_UNICODE */ 
>       nsToolkit::mUseImeApiW = PR_TRUE;
>       // XXX Hack for stopping the crash (125573)
>       if (osversion.dwMajorVersion == 5 && (osversion.dwMinorVersion == 0 || osversion.dwMinorVersion == 1))  { 
>@@ -538,25 +550,6 @@
>         }
>       }
>     }
>-#ifdef MOZ_UNICODE
>-    else  {
>-      // On NT base OS, nsFoo is already pointing to respective W functions
>-      // However, in Windows 9x base OS, we need them to point to A functions
>-      nsToolkit::mDefWindowProc = &DefWindowProcA;
>-      nsToolkit::mCallWindowProc = &CallWindowProcA;
>-      nsToolkit::mSetWindowLong = &SetWindowLongA;
>-      nsToolkit::mSendMessage = &nsSendMessage;
>-      nsToolkit::mDispatchMessage = &DispatchMessageA;
>-      nsToolkit::mGetMessage = &GetMessageA;
>-      nsToolkit::mPeekMessage = &PeekMessageA;
>-
>-      nsToolkit::mGetOpenFileName = &nsGetOpenFileName;
>-      nsToolkit::mGetSaveFileName = &nsGetSaveFileName;
>-      nsToolkit::mGetClassName = &nsGetClassName;
>-      nsToolkit::mCreateWindowEx = &nsCreateWindowEx;
>-      nsToolkit::mRegisterClass = &nsRegisterClass;
>-    }
>-#endif /* MOZ_UNICODE */
>     nsToolkit::mDllInstance = hModule;
> 
>     //

I suggest that for clarity that you move the comment, although you will have to
alter its text slightly, e.g.
// For Windows 9x base OS nsFoo is already pointing to A functions
// However on NT base OS we should point them to respective W functions

Comment 38

[Roy Yokoyama]

neil: thanks, I'll add the suggested comment when I checkin.
shanjian: can you review?

Comment 39

[Shanjian Li]

Comment on attachment 102354 [details] [diff] [review]
Changing default NS_foo to A functions instead of W functions

The run time one time for all assignment for NT should not cause any noticable
different. Looks fine. r=shanjian,

Comment 40

[Roy Yokoyama]

kin: can you super review?

Comment 41

[kinmoz]

Comment on attachment 102354 [details] [diff] [review]
Changing default NS_foo to A functions instead of W functions

sr=[e-mail]

Comment 42

[Roy Yokoyama]

Ok, I have a fix for all of the bugs.
170855: Layout Viewer Test App has no URL bar
171208: Can't type SOME Polish characters
171228: [DHTML] Dragging is broken
170969: Mozilla doesn't start on Windows 95

Yumie is setting up a nightly Unicode for me :)
Once done, then I can ask iQA to start testing the 
unicode version before turn the switch for good.

Comment 43

[Roy Yokoyama]

Attachment: All inclusive patch

I want to keep it here so that I don't loose it.

Comment 44

[Christian :Biesinger (don't email me, ping me on IRC)]

can, after the patch is checked in, the ifdefs be completely removed so that
MOZ_UNICODE is no longer optional, or would that have any disadvantages? ie., is
there a good reason for wanting that off?

Comment 45

[Roy Yokoyama]

>the ifdefs be completely removed so that
>MOZ_UNICODE is no longer optional
Please see 170852

Comment 46

[Asa Dotzler [:asa]]

Comment on attachment 102354 [details] [diff] [review]
Changing default NS_foo to A functions instead of W functions

a=asa for checkin to 1.2beta (on behalf of drivers)

Comment 47

[Roy Yokoyama]

fix checked into the trunk
We will perform more testing before turning the MOZ_UNICODE 
flag.  Until then, I'll keep this bug open.

Comment 48

[Roy Yokoyama]

Attachment: Turning MOZ_UNICODE flag on by _default_

need /r, /sr and /a for this

Comment 49

[Roy Yokoyama]

Kevin: can you review?
Kin: can you super review?
thanks

Comment 50

[kinmoz]

Comment on attachment 104021 [details] [diff] [review]
Turning MOZ_UNICODE flag on by _default_

sr=[e-mail]

Comment 51

[Kevin McCluskey (gone)]

Comment on attachment 104021 [details] [diff] [review]
Turning MOZ_UNICODE flag on by _default_

r=[e-mail]

Comment 52

[Kevin McCluskey (gone)]

Note: When the MOZ_UNICODE flag was turned on previously, it may have caused bug
170272. Bug 170272 is difficult to test because it only shows up on particular
graphics cards with specific driver versions. The MOZ_UNICODE is suspected
because bug 170272 was fixed by an unknown check in which corresponds to a day
that MOZ_UNICODE  flag was backed out, and no other checkins in that time frame
look suspicious.

http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=MozillaTinderboxAll&branch=HEAD&branchtype=match&sortby=Date&date=explicit&mindate=2002%2F09%2F22+07%3A00&maxdate=2002%2F09%2F23+12%3A00&cvsroot=%2Fcvsroot

Comment 53

[Roy Yokoyama]

Thanks for making a note in this bug.

I am going to turn the unicode flag once again; however 
as the impact of turning the flag is big, I am avoiding  
this checkin until the trunk check-in rush hour cools off :)

ETA will be tomorrow or Friday.  Meanwhile, we can continue to test
using the special nightly unicode build available from ftp.mozilla.org

Comment 54

[Roy Yokoyama]

done.
Let's not rollback this again. We need to move forward on this.
I'll be standing by if there are any regressions caused by this. :)

Comment 55

[Greg K.]

Was this bug strictly PC/Windows 95? Were/are there any ramifications for other
platforms/OS's?

Comment 56

[Roy Yokoyama]

>Was this bug strictly PC/Windows 95?
Strickly for Windows; but not limited to Win95.  
All the Windows platforms will be affected.  

Comment 57

[Greg K.]

Thanks, Roy. I take it non-Windows Mozilla builds were already "Unicode apps",
then? (I'm asking these questions because this bug didn't really include any
information at all about why it was needed or what it accomplished.)

Comment 58

[Roy Yokoyama]

Greg: I'm not sure on other platforms. Anybody can answer here?

>why it was needed or what it accomplished.     
this is a big step forward for fixing internation related problems with moz
eg.  Can't type Georgian text, Can't File/Open korean file in English os,
     Can't attach Japanese file to email in Korean OS, 
     browser title doesn't show chinese in English os,  etc....

Windows NT and higher OSs are language mutual OSs. OS supports mixing 
languages in the system level.  You can create Russian/Japanese/Georgian/Arabic
files in Windows; but mozilla can't recognise them :(

Comment 59

[Jungshik Shin]

Greg,

I can't answer about Mac, but as for Unix/Linu/X11, the extent that
Mozilla can support Unicode all depends on which locale users choose
to run Mozilla under. If it's UTF-8 locale and the platform has a good
support for I18N and UTF-8 locale, (different flavors of Unix have
different degree of supports for UTF-8 locale and I18N), Mozilla is a
very good 'Unicode application'. Actually, Mozilla does as much as it
can without depending on the I18N support of the underlying platform
and it's indeed a very good I18Nized/Unicode-enabled browser whichever
flavor of Unix/Linux it happens to run under.

However, there are certain things Mozilla cannot do without the support of
an underlying platform. Perhaps, the most frequesntly asked question is
how to display int'l text (e.g. Russian text under Japanese Unix/Linux
or vice versa) in the title bar regardless of the current locale.
That issue(bug 9449), among others(e.g. file selection, using whatever
names for files when saving, opening, etc), led Roy and others to decide
to convert Win-Mozilla to 'Unicode' application  (In MS Windows jargon,
'Unicode application' is the opposite of 'ANSI' application meaning
it's not bound by many restrictions arising from 'codepage' dependency
of 'ANSI' application). Anyway, in case of Unix/Linux, rendering text
in the title bar is NOT under the control of application programs like
Mozilla. It is the realm of Window manager and the platform support. The
same is true of MS Windows and MacOS.  That's why even after turning
Mozilla into 'Unicode appl.', Japanese/Chinese/Korean title cannot be
displayed in the window title bar under 'Russian/French/German/English'
Windows *9x/ME* while it's possible under Windows 2k/XP. MS IE 5/6 has
been taking advantage of this under Win 2k/XP for a while, but Mozilla
couldn't until it's turned into Unicode appl. And, we can't live with
it, can we? :-)

When it comes to Unix/Linux/X11, it's a matter of what a window manager
and system locale support can do. If a window manager knows nothing
about I18N, Mozilla cannot do anything. It's been almost two years since
Mozilla-Unix/Linux did everything it could do and had to do to render
any title in any language on the window title bar.  However, without
a window manager running under an appropriate locale (usually UTF-8
locale) understanding necessary WM protocol, that's of no use. Many
people got confused about this point and blamed Mozilla or filed many
duplicate bugs on the issue although it was solved almost two years ago
for Mozilla under Unix/Linux/X11.

Anyway, I'm afraid I wrote too much about what must be an kinda off-topic
to this bug. If you want to know details, read my and others' comments
in bug 9449. You can also take a look at bug 150131. RedHat Linux 8 came
with UTF-8 locales for all languages (except for CJK which is unfortunate,
but I expect that to change in next release. Moreover, one can just make
it use UTF-8 locales for CJK like I do) and eventually all other Linux
distros will follow.

Hope this helps..

Comment 60

[Torben]

This seems to have caused bug 179822 (Flash4/Flash% crash...), see bug 179822
comment 18.

Comment 61

[Jun MUTO]

This seems to have caused bug 171468 also.