Closed Bug 239279 (mzlu) Opened 20 years ago Closed 16 years ago

Need Mozilla binary that supports Windows unicode API

Categories

(Core :: Internationalization, enhancement)

x86
Windows 2000
enhancement
Not set
normal

Tracking

()

RESOLVED WORKSFORME

People

(Reporter: andrejohn.mas, Assigned: jshin1987)

References

Details

(Keywords: intl)

Attachments

(2 files, 6 obsolete files)

17.29 KB, text/plain
Details
27.37 KB, text/plain
Details
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7b) Gecko/20040316
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7b) Gecko/20040316

From what I can tell Mozilla is not currently built to support the Windows
unicode binary. This causes issues, such as found in bug 193358:

  http://bugzilla.mozilla.org/show_bug.cgi?id=193358

With access to the source, this can be resolved by compiling from the source and
modifying the necessary build options. This is a pain for most users, especially
on the Windows platform (it is not bundled with necessary build tools).

If the current binary is not built to use the unicode API, because of older
versions of Windows, would it be feasable to have a separate binary build for
Windows. Maybe 'Mozilla UC' or 'Mozilla Intl'?


Reproducible: Always
Steps to Reproduce:
1.
2.
3.
parts of mozilla autodetect whether the unicode functions are available and use
them if so... that seems like a better solution, because that way you don't need
a different binary for NT vs 9x
Sounds fine, if they work. This doesn't seem to work for the issue raised in 193358.
For sure, we do not want to build two separate binaries. We don't have to even
if we make Mozilla a Unicode binary as long as MSLU(Microsoft Layer for Unicode)
is available on Win9x/ME. Among programs that bundle MSLU is MS IE.  The
question is whether we want to keep pretending that Mozilla can be run without
MS IE's presence. Currently we do and that's, I guess,  why Roy and Frank
(ftang) decided to roll out our own mini version of MSLU . Bug 162361 is about
extending it for XPCOM file i/o (there's already a patch), but that has been
held back because I never  heard back from my repeated emails to NSPR owners. 

Recently, I realized that some parts of Mozilla already rely on APIs offerred by
MSIE. That is, Mozilla can't run without MS IE. That being the case, rolling out
our own version of MSLU doesn't seem to do any good although it makes Mozilla
further depend on the presence of MS IE (but it already does as far as I can
tell.) Instead, we'd be better off making Mozilla a Unicode binary as suggested
here. That way, we can cut down a quite lot of code. 

This is also a 'policy issue' and I'm adding brendan and chofmann to cc. 
Assignee: smontagu → jshin
Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: intl
If the heading is anthing to go by, then MSLU appears to be a redistributable:
 
http://www.microsoft.com/downloads/details.aspx?FamilyID=73ba7bd7-ed06-4f0d-80a4-2a7eeaee17e2&displaylang=en

So we could just include it in a regular Mozilla build. Or am I mis-reading MS's
definition of 'redistributable'? Of course the is probably affected by 'policy
issues' that I am not aware of. 
erm, we're backporting support for nt3.5, if you know of some pieces of mozilla
which don't work w/o ie, i'd like to hear of them. (i believe trident import
needs ie, but well, it's a bit hard to import ie cookies w/o an ie to generate
cookies.)
(In reply to comment #3)
> Recently, I realized that some parts of Mozilla already rely on APIs offerred by
> MSIE. That is, Mozilla can't run without MS IE. That being the case, rolling out

Which parts?

And how large is the MSLU redistributable? Depending on licensing, we could
probably make it an optional extra download with installer builds with minimal
trouble.
We use IUniformResourceLocator APIs that I believe are only available with MS IE.

http://lxr.mozilla.org/seamonkey/search?string=IUniformResourceLocator

http://search.microsoft.com/search/results.aspx?View=msdn&st=a&qu=IUniformResourceLocator&c=0&s=1


As for the size of MSLU, I guess it's less than 1MB (the page refered to in
comment #4 indicates that it's 270kB when compressed. )
(In reply to comment #7)
> We use IUniformResourceLocator APIs that I believe are only available with MS IE.

but we fail gracefully when it's not there (we don't support .url files in that
case but that's ok)
what biesi said, although for kicks mozilla could offer an impl of that class
(i'd probably go for it as an xpi perhaps stick it w/ the activex control). note
that without a provider for CLSID_InternetShortcut it should be fairly hard for
someone to manage to get those files onto their system or into memory (just as
it would be hard to need to worry about importing cookies from ie if ie wasn't
installed).

if you want to file a bug reminding people to consider adding support for it,
you could, i think it'd even be reasonable to stick it in the tracking pool for 35.
Blocks: 162361
Blocks: 167128
In bug 162361, Brodie came up with an alternative to MSLU, rolling out our own
version of MSLU ('MZLU'). A number of bugs (including bug 232969, bug 240272,
bug 9449, bug 243618) can be fixed easily (without cluttering the tree with two
branches of code for Win 9x/ME and Win 2k/XP) if we assume or ship MSLU or add
'MZLU'. This issue (Mozilla not being able to take full advantage of Unicode
capability available on Win 2k/XP) is long-standing and I think we have to
resolve this issue before long. It'd be great if we can resolve it in the trunk
soon enough  so that we can back port the change to aviary-1.0 branch as well as
to 1.7branch.

If we decide to add 'MZLU', we can do it gradually (see attachment 150929 [details]. see
also bug 162362 and bug 118013) implementing necessary APIs as we move on to 'W'
APIs through out the tree.

Brodie, would it be possible to write MZLU in such a way that it can be a
drop-in replacement for MSLU (perhaps not)? If that's possible, we can offer it
as an additional download that is needed *only* on Win 9x/ME *without* MSLU
installed. 

Even if it's not, adding MZLU may be a good solution if we're worried about
potential licensing issues with MSLU (bug 162361 comment #30)

Thoughts? 
Status: NEW → ASSIGNED
It is too much effort to try and make a 'drop-in replacement' for MSLU. Either
we roll our own customized minimal version and use it always, absorbing the cost
of associated bugs, maintenance and code size, or we actually use MSLU. Using
MSLU is of course the best solution (the Microsoft approved solution).

Can ANYONE tell me exactly why MSLU is considered taboo? Is it licence? Is it
the build requirements (intimated by bug 162362 comment 4)? I can't find a
definitive history for this decision. 

If the MSLU import library (unicows.lib) build requirements is a problem, then
there is a project at sourceforge (MIT licence) which provides a replacement
import libraries for Microsoft's MSLU which can be used with lots of different
compilers. I'm not sure if this is even required (as I don't know the problem).
http://libunicows.sourceforge.net/
(In reply to comment #11)
> ...
> of associated bugs, maintenance and code size, or we actually use MSLU. Using
> MSLU is of course the best solution (the Microsoft approved solution).

  I wish I could use it right now :-)

> Can ANYONE tell me exactly why MSLU is considered taboo? Is it licence? Is it
> the build requirements (intimated by bug 162362 comment 4)? I can't find a
> definitive history for this decision.

  I also wish to know, but I'm afraid it's 'buried' somewhere between ftang and
roy who are not active any more in Mozilla project. If those currently active on
the project don't see any problem with using (and shipping) MSLU, let's just use it.
 
> If the MSLU import library (unicows.lib) build requirements is a problem, then
> there is a project at sourceforge (MIT licence) which provides a replacement
> import libraries for Microsoft's MSLU which can be used with lots of different
> compilers. I'm not sure if this is even required (as I don't know the problem).
> http://libunicows.sourceforge.net/

  Thanks for the pointer. For some reason, if we can't use MSUL, this seems to
be a good alternative. 
The sourceforge project is not a replacement for unicows. It is only a
replacement loader for unicows. It still uses the original Microsoft supplied
redistributable unicows.dll

Who has to be approached in order to get mozilla.org legal people to check the
MSLU licence and make a decision there?
(In reply to comment #13)
> The sourceforge project is not a replacement for unicows. It is only a
> replacement loader for unicows. It still uses the original Microsoft supplied
> redistributable unicows.dll

Yeah, I realized that after visiting the site and going through the cvs.

> Who has to be approached in order to get mozilla.org legal people to check 
> the MSLU licence and make a decision there?

  I  added Bart to Cc hoping that he may shed some light on the issue or find
someone who can. Bart, can you help us here?  
Blocks: 240272
Blocks: 243618
I can try and figure out if what you want to do is allowed, if someone
summarises exactly what it is that you do want to do :-) I'd also need links to
the relevant license documents.

Gerv
To make Mozilla take the full advantage of Unicode capability on Win 2k/XP while
making it still run on Win 9x/ME are  _without_ adding a whole bunch of |if
(isNT) use 'W' APIs else use 'A' APIs|-like code (see bug 232969 for an
example), we have a few options:

 
  - Ship the whole or a part of MSLU (=unicows.dll + unicows.lib) with Mozilla
Win.  Because we don't want to break compiliation with mingw, we may want to
ship unicows.dll from microsoft (actual implmentation that translates 'W' APIs
to 'A' APIs on Win 9x/ME) but use libunicows (import library that hooks up 'W'
API calls to the 'translator' in unicows.dll if it's on Win 9x/ME. Otherwise,
native 'W' APIs will be used) at http://libunicows.sourceforge.net  (see comment
#13). Taking this way, we don't have to rely on the presence of MS IE or other
applications (e.g. MS Word/Office) that include MSLU 

 - Just assume the presence of MS IE or other applications that include MSLU. In
this case, we have to require MSIE be installed. We already have a couple of
dependencies on MSIE in our code, but according to cbie, we have a graceful
fallback in absence of MSIE (see comment #8)
   
 - Roll out our own version of MSLU (what Brodie dubbed 'MZLU' and had a partial
implementation). In resolving bug 9449, Roy Yokoyama used a similar approach
that can be utilized only in a small portion of Mozilla-code.  
 
  In all three cases, we have to change our code throughout the tree to use 'W'
APIs instead of 'A' APIs. The former use 'UTF16' strings while the latter use
the locale-specific code page (what MS used to call 'ANSI' code page) /
multibyte strings. This  can be done gradually  once we make up our mind about
MSLU. 


   Now as for the MSLU license, I can't find the license terms for MSLU itself
at  the MSDN web site. It's one of what MS called 'redistributable ???'. Brodie,
can you give us the URL of the MSLU license? You quoted a part of the license
(bug 162361 comment #30), which I don't think we can possibly violate.
  
MSLU announcement :
 http://www.microsoft.com/globaldev/handson/dev/mslu_announce.mspx : MSLU 

MSLU download :
    
http://www.microsoft.com/downloads/details.aspx?FamilyId=73BA7BD7-ED06-4F0D-80A4-2A7EEAEE17E2&displaylang=en
 

MSDN article about MSLU:
 http://msdn.microsoft.com/msdnmag/nettop.asp?page=/msdnmag/issues/01/10/MSLU/MSLU.asp&ad=ads.ddj.com/msdnmag/premium.htm

Known MSLU bugs:
http://www.trigeminal.com/usenet/usenet035.asp
Attached file Unicows licence
This attachment is the licence ('licence.txt') from the MSLU download. The
following text is the extra redistribution clause ('redist.txt') from that same
download (MSLU 1.0.4018.0, download link supplied in comment #16).

I couldn't see any plainly obvious reason that we couldn't redistribute MSLU
ourselves with the exception of clause "Distribution Terms" (c). Even this
clause could probably be worked around by including the relevant terms in our
installer (e.g. this package contains software copyright by Microsoft, you must
abide by there terms for all of that software, namely "MSLU"). 

Note that it is highly desirable to distribute unicows.dll ourselves instead of
depending on it being on the machine. This is so that we know what version of
MSLU we are using (as there are a variety of versions out there with a variety
of bugs). 

It is very desirable to fix this problem sooner rather than later, either via
MSLU or via a roll-your-own version like MZLU (see bug 162361). Every day more
people are inserting more roll-your-own functions like MS/ZLU implements, but
at a higher level which leads to greater codesize for everyone.

--redist.txt--

===============================================
Microsoft Layer for Unicode on Windows 95/98/ME
===============================================

In addition to the rights granted in Section 1 of the Agreement ("Agreement"),
with 
respect to UNICOWS.DLL, you have the following non-exclusive, royalty free 
rights subject to the Distribution Requirements detailed in Section 1 of 
the Agreement: 

(1)	You may distribute UNICOWS.DLL with the following: Windows 95, Windows
98, 
Windows 98 Second Edition, Windows Millennium, Windows NT4, Windows 2000,
Windows XP, 
and Windows Server 2003.
Attached patch MZLU 0.1 (by Brodie) (obsolete) — Splinter Review
While it's being decided as to what to do with MSLU, Brodie updated his MZLU
(he uploaded the zip file to bug 162361) from which I've made a patch. 
If we decide to take this route (or are forced to because of licensing issues),
we have to adjust its build procedure. Currently, Brodie's patch has a MS VC++
project file. We need to convert it to Makefile.in.
adding cls to cc for a (potential) build issue (either with MSLU or with
Brodie's MZLU)
Having looked at it, the MSLU license is (fairly predictably) not even vaguely
free-as-in-speech. So we don't want to be shipping it with our products.

So we can either use our own implementation (MZLU) or depend on IE. 

Given that IE has shipped with every version of Windows from 95 up, isn't that
the easiest solution? Or do only newer versions of IE come with MSLU?

Gerv
(In reply to comment #20)
> Having looked at it, the MSLU license is (fairly predictably) not even vaguely
> free-as-in-speech. So we don't want to be shipping it with our products.

thanks for reviewing the license gerv


> Given that IE has shipped with every version of Windows from 95 up, isn't that
> the easiest solution? Or do only newer versions of IE come with MSLU?

it sounds like there are different versions of MSLU floating around, and some
older versions have serious bugs.  i suspect that we might therefore prefer to
use MZLU on older Win32 platforms at least (Win9x in particular).  we might
experience less pain if we simply include MZLU... provided the size is not too
large, which I doubt it will be.

brodie: can you comment on the codesize overhead of MZLU?
> So we can either use our own implementation (MZLU) or depend on IE. 
> 
> Given that IE has shipped with every version of Windows from 95 up, isn't that
> the easiest solution? Or do only newer versions of IE come with MSLU?

MSLU was released only in 2001 so that an old Win9x box with MS IE 3 or 4(?) and
without any relatively new application that includes MSLU wouldn't have MSLU.
I'm sorry I forgot to mention this. Besides, do we really want to rely on MS IE?
Well, the browser war was over, but relying on MS IE appears to giving in to MS'
argument that MS IE is a part of the OS. Even if that's not much a concern, we
still have to figure out how to deal with old Win 9x boxes (there shouldn't be
many of them that fit the description above, but Mozilla is supposed to run on
them, too).

It seems like we have to roll out our own at the cost of increasing the code
size by a few hundred kB's (probably a lot less)


Thanks for the extra info jshin...

> It seems like we have to roll out our own at the cost of increasing the code
> size by a few hundred kB's (probably a lot less)

I'd be surprised is MZLU measures up to even 100 kB.  I'd expect much less.  I
don't think we can decide to take it or leave it until we know exactly how much
codesize increase we are talking about here :-)
we are working on not requiring windows4 and we're also interested in supporting
windows nt 4 without ie. 
The current version of MZLU is about 30Kb. The full version of Microsoft's MSLU
is 240Kb. I expect that once we implement all of the functions that we need in
NSPR and Mozilla we might hit about 80-90Kb in MZLU but I wouldn't expect much
more. This code size will only be experienced on Windows 9x as NT doesn't use
the DLL. 

In addition to this, all code modules (.exe and .dll) need to link to the mslu
static lib and call the initialize function at least once before using any other
MZLU functions. The current version of MZLU intentionally sets all function
pointers to NULL in order to force an error if it is used without
initialization. Linking with this static lib will probably increase sizes by a
couple of Kb at most. 

Since the 'W' version of all wrapped functions are referenced in the static lib,
it will probably increase the system DLL dependencies for all modules which use
the static lib to include all DLLs implementing those functions. e.g. shell32,
commdlg32, winspool. Since they are all system DLLs and are all used somewhere
in the Mozilla code anyhow this shouldn't have any noticeable effect on working
set, but startup times might be slightly increased.

If we use MZLU on any platform we need to use it on all platforms otherwise we
are going to have bug reports caused by MSLU being used in one case and MZLU
used in another. Also with the current implementation it's not a mix and match
situation. MZLU functions are postfixed with 'Mz' instead of 'W' so they aren't
directly interchangeable. 

Note: It might be possible to make it compatible with MSLU (i.e. export 'W'
postfixed functions) and use the libunicows loader from sourceforge to load it,
but I think that will require us to create stubs for the hundreds of functions
that MSLU implements and we don't. Easier to stay with readily identifiable 'Mz'
== MZLU function names, and our own loader to minimize problems.

timeless: windows4
Do you mean IE4? What are the realistic minimum requirements for Moz? Windows 95
or Windows NT4 (no IE required on either)? If someone removes IE altogether from
Windows (http://www.litepc.com/) and installs Mozilla should it work?
Attached file MZLU 0.2 (obsolete) —
Definitive figures for MZLU 0.2. Linking with the static lib increases .exe
file size by 4Kb. The DLL is about 50Kb (49Kb when built with Mozilla build
harness or 56Kb directly with VC7). There are currently 70 functions
implemented. This gives an overhead of about 50 bytes per function added.

It's still a bit early to add MZLU as a patch, so I'll just attach the .zip
file still. Still very much a prototype. This zip includes the Makefile.in
files required for building under the xpcom tree. Not sure if this will be the
final location. Extract the files below xpcom to create the mzlu and mzlu/mzlua
directories. Add xpcom/mzlu directory to xpcom/Makefile.in. Re-run the make at
xpcom.
Attachment #152021 - Attachment is obsolete: true
By Windows 4, I mean Windows 95 or Windows NT 4.

By not requiring Windows 4, I mean supporting Windows NT3.51.
Timeless: who is the "we" in "we are working on..."?

Gerv
biesi, neil, and me.
(In reply to comment #29)
> biesi, neil, and me.

What issues are there in NT 3.5x support if we go ahead with MZLU and using 'W'
APIs throughout the tree?  Apparently, NT 3.5x is not officially supported by
mozilla.org. So if there's no unsurmountable (or rather too-hard-to-reconcilate)
conflict between making Mozilla fully-Unicode-capable and supporting NT 3.5,
that's great.  If there are such problems that it will take us long to resolve,
I'd say that we should give the priority to making Mozilla take the full
advantage of NT4 or later's Unicode capability. 
There is no reason for current and new Mozilla
releases to support NT 3.51.
Actually Mozilla builds and almost runs correctly on NT3.51
I only know of three issues:
System fonts come in cell height sizes instead of character height
Bitmap fonts don't work
Bookmarks don't load (dependency on IE, would you believe...)
codesize overhead of MZLU seems well within reason to me.
However we work out the NSPR API issues, NSPR will need to use MZLU in order to 
support the wchar versions of the API on all platforms (anything else is 
useless). Should MZLU be a standalone library that NSPR is dependent on? I 
would prefer to see it at least stay as a library that can be used separately 
from NSPR to make it useful for other projects that may wish to use it without 
using NSPR. 

Going back to MSLU... Could we get around the licence problems with MSLU by 
creating a new application which abides by the licence terms and then recommend 
that users install it when they don't already have MSLU? Perhaps an extension 
that users on Win9x are 'recommended' to install to actually make Mozilla work 
(if they don't already have MSLU on the system).
Gerv: 
For the future hundreds of developers who are going to wonder why the heck we 
developed our own version of MSLU when the real version was out there begging 
to be used, would you outline exactly where the problems is with redistributing 
MSLU with Mozilla?  Which are the clauses which cause it to be legally 
impossible?
We don't want NSPR, at this late date, to depend directly on some new library. 
wtc can speak authoritatively; I'll venture that we could add callbacks
defaulting to null, not called unless non-null, that could be set at runtime to
point at MZLU exports.  See for an analogue
http://lxr.mozilla.org/mozilla/search?string=JSLocaleCallbacks.

/be
Personally I would still like to see it dependent on Microsoft's 
redistributable component MSLU and not on some new library at all. Otherwise it 
doesn't particularly matter to me how it is accomplished, whatever way is 
considered politically correct is fine. 

As long as the unicode version of the NSPR API calls the unicode windows 
functions on NT platforms and falls back to ANSI functions on non-NT platforms. 
Since MSLU and MZLU implement this automatically it makes sense to use one of 
those libraries.
Would it make any difference if the MSLU binary was downloaded for MS and
installed at install time? If it is legally acceptable then I would imagine the
installer getting the current URL of the file from the Mozilla site (in case the
binary gets moved at MS), downloads it and then get the user to compelete the
installation. Is there any way of getting the installer to check for the
availability of a library call before installing an optional component?
(In reply to comment #38)
> Would it make any difference if the MSLU binary was downloaded for MS and
> installed at install time?

depending on a present internet connection at install time is not acceptable.
What if we just make MSLU a requirement. Similar to the x Mb of memory and x Mb 
of disk? The installer checks to see if it is installed by searching for the 
version we require in common locations, if we can't find it then the installer 
allows download if download is available, otherwise it's bye-bye, you need to 
install MSLU before you can install Mozilla. 

The trouble with this is that there is no standard install location or name for 
the DLL. It is distributed named as Unicows.dll but you can rename it (it's 
recommended by Microsoft too). It's not supposed to be installed into the 
Windows or system directories, etc. There is no real location that we can look 
for it and know that it exists, except perhaps searching for products like IE 
that we know use it.

I guess I don't really understand the problem with redistributing the DLL. Gerv 
says that it isn't free as in speech, but is that important in this situation? 
It is a royalty-free redistributable system DLL provided by Microsoft for this 
exact situation. As long as there are no actual *legal* issues to installing it 
ourselves, we should. It reduces bugs on our side, it is well supported by 
Microsoft, it is the official solution to this problem, it reduces our source 
code size, it has everything going for it and it is redistributable for crying 
out loud. 
Brodie: it's pretty simple. MSLU is not free software, and mozilla.org
distributions contain only free software (except Talkback, for historical and
deeply practical reasons which I wish would go away but probably won't.) 

I don't really need to outline why it's not free software - it's clear to anyone
who knows what the terms means and then looks at the license for 30 seconds that
it's not. :-)

There's nothing wrong with us depending on MSLU being present, or telling people
to install it before they install Mozilla, but we won't ship it ourselves.

Gerv
Attached file MZLU 0.3 (obsolete) —
Gerv: 
Is this being overly moralistic? If we are going to make Mozilla/Win dependent
on a redistributable component of the OS, then isn't it better to distribute
that component in order to reduce the installation problems for users on Win9x.
Does the distribution of MSLU with Mozilla/Win affect the tri-licencing of
Mozilla?

Whoever has the ability to make a decision: 
Is it satisfactory to distribute Mozilla/Win with the requirement on
Win95/98/ME that MSLU has been installed in the windows/system32 directory
first? i.e. unicows.dll of a specific version or better exists in that
directory? On installation we can then either use the file from that location,
or copy it to our installation directory to have the local copy which is
recommended by Microsoft. It is then left up to customers to ponder how to
fulfil this requirement.

And finally, in case MSLU isn't used, attached is MZLU 0.3 with support for
registry functions. Included is the patch for xpcom/Makefile.in
Attachment #152095 - Attachment is obsolete: true
I don't understand why we can't ship 3rd party redistributables in our
distributions. Even the GPL would probably allow this since this Unicode layer
is pretty clearly a piece of the OS and normally distributed with the OS these days.
Gerv: MSLU should be treated the same way as MSVC's
runtime libraries.  If mozilla.org distributions can
contain only free software, mozilla.org will not be
able to ship a distribution compiled with the current
version of MSVC (.NET 2003) because you can't ship
its runtime libraries (msvcr71.dll and msvcp71.dll).
It is obvious to me that this rule must make an
exception for libraries that are considered part of
the programmming language runtime or the platform.
No longer blocks: 243618
Whatever we do, the default Firefox packaging should not include extra stuff
needed only on downrev Windows.  Special package(s) for those OS versions would
be fine with me.

/be
Blocks: 250255
roc: as it happens, the mozilla.org binaries are not distributed under the GPL,
they are distributed under the MPL. 

But, if they were distributed under the GPL, we could not distribute MSLU. The
GPL requires you to distribute the entire source code of your application
(section 3.a). The exception you may be thinking of runs as follows:

"However, as a special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary form) with the
major components (compiler, kernel, and so on) of the operating system on which
the executable runs, unless that component itself accompanies the executable."

The component would be accompanying the executable in this case, and so the
exception does not apply.

Gerv
Wan-Teh: it may well be true that this rule means mozilla.org can't ship
binaries compiled against MSVC .NET 2003. But why would we want to, when we can
compile binaries which work out of the box on all Windows platforms without need
of new system DLLs?

Having said all that, producing "free software" on top of a non-free operating
system is always going to have grey areas. But it seems to be that, if we have
an alternative, introducing a dependency on a proprietary library seems a bad
step. What are the disadvantages of using MZLU?

Gerv
The disadvantage of MZLU is that it is completely new written from scratch. It
has a mildly different API from the system API (i.e. Mz suffix instead of W,
maybe not all API side-effects implemented fully). It doesn't implement the
entire Win32 API and so we need to continually extend and maintain it as
extra/different functions are required. To use it in NSPR (which needs to have a
Unicode API) we need to create a dependency from NSPR to MZLU which isn't popular.

The advantages of MSLU is that it has been in use for a long time, it's stable
and supports everything that we need now and in the future without any
development work. The API is completely specified, documented and tested. NSPR
can just use the W functions, link to the MSLU loader, and leave the problem of
getting the MSLU DLL to the client up to the user of NSPR. 

Brendan: the compressed DLL is about 85Kb and is the entire difference between
NT and 9x versions of Mozilla. I think it would be far more convenient for
customers if we shipped a single distributable for all platforms.

Gerv: you address why the GPL doesn't permit the redistribution of OS components
but you didn't address the MPL which is what we distribute under.
(In reply to comment #47)
> Wan-Teh: it may well be true that this rule means mozilla.org can't ship
> binaries compiled against MSVC .NET 2003. But why would we want to, when we can

We already ship the MSVC6 runtime libraries in seamonkey installer builds,
because some win95 (and NT4?) machines don't come with them.
We should also consider how this will impact extension authors.  If I write an
extension, and want to use Win32 Unicode APIs, it would be nice to utilize the
same solution that Mozilla is using.  If Mozilla is using MZLU, and if MZLU only
consists of entry points for methods that Mozilla is using, then what happens
when my extension needs to use other Win32 APIs?
I'll bring this issue up at the next staff meeting. 

Does anyone have a link to the redistribution license for the MSVC6 runtimes
that we currently distribute? If it's the same, then I guess distributing MSLU
wouldn't be any worse.

Gerv
OK, chaps, this sinks us, surely? From attachment 151916 [details] (Unicows licence):

"(d) you do not permit further redistribution of the Redistributable Components
by your end-user customers;"

We just can't live with that restriction for Firefox builds. (c), which is about
a click-wrap license, is bad too.

Gerv
This issue of the Unicows license is really a big sticking point and seems to be
the main thing holding us back. I have already suggested at the start of all
this, that maybe two versions of Mozilla for Windows might be the best approach,
in the same way that there was a MacOS 9 and MacOS X version (MacOS 9 support
now being discontinued). The first version would be rated 95-NT, and the other
version would be Windows 2000 upwards. This had been rejected, but I am
wondering whether that maybe now it would be the only way to go forward, maybe
as a temporary solution until we can get a complete MZLU finished?

Having a choice of two binaries is not something that is new many Windows users,
since there already a number of programs that a distributed in this way.

Something else:

Out of curosity, how much of the code actually has Windows API hooks that would
be affected by UTF8? I ask this since another approach would be to have two
shared libraries distrubted with the Windows version (one for UTF8 and the other
without) and then depending on the availibity of the Unicode API load the
appropriate shared library, based on an initialisation test.
Yes, it does seem to be a problem. 

How about this solution:

* create a separate product with minor functionality that installs a known
version of unicows.dll into a known location. It has a simple installer, is
completely optional and is licenced using MS happy terms:
  - click-wrap
  - non-redistributable

* Use of Mozilla on Windows 95/98/ME has a requirement that the MSLU must
already exist as 'unicows.dll' version 1.0.4018.0 or greater in the windows
directory. 

* Users of Mozilla on 9x/ME have the option of installing IE, or the minimal app
above, or just download MSLU from the MS site itself and copy it to the windows
folder.

ajmas: two binaries won't really solve anything here. We are trying to get a
single binary that works on all platforms leveraging the Unicode API of NT/2K/XP
and yet falling back to ANSI codepages on 9x/ME. The solution needs to be final
rather than temporary. It should be soon to stop the cruft of ANSI/Unicode
wrappers that is not-so-slowly creeping into the code base all over the place.
UTF-8 is not used internally by Windows, but may be the representation used to
talk to the NSPR library.


Also, simply for interest's sake, here is the licence for VC.NET 2003:
http://msdn.microsoft.com/visualc/vctoolkit2003/eula.aspx
> maybe two versions of Mozilla for Windows might be the best approach,

We can't distribute MSLU with _any_ mozilla.org software, because MSLU forbids
redistribution by the people we give it to, and we can't (or won't) take the
steps they require us to take (clickwrap license etc.) to prevent that.

Also, doing such a thing is basically falling into the Java Trap, or something
like it:
http://www.gnu.org/philosophy/java-trap.html
Our software becomes non-free because it depends on a non-free component being
included.

> create a separate product with minor functionality that installs a known
> version of unicows.dll into a known location.

Note that the MSLU license says that anything that redistributes the DLLs must
add "significant and primary functionality to the Redistributable Components".

Gerv
(In reply to comment #55)
> Note that the MSLU license says that anything that redistributes the DLLs must
> add "significant and primary functionality to the Redistributable Components".

Yes, I'm aware of that. Since MSLU only implements a Unicode API, any program 
that actually does something fulfils that clause. e.g. display the version of 
windows libraries, or run in the background and check a pop mail box. 

It seems like everyone here is acting in an adversarial manner. Instead lets 
work together to find a way to use this software.

Mozilla needs to use Unicode on Windows to solve a heap of problems that are 
plagueing those pesky people that don't use single byte character sets. For the 
reasons supplied in comment #48, the best way to do so is to use MSLU which is 
a non-free redistributable part of the OS, the same as the C runtime libraries.

My suggestions for how to avoid the licencing issue is that we just make MSLU a 
required part of the OS spec. To then make it easier for users, we supply a 
little non-free program (required by MSLU licence) which will add the 
appropriate files to their OS. Their use of the non-free program is optional, 
they can also use any of the MS non-free programs that use MSLU to install it. 
(In reply to comment #55)

> Also, simply for interest's sake, here is the licence for VC.NET 2003:
> http://msdn.microsoft.com/visualc/vctoolkit2003/eula.aspx

This license seems as unsuitable for us as the MSLU license (in light of comment
#52). It has a similar provision for the redistribution. How about the license
of MSVC6 run-time libraries mentioned in comment #49? If it has a similar
provision, according to Gerv, we have to stop including it .....
 
As far as I can tell, the Visual C++ 6 terms are different. I haven't found an
official copy yet, but I found:
http://technology.pitt.edu/software/userights/visual_studio.pdf

Section 3.1.1 of that document has an exception for people re-redistributing
your stuff, which probably covers us, and it doesn't require us to impose a
particular EULA on our customers. So we are probably OK with Visual C++ 6
redistributables.

The VC.NET license, as linked in comment 57, does have a redistribution exception:

"You also agree not to permit further distribution of the Redistributables by
your end users except you may permit further redistribution of the
Redistributables by your distributors to your end-user customers if your
distributors only distribute the Redistributables in conjunction with, and as
part of, the Licensee Software and you and your distributors comply with all
other terms of this EULA."

But it's the last part of that sentence (which seems to tie back to the new
requirement to impose a EULA on your customers) which would present a problem.
We can't ensure that our distributors comply with all the other terms of the MS
EULA.

Time to start polishing up the mingw port? :-)

Gerv
In reply to comment #56)

> plagueing those pesky people that don't use single byte character sets. For 

Just to dispell any misunderstanding, I have to make it clear that the above is
not correct (I know Brodie knew that, but he oversimplified). It's not single
byte character encoding vs double/multibyte character encoding. It's about going
beyond the limit of a single _legacy_ character encoding  to take the full
advantage of Unicode support available on Win NT4/2k/XP (e.g. a Russian user
wanting to have a Greek filename)  while keeping Mozilla still usable (with all
the limit of a single legacy character encoding) on Win 9x/ME. 


> My suggestions for how to avoid the licencing issue is that we just make 
> MSLU a  required part of the OS spec. To then make it easier for users, 
> we supply a  little non-free program (required by MSLU licence) which 
> will add the appropriate files to their OS. Their use of the non-free program 
> is optional,  they can also use any of the MS non-free programs that 
>  use MSLU to install it.

As long as we make Mozilla and friends for Windows, we can't avoid 'Java trap'
completely. That being the case, the above suggestion seems to be workable
although further developing MZLU and offering it to other open source developers
wouldn't be bad, either.


re comment #55:
>> maybe two versions of Mozilla for Windows might be the best approach,

> We can't distribute MSLU with _any_ mozilla.org software, because MSLU 

Gerv, you misunderstood what ajmas suggested. He suggested that to _avoid_ the
licensing issue with MSLU, we make two separate binaries, one for Win 9x/ME that
uses 'A' APIs and the other for Win NT4/2k/XP that uses 'W' APIs. __Neither__ of
them depends on MSLU. In comment #3, I wrote that that's not a good idea because
it makes it very hard to manage and test the code with a lot of #ifdef's.  In
that case, we also need to set up two separate tinderboxes for each of our
products (one for 'A' APIs and the other for 'W' APIs) 

> Gerv, you misunderstood what ajmas suggested. He suggested that to _avoid_ the
> licensing issue with MSLU, we make two separate binaries, one for Win 9x/ME that
> uses 'A' APIs and the other for Win NT4/2k/XP that uses 'W' APIs.

Ah, right. Sorry about that. I thought he meant one with MSLU and one without.

Gerv
I was just doing yet another security update for MS-Windows and something struck
me. Users need to have Internet Explorer installed on their machine to be able
to run Windows update. If they don't have it then their machine is probably a
security nightmare. Windows update will work with nothing else.

Based on that I am wondering whether it is safe to assume that all users will
have Interner Explorer installed in some form. Given this I am wondering to what
extent they are likely to have the unicows.dll on their machine. I know am going
down the dangerous road of assumptions, but maybe a survey in this direction
might help us?
Blocks: 251897
I think that in corporate/enterprise environments some sort of centralised push
solution is usually used to automatically distribute patches, updates and new
applications to the desktops (Zenworks comes to mind), so you can't assume that
IE is installed at all.

On the other hand in such environments older versions of Windows would have long
since been phased out.
Let's see what we can do:

 1.MZLU: Unlike Brodie, I'm not worried that it's not so good as MSLU. It may
be, but we have it under our control so that if there's a problem, we can fix
it. The only downside with this approach is what Darin mentioned in comment #50.
Plugin authors could be impacted similarly. Perhaps, we have to put up a web
page for extension/plugin authors in which it's clearly mentioned what needs to
be done if they want to use 'W' APIs for which there's no wrapper in MZLU. The
page also has to list all Win32 APIs with wrappers available in MZLU and their
naming scheme. 

 2. Shipping MSLU  seems to be out of question (comment #55, comment #52)

Gerv> we can't (or won't) take the
Gerv> steps they require us to take (clickwrap license etc.) to prevent that.

 3. Make it a requirement on Win9x/ME that MSLU be present (by installing
non-free programs that include it or by installing Win SDK kit?). A trouble with
this is that its location and name can vary (I can't find unicows.dll on Windows
ME although MS IE 5(?) is present). Adding details about this (where MSLU should
be present in what name) to the release notes (and perhaps some other very
prominent places)  might solve the problem, but it would be a hurdle for
non-technically oriented people (I wish they all had thrown away Win 9x/ME...). 

 Given all these, I'm inclined to go with the first option. If we decide to go
with it, I'll volunteer to write a short document for extension(/plugin) authors
about using Win32 APIs. 
We could take option 3 by surveying where common apps put their copy (and what
they call it), and trying to load them all in turn. It's a bit hacky, but I
suspect we could get fairly good coverage. 

Gerv
I like option #1 too.  I think it will give us fewer headaches, and at least we
have access to all the knobs.  If it's true that older versions of MSLU are
buggy, then I can only imagine that we are inevitably going to inherit some of
those bugs  in some cases until we eventually get it all ironed out :-(
I also prefer option 1. forcing users to install a seemingly unrelated app just
so they can use mozilla seems like no good idea to me.
(In reply to comment #50)
> consists of entry points for methods that Mozilla is using, then what happens
> when my extension needs to use other Win32 APIs?

I guess they have to extend MZLU to implement their functions and then 
distribute an updated version of the MZLU DLL with their extension. Either they 
update our MZLU DLL with the new one (we need to add version info to the DLL to 
allow this) or implement MZLU such that they can run side by side with 
different DLL names and the extension somehow loads their version of the DLL 
(assuming that the extension calling Win32 API is a DLL this should be 
possible).

If we allow extension authors to upgrade the MZLU DLL then we need to ensure 
that there is only a single instance of that DLL to be upgraded (e.g. all bin, 
bin/components, GRE modules that link to it all load the same DLL). For 
instance by making it a file in Program Files/Common Files.

If going with option 1 (using MZLU) then we need to implement the NSPR 
dependency on MZLU somehow. Callbacks were mentioned in comment #36, but since 
the API of MZLU is already function pointers, this seems somewhat strange. 
Since the NSPR Unicode implementation will be a special mode or API to maintain 
backwards compatibility (currently it's a special API enabled via compile time 
defines), the user of NSPR will need to intentionally use the Unicode enabled 
functions. In this case surely a new library dependency shouldn't be a problem. 
wtc?
Since everyone seems to prefer using MZLU, and it would provide other
open-source projects with a free version of MSLU type functionality, I'm happy
to continue and develop it. I need to know where to place it in the source tree.
It seems to me that it should be a top level component for the following reasons:

a) dependency from NSPR (when MOZILLA_UNICODE or similar defined)
b) dependency from rest of Mozilla
c) we want to provide it as a standalone project for others to use
Brodie: we could put it in mozilla/mzlu, on the theory that a big top-level is
best for finding things (tyranny-of-hierarchy makes any attempt to categorize
modules not scale well over time and people).  But we do try to group like
things, where we have a plurality of such things, or are likely to have a
plurality in due course, under top-levels such as db, gc, and parser.

So should we put this under mozilla/intl, or is that less a container of like
things, and more of a singular module?

Should we call the directory mzlu?

Give me some thoughts, and as top-level mkdir judge, I'll render a quick
verdict.  Thanks,

/be
I think we should decide the NSPR dependency before continuing. If NSPR were to
depend on MZLU, it should live in the NSPR build tree (and we should get rid of
those MOZ_UNICODE #ifdefs. At that point there is no reason I see not to just
integrate the MZLU dynamic loading directly into nspr4.dll instead of having a
separate library.

However, we could alternatively move the NSPR MOZ_UNICODE functions (such as
PR_OpenFileUTF16) into XPCOM or someplace where NSPR wouldn't have to depend on
MZLU.

Let's solve the link-time-dependency issue before settling on a tree location.
We could also make NSPR depend on MZLU not at compile time at all, which is a
much better idea in my book, and probably in wtc's.

Why shouldn't NSPR call through a configurable set of callback functions,
default null, and be free of #ifdefs?  Is it too ugly, due to too many
callbacks?  Or too costly?

bsmedberg: I don't think MZLU is part of NSPR in any sense, so I don't see why
it should live under mozilla/nsprpub.  The source tree issue can be separated
from other build and runtime issues, and should be in general.  Build- and
run-time orders vary over time and across apps.  The tree does not.

/be
> We could also make NSPR depend on MZLU not at compile time at all, which is a

NSPR should drop the FooUTF16 methods altogether, and not pretend to be
unicode-aware. Instead, move the unicode-aware methods into xpcom.dll, and have
xpcom.dll depend on or integrate-in MZLU.

--BDS
Whether NSPR depends on MZLU or not is irrelevant.  MZLU
should be a stand-alone library.  Even if NSPR uses MZLU,
NSPR won't export the MZLU symbols.
Perhaps it should go into whatever location that the separate libraries get 
put. modules or lib? Where is expat? That's in the tree somewhere isn't it?

As for naming, I'm not particularly attached to MZLU if there is some problem 
with it. Suggest another name if necessary. If not then I suggest that we stay 
with it. There won't be any naming issues with Microsoft would there? The full 
name is obviously not mistakable as MS software. The shortened form is arguable.

bsmedberg: I know that you are keen to reduce the DLL footprint of Mozilla and 
this is a good aim. I'm not sure that mzlu can be successfully merged into any 
of the other libraries though. The closest fit might be NSPR, but wtc isn't 
keen for that. At the moment the DLL is only 60-80Kb with 90 functions 
implemented, but if we ever get to MSLU like coverage of 520+ functions, I 
epxect that it will grow to similar proportions of MSLU, about 300-400Kb. 
Perhaps then the overhead of a separate DLL file isn't so important?

In my local copy (after 0.3) I've changed the exported names of the functions 
to match the MSLU DLL (i.e. using xxxW function names instead of xxxWrapped). 
This allows us in future to possible use mzlu as a drop in replacement for the 
MSLU DLL. Combined with the libunicows loader noted above we have a full open-
source mslu replacement.

The static lib that NSPR would link with to provide the unicode API provides a 
number of function pointers implementing the 'W' version of the Windows API. 
This either calls directly into the Windows NT native W API, or it redirects it 
into the wrapped versions implemented by the mzlu DLL when running on 9x 
platforms. If the mzlu library isn't initialized then all of these function 
pointers are NULL. This is already very similar to the callback idea. The major 
difference is that the function pointers etc are the mzlu static library and 
not straight NSPR. Of course NSPR could reimplement the functionality of the 
mzlu static library (or copy it into NSPR code) too, removing the build 
dependency but adding a maintenance dependency.

Similar to bsmedberg, I think that it would be best if the Windows NSPR got rid 
of the UTF16 versions of the functions, and instead implemented a single 
function which switches the meaning of the char* parameters from native to utf8 
encoded. utf8 to/from wchar_t conversions are simple and pretty much foolproof. 
The same API is used, with the difference being the interpretation of the char* 
arguments. When put into Unicode/utf8 mode, NSPR would have to call the W API 
implemented by mzlu. Otherwise it would continue to call the A API natively 
supported on all platforms.

Finally, a note regarding my availability. I have quit my job and will be on an 
extended vacation (anywhere from 1 to 4 months) away from all software 
development in a couple of weeks. It would be good to get some of this 
finalized before I leave if possible, otherwise I'm sure that jshin and others 
can keep moving it along even if it doesn't.
lib is gone, or should be -- an AT&T Unix-ish holdover from the old Netscape tree.

modules is deprecated -- please don't add to it, unless you can convince me that
it is not just a useless container for stuff better located elsewhere, possibly
at top-level.

Again I urge folks to avoid mixing build options, which might do very desirable
things such as combine libraries to cut down on DSO overhead, with source tree
layout.  Source trees don't age or scale well if you try to subsume directories
based on current library and program build bundling.  The source tree should be
more of a module catalog, and at sufficient scale, alphabetical ls listing is
your friend, not any particular non-categorical hierarchy.

/be

/be
The scope of the mzlu project is providing a unicode wrapper around the Win32 
ANSI code-page API used by Windows 9x/ME. It doesn't provide any international 
functionality other than converting text in/out function parameters between 
unicode and the current system codepage. The goal of the project is eventually 
to provide a complete open-source replacement for MSLU, but in the short term 
it is specifically focused on Mozilla's needs.

I am not personally concerned if it exists in the intl directory or at top 
level, however perhaps top-level is more appropriate since 
a) an mzlu-like project will only be created once for Windows (no other 
platforms require it)
b) it is only a character encoding wrapper around an api
c) it is eventually intended for a wider audience then just mozilla

Unless anyone has specific concerns about calling it mzlu (Mozilla Layer for 
Unicode), I suggest that we stay with mzlu as the project name and also use 
that as the main directory name in the build tree. 
Only categorical hierarchy thought that comes to mind: is there any point in a 
mozilla/unicode container?  If not, top-level mzlu is ok with me.  I'm cc'ing 
smontagu.

/be
Blocks: 253035
What would go into the unicode container other than mzlu? Is it just for future
use? Again, I have no preference for where the library is located.
I've been chatting with Vaclav Slavik, the author of the MIT-licenced 
libunicows (http://sourceforge.net/projects/libunicows/) about how we might use 
libunicows with mzlu. 

In summary, they fit together perfectly - if we write mzlu as a drop in 
replacement for the MSLU library "unicows.dll" (i.e. all implemented functions 
use the same export names) then libunicows can load our DLL automatically if it 
can't find the real unicows.dll library. Vaclav is happy to modify libunicows 
to fit any mzlu special requirements as long as it keeps libunicows fully 
interchangeable with the real unicows.dll

In order to use the libunicows/mzlu combination in Mozilla we need to build 
with UNICODE defined (or explicitly call the W functions), link to the 
libunicows static lib (ensuring libunicows comes first in the link list) and 
distribute mzlu.dll with the application. 

What I intend to do now is rewrite the mzlu project so that it is designed 
specifically to work with libunicows. I will get rid of the custom loader that 
I had written and will instead write only the DLL. libunicows will be a 
requirement for use of mzlu. This is a good thing because libunicows is smaller 
and faster than my basic loader. Licencing shouldn't be a problem as the MIT 
licence basically permits anything (Gerv?).

I'll post (with permission) some of the emails that I have exchanged with 
Vaclav.
Extract from Vaclav's reply to my first email explaining the project and asking 
whether we could use libunicows with our own implementation of mzlu. The app 
that he mentions in the email is http://poedit.sf.net

Brodie Thiesfield wrote:
>> My question is: is it possible to use your libunicows loader to
>> load our mzlu.dll

(Why not unicows.dll? That would make it easier to be a complete 
replacement...)

>> instead of the unicows.dll EVEN if we only 
>> have a subset of implemented functionality? i.e. can I use
>> libunicows to load the current mzlu DLL? If so, what would
>> happen to callers that try to call a function not
>> implemented/exported from MZLU?

I think it should work out of the box. You can already change 
UnicowsDllName variable to point to mzlu.dll instead of unicows.dll, 
but it requires special code in the app, which is a Bad Thing (the 
rest of libunicows is completely transparent to the programmer -- the 
*only* thing you have to care about is linking it in, no 
initialization is required (unlike MZLU, I gather?)). But if you 
decide on using mzlu.dll name instead of unicows.dll, I would 
certainly like to include its support directly in libunicows, e.g. by 
trying it if unicows.dll couldn't be found (or the other way around).

As for subset of functionality, it shouldn't be a problem at all. Each 
function is handled separately in libunicows, there aren't 
dependencies an overything (such that ibunicows would, for example, 
try to load all 520+ symbols from unicows.dll). If you don't use FooW 
function, its loader code doesn't get linked into your executable at 
all. If your app uses FooW but MZLU doesn't support it, then loading 
the symbol will fail (on 9x, unicows.dll/mzlu.dll would not be even 
looked at all on NT/2k/XP), libunicows will display error message and 
terminate the app. This is no worse than if native FooW would be 
called (it would return error code and GetLastError would return "not 
implemented" or something), but if this behaviour would be unsuitable 
for you, you can change the UnicowsImportError pointer to point to a 
function that does something better. It's not currently possible to 
use UnicowsImportError callback to provide a replacement no-op 
version of unimplemented function, but if it would be needed, I can 
easily add such a callback to libunicows, it's just that I didn't 
need it myself yet.

So in short, I believe there shouldn't be any problems, and if 
something shows up, I'm more than willing to help with fixing it so 
that you can use libunicows instead of wasting your time reinventing 
the wheel.

(I'll a little selfish here, too -- I use embedded Gecko in my app and 
would like to make sure MZLU won't interfere with NT-only or MSLU 
builds, which it won't if you'd use libunicows  ;-) 
Extract from my reply:

>>>>My question is: is it possible to use your libunicows loader to
>>>>load our mzlu.dll
>
>> 
>> (Why not unicows.dll? That would make it easier to be a complete 
>> replacement...)

I'll leave it up to the user if they want to rename it to unicows.dll,
but I'll release it under mzlu.dll. That avoids any problems with name
clashes and makes it a bit clearer if it is the moz version or the MS
version.

>>>>instead of the unicows.dll EVEN if we only 
>>>>have a subset of implemented functionality? i.e. can I use
>>>>libunicows to load the current mzlu DLL? If so, what would
>>>>happen to callers that try to call a function not
>>>>implemented/exported from MZLU?
>
>> 
>> I think it should work out of the box. You can already change 
>> UnicowsDllName variable to point to mzlu.dll instead of unicows.dll, 
>> but it requires special code in the app, which is a Bad Thing (the 
>> rest of libunicows is completely transparent to the programmer -- the 
>> *only* thing you have to care about is linking it in, no 
>> initialization is required (unlike MZLU, I gather?)). But if you 

My current version is all pretty basic, hence the reason that I was
wanting to use your loader. That would mean that I only need to write
the DLL.

One thing that I could do with the way I was writing mzlu was to wrap
functions that don't seem to be wrapped by mslu (and so aren't supported
by your loader). Things like some CRT functions which I find useful
(e.g. _wgetdcwd).

There are also the IME functions (Imm*) which are natively supported in
Unicode on Win98+ but not on Win95, but don't seem to be wrapped by
MSLU. I don't know if I need to yet, but I was thinking of wrapping them
on Win95 to make them available on all platforms.

>> decide on using mzlu.dll name instead of unicows.dll, I would 
>> certainly like to include its support directly in libunicows, e.g. by 
>> trying it if unicows.dll couldn't be found (or the other way around).

Yes, that would be nice. Probably unicows.dll first since it is a
complete implementation.

>> As for subset of functionality, it shouldn't be a problem at all. Each 
>> function is handled separately in libunicows, there aren't 
>> dependencies an overything (such that ibunicows would, for example, 
>> try to load all 520+ symbols from unicows.dll). If you don't use FooW 
>> function, its loader code doesn't get linked into your executable at 
>> all. If your app uses FooW but MZLU doesn't support it, then loading 
>> the symbol will fail (on 9x, unicows.dll/mzlu.dll would not be even 
>> looked at all on NT/2k/XP), 

Does this mean I don't need to add code to the DLL to ensure that we
pass the function call through to the system if someone calls an MZLU
function on NT? I don't have it at the moment, but MSLU does this so I
was thinking that I probably should too.

>> libunicows will display error message and 
>> terminate the app. This is no worse than if native FooW would be 
>> called (it would return error code and GetLastError would return "not 
>> implemented" or something), but if this behaviour would be unsuitable 
>> for you, you can change the UnicowsImportError pointer to point to a 
>> function that does something better. It's not currently possible to 
>> use UnicowsImportError callback to provide a replacement no-op 
>> version of unimplemented function, but if it would be needed, I can 
>> easily add such a callback to libunicows, it's just that I didn't 
>> need it myself yet.

Personally I feel that bailing out of the app with a message identifying
the missing function is a little drastic. The function may not be
completely necessary, so it would be nice if users were able to call it
and detect if it is implemented in that version of mzlu or not. Not
strictly necessary, just nice.

>> So in short, I believe there shouldn't be any problems, and if 
>> something shows up, I'm more than willing to help with fixing it so 
>> that you can use libunicows instead of wasting your time reinventing 
>> the wheel.

Great to hear. Thanks for the support.

In summary, it seems that as long as I write to the spec and name all of
the exports the same as MSLU, and for the moment set the UnicowsDllName
variable we should be able to use MZLU now right out of the box. In
future we should be able to use it without setting UnicowsDllName.
Vaclav's reply:

>> One thing that I could do with the way I was writing mzlu was to
>> wrap functions that don't seem to be wrapped by mslu (and so aren't
>> supported by your loader). Things like some CRT functions which I
>> find useful (e.g. _wgetdcwd).
>>
>> There are also the IME functions (Imm*) which are natively
>> supported in Unicode on Win98+ but not on Win95, but don't seem to
>> be wrapped by MSLU. 

Again, I certainly don't mind extending libunicows to do more than MS 
version  ;)  I was hit by missing wrappers for CRT functions myself, so 
if you support them, great! In that case, libunicows should try to 
use both unicows.dll _and_ mzlu.dll for best results.

>> Does this mean I don't need to add code to the DLL to ensure that
>> we pass the function call through to the system if someone calls an
>> MZLU function on NT? 

Yes, exactly.

>> I don't have it at the moment, but MSLU does 
>> this so I was thinking that I probably should too.

MSLU does it in unicows.lib. Unicows.dll is only used on win9x 
platforms and unicows.lib (or libunicows) is not a "normal" import 
library; instead, it dynamically loads the symbols when they are 
first used (from unicows.dll on win9x and from system DLLs on 
NT/2k/XP). So you don't have to concern yourself with NT at all in 
mzlu.dll.

>> Personally I feel that bailing out of the app with a message
>> identifying the missing function is a little drastic. 

Right. It's better than outright crash, though.

>> The function 
>> may not be completely necessary, so it would be nice if users were
>> able to call it and detect if it is implemented in that version of
>> mzlu or not. 

I can see two ways to do it: 

1) Introduce libunicows-specific function that would check 
availability. The biggest problem with this is that it's no longer 
interchangable with MS' version of unicows.lib.

2) Let the app check by loading the symbol dynamically using 
GetProcAddress. Unicows.dll hijacks GetProcAddress too, so that when 
you call (pseudocode) GetProcAddress(kernel32.dll, FooW) and FooW is 
implemented in unicows.dll, it returns pointer to (working) FooW 
version from unicows.dll instead of (not working stub only) FooW from 
kernel32.dll. So if mzlu.dll returned NULL from GetProcAdress for 
functions that it knows it should wrap, but doesn't, the apps would 
have a way to recognize this situation while still being compatible 
with MSVC7's standard unicows.lib. There's no harm in returning NULL 
because the function that OS' "real" GetProcAddress would return 
would be useless stub anyway.

The best would be to have both ways, of course.

>> In summary, it seems that as long as I write to the spec and name
>> all of the exports the same as MSLU, and for the moment set the
>> UnicowsDllName variable we should be able to use MZLU now right out
>> of the box. In future we should be able to use it without setting
>> UnicowsDllName.

Yes, exactly. 
(MSLU export names are exactly same as native libs' names, so exported 
names should match names of symbols in system DLLs (e.g. 
"DrawTextW@20" from user32.dll -- see symbols.txt in libunicows). I 
think that you can do some tricks in .def file to export functions 
under different names than what you use in source code...)


--

So basically, assuming that we implement mzlu, we have pretty simple way to 
include Unicode API support into Mozilla already implemented for us. Of course 
there are other problems that need to be sorted out, but this is a very good 
start.

Comments about using libunicows? Are there problems with this that I am naively 
overlooking?
Will this make bug 218242 any easier or harder to implement?
It will want to pass a custom common dialog hook procedure...
It shouldn't make it any harder. You only need to implement the single Unicode 
(W api) version so it should be easier. Microsoft's unicows library supports 
the hook function, so we should be able to as well. However I don't think that 
I support the hook function in the current implementation of these functions in 
mzlu.
Attached file MZLU 0.4 (obsolete) —
MZLU as a drop in replacement DLL for unicows.dll. Currently implementing about
130 functions of the 520 odd that unicows does. Requires the use of the
libunicows loader. Still has the diff and Makefile.in files to build under
xpcom (until a final location has been decided on).
Attachment #152484 - Attachment is obsolete: true
Comments anyone?
Attached file MZLU 0.5 (obsolete) —
This version of MZLU does actually work as a drop-in replacement for
unicows.dll for simple console Win32 programs. The included test program does a
simple FindFirstFileW/FindNextFileW through the current directory. 

The included unicows.lib static library is a custom build of libunicows to
force it to load unicows.dll even on NT platforms (to allow me to test on my XP
development machine). This is still very much development version and there are
definitely a myriad of bugs, but it is now at the stage where it should be able
to be used and tested if anyone has the desire.

I'm starting an extended vacation soon (for a couple of months), so this will
be my last update of MZLU until I get back. If anyone else wants to keep
developing it in the meantime then go ahead. Similarly if anyone wants to
finish the discussion on CVS location, dependencies, etc.
Attachment #154440 - Attachment is obsolete: true
I agree that we should try to eliminate the ifdef MOZ_UNICODE methods in NSPR
and invent a UTF-8 mode for NSPR (unless this proves to be too inefficient).
I stumbled upon a couple of problems while trying to compile attachment 1546638. 

Firstly, MSVC complained that '_stat64' is not found in the global namespace
(MSDN says that it's available even on Win9x and it's in Win64/sys/stat.h of my
copy of MS SDK). This may be due to a problem in my copy of MS SDK... 

The other problem is that 'mzlu.def' (DEFFILE) is not found at the link time
because my build tree is separate from my source tree. I was able to fix this
problem defining 'DEFFILE' as '$(srdcdir)/mzlu.def' instead of 'mzlu.def'.

I've uploaded two patches (one to bug 250255 and the other to bug 251897) that
'rely' on MZLU. While making patches, I identified a couple of functions to add
to MZLU. One of them is EnumPrintersW, which may be a little tough to implement
(apparently MSLU doesn't support this, yet) 
Blocks: 253164
Blocks: 193358
Blocks: 262922
Brendan: any chance of getting the "unicode" and "unicode/mzlu" directories
added to the tree? I've started working on this again and it would be good to be
able to check it in.

The question of using the http://libunicows.sf.net/ library is still open. The
other option is to write a DIY wrapper which is 100% mozilla code which
duplicates the exact same functionality.
In order to get MZLU building with VC6, it seems that I need to use the Platform
SDK. This is because MZLU covers some newer API which isn't included in the
standard VC6 distribution. It doesn't appear that this is a requirement of
Mozilla at the moment. Would requiring the Platform SDK as a requirement for
building MZLU be a problem? It is a big download, but it guarantees the latest
header files, libraries, etc.
Blocks: 100243
Brodie, what platform SDKs do I have to install? Even after installing Win2003
SDK, I'm still getting an error with '_stat64'. Do I have to install platform
SDKs for Win XP SP2? 

 
I shifted development to http://opencow.sourceforge.net/ so that I had CVS and 
so that it is more accessible to non-Mozilla projects. You can download the 
latest version there which has that problem fixed. I'm working on a patch for 
Mozilla to add the libunicows static lib in build time and use the opencow DLL.
Blocks: 231426
A new release of MSLU has been made in response to these EULA problems.  The
licensing terms for MSLU should now be the same as the ones used for the C runtime.
This is the licence.txt file for the latest MSLU download (released
10-Dec-2004). If anything it seems that this licence is more restrictive.
Products should only work with Windows, still need the click-wrap/shrink-wrap
EULA acceptance, and still cannot be further redistributed. It really is a pain
in the ass.
Attachment #154663 - Attachment is obsolete: true
Licence text for MSLU 1.1.379.0 is also readable in a better format online at
http://blogs.msdn.com/michkap/articles/345289.aspx
Comments from Kaplan on the MSLU EULA.
http://weblogs.asp.net/michkap/archive/2005/01/02/345389.aspx#FeedBack

The problems and possible solutions that I see with the licence are the following:

3.1(iii) + 3.1 last paragraph
Requires an addition to the click-wrap licence in the Windows installer such
that redistribution of the Windows software also implies acceptance of the MSLU
EULA.

3.2
Means that anyone wanting to use the Mozilla source code under Windows with MSLU
must be using the MPL (GPL is invalid under this clause). Use of the GPL licence
on Windows is fine if MSLU is not used.
It looks like we lost track of this bug. (no activities for a month)  
Can someone enlighten me on what's going on now?
MZLU, MSLU or other?  Do we have a patch?
opencow is there to be used, and it must be used with the libunicows wrapper. I
haven't had time and won't have time to work on getting the patch working for
this though. MSLU does a hell of a lot that opencow doesn't and never will. 

I still think that we should work on a solution for using MSLU. Mozilla on
non-Windows and Windows NT platforms doesn't require MSLU and so no licence is
required to be accepted there. 

Perhaps the installer for Mozilla on Win9x/ME can download and install MSLU with
the user accepting the licence similar to installing plugins.
Blocks: 131063
Brodie:  Do you have a latest patch for MZLU? (patch format not zip, plz)
         Is MZLU 0.1 good enough? I'd love to give a code review. Thanks
No longer blocks: 250255
Blocks: 211961
Status Request: This is a critical bug for the MS-Windows platform and I see no
activity since March 2005. Could someone please indicate the current status and
how long before we can expect to see this in a release of some sort?
Attachment #198586 - Flags: review?(darin)
Comment on attachment 198586 [details] [diff] [review]
Add --enable-unicode and XPCOM beginnings

needs work
Attachment #198586 - Attachment is obsolete: true
Attachment #198586 - Flags: review?(darin)
Comment on attachment 198586 [details] [diff] [review]
Add --enable-unicode and XPCOM beginnings

<biesi> +    nsString& path = mWorkingPath;
<biesi> +#else
<biesi>      // work in unicode for ease
<biesi>      nsAutoString path;
<biesi>      NS_CopyNativeToUnicode(mWorkingPath, path);
<biesi> bsmedberg, that seems scary
<biesi> bsmedberg, in one of these codepaths, modifying |path| will change
mWorkingPath, and in the other it doesn't
Comment on attachment 198586 [details] [diff] [review]
Add --enable-unicode and XPCOM beginnings

er, nevermind that - the non-unicode codepath copies back the changes anyway.
So, has it been decided that we would build two binaries for Windows, one for
Windows 9x/ME and the other for Windows NT4/2k/XP? 

jshin: nothing has been decided, this is merely a first step to allow a
full-unicode binary. In fact, I think that two binaries is a bad idea, and we
should drop win9x without MSLU support. Adding MSLU linkage would be a secondary
step.
I would be tempted to have Win9x version simply in maintance mode and then concentrate on building a 
version for all current platforms. At the same time it has already been mentioned several times that two 
binaries is unpopular. An extra observation,  from my part, is the number of poorer countries where 
Windows 98 can still be found installed, so supporting this would be useful, since ironically these are the 
guys who would benefit more from a browser which doesn't open up their computer to worms and 
spyware.
(In reply to comment #108)
> jshin: nothing has been decided, this is merely a first step to allow a
> full-unicode binary.

> In fact, I think that two binaries is a bad idea, and we
> should drop win9x without MSLU support. 

That's why I asked. I think keeping two separate binaries would be a lot of
hassle. That being the case, I wonder why you're introducing two code paths
(with #ifdef). Is it  a temporary measure until you introduce MSLU linkage?
Wouldn't it be better to do all the necessary things in a single step without a
lot of '#ifdef's even if they're temporary? 



There is not a chance that I could convert the entire codebase to the unicode
APIs at once, and I want to be able to provide test-builds of unicode before
actually making everyone switch over.
bsmedberg: can you write up a design doc for this?  i'm very interested in
reviewing your approach here, and it seems like we should agree on the approach
before cutting code.  how about something on wiki.mozilla.org?
But ... when will be firefox support Unicode API?
Because there is much trouble in CJK Windows Unicode Platform using ANIS Program....
Let me see if I understand this right, according to Comment #63, to support Windows 98 we either create MZLU, or use MSLU which requires a shrink-wrap, non-redistributable license. 

Apart from that, Comment #111 suggests moving to entire Unicode API needs to be done in stages, not all at once.

If I understand it, are we going to use MSLU and start moving to Unicode API first, then worry about finishing MZLU / opencows later? That is, refer to Comment #63, we implement solution 3 first and then migrate to solution 1 if licensing or other reasons make it necessary to do so, am I correct? 

I think it is a more pragmatic approach right now because we can start the move to Unicode API and get more testing first. If there are enough demands we may see more hackers contributing to opencows in order to continue Win9x support, i.e. if enough users really stick with Win9x.
This out of my league and perhaps slightly off-topic, but I recently posted on this subject at http://forums.mozillazine.org/viewtopic.php?t=347673

I noticed mention here of breakage of building with mingw.

In addition to loading files in unicode paths from within Firefox for example, you also need to be able to do:

Firefox.exe unicode_path_to_file

I'm not sure if mingw and the windows api headers you get from mingw.org have a winmain() that accepts unicode command line arguments.  I do know that mingw's main() does not; even if you define _UNICODE. Plus mingw doesn't have a wmain() for unicode argv support. (Either does STLPORT)

To get around the lack of wmain support in mingw, you have to of course define _UNICODE first and then do this instead:

int argc;
wchar_t** argv = CommandLineToArgvW( GetCommandLineW(), &argc);

Now I know Firefox isn't a command line progam and probably uses winmain either directly or indirectly, but I figured I'd mention the need to pass unicode arguments to the exe and I know little on if you guys forsee a problem implementing that or not, but it's best to bring it up as it's possible.  
NSPR has a method named PR_ImportFile that can be used to create a PRFileDesc from a HANDLE.  This means that we could develop a nsILocalFile implementation for Windows that took care to call either CreateFileW or CreateFileA depending the OS.  In other words, I believe that we could solve this bug without changing NSPR and without introducing dependencies on MZLU or MSLU.
(In reply to comment #115)
>wchar_t** argv = CommandLineToArgvW( GetCommandLineW(), &argc);
CommandLineToArgvW isn't supported on 9x/Me
As a belated comment on the extra observation from comment #109, my personal experience is that the "poorer countries" also tend to have nearly free versions of Windows XP etc widely available.  Hence there are fewer Window 9x systems in use in these countries than you think these days.

What HAS been surprising to me though, living in China, is the number of websites that require IE.  Way more than in the US.  My wife wont even touch firefox because of the confusion.  Well, I was surprised until I started using some public database machines with html interfaces.  Got tired of IE real quick, installed firefox from my usb flash, and then found out one reason why.  Everyone uses Chinese (unicode) filenames and directories.  Bug #162361 is the one I really care about and is probably why almost noone I talk to here knows about much less uses firefox.  But that bug seems to depend on this one.  Dont lose the biggest market for browsers in the world just to keep a few Win9x users!
A few questions:

 - are we at a point yet where Firefox works with the 'W' APIs, if I were to
   compile it from source?

 - where are we with MZLU functionality?

 - has the Mozilla legal team considered contacting Microsoft for a more liberal
   license, with regards to Unicows, that would resolve the "non-redistributable"
   issue?

I would want to have this issue resolved by Firefox 2.0 if possible.
(In reply to comment #119)
>  - are we at a point yet where Firefox works with the 'W' APIs, if I were to
>    compile it from source?

Not at all. The parts that can use W apis probably do so already, w/ fallback to A apis. so, it seems to me that this'd be the major part of the work required for this bug.
(In reply to comment #116)
> NSPR has a method named PR_ImportFile that can be used to create a PRFileDesc
> from a HANDLE.  This means that we could develop a nsILocalFile implementation
> for Windows that took care to call either CreateFileW or CreateFileA depending
> the OS.  In other words, I believe that we could solve this bug without
> changing NSPR and without introducing dependencies on MZLU or MSLU.

Darin, that's a very neat idea. I'll try to give it a shot for bug 162361 after December 31st. However, it's to be noted that although file handling is a very big part of this bug, there are other issues to deal with here. As cbie noted, in many places, we use a 'hand-made custom MSLU-equivalent'. That may not be so bad, but there are some places where we didn't bother to do that expecting this bug to be fixed.  

> there are some places where we didn't bother to do that expecting this bug to be fixed.

Perhaps an early task should be to enumerate those spots in the code and decide which course of action to take: 1) keep using custom failover code, or 2) unify everything behind something like MSLU.
No longer blocks: 193358
Will the descision to drop pre-Windows 2000 support have any effect on this issue?

See /. article: http://developers.slashdot.org/article.pl?sid=06/06/08/1240239&threshold=1
No longer blocks: 211961
No longer blocks: 100243
Blocks: 361543
Cannot we close this? The pre Win2k was dropped already. And new gfx uses Uniscribe now. So, this bug can fix nothing on current trunk, I think.
Does Firefox 3 now provide proper Unicode support when interacting with the OS? If it does then I agree we could properly close this bug.
We fixed a lot of bugs, but there's still bug 413171, for example.
We still generate icons (moz-icon:) using ANSI path names.

We still read version information from plugins in ANSI.
Yeah, we are still using ANSI APIs in some points. They should be replaced to W APIs. However, that is not this bug.
Ah, so this bug is saying "Mozilla should compile with -DUNICODE"?
(In reply to comment #129)
> Ah, so this bug is saying "Mozilla should compile with -DUNICODE"?

And this wants to use MSLU. See:
http://www.microsoft.com.nsatc.net/globaldev/handson/dev/mslu_announce.mspx
If versions of Windows that don't support Unicode natively are no longer part
of the picture, the we should default the Unicode API.

Most of the discussion in this bug was around how to deal with versions of
Windows where Unicode was not available without an extra library. If Windows
2000 support has been dropped, then most of the backwards-compatibility issues
can be ignored and the make Unicode support of Windows default.

The original requirement of this bug was to make a Mozilla binary with support
of the Windows W API, which is the Windows unicode library. I had suggested
this as a separate build, but now if it is not already done in should be made
the default build.
ok, I'm going to claim that this bug has outlived its usefulness. We are using the W APIs in many places now, and will continue to excize A APIs for moz2. There still may be a couple places where we continue to use A APIs for compatibility reasons.
Status: ASSIGNED → RESOLVED
Closed: 16 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.