Closed Bug 239279 (mzlu) Opened 21 years ago Closed 17 years ago

Need Mozilla binary that supports Windows unicode API

Categories

(Core :: Internationalization, enhancement)

Product:
Core
Component:
Internationalization
Platform:
x86
Windows 2000
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: andrejohn.mas, Assigned: jshin1987)

References

Details

(Keywords: intl)

Attachments

(2 files, 6 obsolete files)

Unicows licence
17.29 KB, text/plain
Details
MSLU 1.1.3790.0 licence.txt
27.37 KB, text/plain
Details
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7b) Gecko/20040316 Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7b) Gecko/20040316 From what I can tell Mozilla is not currently built to support the Windows unicode binary. This causes issues, such as found in bug 193358: http://bugzilla.mozilla.org/show_bug.cgi?id=193358 With access to the source, this can be resolved by compiling from the source and modifying the necessary build options. This is a pain for most users, especially on the Windows platform (it is not bundled with necessary build tools). If the current binary is not built to use the unicode API, because of older versions of Windows, would it be feasable to have a separate binary build for Windows. Maybe 'Mozilla UC' or 'Mozilla Intl'? Reproducible: Always Steps to Reproduce: 1. 2. 3.
parts of mozilla autodetect whether the unicode functions are available and use them if so... that seems like a better solution, because that way you don't need a different binary for NT vs 9x
Sounds fine, if they work. This doesn't seem to work for the issue raised in 193358.
For sure, we do not want to build two separate binaries. We don't have to even if we make Mozilla a Unicode binary as long as MSLU(Microsoft Layer for Unicode) is available on Win9x/ME. Among programs that bundle MSLU is MS IE. The question is whether we want to keep pretending that Mozilla can be run without MS IE's presence. Currently we do and that's, I guess, why Roy and Frank (ftang) decided to roll out our own mini version of MSLU . Bug 162361 is about extending it for XPCOM file i/o (there's already a patch), but that has been held back because I never heard back from my repeated emails to NSPR owners. Recently, I realized that some parts of Mozilla already rely on APIs offerred by MSIE. That is, Mozilla can't run without MS IE. That being the case, rolling out our own version of MSLU doesn't seem to do any good although it makes Mozilla further depend on the presence of MS IE (but it already does as far as I can tell.) Instead, we'd be better off making Mozilla a Unicode binary as suggested here. That way, we can cut down a quite lot of code. This is also a 'policy issue' and I'm adding brendan and chofmann to cc.
Assignee: smontagu → jshin
Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: intl
If the heading is anthing to go by, then MSLU appears to be a redistributable: http://www.microsoft.com/downloads/details.aspx?FamilyID=73ba7bd7-ed06-4f0d-80a4-2a7eeaee17e2&displaylang=en So we could just include it in a regular Mozilla build. Or am I mis-reading MS's definition of 'redistributable'? Of course the is probably affected by 'policy issues' that I am not aware of.
erm, we're backporting support for nt3.5, if you know of some pieces of mozilla which don't work w/o ie, i'd like to hear of them. (i believe trident import needs ie, but well, it's a bit hard to import ie cookies w/o an ie to generate cookies.)
(In reply to comment #3) > Recently, I realized that some parts of Mozilla already rely on APIs offerred by > MSIE. That is, Mozilla can't run without MS IE. That being the case, rolling out Which parts? And how large is the MSLU redistributable? Depending on licensing, we could probably make it an optional extra download with installer builds with minimal trouble.
We use IUniformResourceLocator APIs that I believe are only available with MS IE. http://lxr.mozilla.org/seamonkey/search?string=IUniformResourceLocator http://search.microsoft.com/search/results.aspx?View=msdn&st=a&qu=IUniformResourceLocator&c=0&s=1 As for the size of MSLU, I guess it's less than 1MB (the page refered to in comment #4 indicates that it's 270kB when compressed. )
(In reply to comment #7) > We use IUniformResourceLocator APIs that I believe are only available with MS IE. but we fail gracefully when it's not there (we don't support .url files in that case but that's ok)
what biesi said, although for kicks mozilla could offer an impl of that class (i'd probably go for it as an xpi perhaps stick it w/ the activex control). note that without a provider for CLSID_InternetShortcut it should be fairly hard for someone to manage to get those files onto their system or into memory (just as it would be hard to need to worry about importing cookies from ie if ie wasn't installed). if you want to file a bug reminding people to consider adding support for it, you could, i think it'd even be reasonable to stick it in the tracking pool for 35.
Blocks: 162361
Blocks: 167128
In bug 162361, Brodie came up with an alternative to MSLU, rolling out our own version of MSLU ('MZLU'). A number of bugs (including bug 232969, bug 240272, bug 9449, bug 243618) can be fixed easily (without cluttering the tree with two branches of code for Win 9x/ME and Win 2k/XP) if we assume or ship MSLU or add 'MZLU'. This issue (Mozilla not being able to take full advantage of Unicode capability available on Win 2k/XP) is long-standing and I think we have to resolve this issue before long. It'd be great if we can resolve it in the trunk soon enough so that we can back port the change to aviary-1.0 branch as well as to 1.7branch. If we decide to add 'MZLU', we can do it gradually (see attachment 150929 [details]. see also bug 162362 and bug 118013) implementing necessary APIs as we move on to 'W' APIs through out the tree. Brodie, would it be possible to write MZLU in such a way that it can be a drop-in replacement for MSLU (perhaps not)? If that's possible, we can offer it as an additional download that is needed *only* on Win 9x/ME *without* MSLU installed. Even if it's not, adding MZLU may be a good solution if we're worried about potential licensing issues with MSLU (bug 162361 comment #30) Thoughts?
Status: NEW → ASSIGNED
It is too much effort to try and make a 'drop-in replacement' for MSLU. Either we roll our own customized minimal version and use it always, absorbing the cost of associated bugs, maintenance and code size, or we actually use MSLU. Using MSLU is of course the best solution (the Microsoft approved solution). Can ANYONE tell me exactly why MSLU is considered taboo? Is it licence? Is it the build requirements (intimated by bug 162362 comment 4)? I can't find a definitive history for this decision. If the MSLU import library (unicows.lib) build requirements is a problem, then there is a project at sourceforge (MIT licence) which provides a replacement import libraries for Microsoft's MSLU which can be used with lots of different compilers. I'm not sure if this is even required (as I don't know the problem). http://libunicows.sourceforge.net/
(In reply to comment #11) > ... > of associated bugs, maintenance and code size, or we actually use MSLU. Using > MSLU is of course the best solution (the Microsoft approved solution). I wish I could use it right now :-) > Can ANYONE tell me exactly why MSLU is considered taboo? Is it licence? Is it > the build requirements (intimated by bug 162362 comment 4)? I can't find a > definitive history for this decision. I also wish to know, but I'm afraid it's 'buried' somewhere between ftang and roy who are not active any more in Mozilla project. If those currently active on the project don't see any problem with using (and shipping) MSLU, let's just use it. > If the MSLU import library (unicows.lib) build requirements is a problem, then > there is a project at sourceforge (MIT licence) which provides a replacement > import libraries for Microsoft's MSLU which can be used with lots of different > compilers. I'm not sure if this is even required (as I don't know the problem). > http://libunicows.sourceforge.net/ Thanks for the pointer. For some reason, if we can't use MSUL, this seems to be a good alternative.
The sourceforge project is not a replacement for unicows. It is only a replacement loader for unicows. It still uses the original Microsoft supplied redistributable unicows.dll Who has to be approached in order to get mozilla.org legal people to check the MSLU licence and make a decision there?
(In reply to comment #13) > The sourceforge project is not a replacement for unicows. It is only a > replacement loader for unicows. It still uses the original Microsoft supplied > redistributable unicows.dll Yeah, I realized that after visiting the site and going through the cvs. > Who has to be approached in order to get mozilla.org legal people to check > the MSLU licence and make a decision there? I added Bart to Cc hoping that he may shed some light on the issue or find someone who can. Bart, can you help us here?
Blocks: 240272
Blocks: 243618
I can try and figure out if what you want to do is allowed, if someone summarises exactly what it is that you do want to do :-) I'd also need links to the relevant license documents. Gerv
To make Mozilla take the full advantage of Unicode capability on Win 2k/XP while making it still run on Win 9x/ME are _without_ adding a whole bunch of |if (isNT) use 'W' APIs else use 'A' APIs|-like code (see bug 232969 for an example), we have a few options: - Ship the whole or a part of MSLU (=unicows.dll + unicows.lib) with Mozilla Win. Because we don't want to break compiliation with mingw, we may want to ship unicows.dll from microsoft (actual implmentation that translates 'W' APIs to 'A' APIs on Win 9x/ME) but use libunicows (import library that hooks up 'W' API calls to the 'translator' in unicows.dll if it's on Win 9x/ME. Otherwise, native 'W' APIs will be used) at http://libunicows.sourceforge.net (see comment #13). Taking this way, we don't have to rely on the presence of MS IE or other applications (e.g. MS Word/Office) that include MSLU - Just assume the presence of MS IE or other applications that include MSLU. In this case, we have to require MSIE be installed. We already have a couple of dependencies on MSIE in our code, but according to cbie, we have a graceful fallback in absence of MSIE (see comment #8) - Roll out our own version of MSLU (what Brodie dubbed 'MZLU' and had a partial implementation). In resolving bug 9449, Roy Yokoyama used a similar approach that can be utilized only in a small portion of Mozilla-code. In all three cases, we have to change our code throughout the tree to use 'W' APIs instead of 'A' APIs. The former use 'UTF16' strings while the latter use the locale-specific code page (what MS used to call 'ANSI' code page) / multibyte strings. This can be done gradually once we make up our mind about MSLU. Now as for the MSLU license, I can't find the license terms for MSLU itself at the MSDN web site. It's one of what MS called 'redistributable ???'. Brodie, can you give us the URL of the MSLU license? You quoted a part of the license (bug 162361 comment #30), which I don't think we can possibly violate. MSLU announcement : http://www.microsoft.com/globaldev/handson/dev/mslu_announce.mspx : MSLU MSLU download : http://www.microsoft.com/downloads/details.aspx?FamilyId=73BA7BD7-ED06-4F0D-80A4-2A7EEAEE17E2&displaylang=en MSDN article about MSLU: http://msdn.microsoft.com/msdnmag/nettop.asp?page=/msdnmag/issues/01/10/MSLU/MSLU.asp&ad=ads.ddj.com/msdnmag/premium.htm Known MSLU bugs: http://www.trigeminal.com/usenet/usenet035.asp
Attached file Unicows licence —
This attachment is the licence ('licence.txt') from the MSLU download. The following text is the extra redistribution clause ('redist.txt') from that same download (MSLU 1.0.4018.0, download link supplied in comment #16). I couldn't see any plainly obvious reason that we couldn't redistribute MSLU ourselves with the exception of clause "Distribution Terms" (c). Even this clause could probably be worked around by including the relevant terms in our installer (e.g. this package contains software copyright by Microsoft, you must abide by there terms for all of that software, namely "MSLU"). Note that it is highly desirable to distribute unicows.dll ourselves instead of depending on it being on the machine. This is so that we know what version of MSLU we are using (as there are a variety of versions out there with a variety of bugs). It is very desirable to fix this problem sooner rather than later, either via MSLU or via a roll-your-own version like MZLU (see bug 162361). Every day more people are inserting more roll-your-own functions like MS/ZLU implements, but at a higher level which leads to greater codesize for everyone. --redist.txt-- =============================================== Microsoft Layer for Unicode on Windows 95/98/ME =============================================== In addition to the rights granted in Section 1 of the Agreement ("Agreement"), with respect to UNICOWS.DLL, you have the following non-exclusive, royalty free rights subject to the Distribution Requirements detailed in Section 1 of the Agreement: (1) You may distribute UNICOWS.DLL with the following: Windows 95, Windows 98, Windows 98 Second Edition, Windows Millennium, Windows NT4, Windows 2000, Windows XP, and Windows Server 2003.
Attached patch MZLU 0.1 (by Brodie) (obsolete) — — Splinter Review
While it's being decided as to what to do with MSLU, Brodie updated his MZLU (he uploaded the zip file to bug 162361) from which I've made a patch. If we decide to take this route (or are forced to because of licensing issues), we have to adjust its build procedure. Currently, Brodie's patch has a MS VC++ project file. We need to convert it to Makefile.in.
adding cls to cc for a (potential) build issue (either with MSLU or with Brodie's MZLU)
Having looked at it, the MSLU license is (fairly predictably) not even vaguely free-as-in-speech. So we don't want to be shipping it with our products. So we can either use our own implementation (MZLU) or depend on IE. Given that IE has shipped with every version of Windows from 95 up, isn't that the easiest solution? Or do only newer versions of IE come with MSLU? Gerv
(In reply to comment #20) > Having looked at it, the MSLU license is (fairly predictably) not even vaguely > free-as-in-speech. So we don't want to be shipping it with our products. thanks for reviewing the license gerv > Given that IE has shipped with every version of Windows from 95 up, isn't that > the easiest solution? Or do only newer versions of IE come with MSLU? it sounds like there are different versions of MSLU floating around, and some older versions have serious bugs. i suspect that we might therefore prefer to use MZLU on older Win32 platforms at least (Win9x in particular). we might experience less pain if we simply include MZLU... provided the size is not too large, which I doubt it will be. brodie: can you comment on the codesize overhead of MZLU?
> So we can either use our own implementation (MZLU) or depend on IE. > > Given that IE has shipped with every version of Windows from 95 up, isn't that > the easiest solution? Or do only newer versions of IE come with MSLU? MSLU was released only in 2001 so that an old Win9x box with MS IE 3 or 4(?) and without any relatively new application that includes MSLU wouldn't have MSLU. I'm sorry I forgot to mention this. Besides, do we really want to rely on MS IE? Well, the browser war was over, but relying on MS IE appears to giving in to MS' argument that MS IE is a part of the OS. Even if that's not much a concern, we still have to figure out how to deal with old Win 9x boxes (there shouldn't be many of them that fit the description above, but Mozilla is supposed to run on them, too). It seems like we have to roll out our own at the cost of increasing the code size by a few hundred kB's (probably a lot less)
Thanks for the extra info jshin... > It seems like we have to roll out our own at the cost of increasing the code > size by a few hundred kB's (probably a lot less) I'd be surprised is MZLU measures up to even 100 kB. I'd expect much less. I don't think we can decide to take it or leave it until we know exactly how much codesize increase we are talking about here :-)
we are working on not requiring windows4 and we're also interested in supporting windows nt 4 without ie.
The current version of MZLU is about 30Kb. The full version of Microsoft's MSLU is 240Kb. I expect that once we implement all of the functions that we need in NSPR and Mozilla we might hit about 80-90Kb in MZLU but I wouldn't expect much more. This code size will only be experienced on Windows 9x as NT doesn't use the DLL. In addition to this, all code modules (.exe and .dll) need to link to the mslu static lib and call the initialize function at least once before using any other MZLU functions. The current version of MZLU intentionally sets all function pointers to NULL in order to force an error if it is used without initialization. Linking with this static lib will probably increase sizes by a couple of Kb at most. Since the 'W' version of all wrapped functions are referenced in the static lib, it will probably increase the system DLL dependencies for all modules which use the static lib to include all DLLs implementing those functions. e.g. shell32, commdlg32, winspool. Since they are all system DLLs and are all used somewhere in the Mozilla code anyhow this shouldn't have any noticeable effect on working set, but startup times might be slightly increased. If we use MZLU on any platform we need to use it on all platforms otherwise we are going to have bug reports caused by MSLU being used in one case and MZLU used in another. Also with the current implementation it's not a mix and match situation. MZLU functions are postfixed with 'Mz' instead of 'W' so they aren't directly interchangeable. Note: It might be possible to make it compatible with MSLU (i.e. export 'W' postfixed functions) and use the libunicows loader from sourceforge to load it, but I think that will require us to create stubs for the hundreds of functions that MSLU implements and we don't. Easier to stay with readily identifiable 'Mz' == MZLU function names, and our own loader to minimize problems. timeless: windows4 Do you mean IE4? What are the realistic minimum requirements for Moz? Windows 95 or Windows NT4 (no IE required on either)? If someone removes IE altogether from Windows (http://www.litepc.com/) and installs Mozilla should it work?
Attached file MZLU 0.2 (obsolete) —
Definitive figures for MZLU 0.2. Linking with the static lib increases .exe file size by 4Kb. The DLL is about 50Kb (49Kb when built with Mozilla build harness or 56Kb directly with VC7). There are currently 70 functions implemented. This gives an overhead of about 50 bytes per function added. It's still a bit early to add MZLU as a patch, so I'll just attach the .zip file still. Still very much a prototype. This zip includes the Makefile.in files required for building under the xpcom tree. Not sure if this will be the final location. Extract the files below xpcom to create the mzlu and mzlu/mzlua directories. Add xpcom/mzlu directory to xpcom/Makefile.in. Re-run the make at xpcom.
Attachment #152021 - Attachment is obsolete: true
By Windows 4, I mean Windows 95 or Windows NT 4. By not requiring Windows 4, I mean supporting Windows NT3.51.
Timeless: who is the "we" in "we are working on..."? Gerv
biesi, neil, and me.
(In reply to comment #29) > biesi, neil, and me. What issues are there in NT 3.5x support if we go ahead with MZLU and using 'W' APIs throughout the tree? Apparently, NT 3.5x is not officially supported by mozilla.org. So if there's no unsurmountable (or rather too-hard-to-reconcilate) conflict between making Mozilla fully-Unicode-capable and supporting NT 3.5, that's great. If there are such problems that it will take us long to resolve, I'd say that we should give the priority to making Mozilla take the full advantage of NT4 or later's Unicode capability.
There is no reason for current and new Mozilla releases to support NT 3.51.
Actually Mozilla builds and almost runs correctly on NT3.51 I only know of three issues: System fonts come in cell height sizes instead of character height Bitmap fonts don't work Bookmarks don't load (dependency on IE, would you believe...)
codesize overhead of MZLU seems well within reason to me.
However we work out the NSPR API issues, NSPR will need to use MZLU in order to support the wchar versions of the API on all platforms (anything else is useless). Should MZLU be a standalone library that NSPR is dependent on? I would prefer to see it at least stay as a library that can be used separately from NSPR to make it useful for other projects that may wish to use it without using NSPR. Going back to MSLU... Could we get around the licence problems with MSLU by creating a new application which abides by the licence terms and then recommend that users install it when they don't already have MSLU? Perhaps an extension that users on Win9x are 'recommended' to install to actually make Mozilla work (if they don't already have MSLU on the system).
Gerv: For the future hundreds of developers who are going to wonder why the heck we developed our own version of MSLU when the real version was out there begging to be used, would you outline exactly where the problems is with redistributing MSLU with Mozilla? Which are the clauses which cause it to be legally impossible?
We don't want NSPR, at this late date, to depend directly on some new library. wtc can speak authoritatively; I'll venture that we could add callbacks defaulting to null, not called unless non-null, that could be set at runtime to point at MZLU exports. See for an analogue http://lxr.mozilla.org/mozilla/search?string=JSLocaleCallbacks. /be
Personally I would still like to see it dependent on Microsoft's redistributable component MSLU and not on some new library at all. Otherwise it doesn't particularly matter to me how it is accomplished, whatever way is considered politically correct is fine. As long as the unicode version of the NSPR API calls the unicode windows functions on NT platforms and falls back to ANSI functions on non-NT platforms. Since MSLU and MZLU implement this automatically it makes sense to use one of those libraries.
Would it make any difference if the MSLU binary was downloaded for MS and installed at install time? If it is legally acceptable then I would imagine the installer getting the current URL of the file from the Mozilla site (in case the binary gets moved at MS), downloads it and then get the user to compelete the installation. Is there any way of getting the installer to check for the availability of a library call before installing an optional component?
(In reply to comment #38) > Would it make any difference if the MSLU binary was downloaded for MS and > installed at install time? depending on a present internet connection at install time is not acceptable.
What if we just make MSLU a requirement. Similar to the x Mb of memory and x Mb of disk? The installer checks to see if it is installed by searching for the version we require in common locations, if we can't find it then the installer allows download if download is available, otherwise it's bye-bye, you need to install MSLU before you can install Mozilla. The trouble with this is that there is no standard install location or name for the DLL. It is distributed named as Unicows.dll but you can rename it (it's recommended by Microsoft too). It's not supposed to be installed into the Windows or system directories, etc. There is no real location that we can look for it and know that it exists, except perhaps searching for products like IE that we know use it. I guess I don't really understand the problem with redistributing the DLL. Gerv says that it isn't free as in speech, but is that important in this situation? It is a royalty-free redistributable system DLL provided by Microsoft for this exact situation. As long as there are no actual *legal* issues to installing it ourselves, we should. It reduces bugs on our side, it is well supported by Microsoft, it is the official solution to this problem, it reduces our source code size, it has everything going for it and it is redistributable for crying out loud.
Brodie: it's pretty simple. MSLU is not free software, and mozilla.org distributions contain only free software (except Talkback, for historical and deeply practical reasons which I wish would go away but probably won't.) I don't really need to outline why it's not free software - it's clear to anyone who knows what the terms means and then looks at the license for 30 seconds that it's not. :-) There's nothing wrong with us depending on MSLU being present, or telling people to install it before they install Mozilla, but we won't ship it ourselves. Gerv
Attached file MZLU 0.3 (obsolete) —
Gerv: Is this being overly moralistic? If we are going to make Mozilla/Win dependent on a redistributable component of the OS, then isn't it better to distribute that component in order to reduce the installation problems for users on Win9x. Does the distribution of MSLU with Mozilla/Win affect the tri-licencing of Mozilla? Whoever has the ability to make a decision: Is it satisfactory to distribute Mozilla/Win with the requirement on Win95/98/ME that MSLU has been installed in the windows/system32 directory first? i.e. unicows.dll of a specific version or better exists in that directory? On installation we can then either use the file from that location, or copy it to our installation directory to have the local copy which is recommended by Microsoft. It is then left up to customers to ponder how to fulfil this requirement. And finally, in case MSLU isn't used, attached is MZLU 0.3 with support for registry functions. Included is the patch for xpcom/Makefile.in
Attachment #152095 - Attachment is obsolete: true
I don't understand why we can't ship 3rd party redistributables in our distributions. Even the GPL would probably allow this since this Unicode layer is pretty clearly a piece of the OS and normally distributed with the OS these days.
Gerv: MSLU should be treated the same way as MSVC's runtime libraries. If mozilla.org distributions can contain only free software, mozilla.org will not be able to ship a distribution compiled with the current version of MSVC (.NET 2003) because you can't ship its runtime libraries (msvcr71.dll and msvcp71.dll). It is obvious to me that this rule must make an exception for libraries that are considered part of the programmming language runtime or the platform.
No longer blocks: 243618
Whatever we do, the default Firefox packaging should not include extra stuff needed only on downrev Windows. Special package(s) for those OS versions would be fine with me. /be
Blocks: 250255
roc: as it happens, the mozilla.org binaries are not distributed under the GPL, they are distributed under the MPL. But, if they were distributed under the GPL, we could not distribute MSLU. The GPL requires you to distribute the entire source code of your application (section 3.a). The exception you may be thinking of runs as follows: "However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable." The component would be accompanying the executable in this case, and so the exception does not apply. Gerv
Wan-Teh: it may well be true that this rule means mozilla.org can't ship binaries compiled against MSVC .NET 2003. But why would we want to, when we can compile binaries which work out of the box on all Windows platforms without need of new system DLLs? Having said all that, producing "free software" on top of a non-free operating system is always going to have grey areas. But it seems to be that, if we have an alternative, introducing a dependency on a proprietary library seems a bad step. What are the disadvantages of using MZLU? Gerv
The disadvantage of MZLU is that it is completely new written from scratch. It has a mildly different API from the system API (i.e. Mz suffix instead of W, maybe not all API side-effects implemented fully). It doesn't implement the entire Win32 API and so we need to continually extend and maintain it as extra/different functions are required. To use it in NSPR (which needs to have a Unicode API) we need to create a dependency from NSPR to MZLU which isn't popular. The advantages of MSLU is that it has been in use for a long time, it's stable and supports everything that we need now and in the future without any development work. The API is completely specified, documented and tested. NSPR can just use the W functions, link to the MSLU loader, and leave the problem of getting the MSLU DLL to the client up to the user of NSPR. Brendan: the compressed DLL is about 85Kb and is the entire difference between NT and 9x versions of Mozilla. I think it would be far more convenient for customers if we shipped a single distributable for all platforms. Gerv: you address why the GPL doesn't permit the redistribution of OS components but you didn't address the MPL which is what we distribute under.
(In reply to comment #47) > Wan-Teh: it may well be true that this rule means mozilla.org can't ship > binaries compiled against MSVC .NET 2003. But why would we want to, when we can We already ship the MSVC6 runtime libraries in seamonkey installer builds, because some win95 (and NT4?) machines don't come with them.
We should also consider how this will impact extension authors. If I write an extension, and want to use Win32 Unicode APIs, it would be nice to utilize the same solution that Mozilla is using. If Mozilla is using MZLU, and if MZLU only consists of entry points for methods that Mozilla is using, then what happens when my extension needs to use other Win32 APIs?
I'll bring this issue up at the next staff meeting. Does anyone have a link to the redistribution license for the MSVC6 runtimes that we currently distribute? If it's the same, then I guess distributing MSLU wouldn't be any worse. Gerv
OK, chaps, this sinks us, surely? From attachment 151916 [details] (Unicows licence): "(d) you do not permit further redistribution of the Redistributable Components by your end-user customers;" We just can't live with that restriction for Firefox builds. (c), which is about a click-wrap license, is bad too. Gerv
This issue of the Unicows license is really a big sticking point and seems to be the main thing holding us back. I have already suggested at the start of all this, that maybe two versions of Mozilla for Windows might be the best approach, in the same way that there was a MacOS 9 and MacOS X version (MacOS 9 support now being discontinued). The first version would be rated 95-NT, and the other version would be Windows 2000 upwards. This had been rejected, but I am wondering whether that maybe now it would be the only way to go forward, maybe as a temporary solution until we can get a complete MZLU finished? Having a choice of two binaries is not something that is new many Windows users, since there already a number of programs that a distributed in this way. Something else: Out of curosity, how much of the code actually has Windows API hooks that would be affected by UTF8? I ask this since another approach would be to have two shared libraries distrubted with the Windows version (one for UTF8 and the other without) and then depending on the availibity of the Unicode API load the appropriate shared library, based on an initialisation test.
Yes, it does seem to be a problem. How about this solution: * create a separate product with minor functionality that installs a known version of unicows.dll into a known location. It has a simple installer, is completely optional and is licenced using MS happy terms: - click-wrap - non-redistributable * Use of Mozilla on Windows 95/98/ME has a requirement that the MSLU must already exist as 'unicows.dll' version 1.0.4018.0 or greater in the windows directory. * Users of Mozilla on 9x/ME have the option of installing IE, or the minimal app above, or just download MSLU from the MS site itself and copy it to the windows folder. ajmas: two binaries won't really solve anything here. We are trying to get a single binary that works on all platforms leveraging the Unicode API of NT/2K/XP and yet falling back to ANSI codepages on 9x/ME. The solution needs to be final rather than temporary. It should be soon to stop the cruft of ANSI/Unicode wrappers that is not-so-slowly creeping into the code base all over the place. UTF-8 is not used internally by Windows, but may be the representation used to talk to the NSPR library. Also, simply for interest's sake, here is the licence for VC.NET 2003: http://msdn.microsoft.com/visualc/vctoolkit2003/eula.aspx
> maybe two versions of Mozilla for Windows might be the best approach, We can't distribute MSLU with _any_ mozilla.org software, because MSLU forbids redistribution by the people we give it to, and we can't (or won't) take the steps they require us to take (clickwrap license etc.) to prevent that. Also, doing such a thing is basically falling into the Java Trap, or something like it: http://www.gnu.org/philosophy/java-trap.html Our software becomes non-free because it depends on a non-free component being included. > create a separate product with minor functionality that installs a known > version of unicows.dll into a known location. Note that the MSLU license says that anything that redistributes the DLLs must add "significant and primary functionality to the Redistributable Components". Gerv
(In reply to comment #55) > Note that the MSLU license says that anything that redistributes the DLLs must > add "significant and primary functionality to the Redistributable Components". Yes, I'm aware of that. Since MSLU only implements a Unicode API, any program that actually does something fulfils that clause. e.g. display the version of windows libraries, or run in the background and check a pop mail box. It seems like everyone here is acting in an adversarial manner. Instead lets work together to find a way to use this software. Mozilla needs to use Unicode on Windows to solve a heap of problems that are plagueing those pesky people that don't use single byte character sets. For the reasons supplied in comment #48, the best way to do so is to use MSLU which is a non-free redistributable part of the OS, the same as the C runtime libraries. My suggestions for how to avoid the licencing issue is that we just make MSLU a required part of the OS spec. To then make it easier for users, we supply a little non-free program (required by MSLU licence) which will add the appropriate files to their OS. Their use of the non-free program is optional, they can also use any of the MS non-free programs that use MSLU to install it.
(In reply to comment #55) > Also, simply for interest's sake, here is the licence for VC.NET 2003: > http://msdn.microsoft.com/visualc/vctoolkit2003/eula.aspx This license seems as unsuitable for us as the MSLU license (in light of comment #52). It has a similar provision for the redistribution. How about the license of MSVC6 run-time libraries mentioned in comment #49? If it has a similar provision, according to Gerv, we have to stop including it .....
As far as I can tell, the Visual C++ 6 terms are different. I haven't found an official copy yet, but I found: http://technology.pitt.edu/software/userights/visual_studio.pdf Section 3.1.1 of that document has an exception for people re-redistributing your stuff, which probably covers us, and it doesn't require us to impose a particular EULA on our customers. So we are probably OK with Visual C++ 6 redistributables. The VC.NET license, as linked in comment 57, does have a redistribution exception: "You also agree not to permit further distribution of the Redistributables by your end users except you may permit further redistribution of the Redistributables by your distributors to your end-user customers if your distributors only distribute the Redistributables in conjunction with, and as part of, the Licensee Software and you and your distributors comply with all other terms of this EULA." But it's the last part of that sentence (which seems to tie back to the new requirement to impose a EULA on your customers) which would present a problem. We can't ensure that our distributors comply with all the other terms of the MS EULA. Time to start polishing up the mingw port? :-) Gerv
In reply to comment #56) > plagueing those pesky people that don't use single byte character sets. For Just to dispell any misunderstanding, I have to make it clear that the above is not correct (I know Brodie knew that, but he oversimplified). It's not single byte character encoding vs double/multibyte character encoding. It's about going beyond the limit of a single _legacy_ character encoding to take the full advantage of Unicode support available on Win NT4/2k/XP (e.g. a Russian user wanting to have a Greek filename) while keeping Mozilla still usable (with all the limit of a single legacy character encoding) on Win 9x/ME. > My suggestions for how to avoid the licencing issue is that we just make > MSLU a required part of the OS spec. To then make it easier for users, > we supply a little non-free program (required by MSLU licence) which > will add the appropriate files to their OS. Their use of the non-free program > is optional, they can also use any of the MS non-free programs that > use MSLU to install it. As long as we make Mozilla and friends for Windows, we can't avoid 'Java trap' completely. That being the case, the above suggestion seems to be workable although further developing MZLU and offering it to other open source developers wouldn't be bad, either. re comment #55: >> maybe two versions of Mozilla for Windows might be the best approach, > We can't distribute MSLU with _any_ mozilla.org software, because MSLU Gerv, you misunderstood what ajmas suggested. He suggested that to _avoid_ the licensing issue with MSLU, we make two separate binaries, one for Win 9x/ME that uses 'A' APIs and the other for Win NT4/2k/XP that uses 'W' APIs. __Neither__ of them depends on MSLU. In comment #3, I wrote that that's not a good idea because it makes it very hard to manage and test the code with a lot of #ifdef's. In that case, we also need to set up two separate tinderboxes for each of our products (one for 'A' APIs and the other for 'W' APIs)
> Gerv, you misunderstood what ajmas suggested. He suggested that to _avoid_ the > licensing issue with MSLU, we make two separate binaries, one for Win 9x/ME that > uses 'A' APIs and the other for Win NT4/2k/XP that uses 'W' APIs. Ah, right. Sorry about that. I thought he meant one with MSLU and one without. Gerv
I was just doing yet another security update for MS-Windows and something struck me. Users need to have Internet Explorer installed on their machine to be able to run Windows update. If they don't have it then their machine is probably a security nightmare. Windows update will work with nothing else. Based on that I am wondering whether it is safe to assume that all users will have Interner Explorer installed in some form. Given this I am wondering to what extent they are likely to have the unicows.dll on their machine. I know am going down the dangerous road of assumptions, but maybe a survey in this direction might help us?
Blocks: 251897
I think that in corporate/enterprise environments some sort of centralised push solution is usually used to automatically distribute patches, updates and new applications to the desktops (Zenworks comes to mind), so you can't assume that IE is installed at all. On the other hand in such environments older versions of Windows would have long since been phased out.
Let's see what we can do: 1.MZLU: Unlike Brodie, I'm not worried that it's not so good as MSLU. It may be, but we have it under our control so that if there's a problem, we can fix it. The only downside with this approach is what Darin mentioned in comment #50. Plugin authors could be impacted similarly. Perhaps, we have to put up a web page for extension/plugin authors in which it's clearly mentioned what needs to be done if they want to use 'W' APIs for which there's no wrapper in MZLU. The page also has to list all Win32 APIs with wrappers available in MZLU and their naming scheme. 2. Shipping MSLU seems to be out of question (comment #55, comment #52) Gerv> we can't (or won't) take the Gerv> steps they require us to take (clickwrap license etc.) to prevent that. 3. Make it a requirement on Win9x/ME that MSLU be present (by installing non-free programs that include it or by installing Win SDK kit?). A trouble with this is that its location and name can vary (I can't find unicows.dll on Windows ME although MS IE 5(?) is present). Adding details about this (where MSLU should be present in what name) to the release notes (and perhaps some other very prominent places) might solve the problem, but it would be a hurdle for non-technically oriented people (I wish they all had thrown away Win 9x/ME...). Given all these, I'm inclined to go with the first option. If we decide to go with it, I'll volunteer to write a short document for extension(/plugin) authors about using Win32 APIs.
We could take option 3 by surveying where common apps put their copy (and what they call it), and trying to load them all in turn. It's a bit hacky, but I suspect we could get fairly good coverage. Gerv
I like option #1 too. I think it will give us fewer headaches, and at least we have access to all the knobs. If it's true that older versions of MSLU are buggy, then I can only imagine that we are inevitably going to inherit some of those bugs in some cases until we eventually get it all ironed out :-(
I also prefer option 1. forcing users to install a seemingly unrelated app just so they can use mozilla seems like no good idea to me.
(In reply to comment #50) > consists of entry points for methods that Mozilla is using, then what happens > when my extension needs to use other Win32 APIs? I guess they have to extend MZLU to implement their functions and then distribute an updated version of the MZLU DLL with their extension. Either they update our MZLU DLL with the new one (we need to add version info to the DLL to allow this) or implement MZLU such that they can run side by side with different DLL names and the extension somehow loads their version of the DLL (assuming that the extension calling Win32 API is a DLL this should be possible). If we allow extension authors to upgrade the MZLU DLL then we need to ensure that there is only a single instance of that DLL to be upgraded (e.g. all bin, bin/components, GRE modules that link to it all load the same DLL). For instance by making it a file in Program Files/Common Files. If going with option 1 (using MZLU) then we need to implement the NSPR dependency on MZLU somehow. Callbacks were mentioned in comment #36, but since the API of MZLU is already function pointers, this seems somewhat strange. Since the NSPR Unicode implementation will be a special mode or API to maintain backwards compatibility (currently it's a special API enabled via compile time defines), the user of NSPR will need to intentionally use the Unicode enabled functions. In this case surely a new library dependency shouldn't be a problem. wtc?
Since everyone seems to prefer using MZLU, and it would provide other open-source projects with a free version of MSLU type functionality, I'm happy to continue and develop it. I need to know where to place it in the source tree. It seems to me that it should be a top level component for the following reasons: a) dependency from NSPR (when MOZILLA_UNICODE or similar defined) b) dependency from rest of Mozilla c) we want to provide it as a standalone project for others to use
Brodie: we could put it in mozilla/mzlu, on the theory that a big top-level is best for finding things (tyranny-of-hierarchy makes any attempt to categorize modules not scale well over time and people). But we do try to group like things, where we have a plurality of such things, or are likely to have a plurality in due course, under top-levels such as db, gc, and parser. So should we put this under mozilla/intl, or is that less a container of like things, and more of a singular module? Should we call the directory mzlu? Give me some thoughts, and as top-level mkdir judge, I'll render a quick verdict. Thanks, /be
I think we should decide the NSPR dependency before continuing. If NSPR were to depend on MZLU, it should live in the NSPR build tree (and we should get rid of those MOZ_UNICODE #ifdefs. At that point there is no reason I see not to just integrate the MZLU dynamic loading directly into nspr4.dll instead of having a separate library. However, we could alternatively move the NSPR MOZ_UNICODE functions (such as PR_OpenFileUTF16) into XPCOM or someplace where NSPR wouldn't have to depend on MZLU. Let's solve the link-time-dependency issue before settling on a tree location.
We could also make NSPR depend on MZLU not at compile time at all, which is a much better idea in my book, and probably in wtc's. Why shouldn't NSPR call through a configurable set of callback functions, default null, and be free of #ifdefs? Is it too ugly, due to too many callbacks? Or too costly? bsmedberg: I don't think MZLU is part of NSPR in any sense, so I don't see why it should live under mozilla/nsprpub. The source tree issue can be separated from other build and runtime issues, and should be in general. Build- and run-time orders vary over time and across apps. The tree does not. /be
> We could also make NSPR depend on MZLU not at compile time at all, which is a NSPR should drop the FooUTF16 methods altogether, and not pretend to be unicode-aware. Instead, move the unicode-aware methods into xpcom.dll, and have xpcom.dll depend on or integrate-in MZLU. --BDS
Whether NSPR depends on MZLU or not is irrelevant. MZLU should be a stand-alone library. Even if NSPR uses MZLU, NSPR won't export the MZLU symbols.
Perhaps it should go into whatever location that the separate libraries get put. modules or lib? Where is expat? That's in the tree somewhere isn't it? As for naming, I'm not particularly attached to MZLU if there is some problem with it. Suggest another name if necessary. If not then I suggest that we stay with it. There won't be any naming issues with Microsoft would there? The full name is obviously not mistakable as MS software. The shortened form is arguable. bsmedberg: I know that you are keen to reduce the DLL footprint of Mozilla and this is a good aim. I'm not sure that mzlu can be successfully merged into any of the other libraries though. The closest fit might be NSPR, but wtc isn't keen for that. At the moment the DLL is only 60-80Kb with 90 functions implemented, but if we ever get to MSLU like coverage of 520+ functions, I epxect that it will grow to similar proportions of MSLU, about 300-400Kb. Perhaps then the overhead of a separate DLL file isn't so important? In my local copy (after 0.3) I've changed the exported names of the functions to match the MSLU DLL (i.e. using xxxW function names instead of xxxWrapped). This allows us in future to possible use mzlu as a drop in replacement for the MSLU DLL. Combined with the libunicows loader noted above we have a full open- source mslu replacement. The static lib that NSPR would link with to provide the unicode API provides a number of function pointers implementing the 'W' version of the Windows API. This either calls directly into the Windows NT native W API, or it redirects it into the wrapped versions implemented by the mzlu DLL when running on 9x platforms. If the mzlu library isn't initialized then all of these function pointers are NULL. This is already very similar to the callback idea. The major difference is that the function pointers etc are the mzlu static library and not straight NSPR. Of course NSPR could reimplement the functionality of the mzlu static library (or copy it into NSPR code) too, removing the build dependency but adding a maintenance dependency. Similar to bsmedberg, I think that it would be best if the Windows NSPR got rid of the UTF16 versions of the functions, and instead implemented a single function which switches the meaning of the char* parameters from native to utf8 encoded. utf8 to/from wchar_t conversions are simple and pretty much foolproof. The same API is used, with the difference being the interpretation of the char* arguments. When put into Unicode/utf8 mode, NSPR would have to call the W API implemented by mzlu. Otherwise it would continue to call the A API natively supported on all platforms. Finally, a note regarding my availability. I have quit my job and will be on an extended vacation (anywhere from 1 to 4 months) away from all software development in a couple of weeks. It would be good to get some of this finalized before I leave if possible, otherwise I'm sure that jshin and others can keep moving it along even if it doesn't.
lib is gone, or should be -- an AT&T Unix-ish holdover from the old Netscape tree. modules is deprecated -- please don't add to it, unless you can convince me that it is not just a useless container for stuff better located elsewhere, possibly at top-level. Again I urge folks to avoid mixing build options, which might do very desirable things such as combine libraries to cut down on DSO overhead, with source tree layout. Source trees don't age or scale well if you try to subsume directories based on current library and program build bundling. The source tree should be more of a module catalog, and at sufficient scale, alphabetical ls listing is your friend, not any particular non-categorical hierarchy. /be /be
The scope of the mzlu project is providing a unicode wrapper around the Win32 ANSI code-page API used by Windows 9x/ME. It doesn't provide any international functionality other than converting text in/out function parameters between unicode and the current system codepage. The goal of the project is eventually to provide a complete open-source replacement for MSLU, but in the short term it is specifically focused on Mozilla's needs. I am not personally concerned if it exists in the intl directory or at top level, however perhaps top-level is more appropriate since a) an mzlu-like project will only be created once for Windows (no other platforms require it) b) it is only a character encoding wrapper around an api c) it is eventually intended for a wider audience then just mozilla Unless anyone has specific concerns about calling it mzlu (Mozilla Layer for Unicode), I suggest that we stay with mzlu as the project name and also use that as the main directory name in the build tree.
Only categorical hierarchy thought that comes to mind: is there any point in a mozilla/unicode container? If not, top-level mzlu is ok with me. I'm cc'ing smontagu. /be
Blocks: 253035
What would go into the unicode container other than mzlu? Is it just for future use? Again, I have no preference for where the library is located.
I've been chatting with Vaclav Slavik, the author of the MIT-licenced libunicows (http://sourceforge.net/projects/libunicows/) about how we might use libunicows with mzlu. In summary, they fit together perfectly - if we write mzlu as a drop in replacement for the MSLU library "unicows.dll" (i.e. all implemented functions use the same export names) then libunicows can load our DLL automatically if it can't find the real unicows.dll library. Vaclav is happy to modify libunicows to fit any mzlu special requirements as long as it keeps libunicows fully interchangeable with the real unicows.dll In order to use the libunicows/mzlu combination in Mozilla we need to build with UNICODE defined (or explicitly call the W functions), link to the libunicows static lib (ensuring libunicows comes first in the link list) and distribute mzlu.dll with the application. What I intend to do now is rewrite the mzlu project so that it is designed specifically to work with libunicows. I will get rid of the custom loader that I had written and will instead write only the DLL. libunicows will be a requirement for use of mzlu. This is a good thing because libunicows is smaller and faster than my basic loader. Licencing shouldn't be a problem as the MIT licence basically permits anything (Gerv?). I'll post (with permission) some of the emails that I have exchanged with Vaclav.
Extract from Vaclav's reply to my first email explaining the project and asking whether we could use libunicows with our own implementation of mzlu. The app that he mentions in the email is http://poedit.sf.net Brodie Thiesfield wrote: >> My question is: is it possible to use your libunicows loader to >> load our mzlu.dll (Why not unicows.dll? That would make it easier to be a complete replacement...) >> instead of the unicows.dll EVEN if we only >> have a subset of implemented functionality? i.e. can I use >> libunicows to load the current mzlu DLL? If so, what would >> happen to callers that try to call a function not >> implemented/exported from MZLU? I think it should work out of the box. You can already change UnicowsDllName variable to point to mzlu.dll instead of unicows.dll, but it requires special code in the app, which is a Bad Thing (the rest of libunicows is completely transparent to the programmer -- the *only* thing you have to care about is linking it in, no initialization is required (unlike MZLU, I gather?)). But if you decide on using mzlu.dll name instead of unicows.dll, I would certainly like to include its support directly in libunicows, e.g. by trying it if unicows.dll couldn't be found (or the other way around). As for subset of functionality, it shouldn't be a problem at all. Each function is handled separately in libunicows, there aren't dependencies an overything (such that ibunicows would, for example, try to load all 520+ symbols from unicows.dll). If you don't use FooW function, its loader code doesn't get linked into your executable at all. If your app uses FooW but MZLU doesn't support it, then loading the symbol will fail (on 9x, unicows.dll/mzlu.dll would not be even looked at all on NT/2k/XP), libunicows will display error message and terminate the app. This is no worse than if native FooW would be called (it would return error code and GetLastError would return "not implemented" or something), but if this behaviour would be unsuitable for you, you can change the UnicowsImportError pointer to point to a function that does something better. It's not currently possible to use UnicowsImportError callback to provide a replacement no-op version of unimplemented function, but if it would be needed, I can easily add such a callback to libunicows, it's just that I didn't need it myself yet. So in short, I believe there shouldn't be any problems, and if something shows up, I'm more than willing to help with fixing it so that you can use libunicows instead of wasting your time reinventing the wheel. (I'll a little selfish here, too -- I use embedded Gecko in my app and would like to make sure MZLU won't interfere with NT-only or MSLU builds, which it won't if you'd use libunicows ;-)
Extract from my reply: >>>>My question is: is it possible to use your libunicows loader to >>>>load our mzlu.dll > >> >> (Why not unicows.dll? That would make it easier to be a complete >> replacement...) I'll leave it up to the user if they want to rename it to unicows.dll, but I'll release it under mzlu.dll. That avoids any problems with name clashes and makes it a bit clearer if it is the moz version or the MS version. >>>>instead of the unicows.dll EVEN if we only >>>>have a subset of implemented functionality? i.e. can I use >>>>libunicows to load the current mzlu DLL? If so, what would >>>>happen to callers that try to call a function not >>>>implemented/exported from MZLU? > >> >> I think it should work out of the box. You can already change >> UnicowsDllName variable to point to mzlu.dll instead of unicows.dll, >> but it requires special code in the app, which is a Bad Thing (the >> rest of libunicows is completely transparent to the programmer -- the >> *only* thing you have to care about is linking it in, no >> initialization is required (unlike MZLU, I gather?)). But if you My current version is all pretty basic, hence the reason that I was wanting to use your loader. That would mean that I only need to write the DLL. One thing that I could do with the way I was writing mzlu was to wrap functions that don't seem to be wrapped by mslu (and so aren't supported by your loader). Things like some CRT functions which I find useful (e.g. _wgetdcwd). There are also the IME functions (Imm*) which are natively supported in Unicode on Win98+ but not on Win95, but don't seem to be wrapped by MSLU. I don't know if I need to yet, but I was thinking of wrapping them on Win95 to make them available on all platforms. >> decide on using mzlu.dll name instead of unicows.dll, I would >> certainly like to include its support directly in libunicows, e.g. by >> trying it if unicows.dll couldn't be found (or the other way around). Yes, that would be nice. Probably unicows.dll first since it is a complete implementation. >> As for subset of functionality, it shouldn't be a problem at all. Each >> function is handled separately in libunicows, there aren't >> dependencies an overything (such that ibunicows would, for example, >> try to load all 520+ symbols from unicows.dll). If you don't use FooW >> function, its loader code doesn't get linked into your executable at >> all. If your app uses FooW but MZLU doesn't support it, then loading >> the symbol will fail (on 9x, unicows.dll/mzlu.dll would not be even >> looked at all on NT/2k/XP), Does this mean I don't need to add code to the DLL to ensure that we pass the function call through to the system if someone calls an MZLU function on NT? I don't have it at the moment, but MSLU does this so I was thinking that I probably should too. >> libunicows will display error message and >> terminate the app. This is no worse than if native FooW would be >> called (it would return error code and GetLastError would return "not >> implemented" or something), but if this behaviour would be unsuitable >> for you, you can change the UnicowsImportError pointer to point to a >> function that does something better. It's not currently possible to >> use UnicowsImportError callback to provide a replacement no-op >> version of unimplemented function, but if it would be needed, I can >> easily add such a callback to libunicows, it's just that I didn't >> need it myself yet. Personally I feel that bailing out of the app with a message identifying the missing function is a little drastic. The function may not be completely necessary, so it would be nice if users were able to call it and detect if it is implemented in that version of mzlu or not. Not strictly necessary, just nice. >> So in short, I believe there shouldn't be any problems, and if >> something shows up, I'm more than willing to help with fixing it so >> that you can use libunicows instead of wasting your time reinventing >> the wheel. Great to hear. Thanks for the support. In summary, it seems that as long as I write to the spec and name all of the exports the same as MSLU, and for the moment set the UnicowsDllName variable we should be able to use MZLU now right out of the box. In future we should be able to use it without setting UnicowsDllName.
Vaclav's reply: >> One thing that I could do with the way I was writing mzlu was to >> wrap functions that don't seem to be wrapped by mslu (and so aren't >> supported by your loader). Things like some CRT functions which I >> find useful (e.g. _wgetdcwd). >> >> There are also the IME functions (Imm*) which are natively >> supported in Unicode on Win98+ but not on Win95, but don't seem to >> be wrapped by MSLU. Again, I certainly don't mind extending libunicows to do more than MS version ;) I was hit by missing wrappers for CRT functions myself, so if you support them, great! In that case, libunicows should try to use both unicows.dll _and_ mzlu.dll for best results. >> Does this mean I don't need to add code to the DLL to ensure that >> we pass the function call through to the system if someone calls an >> MZLU function on NT? Yes, exactly. >> I don't have it at the moment, but MSLU does >> this so I was thinking that I probably should too. MSLU does it in unicows.lib. Unicows.dll is only used on win9x platforms and unicows.lib (or libunicows) is not a "normal" import library; instead, it dynamically loads the symbols when they are first used (from unicows.dll on win9x and from system DLLs on NT/2k/XP). So you don't have to concern yourself with NT at all in mzlu.dll. >> Personally I feel that bailing out of the app with a message >> identifying the missing function is a little drastic. Right. It's better than outright crash, though. >> The function >> may not be completely necessary, so it would be nice if users were >> able to call it and detect if it is implemented in that version of >> mzlu or not. I can see two ways to do it: 1) Introduce libunicows-specific function that would check availability. The biggest problem with this is that it's no longer interchangable with MS' version of unicows.lib. 2) Let the app check by loading the symbol dynamically using GetProcAddress. Unicows.dll hijacks GetProcAddress too, so that when you call (pseudocode) GetProcAddress(kernel32.dll, FooW) and FooW is implemented in unicows.dll, it returns pointer to (working) FooW version from unicows.dll instead of (not working stub only) FooW from kernel32.dll. So if mzlu.dll returned NULL from GetProcAdress for functions that it knows it should wrap, but doesn't, the apps would have a way to recognize this situation while still being compatible with MSVC7's standard unicows.lib. There's no harm in returning NULL because the function that OS' "real" GetProcAddress would return would be useless stub anyway. The best would be to have both ways, of course. >> In summary, it seems that as long as I write to the spec and name >> all of the exports the same as MSLU, and for the moment set the >> UnicowsDllName variable we should be able to use MZLU now right out >> of the box. In future we should be able to use it without setting >> UnicowsDllName. Yes, exactly. (MSLU export names are exactly same as native libs' names, so exported names should match names of symbols in system DLLs (e.g. "DrawTextW@20" from user32.dll -- see symbols.txt in libunicows). I think that you can do some tricks in .def file to export functions under different names than what you use in source code...) -- So basically, assuming that we implement mzlu, we have pretty simple way to include Unicode API support into Mozilla already implemented for us. Of course there are other problems that need to be sorted out, but this is a very good start. Comments about using libunicows? Are there problems with this that I am naively overlooking?
Will this make bug 218242 any easier or harder to implement? It will want to pass a custom common dialog hook procedure...
It shouldn't make it any harder. You only need to implement the single Unicode (W api) version so it should be easier. Microsoft's unicows library supports the hook function, so we should be able to as well. However I don't think that I support the hook function in the current implementation of these functions in mzlu.
Attached file MZLU 0.4 (obsolete) —
MZLU as a drop in replacement DLL for unicows.dll. Currently implementing about 130 functions of the 520 odd that unicows does. Requires the use of the libunicows loader. Still has the diff and Makefile.in files to build under xpcom (until a final location has been decided on).
Attachment #152484 - Attachment is obsolete: true
Comments anyone?
Attached file MZLU 0.5 (obsolete) —
This version of MZLU does actually work as a drop-in replacement for unicows.dll for simple console Win32 programs. The included test program does a simple FindFirstFileW/FindNextFileW through the current directory. The included unicows.lib static library is a custom build of libunicows to force it to load unicows.dll even on NT platforms (to allow me to test on my XP development machine). This is still very much development version and there are definitely a myriad of bugs, but it is now at the stage where it should be able to be used and tested if anyone has the desire. I'm starting an extended vacation soon (for a couple of months), so this will be my last update of MZLU until I get back. If anyone else wants to keep developing it in the meantime then go ahead. Similarly if anyone wants to finish the discussion on CVS location, dependencies, etc.
Attachment #154440 - Attachment is obsolete: true
I agree that we should try to eliminate the ifdef MOZ_UNICODE methods in NSPR and invent a UTF-8 mode for NSPR (unless this proves to be too inefficient).
I stumbled upon a couple of problems while trying to compile attachment 1546638. Firstly, MSVC complained that '_stat64' is not found in the global namespace (MSDN says that it's available even on Win9x and it's in Win64/sys/stat.h of my copy of MS SDK). This may be due to a problem in my copy of MS SDK... The other problem is that 'mzlu.def' (DEFFILE) is not found at the link time because my build tree is separate from my source tree. I was able to fix this problem defining 'DEFFILE' as '$(srdcdir)/mzlu.def' instead of 'mzlu.def'.
I've uploaded two patches (one to bug 250255 and the other to bug 251897) that 'rely' on MZLU. While making patches, I identified a couple of functions to add to MZLU. One of them is EnumPrintersW, which may be a little tough to implement (apparently MSLU doesn't support this, yet)
Blocks: 253164
Blocks: 193358
Blocks: 262922
Brendan: any chance of getting the "unicode" and "unicode/mzlu" directories added to the tree? I've started working on this again and it would be good to be able to check it in. The question of using the http://libunicows.sf.net/ library is still open. The other option is to write a DIY wrapper which is 100% mozilla code which duplicates the exact same functionality.
In order to get MZLU building with VC6, it seems that I need to use the Platform SDK. This is because MZLU covers some newer API which isn't included in the standard VC6 distribution. It doesn't appear that this is a requirement of Mozilla at the moment. Would requiring the Platform SDK as a requirement for building MZLU be a problem? It is a big download, but it guarantees the latest header files, libraries, etc.
Blocks: 100243
Brodie, what platform SDKs do I have to install? Even after installing Win2003 SDK, I'm still getting an error with '_stat64'. Do I have to install platform SDKs for Win XP SP2?
I shifted development to http://opencow.sourceforge.net/ so that I had CVS and so that it is more accessible to non-Mozilla projects. You can download the latest version there which has that problem fixed. I'm working on a patch for Mozilla to add the libunicows static lib in build time and use the opencow DLL.
Blocks: 231426
A new release of MSLU has been made in response to these EULA problems. The licensing terms for MSLU should now be the same as the ones used for the C runtime.
Attached file MSLU 1.1.3790.0 licence.txt —
This is the licence.txt file for the latest MSLU download (released 10-Dec-2004). If anything it seems that this licence is more restrictive. Products should only work with Windows, still need the click-wrap/shrink-wrap EULA acceptance, and still cannot be further redistributed. It really is a pain in the ass.
Attachment #154663 - Attachment is obsolete: true
Licence text for MSLU 1.1.379.0 is also readable in a better format online at http://blogs.msdn.com/michkap/articles/345289.aspx
Comments from Kaplan on the MSLU EULA. http://weblogs.asp.net/michkap/archive/2005/01/02/345389.aspx#FeedBack The problems and possible solutions that I see with the licence are the following: 3.1(iii) + 3.1 last paragraph Requires an addition to the click-wrap licence in the Windows installer such that redistribution of the Windows software also implies acceptance of the MSLU EULA. 3.2 Means that anyone wanting to use the Mozilla source code under Windows with MSLU must be using the MPL (GPL is invalid under this clause). Use of the GPL licence on Windows is fine if MSLU is not used.
It looks like we lost track of this bug. (no activities for a month) Can someone enlighten me on what's going on now? MZLU, MSLU or other? Do we have a patch?
opencow is there to be used, and it must be used with the libunicows wrapper. I haven't had time and won't have time to work on getting the patch working for this though. MSLU does a hell of a lot that opencow doesn't and never will. I still think that we should work on a solution for using MSLU. Mozilla on non-Windows and Windows NT platforms doesn't require MSLU and so no licence is required to be accepted there. Perhaps the installer for Mozilla on Win9x/ME can download and install MSLU with the user accepting the licence similar to installing plugins.
Blocks: 131063
Brodie: Do you have a latest patch for MZLU? (patch format not zip, plz) Is MZLU 0.1 good enough? I'd love to give a code review. Thanks
No longer blocks: 250255
Blocks: 211961
Status Request: This is a critical bug for the MS-Windows platform and I see no activity since March 2005. Could someone please indicate the current status and how long before we can expect to see this in a release of some sort?
Attached patch Add --enable-unicode and XPCOM beginnings (obsolete) — — Splinter Review
Attachment #198586 - Flags: review?(darin)
Comment on attachment 198586 [details] [diff] [review] Add --enable-unicode and XPCOM beginnings needs work
Attachment #198586 - Attachment is obsolete: true
Attachment #198586 - Flags: review?(darin)
Comment on attachment 198586 [details] [diff] [review] Add --enable-unicode and XPCOM beginnings <biesi> + nsString& path = mWorkingPath; <biesi> +#else <biesi> // work in unicode for ease <biesi> nsAutoString path; <biesi> NS_CopyNativeToUnicode(mWorkingPath, path); <biesi> bsmedberg, that seems scary <biesi> bsmedberg, in one of these codepaths, modifying |path| will change mWorkingPath, and in the other it doesn't
Comment on attachment 198586 [details] [diff] [review] Add --enable-unicode and XPCOM beginnings er, nevermind that - the non-unicode codepath copies back the changes anyway.
So, has it been decided that we would build two binaries for Windows, one for Windows 9x/ME and the other for Windows NT4/2k/XP?
jshin: nothing has been decided, this is merely a first step to allow a full-unicode binary. In fact, I think that two binaries is a bad idea, and we should drop win9x without MSLU support. Adding MSLU linkage would be a secondary step.
I would be tempted to have Win9x version simply in maintance mode and then concentrate on building a version for all current platforms. At the same time it has already been mentioned several times that two binaries is unpopular. An extra observation, from my part, is the number of poorer countries where Windows 98 can still be found installed, so supporting this would be useful, since ironically these are the guys who would benefit more from a browser which doesn't open up their computer to worms and spyware.
(In reply to comment #108) > jshin: nothing has been decided, this is merely a first step to allow a > full-unicode binary. > In fact, I think that two binaries is a bad idea, and we > should drop win9x without MSLU support. That's why I asked. I think keeping two separate binaries would be a lot of hassle. That being the case, I wonder why you're introducing two code paths (with #ifdef). Is it a temporary measure until you introduce MSLU linkage? Wouldn't it be better to do all the necessary things in a single step without a lot of '#ifdef's even if they're temporary?
There is not a chance that I could convert the entire codebase to the unicode APIs at once, and I want to be able to provide test-builds of unicode before actually making everyone switch over.
bsmedberg: can you write up a design doc for this? i'm very interested in reviewing your approach here, and it seems like we should agree on the approach before cutting code. how about something on wiki.mozilla.org?
But ... when will be firefox support Unicode API? Because there is much trouble in CJK Windows Unicode Platform using ANIS Program....
Let me see if I understand this right, according to Comment #63, to support Windows 98 we either create MZLU, or use MSLU which requires a shrink-wrap, non-redistributable license. Apart from that, Comment #111 suggests moving to entire Unicode API needs to be done in stages, not all at once. If I understand it, are we going to use MSLU and start moving to Unicode API first, then worry about finishing MZLU / opencows later? That is, refer to Comment #63, we implement solution 3 first and then migrate to solution 1 if licensing or other reasons make it necessary to do so, am I correct? I think it is a more pragmatic approach right now because we can start the move to Unicode API and get more testing first. If there are enough demands we may see more hackers contributing to opencows in order to continue Win9x support, i.e. if enough users really stick with Win9x.
This out of my league and perhaps slightly off-topic, but I recently posted on this subject at http://forums.mozillazine.org/viewtopic.php?t=347673 I noticed mention here of breakage of building with mingw. In addition to loading files in unicode paths from within Firefox for example, you also need to be able to do: Firefox.exe unicode_path_to_file I'm not sure if mingw and the windows api headers you get from mingw.org have a winmain() that accepts unicode command line arguments. I do know that mingw's main() does not; even if you define _UNICODE. Plus mingw doesn't have a wmain() for unicode argv support. (Either does STLPORT) To get around the lack of wmain support in mingw, you have to of course define _UNICODE first and then do this instead: int argc; wchar_t** argv = CommandLineToArgvW( GetCommandLineW(), &argc); Now I know Firefox isn't a command line progam and probably uses winmain either directly or indirectly, but I figured I'd mention the need to pass unicode arguments to the exe and I know little on if you guys forsee a problem implementing that or not, but it's best to bring it up as it's possible.
NSPR has a method named PR_ImportFile that can be used to create a PRFileDesc from a HANDLE. This means that we could develop a nsILocalFile implementation for Windows that took care to call either CreateFileW or CreateFileA depending the OS. In other words, I believe that we could solve this bug without changing NSPR and without introducing dependencies on MZLU or MSLU.
(In reply to comment #115) >wchar_t** argv = CommandLineToArgvW( GetCommandLineW(), &argc); CommandLineToArgvW isn't supported on 9x/Me
As a belated comment on the extra observation from comment #109, my personal experience is that the "poorer countries" also tend to have nearly free versions of Windows XP etc widely available. Hence there are fewer Window 9x systems in use in these countries than you think these days. What HAS been surprising to me though, living in China, is the number of websites that require IE. Way more than in the US. My wife wont even touch firefox because of the confusion. Well, I was surprised until I started using some public database machines with html interfaces. Got tired of IE real quick, installed firefox from my usb flash, and then found out one reason why. Everyone uses Chinese (unicode) filenames and directories. Bug #162361 is the one I really care about and is probably why almost noone I talk to here knows about much less uses firefox. But that bug seems to depend on this one. Dont lose the biggest market for browsers in the world just to keep a few Win9x users!
A few questions: - are we at a point yet where Firefox works with the 'W' APIs, if I were to compile it from source? - where are we with MZLU functionality? - has the Mozilla legal team considered contacting Microsoft for a more liberal license, with regards to Unicows, that would resolve the "non-redistributable" issue? I would want to have this issue resolved by Firefox 2.0 if possible.
(In reply to comment #119) > - are we at a point yet where Firefox works with the 'W' APIs, if I were to > compile it from source? Not at all. The parts that can use W apis probably do so already, w/ fallback to A apis. so, it seems to me that this'd be the major part of the work required for this bug.
(In reply to comment #116) > NSPR has a method named PR_ImportFile that can be used to create a PRFileDesc > from a HANDLE. This means that we could develop a nsILocalFile implementation > for Windows that took care to call either CreateFileW or CreateFileA depending > the OS. In other words, I believe that we could solve this bug without > changing NSPR and without introducing dependencies on MZLU or MSLU. Darin, that's a very neat idea. I'll try to give it a shot for bug 162361 after December 31st. However, it's to be noted that although file handling is a very big part of this bug, there are other issues to deal with here. As cbie noted, in many places, we use a 'hand-made custom MSLU-equivalent'. That may not be so bad, but there are some places where we didn't bother to do that expecting this bug to be fixed.
> there are some places where we didn't bother to do that expecting this bug to be fixed. Perhaps an early task should be to enumerate those spots in the code and decide which course of action to take: 1) keep using custom failover code, or 2) unify everything behind something like MSLU.
No longer blocks: 193358
Will the descision to drop pre-Windows 2000 support have any effect on this issue? See /. article: http://developers.slashdot.org/article.pl?sid=06/06/08/1240239&threshold=1
No longer blocks: 211961
No longer blocks: 100243
Blocks: 361543
Cannot we close this? The pre Win2k was dropped already. And new gfx uses Uniscribe now. So, this bug can fix nothing on current trunk, I think.
Does Firefox 3 now provide proper Unicode support when interacting with the OS? If it does then I agree we could properly close this bug.
We fixed a lot of bugs, but there's still bug 413171, for example.
We still generate icons (moz-icon:) using ANSI path names. We still read version information from plugins in ANSI.
Yeah, we are still using ANSI APIs in some points. They should be replaced to W APIs. However, that is not this bug.
Ah, so this bug is saying "Mozilla should compile with -DUNICODE"?
(In reply to comment #129) > Ah, so this bug is saying "Mozilla should compile with -DUNICODE"? And this wants to use MSLU. See: http://www.microsoft.com.nsatc.net/globaldev/handson/dev/mslu_announce.mspx
If versions of Windows that don't support Unicode natively are no longer part of the picture, the we should default the Unicode API. Most of the discussion in this bug was around how to deal with versions of Windows where Unicode was not available without an extra library. If Windows 2000 support has been dropped, then most of the backwards-compatibility issues can be ignored and the make Unicode support of Windows default. The original requirement of this bug was to make a Mozilla binary with support of the Windows W API, which is the Windows unicode library. I had suggested this as a separate build, but now if it is not already done in should be made the default build.
ok, I'm going to claim that this bug has outlived its usefulness. We are using the W APIs in many places now, and will continue to excize A APIs for moz2. There still may be a couple places where we continue to use A APIs for compatibility reasons.
Status: ASSIGNED → RESOLVED
Closed: 17 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: