197051 - codeset name mismatch between GNU libiconv and nl_langinfo makes nsNativeCharsetUtil fail

Status: RESOLVED INCOMPLETE

(Core :: XPCOM, defect)

Description

[Martin Mokrejs]

User-Agent:       Mozilla/4.78 [en] (X11; U; OSF1 V5.1 alpha)
Build Identifier: Mozilla 1.4a

On Tru64Unix 5.1A I see with build mozilla-1.4a (current cvs checkout) the
following:

nsNativeComponentLoader: autoregistering begins.
###!!! ASSERTION: no native to utf-8 converter: 'gNativeToUTF8 !=
INVALID_ICONV_T', file nsNativeCharsetUtils.cpp, line 275
Break: at file nsNativeCharsetUtils.cpp, line 275
###!!! ASSERTION: no utf-8 to native converter: 'gUTF8ToNative !=
INVALID_ICONV_T', file nsNativeCharsetUtils.cpp, line 282
Break: at file nsNativeCharsetUtils.cpp, line 282
*** Registering nsTestDynamicModule components (all right -- a generic module!)
*** Registering MyService components (all right -- a generic module!)


I have latest/cvs version of gettext/libtool/libintl/liniconv installed.

Could it be similar to this problem?:

From: Bruno Haible <[e-mail]>
To: Martin MOKREJ© <[e-mail]>
Date: Mon, 24 Feb 2003 20:49:03 +0100 (CET)
Subject: Re: gettext: configure in current cvs

Hi,

> BTW, here is a bit more:
>   Actually, here are more details:
> 
> configure:3067: checking for Java to native code compiler
> configure:3090: found /software/@sys/usr/bin/gcj
> configure:3131: gcj  conftest.java --main=conftest -o conftest
> conftest.java:0: unknown encoding: `ISO8859-1'
>    This might mean that your locale's encoding is not supported
>    by your system's iconv(3) implementation.  If you aren't trying
>    to use a particular encoding for your input file, try the
>    `--encoding=UTF-8' option
> compilation terminated.
> configure:3134: $? = 1
> configure:3150: result: none
> 
> $ /software/@sys/usr/bin/gcj --version
> gcj (GCC) 3.2.1

Looking at the sources of gcc-3.2.1/gcc/java/lex.c, we see:

#if defined (HAVE_LOCALE_H) && defined (HAVE_ICONV) && defined
(HAVE_NL_LANGINFO)
            setlocale (LC_CTYPE, "");
            if (encoding == NULL)
              encoding = nl_langinfo (CODESET);
#endif
            if (encoding == NULL || *encoding == '\0')
              encoding = DEFAULT_ENCODING;

            java_init_lex (finput, encoding);

I.e. gcc assumes that the 'nl_langinfo (CODESET)' result can be passed
to iconv(). Now this is "ISO8859-1", which is a nonstandard encoding
name, and libiconv - which is your iconv implementation in this - case,
doesn't understand it.

The 'localcharset' module in gnulib solves this problem; it would have
to be integrated into gcc.



Reproducible: Always

Steps to Reproduce:
1.
2.
3.

Comment 1

[Martin Mokrejs]

The duplicate of this bug is http://bugzilla.mozilla.org/show_bug.cgi?id=202747

Comment 2

[Christian :Biesinger (don't email me, ping me on IRC)]

marking as duplicate of that bug, because it has the relevant people already
cc'd  and has some developer discussion

*** This bug has been marked as a duplicate of 202747 ***

Comment 3

[Martin Mokrejs]

Upgrading from GNU libiconv 1.8 to 1.9.1 fixed the problem. That was guessed by
Bruno Haible:

"libiconv 1.9.1 understands the name "ISO8859-1", libiconv 1.8 does not.
You might try the newer libiconv. There are high chances that it fixes
the problem."


I suggest improving configure to either accept GNU libiconv if 1.9.1 or above or
ignore it. And this is not a duplicate of bug #202747 (that one focuses why
system-wide iconv(3) from HP does not work).



His testcode on my platform(Tru64Unxi 5.1A) has shown:

$ ./locale_codeset
ISO8859-1
$

A bit more details for completeness:

Some details you probably already know:

  - libiconv.so is made for linking with the -l option, not for _RLD_LIST.
    libiconv_plug_osf.so is made only for use with _RLD_LIST.
    Both contain the same code, but with different function names:
    libiconv.so defines 'libiconv_open', whereas libiconv_plug_osf.so
    defines 'iconv_open'.
    > The only explanation I have is that libc is used before libiconv and
    > therefore symbols override.
    Very improbable: libc doesn't define 'libiconv_open'.

  - Whether the code calls 'iconv_open' or 'libiconv_open', depends on
    how it was compiled. More exactly, which <iconv.h> was found during
    compilation.

Some details Martin should provide:
  1) What iconv implementation is your mozilla using?
     a) the ldd output [with no _RLD_LIST in effect!]
     b) nm ... | grep iconv_open
     c) .../usr/bin/iconv --version
  2) What's the result of nl_langinfo(CODESET) in your environment?
     Compile and run this program.
============================= locale_codeset.c ===========================
/* Prints the system dependent name for the current locale's codeset. */

#define _XOPEN_SOURCE 500  /* Needed on AIX 3.2.5 */

#include <stdio.h>
#include <stdlib.h>
#include <locale.h>
#include <langinfo.h>

int main ()
{
  setlocale(LC_ALL, "");
  printf("%s\n", nl_langinfo(CODESET));
  exit(0);
}
==========================================================================

Comment 4

[Jungshik Shin]

> His testcode on my platform(Tru64Unxi 5.1A) has shown:

> $ ./locale_codeset
> ISO8859-1

We already knew that. See bug 202747 comment 15 (you wrote) after running
attachment 135656 [details] [diff] [review]

What made me confused was that you had no problem running attachment 135656 [details] [diff] [review] with
GNU libiconv installed on your system (that is, ISO8859-1 worked) so that I
thought your version of GNU libiconv had no problem recognizing ISO8859-1. As it
turned out, you have two versions and your test program was linked with 1.9.x
(that understands ISO8859-1) and your mozilla binary was with the other (1.8). 

seawood, is it possible to tweak configure to detect the codeset name
mismatch(or better, GNU libiconv version?) at the compile time (although it
wouldn't solve the run-time problem when a binary compiled on one machine is run
on another machine with a different version of libiconv)?

darin, what do you think of integrating 'localcharset' module in gnulib as
suggested by Bruno (the suggestion was made to gcc) as quoted by Martin in
comment #0?

> The 'localcharset' module in gnulib solves this problem; it would have to be
integrated into gcc.

In the meantime, we may add to the intl' known issues that to avoid this
mismatch problem, GNU libiconv 1.9.1 be used on Irix and Tru64 if it's used to
override the system-provided iconv(3).

Comment 5

[Bruno Haible]

While upgrading from libiconv 1.8 to 1.9.1 happens to fix Martin's immediate problem - 
encoding name "ISO8859-1" now being recognized by libiconv - the problem is still present for 
"dechanyu", "dechanzi", "deckanji", "deckorean", "KSC5601", "sdeckanji", "TACTIS" alone on 
OSF/1, and many others on other platforms. The real fix is to use the locale_charset() function 
instead of just nl_langinfo(CODESET) when you're using GNU libiconv. You can detect the 
latter through #ifdef _LIBICONV_VERSION. 
 

Comment 6

[Jungshik Shin]

Bruno, thanks for the tip.  A problem with that approach is that we
have to make (well, some volunteers need to) two binaries, one built
with the system iconv(3) and the other with GNU libiconv. A possible
workaround is run-time detection of locale_charset() (hope it's not a
macro) on non-Linux Unix. Life would be a lot simpler if every Unix on
the planet used your iconv(3) implementation (or equivalent as in Glibc
on linux). Actually, Tru64's iconv(3) appears (judging from its online
description) pretty good but most system administrators don't seem
to  bother to install most modules (other than the minimal ISO-8859-1) [1]
although all it may take is at most a few tens of megabytes of diks space
(I'm just making a wild guess about the disk footprint).

[1] It seems like even in Japan, sys. admins don't bother to install iconv(3).
See bug 202747 although I haven't gotten the confirmation from any
of those who reported the problem in the bugzilla of the
Japanese Mozilla user group.

Comment 7

[Martin Mokrejs]

Update the documentaion clearly stating the consequences of older version of
libiconv below 1.9.1.

Second, that on some OSes it's better to compile using native iconv()
implementation rather then GNU libiconv.

Third, try to document that on OSF1 several system packages are required for
mozilla to function properly if binary uses system iconv() implementation. I
believe Bruno Haible "Additional Comment #5" could help with the exact
requirements for each platform.

Fourth, mention that there might be handy two different binaries - depending on
the iconv() used to build binaries. ;)

Fifth, make configure do extra check for such packages installed on the system
before trying to compile the stuff. It might use "setld -i" on OSF1 and grep for
those packages(see the related bugreports in bugzilla).

Comment 8

[Gervase Markham [:gerv]]

This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

Comment 9

[Gervase Markham [:gerv]]

This bug has been automatically resolved after a period of inactivity (see above
comment). If anyone thinks this is incorrect, they should feel free to reopen it.

Comment 10

[Martin Mokrejs]

I think the comments #5 and #6 should not be forgotten.