Closed Bug 126453 Opened 22 years ago Closed 19 years ago

Chinese newsgroup names are not readable

Categories

(MailNews Core :: Internationalization, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ji, Assigned: jshin1987)

References

Details

(Keywords: intl, Whiteboard: [landed, needs testing])

Attachments

(9 files, 12 obsolete files)

32.91 KB, image/jpeg
Details
9.27 KB, patch
Details | Diff | Splinter Review
166.67 KB, text/plain
Details
16.04 KB, patch
Details | Diff | Splinter Review
67.14 KB, image/png
Details
66.97 KB, patch
Details | Diff | Splinter Review
138.01 KB, patch
Details | Diff | Splinter Review
128.65 KB, patch
Bienvenu
: review+
mscott
: superreview+
Details | Diff | Splinter Review
90.35 KB, image/jpeg
Details
This problem is original posted in China linux forum.
A user complained that the Chinese newsgroup names are not readable by using
mozilla.
The news server is 61.156.20.89.
And there are some other news servers that have Chinese newsgroup names:
news://news.newsfan.net (IP: 210.12.195.2)
news://news.cnnb.net
news://zhnews.net (IP: 210.73.88.194)
Do you know what charset is used?
Target Milestone: --- → mozilla1.2
Attached image A screenshot.
Some info from Chinese linux forum:
One of the news servers (61.156.20.89) that hosts Chinese newsgroup names are
developed by DNEWS during 1997-1998. They used WinNT4.0, it could be gb2312. I
sent out an email to support-dnews@netwinsite.com
no reply yet.

The one high lighted in the screen shot looks like have 8 bytes (look at cp1252
mapping table and find the byte)
3/4 in cp1252 == 0xbe
<< in cp1252 == 0xab
>> in cp1252 == 0xbb
^a in cp1252 == 0xaa
D- in cp1252 == 0xd0
A ring in cp1252 == 0xc5
C ced in cp1252 = 0xc7
o/ in cp1252 == f8

if we treate these 8 bytes as GB18030 instead of cp1252, then it mean 4
characters (look at my GB18030 / GB2312 map and figure out it's glyph and unicode)
beab bbaa d0c5 c7f8
U+7ebe U+564e U+4fe1 U+533a
which make sense in Chinese.

where is the code which we convert a newsgroup name into Unichar ?

sspitzer@netscape.com:
where is the code which convert NNTP group name into PRUnichar ?
Keywords: intl, nsbeta1
The Chinese newsgroup name is in GB2312 (confirm from a user in Chinese linux forum)
nsbeta1- per i18n triage
Keywords: nsbeta1nsbeta1-
Status: NEW → ASSIGNED
Does the server tell us what encoding the newsgroups are in?

How would we even know what to do with them?
I have chinese newsgroup names working.

Right now I have hardcoded the encoding to GB2312.

I'm trying to decide how to get the right encoding. We can either assume the
system locale, or add UI to set the encoding on a news server.

Incidentally, should Chinese newsgroup items work?
Composing or replying displays corrupt characters in the To: Field of composer. 
I'm not sure where this comes from, but I am looking at it.
If the server doesn't tell us which encoding to use, we should make them
settable by the user.
Should they be per news server, per newsgroup, or both?
If a server or newsgroup is set to a particular encoding, should we always then
assume all posts are also in that encoding?
Blocks: 157673
*** Bug 162255 has been marked as a duplicate of this bug. ***
Target Milestone: mozilla1.2alpha → ---
i18n triage team: nsbeta1-
*** Bug 196747 has been marked as a duplicate of this bug. ***
I subcribe to a chinese newsgroup, It display chinese name incorrectly!
     in linux version 1.4 final, this problem is still exist! mozilla mailnews
can't display chinese newsgroup correctly! you can't subscribe to   
news.newsfan.net  to test this bug! 
Dear Michael Kaply,

   Are you still watching this bug?
   I think that we can add a new preference value,the default value is
"UTF-8".The user can set it from the about:config or some other script file.What
we need to do 
is just add some code to get the preference value.
   Please let me know your idea.
   Thanks.

Bye,Holy
   
I'm thinking about a UI for this.

What if mail/news prefs - server settings had a dropdown for the default
character coding for a server?

And then when you open the properties for a particular newsgroup, at the top of
the encoding pulldown (and the default) would say "Same as server" or something
like that.

What do people think of this?
Firstly, I wish Chinese had not been so 'brave' to use non-ASCII names in
newsgroup names. Sure, NNTP needs to be updated for better I18N (how old is RFC
1036? A couple of attempts have been made to update it, but until the last time
I checked, new RFC hadn't been released) but using gb2312(EUC-CN) is ....

Anyway, as for mkaply's  question at hand, for mail server, it would work well.
 For news server, what we need is not per-server default charset but
per-hierarchy default charset that would be applied _across_ news servers. (this
was implemented in Emacs/Gnus in the mid-to-late 1990's and tin may also have
implemented it). For instance, Chinese news servers carrying newsgroups in
Chinese also carry other hierarchies in which most postings are in scripts other
than Chinese. Of course, we'd not have all these options if everybody follows
RFC 2047 in their mail/news header. 
I have no idea how I could handle the hiearchy thing in the subscribe dialog.

Basically we get a to of newsgroups from the server.

Is the only issue here Chinese newsgroups? Or are there Japanese named
newsgroups, etc.
No, there isn't. As far as I know, Japanese and Korean were not so 'brave' to
make newsgroup names in Shift_JIS/EUC-JP/ISO-2022-JP or EUC-KR. 

Anyway, note that it's not only newsgroup names that is at stake. If that was
the case, per-server setting could  work better(because only Chinese Usenet
admins. made newsgroup names under their control in GB2312 as far as I know).
The problem is that we also need to set the default charset to convert properly
to Unicode 'unlabelled' characters in __legacy__ encodings. That is, in
alt.chinese.* hierarchy, there are lots of postings with _untagged_ headers in
GB2312. Similarly, in han.*, there are a number of postings with _untagged_
headers in EUC-KR. The same is true of tw.* (Big5), de.*, fr.* (ISO-8859-1), and
so forth. 

As for the UI, about:config (or prefs.js) could work. It's not something
end-users have to configure themselves. Which character encoding is used on
which hierarchy is well known and we (or localizers) can set the appropriate
default values. 
Sorry I realized that what i wrote is kinda off-topic depending on how you look
at it.  Perhaps, your idea can work with per-hierarchy (rathar static default
charsets configurable only via about:config/prefs.js) provided that the latter
is made to override the former. 
So if I am to understand you correctly, there are certain common hierarchies
where the newsgroup names are always in a particular encoding.

So rather than attempting to fix this in a server way, I should see about
figuring out how to tell Mozilla, all newsgroups under x.y.z are Chinese names
regardless of the server...
Dear Michael and  Jungshik,

  I think that to the end user,per-hierarchy default charset that would be
applied _across_ news servers will be the best solution,but per-client default
charset is a good enough solution by now :).
  Maybe we can implement per-hierarchy default charset by steps?

Bye,Holy
news.cnnb.net uses chinese newsgroup names for EVERY newsgroup, not just a
certain hierarchy.

The same is true of news.newsfan.net

In these cases, setting the defaults character set of the news server makes much
more sense.

Can you point me to a public news server that only uses Chinese newsgroup names
in one of the hierarchies?

Or are we talking about two different problems here?
Yeah, we sorta two different problems. That's why I wrote comment #22. 

So, did  they translate all newsgroup names to Chinese? I lost my word.

Anyway, that's only Chinese. I'd stand corrected but no other news admins in any
other country does that AFAIK.  In all other cases, newsgroup names are all in
ASCII (hmm... I may as well not be so sure here given what Chinese did. Who
knows what others do...) regardless of whether they belong to Big8/global
hierarchy or regional/national hierarchies. Per-hierarchy default charset is
necessary NOT for newsgroup names BUT for correctly decoding RFC-2047 violating
non-tagged message headers. For instance, in han.* (Korean) hierarchy, a lot of
articles are posted like this (at one time, we recommend this practice, but not
any more).

Subject: Korean string in EUC-KR 
From: Korean name in EUC-KR <....@.....>
Newsgroup: han.comp.os.linux.yyyy       <== note that newsgroup name is in ASCII
Content-Type: text/plain; charset=EUC-KR
Content-Transfer-Encoding: 8bit

instead of 

Subject: =?EUC-KR?B?......?=
From: =?EUC-KR?B?......?= <...@....>
Content-Type: text/plain; charset=EUC-KR
Content-Transfer-Encoding: 8bit

Even worse is that some messages (although must be very rare nowadays) don't
have C-T header. Per-hierarchy default charset is to cope with situations like
these. 

What we can do is as following:

 - add UI for per-server default charset
 - For newsgroups, make per-hiearachy default charset (configurable 
    via about:config and prefs.js) should override as long as newsgroup
   names are all in ASCII. For non-ASCII newsgroup names (Chinese newsgroup
   names in GB2312), always use per-server setting (that's the only option in
   that case) 

 - For mail folders, likewise, per-folder default charset 
  (if different from per-server setting) overrides 
   per-server setting.
    per-server     setting

I think it best if we pay attention to what USEFOR guidelines are 
on newsgroup names. There is one proposal fro internationalized
newsgroups names:

http://www.ietf.org/internet-drafts/draft-faerber-i18n-email-netnews-names-00.txt

This draft expired in March 2003 but it would be worthwhile finding out 
what the next plan is. 
The following two documents, "Usenet Best Practice": section 7.2, and "News
Article Format": section 5.5 where newsgroup name synatx is defined.

http://www.landfield.com/usefor/drafts/draft-ietf-usefor-useage--1.03.txt
http://www.landfield.com/usefor/drafts/draft-ietf-usefor-article-11.txt

I would think that a good approach would be go a step toward using Punycode
as suggested in the 1st document cited above.
Sorry for my misunderstanding of "per-hierarchy ".
I think in the comment#26,the "per-hierarchy default charset" is just a default
value of "per-server charset",am I right?
For a localized mail/news client,if the localizer set this "per-hierarchy
default charset" to "GB2312" in pref.js,then all the "per-server charset" are
set to "GB2312".
but when the user visiting some newsgroup server which is using "EUC-KR",now
he/she can change the "per-server charset" to "EUC-KR".

But I don't understand why "per-hiearachy default charset (configurable 
    via about:config and prefs.js) should override as long as newsgroup
   names are all in ASCII" cause even the newsgroup names are in ASCII,after we
decoding the names as "GB2312" or "EUC-KR" ,this names is still correct.

There is NO newsgroup name whose name is in EUC-KR. Koreans are not so 
'brave' as Chinese to make newsgroup names in EUC-KR.(if we had wanted 
that, we would have used UTF-8  - or punycode as Kat pointed out - instead of 
EUC-KR which should be put into disuse as soon as possible) 
 
See my comment #26 as to what per-hierarch default charset is for. It's NOT for 
newsgroup name decoding (how can it be?) but for decoding of __untagged_  
headers and message body.  As it stands now, it's __impossible__ to set 
per-hiearachy charset if newsgroup names are non-ASCII (and non-Unicode 
based like GB2312). So, per-hierarchy charset would be only applied to 
newsgroup names in ASCII. 
 
Just out of the curiosity, how would you sync Chinese translation of 
newsgroup names with the original ASCII names? How about news exchanges 
with other news servers? Have you formed an exclusive-Chinese-server only 
network?  
Dear Jungshik,
  Thanks for your reply.
  Actually there are some "stand alone" news servers in China,and they never
sync with other servers :(. I never hosted a server like this :).
  But,the Chinese newsgroup name will help a lot to those people who don't know
english,Do you think so?
  To my own opinion,I'd like to use UTF-8 at verywhere,but there are some policy
defined by some government department let us use GB2312 and now GB18030.

Bye,Holy
Let me clarify. I have nothing against using Chinese names for Chinese
newsgroups. However, that has to be done in a proper and standard-compliant way
rather than in an ad-hoc manner as was done on some Chinese news servers.
Exactly for the same reason, I didn't like what 'Netpia' (a Koeran company) did
with 'domain names' in Korean. Their solution was a hack at best that's not
interoperable and works ONLY with MS IE. Eventually, IETF adopted a sound and
backward compatibile solution (IDN) and national and international domain
registrars implemented IDN rather quickly (there's not much to do on the
server-side because punycode used in IDN just works with old name servers). Now
the situation is reversed in terms of Korean domain name support because MS IE
doesn't yet support IDN while Mozilla and Opera support them very well. 

As for GB 18030 vs UTF-8, please don't make PRC government (your own) silly by
saying that it imposes what it does NOT. As far as I know, all PRC government
asks for is to support the conversion between GB18030 and other representations.
If PRC government did what you said it does, IDN (mentioned above) that uses
punycode (a specially-designed way of representing Unicode characters) wouldn't
have been adopted in China. However, cn domain already has a lot of domain names
in Chinese registered that are stored on BIND and other name servers as punycode
(NOT GB18030). 
  Do you know that in China There is a policy (from government obviously) that
any IT products should support GB18030 or it can not be sold in China.
  I love my Country but not the government.
Please, don't make me repeat myself. Your understanding of your government's
policy is wrong. Mozilla does support GB18030 and there's no problem with the
compliance to the policy. This bug has nothing to do with that.
  Nobody will sell Mozilla in China,So this never happens.
and I don't think that Mozilla has any problem with this policy either.
As you said ,the topic is nothing about this bug,so lets talk something
about how to resolve this bug.
I added a dropdown list to Mail & Newsgroup Account Settings dialog under the
newsgroup server panel. This dropdown list is defaulted to ISO-8859-1. Here you
can change the charset used for the server. When subscribing to a newsgroup,
they will be displayed in that charset. When newsgroups get added, the default
charset that the messages will be displayed in is the defined charset for the
server.
What do people think of Jessica's patch?
So, when the new drop down list is added, we have the following:

1. The Main encoding setting option (global) in the Message Display section of
the Mail pref. window.
2. The new drop down list per server in Mail & Newsgroup pref settings, (which
defaults to ISO-8859-1?).
3. Folder proprties dialog under which encoding choice is listed.

Since there is already #3, the new drop down is mainly for discoverability? 

One thing we should not do is to default to ISO-8859-1. Instead, the new option
should default to the global Message Display encoding in #1. 
I agree with Kats if his #1 is 'intl.charset.default' configurable in Edit | 
Preference | Languages | Default Character Coding. I'll comment more later.
jungshik, it is actually "mailnews.view_default_charset", configurable via 

Edit > Preferences > Mail & Newsgroup > Languages > Display

Note that all mail folder charsets are initially set to
"mailnews.view_default_charset",
which then you can override with per folder charsets via Folder Properties menu.

This is why I mentioned that creating this new drop down list would be only
adding a bit more discoverability to the folder charset menu via Edit > (Folder)
Properties menu. I don't see that this menu will be doing anything more than
what we already do with the folder charset drop down list.


> Edit > Preferences > Mail & Newsgroup > Languages > Display

Sorry, actually the menu is:

Edit > Preferences > Mail & Newsgroups > Message Display > Languages > Character
Coding
I take back part of what I said in Comment #37 and Comment #39. 
The new menu is distinct from the folder charset menu. It sets the server
charset and this may be needed to display newsgroup folder names correctly. 

However, I still believe the default for the new menu should be the global
Message Display encoding. 

We need to make sure that any folders for newsgroups and mail created under a
server should now listen to the server default encoding to create a new folder
rather than to the global Message Display default.
I believe the patch also currently makes the folder charset for newsgroups that
are on a server with a different charset default to the server charset.

Are you sure about defaulting to the global mail charset?

I thought chinese newsgroup servers was the exception not the rule.
> Are you sure about defaulting to the global mail charset?

I think we should look at 2 factors in answering this question.

1. Will defaulting to the global mail charset cause any serious problem?

The lisf of mail default charsets is not huge. It is a limited set. And all the
charsets contain ASCII as a subset. Since almost all newsgroups names are in
ASCII. This then would not  cause a problem. 

2. From user experience point of view, increasing the number of default charsets
may contribute to confusion. Currently loclizers set 3 default encodings for a
target language. 1) browser default, 2) mail view default, 3) mail send default.

The mail global default applies to the default mail folders such as Inbox and
any new folders. Taking Simplified Chinese as an example, the global mail
charset will be GB 18030. But if the server default is set to ISO-8859-1 and all
the folders under a server also gets set to ISO-8859-1, that would in effect
nullify the effect of the global mail charset even before the user does
anything. Unless I am mistaken about this, setting the server default to the
mail global charset seems to the only sane solution.
Comment on attachment 137265 [details] [diff] [review]
using mkaply's patch above adding a pref for server charset

As Kats gave exactly the same answer as I was about to give, I'll go through
the patch. Firstly, it'd be nice if you make a patch with 'diff -u8 -p' ('8' is
arbitrary but the point is that giving more context would help. And '-p' is
critical in making your patch readable.).

>Index: mailnews/base/prefs/resources/content/am-server.xul

>+  <hbox align="center" valign="middle" iscontrolcontainer="true">
>+    <label value="&serverDefaultCharset.label;" control="nntp.charset"/>
>+    <menulist hidable="true" hidefor="imap,pop3,movemail" wsm_persist="true" id="nntp.charset" 

  It might be a good idea to make this menu available for imap4/pop3 as well.
Perhaps, we can deal with that issue in another bug. For that and consistency
with the pref. entry name (mail.string.server......charset'), you may consider
using 'mailnews.server.charset' instead of 'nntp.charset'. 

>+                ref="NC:DecodersRoot" datasources="rdf:charset-menu"
>+                preftype="string" prefstring="mail.server.%serverkey%.charset"> 


>Index: mailnews/news/public/nsINntpIncomingServer.idl

>+    /* what charset do we use for the newsgroup server? */
>+    attribute string charset;

  A bit more extensive comment describing the role of the server default
charset would be helpful especially in light of Kats' comment. 


>Index: mailnews/news/src/nsNewsFolder.cpp

>@@ -214,7 +217,34 @@

>+  if (NS_FAILED(rv)) {
>+    newsgroupName.AssignWithConversion(name);

    |CopyASCIItoUTF16(nsDependentCString(name), newsgroupName)| would be
better. (there's a patch to add CopyASCIItoUTF16() that accepts 'char *', but
it's not yet landed).

>@@ -548,7 +578,39 @@
>   if (!*uninewsgroupname) return NS_ERROR_FAILURE;
>   
>   nsCAutoString newsgroupname; 
>-  newsgroupname.AssignWithConversion(uninewsgroupname);
>+
>+  nsCOMPtr <nsINntpIncomingServer> nntpServer;
>+  rv = GetNntpServer(getter_AddRefs(nntpServer));
>+  if (NS_FAILED(rv)) return rv;
>+
>+  char *dataCharset; 
>+  rv = nntpServer->GetCharset(&dataCharset);
>+
>+  nsCOMPtr<nsICharsetConverterManager> ccm = 
>+           do_GetService("@mozilla.org/charset-converter-manager;1", &rv); 
>+  if (NS_SUCCEEDED(rv)) {
>+    nsCOMPtr<nsIUnicodeEncoder> encoder;
>+    rv = ccm->GetUnicodeEncoder(dataCharset, getter_AddRefs(encoder));
..... snipped conversion .....
>+
>+  if (NS_FAILED(rv)) {
>+    newsgroupname.AssignWithConversion(uninewsgroupname);
>+  }
>   
>   nsFileSpec path;
>   nsCOMPtr<nsIFileSpec> pathSpec;

newsgroupname obtained here(nsMsgNewsFolder::CreateSubfolder) is used to create
a filename on the local file system. Therefore, you should not convert to the
server default charset but have to convert to the file system encoding. There's
an utility function somewhere in mailnews for that. This is a bit debatable,
but I guess that's the best we can do.


>@@ -659,8 +723,9 @@
>     rv = GetNntpServer(getter_AddRefs(nntpServer));
>     if (NS_FAILED(rv)) return rv;
>     
>-    PRBool abbreviate = PR_TRUE;
>-    rv = nntpServer->GetAbbreviate(&abbreviate);
>+//    PRBool abbreviate = PR_TRUE;
>+    PRBool abbreviate = PR_FALSE; //@MAK 
>+//    rv = nntpServer->GetAbbreviate(&abbreviate);
>     if (NS_FAILED(rv)) return rv;

Any reason to turn off the abbreviation here? We're working with PRUnichar[] so
that I don't see any risk of chopping names in the middle (well, if we consider
Unicode combining character sequence and surrogate pairs, we have to fix 
nsMsgNewsFolder::AbbreviatePrettyName, but at the moment, there's no need).


>@@ -1595,7 +1660,39 @@   GetAsciiName()
>     
>     // convert to ASCII
>     nsCAutoString tmpStr;
>-    tmpStr.AssignWithConversion(name);

 <conversion code snipped>
>
>+    }
>+
>+    if (NS_FAILED(rv)) {
>+      tmpStr.AssignWithConversion(name.get());
>+    }

GetAsciiName() is invoked in places (I was just surprised to know that there's
a method whose only role is to convert an nsCString member variable to |char
*|) like the following:

http://lxr.mozilla.org/seamonkey/source/mailnews/news/src/nsNNTPProtocol.cpp#20
04

2009		     rv =
m_newsFolder->GetAsciiName(getter_Copies(newsgroupName));
2010		     NS_ENSURE_SUCCESS(rv,rv);
2011		 }
2012 
2013		 PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) current group = %s, desired
group = %s", this, m_currentGroup.get(), newsgroupName.get()));
2014		 // if the current group is the desired group, we can just
issue the ARTICLE command
2015		 // if not, we have to do a GROUP first
2016		 if (!PL_strcmp(m_currentGroup.get(), newsgroupName.get()))
2017		   m_nextState = NNTP_SEND_ARTICLE_NUMBER;

If you change the way |m_currentGroup| (of which class? I have to check) is
stored, you can avoid the conversion here. It's quite a lot more work, but I
think we must do it while we're at this. Besides, GetAsciiName() is a misnomer
with your change, isn't it? 

>Index: mailnews/news/src/nsNntpIncomingServer.cpp


>+struct GetNewsgroupData {
>+   nsIOFileStream*        mHostInfoStream;
>+   nsINntpIncomingServer* nntpServer;
>+};

Wouldn'it better to use a noun instead of a 'verb' here?



>@@ -801,7 +811,34 @@  writeGroupToHostInfoFile
>
>+    char *dataCharset; 
>+    rv = data->nntpServer->GetCharset(&dataCharset); 
....
<conversion snipped>

>+    if (NS_FAILED(rv)) {
>+      nameOnDisk.AssignWithConversion(name.get());
>+    }

  Here, too, I doubt converting to the server charset is the 'right thing'. In
this case, I guess you have to convert to UTF-8.

>@@ -856,8 +893,21 @@
>          << "begingroups" << MSG_LINEBREAK;
> 
>     // XXX todo, sort groups first?
>+    nsCOMPtr<nsIMsgFolder> rootFolder;
>+    rv = GetRootFolder(getter_AddRefs(rootFolder));
>+    NS_ENSURE_SUCCESS(rv,rv);
>+    if (!rootFolder) return NS_ERROR_FAILURE;
>+
>+    nsCOMPtr <nsIMsgNewsFolder> newsFolder = do_QueryInterface(rootFolder, &rv);
>+    NS_ENSURE_SUCCESS(rv,rv);
>+    if (!newsFolder) return NS_ERROR_FAILURE;

  You don't have to check both conditions. NS_ENSURE_SUCCESS() would suffice.

>+    nsCOMPtr <nsINntpIncomingServer> nntpServer;
>+    rv = newsFolder->GetNntpServer(getter_AddRefs(nntpServer));
>+    if (NS_FAILED(rv)) return rv;

    The consistency wouldn't be bad here. Pls, use NS_ENSURE_SUCCESS(rv, rv).


>@@ -1123,9 +1173,51 @@


>+          rv = decoder->Convert(aName, &inLength, outString, &outLength);
>+          if (NS_SUCCEEDED(rv)) {
>+            outString[outLength] = 0;
>+           escapedName = nsEscape(NS_ConvertUCS2toUTF8(outString).get(), url_Path);

  NS_ConvertUCS2toUTF8 is now NS_ConvertUTF16toUTF8. Please, use that name. The
same is true of all other similar utility methods/functions/classes in
xpcom/string. 

>+    if (NS_FAILED(rv)) {
>+      newsgroupName.AssignWithConversion(aName);
>+      escapedName = nsEscape(NS_ConvertUCS2toUTF8(newsgroupName.get()).get(), url_Path);

    I'm not sure if this round-tripping is necessary. Why not just pass |aName|
to nsEscape? 

>Index: modules/libpref/src/init/mailnews.js
>
>+pref("mail.server.default.charset", "ISO-8859-1");

Per Kats' comment, we don't need this entry.
Product: MailNews → Core
can we use the method nsMsgI18NConvertFromUnicode and nsMsgI18NConvertToUnicode
in nsMsgI18n.cpp instead for the conversion?
Attached patch new patch (obsolete) — Splinter Review
based on the previous patch ,but used  nsMsgI18NConvertFromUnicode/
nsMsgI18NConvertToUnicode instead for the conversion.
and get the default charset from the mailnews.view_default_charset preference.
Hi Jungshik Shin,
  Could you please give some comments about the new patch?
  Thank you very much!
Best Regards,Holy
Dear all,
  I have one question about this bug:
  Should we change the type of asciiName in interface nsIMsgNewsFolder to a
AString instead of a string (rename it to name or something else)?
Attached patch a patch with more fixes. (obsolete) — Splinter Review
Merry Christmas!

this patch based on the previous patch.
fixed the newsgroup name in the news download dialog.
fixed when the user click the message but not able to view the message content
when subscribe to a Chinese news group.

could anyone give it a review? I hope it can land to the trunk before 1.8
release .
Many people complain to me that they don't use mozilla mail client because it
cannot support chinese newsgroup.Now I am not only a localizer but also a bug
fixer,haha :)

Best Regards,Holy
Attachment #169045 - Attachment is obsolete: true
Attachment #169507 - Flags: review?(mkaply)
Comment on attachment 169507 [details] [diff] [review]
a patch with more fixes.

>diff -U 8 -r 

Please, use 'diff -u8 -p'


>+++ mozilla/mailnews/news/public/nsIMsgNewsFolder.idl	2004-12-22 
>+  readonly attribute AString newsgroupName;
>   readonly attribute string asciiName;

Why do you still need 'asciiName'?


>-  newsgroupName.AssignWithConversion(name);
>+  nsXPIDLCString dataCharset;
>+  rv = nntpServer->GetCharset(getter_Copies(dataCharset));

You can avoid getter_Copies if you declare 'charset' as ACString instead of
'string'


>+  rv = nsMsgI18NConvertFromUnicode(dataCharset, nsDependentString(uninewsgroupname), newsgroupname);
>+  if (NS_FAILED(rv)) {
>+    newsgroupname.AssignWithConversion(uninewsgroupname);

I've already told you that you MUST use the file system charset instead of the
server charset when you deal with local files. Please, use
NS_CopyUnicodeToNative or equivalent. In this particular case, refer to 
http://lxr.mozilla.org/seamonkey/source/mailnews/local/src/nsLocalMailFolder.cp
p#780
and do a similar thing with 'hashedNames'. There's one more place with the same
problem.


>+#include "nsIPref.h"

nsIPref has been obsolete for almost 4years. Please, use
nsIPrefBranch/nsIPrefService.


>+  rv = nsMsgI18NConvertToUnicode(NS_LITERAL_CSTRING("UTF-8"),nsDependentCString(unescapedPath),newsgroupName);
>+  NS_ENSURE_SUCCESS(rv,rv);

You can simply use CopyUTF8toUTF16 here.

Also, don't use old names like '..CopyUCS2toUTF8/ASCII' but use
'..CopyUTF16toUTF8/ASCII'.
Attachment #169507 - Flags: review?(mkaply) → review-
Hi Jungshik Shin,
I've tried with diff -u8 -p but it's not work.
holywen@holy-note mozsrc $ diff -u8 -p  mailnews.old/ mozilla/mailnews/ >
newgroupname.testclean.patch
diff: `-8' option is obsolete; use `-U 8'
diff: Try `diff --help' for more information.

>+  rv = nsMsgI18NConvertFromUnicode(dataCharset,
nsDependentString(uninewsgroupname), newsgroupname);
>+  if (NS_FAILED(rv)) {
>+    newsgroupname.AssignWithConversion(uninewsgroupname);
Here I also tried to use  the filesystem charset here,but it does not work.

I kept asciiName in the interface because so many places they calling the
getAsciiName.


(In reply to comment #51)

> holywen@holy-note mozsrc $ diff -u8 -p  mailnews.old/ mozilla/mailnews/ 
> diff: `-8' option is obsolete; use `-U 8'
> diff: Try `diff --help' for more information.

My point is that you should use '-p'. Btw, why don't you use cvs? 
 
> >+  rv = nsMsgI18NConvertFromUnicode(dataCharset,
> nsDependentString(uninewsgroupname), newsgroupname);
> >+  if (NS_FAILED(rv)) {
> >+    newsgroupname.AssignWithConversion(uninewsgroupname);
> Here I also tried to use  the filesystem charset here,but it does not work.

What did you mean by not working? Did you try NS_CopyUnicodeToNative? It's
declaredin nsNativeCharsetUtils.h

> I kept asciiName in the interface because so many places they calling the
> getAsciiName.

 All right (even then, changing names would be a good idea), but I don't feel
easy about using 'gb18030' in places like
http://lxr.mozilla.org/seamonkey/source/mailnews/news/src/nsNewsFolder.cpp#521
 

I cannot use cvs diff because I cannot access directly to the cvs server (I am
inside a intranet).

  There is a method nsMsgI18NFileSystemCharset in nsMsgI18N.cpp,I tried to use
the return value of this method instead of the dataCharset,then the newsgroup
names in the left tree becomes to unknown chars,and when I click an newsgroup it
says like "the newsgroup doesnot exists in the server,do you want to unsubscribe?"

please, whatever you do, do not introduce more AssignWithConversion callers. it
is deprecated. Use AssignASCII/CopyASCIItoUTF16 if you must.
(In reply to comment #53)

>   There is a method nsMsgI18NFileSystemCharset in nsMsgI18N.cpp,I tried to 

You shouldn't use that to convert a  newsgroup name stored in UTF-16 to GB18030
to communicate with a news server. You only need to use NS_CopyUnicodeToNative
(the  opposite way) when making up a file name to use on the disk. 
OS: Windows 2000 → All
Hardware: PC → All
(In reply to comment #53)

>   There is a method nsMsgI18NFileSystemCharset in nsMsgI18N.cpp,I tried to 

You shouldn't use that to convert a  newsgroup name stored in UTF-16 to GB18030
to communicate with a news server. You only need to use NS_CopyUnicodeToNative
(the  opposite way) when making up a file name to use on the disk. 
(In reply to comment #56)
> (In reply to comment #53)
> 
> >   There is a method nsMsgI18NFileSystemCharset in nsMsgI18N.cpp,I tried to 
> 
> You shouldn't use that to convert a  newsgroup name stored in UTF-16 to GB18030
> to communicate with a news server. You only need to use NS_CopyUnicodeToNative
> (the  opposite way) when making up a file name to use on the disk. 
> 
I'm not use this as charset everywhere,just in the createsubfolder function and
it give me those error messages.
Now I know why,because it calles AddNewsgroup(...) after the conversion.
Actually I set my locale to zh_CN.utf-8 and the file names in my mozilla profile
are correct.

for the GetAsciiName in the GetFolderURL(),I think that we should not call
getAsciiName here ,but ,should we use escaped name or we just use utf8 string in
the url?



use escaped name for the URL in method GetFolderURL .
use LossyCopyUTF16toASCII instead of assignWithConversion.
use nsIPrefBranch Instead of nsIPref .
use ACString for charset.
use -p when diff.
Thank Shin very much!
Attachment #169507 - Attachment is obsolete: true
I read through the nsMsgNewsFolder::CreateSubfolder(...) and found that.
the nsFileSpec path is initiated but never used after: path += hashedName.get();
I tried to assign different encoded newsgroupname for the hashedName and do
subscribe&unsubscribe a Chinese group and check the creation of <groupname>.dat
and <groupname>.msf file,it doesn't make any sense.
Is it just a piece of code that do nothing?
can we remove it?
Attached patch patch for the trunk(1.8a6). (obsolete) — Splinter Review
replacement of the previous patch for the trunk.
the previous patch is for mozilla 1.7.3
Attachment #169887 - Attachment is obsolete: true
Comment on attachment 170025 [details] [diff] [review]
patch for the trunk(1.8a6).

> +++ mozilla/mailnews/news/public/nsIMsgNewsFolder.idl 2005-01-02 05:18:12.375209792 +0800
> -  /** the ascii name for the newsgroup.
> -   *  newsgroup names are always ascii, and this can help simply
> -   *  the code, to prevent use from having to get the name (as a wstring)
> -   *  and then converting it back to ascii.
> -   */
> +  readonly attribute AString newsgroupName;
>    readonly attribute string asciiName;

  You'd better document the difference between 'newsgroupName' and 'asciiName'.
'asciiName' is not ASCII-only any more. 'unicodeName' may be better
than 'newsgroupName'.


>  [scriptable, uuid(b5037f06-1dd1-11b2-bf61-fcab5b5837a7)]

When you change an interface, you have to change the UUID. use 'uuidgen'
to generate a new uuid.

> +++ mozilla/mailnews/news/src/nsNewsDownloadDialogArgs.h      2005-01-02 04:15:48.507364216 +0800
> +++ mozilla/mailnews/news/src/nsNewsFolder.cpp        2005-01-02 05:17:54.805880736 +0800

> +  nsXPIDLCString folderAsciiName;

 You don't seem to use this variable anywhere.

> +  rv = nsMsgI18NConvertToUnicode(dataCharset.get(),nsDependentCString(name),newsgroupName);

     rv = ConvertToUnicode(dataCharset.get(), name, newsgroupName);

> +  if (NS_FAILED(rv)) return rv;

To be consistent, it's better to use one or the other.
     NS_ENSURE_SUCCESS(rv,rv);

> +  nsCAutoString dataCharset;
> +  rv = nntpServer->GetCharset(dataCharset);
> +  NS_ENSURE_SUCCESS(rv,rv);
> +  rv = nsMsgI18NConvertFromUnicode(dataCharset.get(), nsDependentString(uninewsgroupname), newsgroupname);

Now I understand why you're doing this. You're using  GB2312 newsgroup
names for newsrc/hostinfo files. I think it's not so good an idea.
Using UTF-8 is better, which require some more changes elsewhere.


> +  NS_CopyUnicodeToNative(nsDependentString(uninewsgroupname),filename);

> +  nsCAutoString hashedName = filename;

>    rv = NS_MsgHashIfNecessary(hashedName);
>    path += hashedName.get();

 You have to do this way (see comment #50)

    nsAutoString hashedName = uniNewsgroupName;
    rv = NS_MsgHashIfNecessary(hashedName);
    nsAutoString hashedName = uniNewsgroupName;
    rv = NS_MsgHashIfNecessary(hashedName);
    NS_CopyUnicodeToNative(hashedName, fileName);
    path += fileName;


> +NS_IMETHODIMP
> +nsMsgNewsFolder::GetNewsgroupName(nsAString & name)
> +{
> +  nsresult rv;
> +  
> +  nsXPIDLString newsgroupName;
> +  rv = GetName(getter_Copies(newsgroupName));
> +  NS_ENSURE_SUCCESS(rv,rv);
> +  name = newsgroupName;
> +  return NS_OK;
> +}

Why do we need both |GetNewsgroupName| and |GetName|?  



> +++ mozilla/mailnews/news/src/nsNntpIncomingServer.cpp        2005-01-02 05:17:23.239679528 +0800

> +nsNntpIncomingServer::SetCharset(const nsACString & aCharset)
> +{ 
> +  nsresult rv;
> +  rv = SetCharValue("charset",ToNewCString(aCharset));

You're leaking here. 


     rv = SetCharValue("charset", PromiseFlatCString(aCharset).get());


    You don't have to check both.

> +   if(!serverCharset || serverCharset.IsEmpty())

      if(serverCharset.IsEmpty())


> +      nsCOMPtr<nsIPrefBranch> prefBranch(do_GetService(NS_PREFSERVICE_CONTRACTID, &rv));
> +
> +      if (NS_SUCCEEDED(rv))
> +      {
> +         rv = prefBranch->GetCharPref(PREF_MAILNEWS_VIEW_DEFAULT_CHARSET, getter_Copies(defaultCharset));
> +         serverCharset.Assign(defaultCharset ? defaultCharset.get() : NS_LITERAL_CSTRING("ISO-8859-1").get());

  Can you just use |GetCharValue()|?


> +      }else{
> +                serverCharset.Assign(NS_LITERAL_CSTRING("ISO-8859-1").get());

	     serverCharset = "ISO-8859-1";

> +  aCharset.Assign(serverCharset.get());


     aCharset = serverCharset;

As mentioned above, if you use UTF-8 instead of the 'server encoding'
(GB2312/GB18030),  you can simplify code here.

> +struct NewsgroupData {
> +   nsIOFileStream*        mHostInfoStream;
> +   nsINntpIncomingServer* nntpServer;
> +};
> +
>  PRBool
>  writeGroupToHostInfoFile(nsCString &aElement, void *aData)
>  {
> +    NewsgroupData *data = (NewsgroupData*)aData;
>      nsIOFileStream *stream;
> -    stream = (nsIOFileStream *)aData;
> +    stream = data->mHostInfoStream;
....

> +    nsCAutoString dataCharset;
> +    rv =  data->nntpServer->GetCharset(dataCharset);
> +    NS_ENSURE_SUCCESS(rv,rv);
> +    rv = nsMsgI18NConvertFromUnicode(dataCharset.get(), nsDependentString(name.get()), nameOnDisk);
....
>      // XXX todo ",,1,0,0" is a temporary hack, fix it
>      *stream << nameOnDisk.get() << ",,1,0,0" << MSG_LINEBREAK;
>      return PR_TRUE;



+    rv = GetRootFolder(getter_AddRefs(rootFolder));
+    NS_ENSURE_SUCCESS(rv,rv);
+    if (!rootFolder) return NS_ERROR_FAILURE;
+    nsCOMPtr <nsIMsgNewsFolder> newsFolder = do_QueryInterface(rootFolder,
&rv);
+    NS_ENSURE_SUCCESS(rv,rv);
+    if (!newsFolder) return NS_ERROR_FAILURE;

  Just |NS_ENSURE_SUCCESS(rv,rv)| suffices. |if (!...) return NS_ERROR....| is
redudant.
(In reply to comment #61)
Thank you very much!

>   You'd better document the difference between 'newsgroupName' and 'asciiName'.
> 'asciiName' is not ASCII-only any more. 'unicodeName' may be better
> than 'newsgroupName'.
> 
How about change asciiName to encodedName?
Then we need to change all the GetAsciiName to GetEncodedName in the
nsNNTPProtocol.cpp and nsNewsFolder.cpp

>  You have to do this way (see comment #50)
> 
>     nsAutoString hashedName = uniNewsgroupName;
>     rv = NS_MsgHashIfNecessary(hashedName);
>     nsAutoString hashedName = uniNewsgroupName;
>     rv = NS_MsgHashIfNecessary(hashedName);
>     NS_CopyUnicodeToNative(hashedName, fileName);
>     path += fileName;
> 
As the comment#59,the path is not used later,should we remove it?

> 
> > +NS_IMETHODIMP
> > +nsMsgNewsFolder::GetNewsgroupName(nsAString & name)
> > +{
> > +  nsresult rv;
> > +  
> > +  nsXPIDLString newsgroupName;
> > +  rv = GetName(getter_Copies(newsgroupName));
> > +  NS_ENSURE_SUCCESS(rv,rv);
> > +  name = newsgroupName;
> > +  return NS_OK;
> > +}
> 
> Why do we need both |GetNewsgroupName| and |GetName|?  

in the nsNNTPNewsgroupList.cpp,it need to get the newsgroup name from the
interface nsIMsgNewsFolder but not from the class nsNewsFolder,So we need to 
add a method to the interface for get the newsgroup name.

> 
> > +      nsCOMPtr<nsIPrefBranch>
prefBranch(do_GetService(NS_PREFSERVICE_CONTRACTID, &rv));
> > +
> > +      if (NS_SUCCEEDED(rv))
> > +      {
> > +         rv = prefBranch->GetCharPref(PREF_MAILNEWS_VIEW_DEFAULT_CHARSET,
getter_Copies(defaultCharset));
> > +         serverCharset.Assign(defaultCharset ? defaultCharset.get() :
NS_LITERAL_CSTRING("ISO-8859-1").get());
> 
>   Can you just use |GetCharValue()|?
GetCharValue is used to get value of mail.server.%serverkey%.prefname.
So we cannot use it here.
use utf8 in the newsrc/hostinfo file.
(actually If we simplify the code in writing to the newsrc/hostinfo file.we
need to add some code to convert the names to encoded string when loading the
two file)
change newsgroupName and asciiName to  unicodeName and encodedName in
nsIMsgNewsFolder interface.
modified other places according to the new comments.
Attachment #170025 - Attachment is obsolete: true
Attachment #170502 - Attachment is obsolete: true
Attachment #170507 - Flags: review?(jshin)
Comment on attachment 170507 [details] [diff] [review]
patch use utf8 in the newsrc/hostinfo file

Please, read my comment more carefully. You didn't take care of unnecessary
uses of '.get()' methods in a few places. Here's an example.

+      {
+	  rv = prefBranch->GetCharPref(PREF_MAILNEWS_VIEW_DEFAULT_CHARSET,
getter_Copies(defaultCharset));
+	  serverCharset.Assign(defaultCharset ? defaultCharset.get() :
NS_LITERAL_CSTRING("ISO-8859-1").get());

Btw, I don't like 'encodedName'. Perhaps, 'nativeName' would be better.

I'll get to the rest of the patch later.
(In reply to comment #65)
> (From update of attachment 170507 [details] [diff] [review] [edit])
> Please, read my comment more carefully. You didn't take care of unnecessary
> uses of '.get()' methods in a few places. Here's an example.
> 
> +      {
> +	  rv = prefBranch->GetCharPref(PREF_MAILNEWS_VIEW_DEFAULT_CHARSET,
> getter_Copies(defaultCharset));
> +	  serverCharset.Assign(defaultCharset ? defaultCharset.get() :
> NS_LITERAL_CSTRING("ISO-8859-1").get());
> 
these piece of code I just copy from the 
mozilla\mailnews\import\eudora\src\nsEudoraCompose.cpp#628

> Btw, I don't like 'encodedName'. Perhaps, 'nativeName' would be better.
> 
> I'll get to the rest of the patch later.
> 
OK,I will change the encodedName to nativeName,but a nativeName should be a
filesystem encoded name?
(In reply to comment #66)
> (In reply to comment #65)


> > Please, read my comment more carefully. You didn't take care of unnecessary
> > uses of '.get()' methods in a few places. Here's an example.
 ...
> these piece of code I just copy from the 
> mozilla\mailnews\import\eudora\src\nsEudoraCompose.cpp#628

That doesn't justify your unnecessary use of 'get()'. Instead, it means that
nsEudoraCompose.cpp needs to be fixed as well. 
(In reply to comment #66)

> OK,I will change the encodedName to nativeName,but a nativeName should be a
> filesystem encoded name?

Hmm. yeah... That method belongs to |nsIMsgNewsFolder| so that one may ask
'native to what?'
How about 'serversideName' ? 

This makes me even less happy about adding this to support  ugly hacks of some
Chinese news servers.
 

> Hmm. yeah... That method belongs to |nsIMsgNewsFolder| so that one may ask
> 'native to what?'
> How about 'serversideName' ? 
> 
> This makes me even less happy about adding this to support  ugly hacks of some
> Chinese news servers.
>  
What about rawName?

about Chinese newsgroup names,my opinion is that the news protocol should be 
upgraded to supply not only chinese newsgroup names but also other names.
*** Bug 272753 has been marked as a duplicate of this bug. ***
let's try and get this for 1.1.
Flags: blocking-aviary1.1+
Is there anything I can help in the next step?
I'm sorry for delay. Can you see if your patch hasn't bit-rotten? If it has,
please update it and I'll review it over the weekend. (btw, 'rawName' might be a
reasonable choice)
Comment on attachment 170507 [details] [diff] [review]
patch use utf8 in the newsrc/hostinfo file

diff -U 8 -p -r mailnews.old/base/prefs/resources/content/am-server.xul
mozilla/mailnews/base/prefs/resources/content/am-server.xul
--- mailnews.old/base/prefs/resources/content/am-server.xul	2005-03-04
02:08:45.000000000 +0800
+++ mozilla/mailnews/base/prefs/resources/content/am-server.xul 2005-04-13
13:20:05.000000000 +0800
@@ -47,16 +47,20 @@

 <page xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
       onload="parent.onPanelLoaded('am-server.xul');">

   <script type="application/x-javascript"
src="chrome://messenger/content/AccountManager.js"/>
   <script type="application/x-javascript"
src="chrome://messenger/content/am-server.js"/>
   <script type="application/x-javascript"
src="chrome://messenger/content/am-prefs.js"/>
   <script type="application/x-javascript"
src="chrome://messenger/content/amUtils.js"/>
+  <script type="application/x-javascript"><![CDATA[
+	var observerService =
Components.classes["@mozilla.org/observer-service;1"].getService(Components.int
erfaces.nsIObserverService);
+	observerService.notifyObservers(null, "charsetmenu-selected", "other");
+  ]]></script>
   <stringbundle id="bundle_messenger"
src="chrome://messenger/locale/messenger.properties"/>

   <broadcaster id="broadcaster_doBiff"/>
   <broadcaster id="broadcaster_notify"/>

   <label hidden="true" wsm_persist="true" id="server.type"/>

   <dialogheader title="&serverSettings.label;"/>
@@ -325,9 +329,22 @@
	    accesskey="&localPath.accesskey;"/>
     <hbox align="center">
       <textbox readonly="true" wsm_persist="true" flex="1"
id="server.localPath" datatype="nsIFileSpec"
		prefstring="mail.server.%serverkey%.directory"
class="uri-element"/>
       <button id="browseForLocalFolder" label="&browseFolder.label;"
filepickertitle="&localFolderPicker.label;"
	       accesskey="&browseFolder.accesskey;"
oncommand="BrowseForLocalFolders()"/>
     </hbox>
   </vbox>
+  <separator class="thin"/>
+   <hbox hidefor="imap,pop3,movemail" align="center" valign="middle"
iscontrolcontainer="true">
+     <label value="&serverDefaultCharset.label;" control="nntp.charset"/>
+     <menulist hidable="true" hidefor="imap,pop3,movemail" wsm_persist="true"
id="nntp.charset" 
+		  ref="NC:DecodersRoot" datasources="rdf:charset-menu"
+		  preftype="string"
prefstring="mail.server.%serverkey%.charset"> 
+	<template>
+	  <menupopup>
+	    <menuitem label="rdf:http://home.netscape.com/NC-rdf#Name"
value="..." uri="..."/>
+	  </menupopup>
+	</template>
+     </menulist>
+   </hbox> 
 </page>
Only in mozilla/mailnews/base/prefs/resources/content: am-server.xul~
diff -U 8 -p -r
mailnews.old/base/prefs/resources/locale/en-US/am-server-top.dtd
mozilla/mailnews/base/prefs/resources/locale/en-US/am-server-top.dtd
--- mailnews.old/base/prefs/resources/locale/en-US/am-server-top.dtd   
2005-02-09 01:01:47.000000000 +0800
+++ mozilla/mailnews/base/prefs/resources/locale/en-US/am-server-top.dtd       
2005-04-13 13:19:30.000000000 +0800
@@ -79,15 +79,16 @@
 <!ENTITY newsrcFilePath.label "newsrc file:">
 <!ENTITY newsrcFilePath.accesskey "f">
 <!ENTITY localPath.label "Local directory:">
 <!ENTITY localPath.accesskey "L">
 <!ENTITY abbreviate.label "Show newsgroup names in the Mail Folder pane as:">
 <!ENTITY abbreviateOn.label "Full names (For example,
'netscape.public.mozilla.mail-news')">
 <!ENTITY abbreviateOff.label "Abbreviate names (For example,
'n.p.m.mail-news')">
 <!ENTITY advancedButton.label "Advanced...">
+<!ENTITY serverDefaultCharset.label "Default Character Encoding:">
 <!ENTITY advancedButton.accesskey "v">
 <!ENTITY localFolderPicker.label "Select Local Directory">
 <!ENTITY browseFolder.label "Browse...">
 <!ENTITY browseFolder.accesskey "B">
 <!ENTITY newsrcPicker.label "Select newsrc File">
 <!ENTITY browseNewsrc.label "Browse...">
 <!ENTITY browseNewsrc.accesskey "e">
diff -U 8 -p -r mailnews.old/news/public/nsIMsgNewsFolder.idl
mozilla/mailnews/news/public/nsIMsgNewsFolder.idl
--- mailnews.old/news/public/nsIMsgNewsFolder.idl	2004-04-18
02:33:14.000000000 +0800
+++ mozilla/mailnews/news/public/nsIMsgNewsFolder.idl	2005-04-19
03:37:37.000000000 +0800
@@ -36,27 +36,28 @@
  * ***** END LICENSE BLOCK ***** */

 #include "nsISupports.idl"
 #include "nsIMsgFolder.idl"

 interface nsIMsgWindow;
 interface nsINntpIncomingServer;

-[scriptable, uuid(3716abe4-f6d4-11d2-86d5-004005263078)]
+[scriptable, uuid(02699533-7ba8-4e78-94dc-a20374fb2711)]
 interface nsIMsgNewsFolder : nsISupports {
   attribute string groupUsername;
   attribute string groupPassword;

-  /** the ascii name for the newsgroup.  
-   *  newsgroup names are always ascii, and this can help simply
-   *  the code, to prevent use from having to get the name (as a wstring)
-   *  and then converting it back to ascii.
+  readonly attribute AString unicodeName;
+  /** the raw name name for the newsgroup (server side name).	
+   *  newsgroup names are might not always ascii(bug#126453), and this raw
name name 
+   *  can help simply the code, to prevent use from having to get the unicode
name 
+   *  (as a wstring) and then converting it back to server side raw name.
    */
-  readonly attribute string asciiName;
+  readonly attribute string rawName;
   readonly attribute nsINntpIncomingServer nntpServer;
   attribute boolean saveArticleOffline;

   string getGroupPasswordWithUI(in wstring aPromptString, in wstring
aPromptTitle, in nsIMsgWindow aMsgWindow);
   string getGroupUsernameWithUI(in wstring aPromptString, in wstring
aPromptTitle, in nsIMsgWindow aMsgWindow);

   void forgetGroupUsername();
   void forgetGroupPassword();
Only in mozilla/mailnews/news/public: nsIMsgNewsFolder.idl~
diff -U 8 -p -r mailnews.old/news/public/nsINewsDownloadDialogArgs.idl
mozilla/mailnews/news/public/nsINewsDownloadDialogArgs.idl
--- mailnews.old/news/public/nsINewsDownloadDialogArgs.idl	2004-04-18
02:33:14.000000000 +0800
+++ mozilla/mailnews/news/public/nsINewsDownloadDialogArgs.idl	2005-04-13
13:22:00.000000000 +0800
@@ -33,19 +33,19 @@
  * and other provisions required by the GPL or the LGPL. If you do not delete
  * the provisions above, a recipient may use your version of this file under
  * the terms of any one of the MPL, the GPL or the LGPL.
  *
  * ***** END LICENSE BLOCK ***** */

 #include "nsISupports.idl"

-[scriptable, uuid(b5037f06-1dd1-11b2-bf61-fcab5b5837a7)]
+[scriptable, uuid(3634327c-392b-4686-adf5-576e6cef9196)]
 interface nsINewsDownloadDialogArgs: nsISupports {
-  attribute string groupName;
+  attribute AString groupName;
   attribute long articleCount;
   attribute string serverKey;
   attribute boolean hitOK;
   attribute boolean downloadAll;
 };

 %{ C++
 #define DOWNLOAD_HEADERS_URL "chrome://messenger/content/downloadheaders.xul"
diff -U 8 -p -r mailnews.old/news/public/nsINntpIncomingServer.idl
mozilla/mailnews/news/public/nsINntpIncomingServer.idl
--- mailnews.old/news/public/nsINntpIncomingServer.idl	2004-04-18
02:33:14.000000000 +0800
+++ mozilla/mailnews/news/public/nsINntpIncomingServer.idl	2005-04-13
13:22:54.000000000 +0800
@@ -61,16 +61,19 @@ interface nsINntpIncomingServer : nsISup
	attribute boolean markOldRead;

	/* abbreviate the newsgroup names in the folder pane? */
	attribute boolean abbreviate;

     /* do we use a single login per server or do we login per group */
     attribute boolean singleSignon;

+    /*  the server charset and it may be needed to display newsgroup folder
names correctly */
+    attribute ACString charset;
+
	/* the server keeps track of all the newsgroups we are subscribed to */
	void addNewsgroup(in string name);
	void removeNewsgroup(in string name);

     void writeNewsrcFile();

     attribute boolean newsrcHasChanged;

diff -U 8 -p -r mailnews.old/news/src/nsNewsDownloadDialogArgs.cpp
mozilla/mailnews/news/src/nsNewsDownloadDialogArgs.cpp
--- mailnews.old/news/src/nsNewsDownloadDialogArgs.cpp	2004-04-18
02:33:16.000000000 +0800
+++ mozilla/mailnews/news/src/nsNewsDownloadDialogArgs.cpp	2005-04-13
13:24:48.000000000 +0800
@@ -37,45 +37,41 @@
  * ***** END LICENSE BLOCK ***** */

 #include "nsNewsDownloadDialogArgs.h"
 #include "nsCRT.h"
 #include "nsReadableUtils.h"

 nsNewsDownloadDialogArgs::nsNewsDownloadDialogArgs()
 {
-    mGroupName = "";
     mArticleCount = 0;
     mServerKey = "";
     mHitOK = PR_FALSE;
     mDownloadAll = PR_FALSE;
 }

 nsNewsDownloadDialogArgs::~nsNewsDownloadDialogArgs()
 {
 }

 NS_IMPL_ISUPPORTS1(nsNewsDownloadDialogArgs, nsINewsDownloadDialogArgs)

-NS_IMETHODIMP nsNewsDownloadDialogArgs::GetGroupName(char * *aGroupName)
-{
-    NS_ENSURE_ARG_POINTER(aGroupName);
-
-    *aGroupName = ToNewCString(mGroupName);
-
-    return NS_OK;
-}
-NS_IMETHODIMP nsNewsDownloadDialogArgs::SetGroupName(const char * aGroupName)
-{
-    NS_ENSURE_ARG_POINTER(aGroupName);
-
-    mGroupName = aGroupName;
-
-    return NS_OK;
-}
+NS_IMETHODIMP nsNewsDownloadDialogArgs::GetGroupName(nsAString & aGroupName)
+ {
+    aGroupName = mGroupName;
+ 
+     return NS_OK;
+ }
+NS_IMETHODIMP nsNewsDownloadDialogArgs::SetGroupName(const nsAString &
aGroupName)
+ {
+ 
+     mGroupName = aGroupName;
+ 
+     return NS_OK;
+ }
 NS_IMETHODIMP nsNewsDownloadDialogArgs::GetArticleCount(PRInt32
*aArticleCount)
 {
     NS_ENSURE_ARG_POINTER(aArticleCount);

     *aArticleCount = mArticleCount;

     return NS_OK;
 }
diff -U 8 -p -r mailnews.old/news/src/nsNewsDownloadDialogArgs.h
mozilla/mailnews/news/src/nsNewsDownloadDialogArgs.h
--- mailnews.old/news/src/nsNewsDownloadDialogArgs.h	2004-04-18
02:33:16.000000000 +0800
+++ mozilla/mailnews/news/src/nsNewsDownloadDialogArgs.h	2005-04-13
13:25:12.000000000 +0800
@@ -47,16 +47,16 @@ class nsNewsDownloadDialogArgs : public 
 public:
   nsNewsDownloadDialogArgs();
   virtual ~nsNewsDownloadDialogArgs();

   NS_DECL_ISUPPORTS
   NS_DECL_NSINEWSDOWNLOADDIALOGARGS

 private:
-  nsCString mGroupName;
+  nsString mGroupName;
   PRInt32 mArticleCount;
   nsCString mServerKey;
   PRBool mHitOK;
   PRBool mDownloadAll;
 };

 #endif // nsNewsDownloadDialogArgs_h__
diff -U 8 -p -r mailnews.old/news/src/nsNewsFolder.cpp
mozilla/mailnews/news/src/nsNewsFolder.cpp
--- mailnews.old/news/src/nsNewsFolder.cpp	2005-02-15 08:13:35.000000000
+0800
+++ mozilla/mailnews/news/src/nsNewsFolder.cpp	2005-04-19 08:33:14.976769560
+0800
@@ -88,16 +88,18 @@

 #include "nsIInterfaceRequestor.h"
 #include "nsIInterfaceRequestorUtils.h"

 #include "nsReadableUtils.h"
 #include "nsNewsDownloader.h"
 #include "nsIStringBundle.h"
 #include "nsEscape.h"
+#include "nsMsgI18N.h"
+#include "nsNativeCharsetUtils.h"

 // we need this because of an egcs 1.0 (and possibly gcc) compiler bug
 // that doesn't allow you to call ::nsISupports::GetIID() inside of a class
 // that multiply inherits from nsISupports
 static NS_DEFINE_CID(kRDFServiceCID, NS_RDFSERVICE_CID);
 static NS_DEFINE_CID(kCNewsDB, NS_NEWSDB_CID);

 // ###tw  This really ought to be the most
@@ -112,33 +114,33 @@ static NS_DEFINE_CID(kCNewsDB, NS_NEWSDB



///////////////////////////////////////////////////////////////////////////////
/

 nsMsgNewsFolder::nsMsgNewsFolder(void) : nsMsgLineBuffer(nsnull, PR_FALSE),
      mExpungedBytes(0), mGettingNews(PR_FALSE),
     mInitialized(PR_FALSE), mOptionLines(""), mUnsubscribedNewsgroupLines(""), 
     m_downloadMessageForOfflineUse(PR_FALSE),
m_downloadingMultipleMessages(PR_FALSE), 
-    mReadSet(nsnull), mGroupUsername(nsnull), mGroupPassword(nsnull),
mAsciiName(nsnull)
+    mReadSet(nsnull), mGroupUsername(nsnull), mGroupPassword(nsnull),
mRawName(nsnull)
 {
   MOZ_COUNT_CTOR(nsNewsFolder); // double count these for now.
   /* we're parsing the newsrc file, and the line breaks are platform specific.
    * if MSG_LINEBREAK != CRLF, then we aren't looking for CRLF 
    */
   if (PL_strcmp(MSG_LINEBREAK, CRLF))
     SetLookingForCRLF(PR_FALSE);
 }

 nsMsgNewsFolder::~nsMsgNewsFolder(void)
 {
   MOZ_COUNT_DTOR(nsNewsFolder);
   delete mReadSet;
   PR_Free(mGroupUsername);
   PR_Free(mGroupPassword);
-  PR_Free(mAsciiName);
+  PR_Free(mRawName);
 }

 NS_IMPL_ADDREF_INHERITED(nsMsgNewsFolder, nsMsgDBFolder)
 NS_IMPL_RELEASE_INHERITED(nsMsgNewsFolder, nsMsgDBFolder)

 NS_IMETHODIMP nsMsgNewsFolder::QueryInterface(REFNSIID aIID, void**
aInstancePtr)
 {
   if (!aInstancePtr) return NS_ERROR_NULL_POINTER;
@@ -205,20 +207,24 @@ nsMsgNewsFolder::AddNewsgroup(const char
   nsCOMPtr <nsINntpIncomingServer> nntpServer;
   rv = GetNntpServer(getter_AddRefs(nntpServer));
   if (NS_FAILED(rv)) return rv;

   nsCAutoString uri(mURI);
   uri.Append('/');
   // URI should use UTF-8
   // (see RFC2396 Uniform Resource Identifiers (URI): Generic Syntax)
-  // since we are forcing it to be latin-1 (IS0-8859-1)
-  // we can just assign with conversion
+  nsCAutoString dataCharset;
+  rv = nntpServer->GetCharset(dataCharset);
+  if (NS_FAILED(rv)) return rv;
   nsAutoString newsgroupName;
-  newsgroupName.AssignWithConversion(name);
+  rv =
nsMsgI18NConvertToUnicode(dataCharset.get(),nsDependentCString(name),newsgroupN
ame);
+  if (NS_FAILED(rv)) {
+    CopyASCIItoUTF16(name, newsgroupName);
+  }

   nsCAutoString escapedName;
   rv = NS_MsgEscapeEncodeURLPath(newsgroupName, escapedName);
   if (NS_FAILED(rv)) return rv;

   rv = nntpServer->AddNewsgroup(escapedName.get());
   if (NS_FAILED(rv)) return rv;

@@ -511,32 +517,35 @@ nsMsgNewsFolder::GetMessages(nsIMsgWindo
 }

 NS_IMETHODIMP nsMsgNewsFolder::GetFolderURL(char **url)
 {
   NS_ENSURE_ARG_POINTER(url);

   nsXPIDLCString hostName;
   nsresult rv = GetHostname(getter_Copies(hostName));
-  nsXPIDLCString groupName;
-  rv = GetAsciiName(getter_Copies(groupName));
+  nsXPIDLString groupName;
+  rv = GetName(getter_Copies(groupName));
   NS_ENSURE_SUCCESS(rv, rv);

   nsCOMPtr<nsIMsgIncomingServer> server;
   rv = GetServer(getter_AddRefs(server));
   NS_ENSURE_SUCCESS(rv, rv);

   PRInt32 port;
   PRBool isSecure = PR_FALSE;
   rv = server->GetIsSecure(&isSecure);
   NS_ENSURE_SUCCESS(rv, rv);
   rv = server->GetPort(&port);
   NS_ENSURE_SUCCESS(rv, rv);
   const char *newsScheme = (isSecure) ? SNEWS_SCHEME : NEWS_SCHEME;
-  *url = PR_smprintf("%s//%s:%ld/%s", newsScheme, hostName.get(), port,
groupName.get());
+  nsXPIDLCString escapedName;
+  rv = NS_MsgEscapeEncodeURLPath(groupName, escapedName);
+  NS_ENSURE_SUCCESS(rv, rv);
+  *url = PR_smprintf("%s//%s:%ld/%s", newsScheme, hostName.get(), port,
escapedName.get());
   return NS_OK;

 }

 NS_IMETHODIMP nsMsgNewsFolder::SetNewsrcHasChanged(PRBool newsrcHasChanged)
 {
     nsresult rv;

@@ -549,18 +558,28 @@ NS_IMETHODIMP nsMsgNewsFolder::SetNewsrc

 NS_IMETHODIMP nsMsgNewsFolder::CreateSubfolder(const PRUnichar
*uninewsgroupname, nsIMsgWindow *msgWindow)
 {
   nsresult rv = NS_OK;

   NS_ENSURE_ARG_POINTER(uninewsgroupname);
   if (!*uninewsgroupname) return NS_ERROR_FAILURE;

-  nsCAutoString newsgroupname; 
-  newsgroupname.AssignWithConversion(uninewsgroupname);
+  nsCAutoString newsgroupname;
+  nsCOMPtr <nsINntpIncomingServer> nntpServer;
+  rv = GetNntpServer(getter_AddRefs(nntpServer));
+  if (NS_FAILED(rv)) return rv;
+  nsCAutoString dataCharset;
+  rv = nntpServer->GetCharset(dataCharset);
+  if (NS_FAILED(rv)) return rv;
+
+  rv = nsMsgI18NConvertFromUnicode(dataCharset.get(),
nsDependentString(uninewsgroupname), newsgroupname);
+  if (NS_FAILED(rv)) {
+    LossyCopyUTF16toASCII(uninewsgroupname,newsgroupname);
+  }

   nsFileSpec path;
   nsCOMPtr<nsIFileSpec> pathSpec;
   rv = GetPath(getter_AddRefs(pathSpec));
   if (NS_FAILED(rv)) return rv;

   rv = pathSpec->GetFileSpec(&path);

@@ -569,29 +588,32 @@ NS_IMETHODIMP nsMsgNewsFolder::CreateSub
   // Create an empty database for this mail folder, set its name from the user 
   nsCOMPtr<nsIMsgDatabase> newsDBFactory;
   nsCOMPtr <nsIMsgDatabase> newsDB;

   //Now we have a valid directory or we have returned.
   //Make sure the new folder name is valid

   // remember, some file systems (like mac) can't handle long file names
-  nsCAutoString hashedName = newsgroupname;
+  nsCAutoString fileName;
+  nsAutoString hashedName(uninewsgroupname);
   rv = NS_MsgHashIfNecessary(hashedName);
-  path += hashedName.get();
+  NS_CopyUnicodeToNative(hashedName, fileName);
+  path += fileName.get();

   //Now let's create the actual new folder
   rv = AddNewsgroup(newsgroupname.get(), "", getter_AddRefs(child));

   if (NS_SUCCEEDED(rv))
     SetNewsrcHasChanged(PR_TRUE); // subscribe UI does this - but maybe we got
here through auto-subscribe

-  if(NS_SUCCEEDED(rv) && child)
+  if(NS_SUCCEEDED(rv) && child){
+    child->SetCharset(dataCharset.get());
     NotifyItemAdded(child);
-
+  }
   return rv;
 }

 NS_IMETHODIMP nsMsgNewsFolder::Delete()
 {
   nsresult rv = GetDatabase(nsnull);

   if(NS_SUCCEEDED(rv)) 
@@ -1124,27 +1146,42 @@ nsMsgNewsFolder::HandleNewsrcLine(char* 
   //
   // So lines like this in a newsrc file should be ignored:
   // 3746EF3F.6080309@netscape.com:
   // 3746EF3F.6080309%40netscape.com:
   if (PL_strstr(line,"@") || PL_strstr(line,"%40")) 
     // skipping, it contains @ or %40
     subscribed = PR_FALSE;

+    nsCOMPtr<nsINntpIncomingServer> nntpServer;
+    rv = GetNntpServer(getter_AddRefs(nntpServer));
+    if (NS_FAILED(rv)) return rv;
+    //now we convert from UTF8 to the server charset encoding because the
newsrc file is always saved in UTF8
+    nsAutoString unicodeLine;
+    nsCAutoString rawLine;
+    CopyUTF8toUTF16(line,unicodeLine);
+
+    nsCAutoString dataCharset;
+    rv = nntpServer->GetCharset(dataCharset);
+    NS_ASSERTION(NS_SUCCEEDED(rv),"failed get sever charset");
+    rv = nsMsgI18NConvertFromUnicode(dataCharset.get(), unicodeLine, rawLine);
+    if (NS_FAILED(rv))
+	LossyCopyUTF16toASCII(unicodeLine,rawLine);
+
   if (subscribed) 
   {
     // we're subscribed, so add it
     nsCOMPtr <nsIMsgFolder> child;

-    rv = AddNewsgroup(line, setStr, getter_AddRefs(child));
+    rv = AddNewsgroup(rawLine.get(), setStr, getter_AddRefs(child));

     if (NS_FAILED(rv)) return -1;
   }
   else {
-    rv = RememberUnsubscribedGroup(line, setStr);
+    rv = RememberUnsubscribedGroup(rawLine.get(), setStr);
     if (NS_FAILED(rv)) return -1;
   }

   return 0;
 }


 nsresult
@@ -1503,20 +1540,22 @@ nsresult nsMsgNewsFolder::CreateBaseMess

 NS_IMETHODIMP
 nsMsgNewsFolder::GetNewsrcLine(char **newsrcLine)
 {
   nsresult rv;

   if (!newsrcLine) return NS_ERROR_NULL_POINTER;

+  nsXPIDLString uninewsgroupname;
   nsXPIDLCString newsgroupname;
-  rv = GetAsciiName(getter_Copies(newsgroupname));
+  rv = GetName(getter_Copies(uninewsgroupname));
   if (NS_FAILED(rv)) return rv;
-  
+  CopyUTF16toUTF8(uninewsgroupname, newsgroupname);
+
   nsCAutoString newsrcLineStr;
   newsrcLineStr = newsgroupname;
   newsrcLineStr += ':';

   if (mReadSet) {
     nsXPIDLCString setStr;
     mReadSet->Output(getter_Copies(setStr));
     if (NS_SUCCEEDED(rv)) 
@@ -1575,38 +1614,62 @@ nsMsgNewsFolder::GetOptionLines(char **o
 }

 NS_IMETHODIMP
 nsMsgNewsFolder::OnReadChanged(nsIDBChangeListener * aInstigator)
 {
     return SetNewsrcHasChanged(PR_TRUE);
 }

+NS_IMETHODIMP
+nsMsgNewsFolder::GetUnicodeName(nsAString & name)
+{
+  nsresult rv;
+  if(mUnicodeName.IsEmpty()){
+    nsXPIDLString newsgroupName;
+    rv = GetName(getter_Copies(newsgroupName));
+    NS_ENSURE_SUCCESS(rv,rv);
+    mUnicodeName = newsgroupName;
+  }
+  name = mUnicodeName;
+  return NS_OK;
+}

 NS_IMETHODIMP
-nsMsgNewsFolder::GetAsciiName(char **asciiName)
+nsMsgNewsFolder::GetRawName(char **rawName)
 {
   nsresult rv;
-  NS_ENSURE_ARG_POINTER(asciiName);
-  if (!mAsciiName) 
+  NS_ENSURE_ARG_POINTER(rawName);
+  if (!mRawName) 
   {
     nsXPIDLString name;
     rv = GetName(getter_Copies(name));
     NS_ENSURE_SUCCESS(rv,rv);

     // convert to ASCII
     nsCAutoString tmpStr;
-    tmpStr.AssignWithConversion(name);
+    nsCOMPtr <nsINntpIncomingServer> nntpServer;
+    rv = GetNntpServer(getter_AddRefs(nntpServer));
+    NS_ENSURE_SUCCESS(rv,rv);
+    
+    nsCAutoString dataCharset;
+    rv = nntpServer->GetCharset(dataCharset);
+    NS_ENSURE_SUCCESS(rv,rv);
+    rv = nsMsgI18NConvertFromUnicode(dataCharset.get(), name, tmpStr);
+
+    if (NS_FAILED(rv)) {
+      LossyCopyUTF16toASCII(name,tmpStr);
+    }

-    mAsciiName = nsCRT::strdup(tmpStr.get());
-    if (!mAsciiName) return NS_ERROR_OUT_OF_MEMORY;
+    mRawName = nsCRT::strdup(tmpStr.get());
+    if (!mRawName) return NS_ERROR_OUT_OF_MEMORY;
   }

-  *asciiName = nsCRT::strdup(mAsciiName);
-  if (!*asciiName) return NS_ERROR_OUT_OF_MEMORY;
+  *rawName = nsCRT::strdup(mRawName);
+  if (!*rawName) return NS_ERROR_OUT_OF_MEMORY;

   return NS_OK;
 }

 NS_IMETHODIMP
 nsMsgNewsFolder::GetNntpServer(nsINntpIncomingServer **result)
 {
     nsresult rv;
Only in mozilla/mailnews/news/src: nsNewsFolder.cpp~
diff -U 8 -p -r mailnews.old/news/src/nsNewsFolder.h
mozilla/mailnews/news/src/nsNewsFolder.h
--- mailnews.old/news/src/nsNewsFolder.h	2004-04-18 02:33:16.000000000
+0800
+++ mozilla/mailnews/news/src/nsNewsFolder.h	2005-04-19 08:33:47.599810104
+0800
@@ -154,18 +154,18 @@ protected:
   nsMsgKeySet *mReadSet; 

   nsCOMPtr<nsIFileSpec> mNewsrcFilePath; 

   // used for auth news
   char 		*mGroupUsername;
   char 	*mGroupPassword;

-  // the name of the newsgroup.
-  char    *mAsciiName;
+  char    *mRawName;
+  nsString mUnicodeName;
   PRInt32 mSortOrder;

 private:
     nsresult CreateNewsgroupUsernameUrlForSignon(const char *inUriStr, char
**result);
     nsresult CreateNewsgroupPasswordUrlForSignon(const char *inUriStr, char
**result);
     nsresult CreateNewsgroupUrlForSignon(const char *inUriStr, const char
*ref, char **result);

     nsCOMPtr <nsIFileSpec> mFilterFile;
Only in mozilla/mailnews/news/src: nsNewsFolder.h~
diff -U 8 -p -r mailnews.old/news/src/nsNntpIncomingServer.cpp
mozilla/mailnews/news/src/nsNntpIncomingServer.cpp
--- mailnews.old/news/src/nsNntpIncomingServer.cpp	2004-08-27
06:16:52.000000000 +0800
+++ mozilla/mailnews/news/src/nsNntpIncomingServer.cpp	2005-04-19
08:33:31.246296216 +0800
@@ -57,22 +57,25 @@
 #include "nsMsgUtils.h"
 #include "nsIPrompt.h"
 #include "nsIStringBundle.h"
 #include "nntpCore.h"
 #include "nsIWindowWatcher.h"
 #include "nsITreeColumns.h"
 #include "nsIDOMElement.h"
 #include "nsMsgFolderFlags.h"
+#include "nsMsgI18N.h"
+#include "nsIPrefLocalizedString.h"

 #define INVALID_VERSION	 0
 #define VALID_VERSION		 1
 #define NEW_NEWS_DIR_NAME	 "News"
 #define PREF_MAIL_NEWSRC_ROOT	 "mail.newsrc_root"
 #define PREF_MAIL_NEWSRC_ROOT_REL "mail.newsrc_root-rel"
+#define PREF_MAILNEWS_VIEW_DEFAULT_CHARSET "mailnews.view_default_charset"
 #define HOSTINFO_FILE_NAME	 "hostinfo.dat"

 #define NEWS_DELIMITER 	 '.'

 // this platform specific junk is so the newsrc filenames we create 
 // will resemble the migrated newsrc filenames.
 #if defined(XP_UNIX) || defined(XP_BEOS)
 #define NEWSRC_FILE_PREFIX "newsrc-"
@@ -304,16 +307,70 @@ nsresult nsNntpIncomingServer::SetupNews
     mNewsrcSaveTimer = do_CreateInstance("@mozilla.org/timer;1");
	mNewsrcSaveTimer->InitWithFuncCallback(OnNewsrcSaveTimer, (void*)this,
timeInMSUint32, 
					    nsITimer::TYPE_REPEATING_SLACK);

     return NS_OK;
 }

 NS_IMETHODIMP
+nsNntpIncomingServer::SetCharset(const nsACString & aCharset)
+{
+	nsresult rv;
+	rv = SetCharValue("charset", PromiseFlatCString(aCharset).get());
+	return rv;
+}
+
+NS_IMETHODIMP
+nsNntpIncomingServer::GetCharset(nsACString & aCharset)
+{
+	nsresult rv; 
+	nsXPIDLCString serverCharset;
+	//first we get the per-server settings mail.server.<serverkey>.charset
+	rv = GetCharValue("charset",getter_Copies(serverCharset));
+
+	//if the per-server setting is empty,we get the default charset from 
+	//mailnews.view_default_charset setting and set it as per-server
preference.
+	if(serverCharset.IsEmpty()){
+		nsXPIDLCString defaultCharset;
+		nsCOMPtr<nsIPrefBranch>
prefBranch(do_GetService(NS_PREFSERVICE_CONTRACTID, &rv));
+
+		if (NS_SUCCEEDED(rv)){
+			nsCOMPtr<nsIPrefLocalizedString> pls;
+			rv =
prefBranch->GetComplexValue(PREF_MAILNEWS_VIEW_DEFAULT_CHARSET,
+		       NS_GET_IID(nsIPrefLocalizedString),
getter_AddRefs(pls));
+			if (NS_SUCCEEDED(rv)) 
+			{
+				nsXPIDLString ucsval;
+				pls->ToString(getter_Copies(ucsval));
+				if (ucsval)
+				{
+					LossyCopyUTF16toASCII(ucsval,
defaultCharset);
+				}
+			}
+		}
+
+		if(defaultCharset.IsEmpty()){
+			serverCharset = "ISO-8859-1";
+		}else{
+			serverCharset = defaultCharset;
+		}
+
+#ifdef DEBUG_NEWS
+		printf("default charset for the server is %s\n", (const char
*)defaultCharset);
+#endif
+
+		SetCharset(serverCharset);
+	}
+
+	aCharset = serverCharset;
+	return NS_OK;
+}
+
+NS_IMETHODIMP
 nsNntpIncomingServer::WriteNewsrcFile()
 {
     nsresult rv;

     PRBool newsrcHasChanged;
     rv = GetNewsrcHasChanged(&newsrcHasChanged);
     if (NS_FAILED(rv)) return rv;

@@ -806,17 +863,17 @@ writeGroupToHostInfoFile(nsCString &aEle
     nsAutoString name;
     nsresult rv = NS_MsgDecodeUnescapeURLPath(aElement, name); 
     if (NS_FAILED(rv)) {
	 // stop, something is bad.
	 return PR_FALSE;
     }

     nsCAutoString nameOnDisk;
-    nameOnDisk.AssignWithConversion(name.get());
+    CopyUTF16toUTF8(name, nameOnDisk);

     // XXX todo ",,1,0,0" is a temporary hack, fix it
     *stream << nameOnDisk.get() << ",,1,0,0" << MSG_LINEBREAK;
     return PR_TRUE;
 }

 nsresult
 nsNntpIncomingServer::WriteHostInfoFile()
@@ -1128,19 +1185,41 @@ nsNntpIncomingServer::UpdateSubscribed()

 NS_IMETHODIMP
 nsNntpIncomingServer::AddTo(const char *aName, PRBool addAsSubscribed, PRBool
aSubscribable, PRBool changeIfExists)
 {
     nsresult rv = EnsureInner();
     NS_ENSURE_SUCCESS(rv,rv);

     nsAutoString newsgroupName;
-    newsgroupName.AssignWithConversion(aName);
+    char *escapedName = nsnull;
+
+    nsCOMPtr<nsIMsgFolder> rootFolder;
+    rv = GetRootFolder(getter_AddRefs(rootFolder));
+    NS_ENSURE_SUCCESS(rv,rv);
+    if (!rootFolder) return NS_ERROR_FAILURE;
+    nsCOMPtr <nsIMsgNewsFolder> newsFolder = do_QueryInterface(rootFolder,
&rv);
+    NS_ENSURE_SUCCESS(rv,rv);
+    if (!newsFolder) return NS_ERROR_FAILURE;
+    nsCOMPtr <nsINntpIncomingServer> nntpServer;
+    rv = newsFolder->GetNntpServer(getter_AddRefs(nntpServer));
+    NS_ENSURE_SUCCESS(rv,rv);
+    nsCAutoString dataCharset;
+    rv = nntpServer->GetCharset(dataCharset);
+    NS_ENSURE_SUCCESS(rv,rv);
+    rv =
nsMsgI18NConvertToUnicode(dataCharset.get(),nsDependentCString(aName),newsgroup
Name);
+    if (NS_SUCCEEDED(rv)) {
+	 escapedName = nsEscape(NS_ConvertUTF16toUTF8(newsgroupName).get(),
url_Path);
+     }
+
+    if (NS_FAILED(rv)) {
+      CopyASCIItoUTF16(aName, newsgroupName);
+      escapedName = nsEscape(aName, url_Path);
+    }

-    char *escapedName =
nsEscape(NS_ConvertUCS2toUTF8(newsgroupName.get()).get(), url_Path);
     if (!escapedName) return NS_ERROR_OUT_OF_MEMORY;

     rv = AddGroupOnServer(escapedName);
     NS_ENSURE_SUCCESS(rv,rv);

     rv = mInner->AddTo(escapedName,addAsSubscribed, aSubscribable,
changeIfExists);
     NS_ENSURE_SUCCESS(rv,rv);

@@ -1245,17 +1324,27 @@ nsNntpIncomingServer::HandleLine(char* l
	if (line[0] == '#' || line[0] == '\0') return 0;

	line[line_size] = 0;

	if (mHasSeenBeginGroups) {
		char *commaPos = PL_strchr(line,',');
		if (commaPos) *commaPos = 0;

-		nsresult rv = AddTo(line, PR_FALSE, PR_TRUE, PR_TRUE);
+		//now we convert from UTF8 to the server charset encoding
because the hostInfo.dat is always saved in UTF8
+	 nsAutoString unicodeLine;
+	 nsCAutoString encodedLine;
+	 CopyUTF8toUTF16(line,unicodeLine);
+	 nsCAutoString dataCharset;
+	 nsresult rv = GetCharset(dataCharset);
+	 NS_ASSERTION(NS_SUCCEEDED(rv),"failed to get sever charset");
+	 rv = nsMsgI18NConvertFromUnicode(dataCharset.get(), unicodeLine,
encodedLine);
+	 if (NS_FAILED(rv))
+	   LossyCopyUTF16toASCII(unicodeLine,encodedLine);
+		rv = AddTo(encodedLine.get(), PR_FALSE, PR_TRUE, PR_TRUE);
		NS_ASSERTION(NS_SUCCEEDED(rv),"failed to add line");
		if (NS_SUCCEEDED(rv)) {
	   // since we've seen one group, we can claim we've loaded the
hostinfo file
	   mHostInfoLoaded = PR_TRUE;
		}
	}
	else {
		if (nsCRT::strncmp(line,"begingroups", 11) == 0) {
Only in mozilla/mailnews/news/src: nsNntpIncomingServer.cpp~
diff -U 8 -p -r mailnews.old/news/src/nsNNTPNewsgroupList.cpp
mozilla/mailnews/news/src/nsNNTPNewsgroupList.cpp
--- mailnews.old/news/src/nsNNTPNewsgroupList.cpp	2005-02-01
05:53:40.000000000 +0800
+++ mozilla/mailnews/news/src/nsNNTPNewsgroupList.cpp	2005-04-13
13:55:22.000000000 +0800
@@ -319,21 +319,21 @@ nsNNTPNewsgroupList::GetRangeOfArtsToDow

		 nsCOMPtr<nsINewsDownloadDialogArgs> args =
do_CreateInstance("@mozilla.org/messenger/newsdownloaddialogargs;1", &rv);
		 if (NS_FAILED(rv)) return rv;
		 NS_ENSURE_SUCCESS(rv,rv);

		 rv = args->SetArticleCount(*last - *first + 1);
		 NS_ENSURE_SUCCESS(rv,rv);

-		 nsXPIDLCString groupName;
-		 rv = m_newsFolder->GetAsciiName(getter_Copies(groupName));
+		 nsXPIDLString groupName;
+		 rv = m_newsFolder->GetUnicodeName(groupName);
		 NS_ENSURE_SUCCESS(rv,rv);

-		 rv = args->SetGroupName((const char *)groupName);
+		 rv = args->SetGroupName(groupName);
		 NS_ENSURE_SUCCESS(rv,rv);

				// get the server key
				nsXPIDLCString serverKey;
				rv = server->GetKey(getter_Copies(serverKey));
		 NS_ENSURE_SUCCESS(rv,rv);

		 rv = args->SetServerKey((const char *)serverKey);
Only in mozilla/mailnews/news/src: nsNNTPNewsgroupList.cpp~
diff -U 8 -p -r mailnews.old/news/src/nsNNTPProtocol.cpp
mozilla/mailnews/news/src/nsNNTPProtocol.cpp
--- mailnews.old/news/src/nsNNTPProtocol.cpp	2005-02-16 11:18:46.000000000
+0800
+++ mozilla/mailnews/news/src/nsNNTPProtocol.cpp	2005-04-19
03:36:57.000000000 +0800
@@ -1934,17 +1934,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
	char *command=0;
	PRInt32 status = 0;

	if (m_typeWanted == ARTICLE_WANTED) {
		if (m_key != nsMsgKey_None) {
		    nsresult rv;
	     nsXPIDLCString newsgroupName;
	     if (m_newsFolder) {
-		 rv = m_newsFolder->GetAsciiName(getter_Copies(newsgroupName));
+		 rv = m_newsFolder->GetRawName(getter_Copies(newsgroupName));
		 NS_ENSURE_SUCCESS(rv,rv);
	     }

		    PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) current group = %s,
desired group = %s", this, m_currentGroup.get(), newsgroupName.get()));
	     // if the current group is the desired group, we can just issue
the ARTICLE command
	     // if not, we have to do a GROUP first
			if (!PL_strcmp(m_currentGroup.get(),
newsgroupName.get()))
			  m_nextState = NNTP_SEND_ARTICLE_NUMBER;
@@ -2045,17 +2045,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
	else if(m_typeWanted == GROUP_WANTED) 
     {
	 nsresult rv = NS_ERROR_NULL_POINTER;

	 NS_ASSERTION(m_newsFolder, "m_newsFolder is null, panic!");
	 if (!m_newsFolder) return -1;

	 nsXPIDLCString group_name;
-		rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+		rv = m_newsFolder->GetRawName(getter_Copies(group_name));
	 NS_ASSERTION(NS_SUCCEEDED(rv),"failed to get newsgroup name");
	 if (NS_FAILED(rv)) return -1;

	 m_firstArticle = 0;
	 m_lastArticle = 0;

	 NS_MsgSACopy(&command, "GROUP ");
	 NS_MsgSACat(&command, (const char *)group_name);
@@ -2090,17 +2090,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
	     PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) doing GROUP for XPAT", this));
	     nsXPIDLCString group_name;

			/* for XPAT, we have to GROUP into the group before
searching */
			if (!m_newsFolder) {
				NNTP_LOG_NOTE("m_newsFolder is null, panic!");
				return -1;
			}
-	     rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+	     rv = m_newsFolder->GetRawName(getter_Copies(group_name));
	     if (NS_FAILED(rv)) return -1;

			NS_MsgSACopy(&command, "GROUP ");
	     NS_MsgSACat (&command, group_name);

	     // force a GROUP next time
	     m_currentGroup.Truncate();
			m_nextState = NNTP_RESPONSE;
@@ -2201,17 +2201,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
   }
   else
   {
     nsresult rv = NS_OK;

     nsXPIDLCString group_name;
     NS_ASSERTION(m_newsFolder, "no newsFolder");
     if (m_newsFolder) {
-      rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+      rv = m_newsFolder->GetRawName(getter_Copies(group_name));
     }

     if (m_responseCode == MK_NNTP_RESPONSE_GROUP_NO_GROUP &&
       m_typeWanted == GROUP_WANTED) {
       PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) group (%s) not found, so unset
m_currentGroup",this,(const char *)group_name));
       m_currentGroup.Truncate();

       m_nntpServer->GroupNotFound(m_msgWindow, group_name.get(), PR_TRUE /*
opening */);
@@ -2316,17 +2316,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
 }

 PRInt32 nsNNTPProtocol::SendGroupForArticle()
 {
   nsresult rv;
   PRInt32 status = 0; 

   nsXPIDLCString groupname;
-  rv = m_newsFolder->GetAsciiName(getter_Copies(groupname));
+  rv = m_newsFolder->GetRawName(getter_Copies(groupname));
   NS_ASSERTION(NS_SUCCEEDED(rv) && groupname.get() && groupname.get()[0], "no
group name");

   char outputBuffer[OUTPUT_BUFFER_SIZE];

   PR_snprintf(outputBuffer, 
			OUTPUT_BUFFER_SIZE, 
			"GROUP %.512s" CRLF, 
			(const char *)groupname);
@@ -2348,17 +2348,17 @@ nsNNTPProtocol::SetCurrentGroup()
   nsresult rv;
   nsXPIDLCString groupname;
   NS_ASSERTION(m_newsFolder, "no news folder");
   if (!m_newsFolder) {
     m_currentGroup.Truncate();
     return NS_ERROR_UNEXPECTED;
   }

-  rv = m_newsFolder->GetAsciiName(getter_Copies(groupname));
+  rv = m_newsFolder->GetRawName(getter_Copies(groupname));
   NS_ASSERTION(NS_SUCCEEDED(rv) && groupname.get()[0], "no group name");
   PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) SetCurrentGroup to %s",this,(const char
*)groupname));
   m_currentGroup = groupname;
   return NS_OK;
 }

 PRInt32 nsNNTPProtocol::SendGroupForArticleResponse()
 {
@@ -3873,17 +3873,17 @@ PRInt32 nsNNTPProtocol::DisplayNewsRC()
     if (NS_FAILED(rv)) return -1;
     if (!currFolder) return -1;

     m_newsFolder = do_QueryInterface(currFolder, &rv);
     if (NS_FAILED(rv)) return -1;
     if (!m_newsFolder) return -1;

     nsXPIDLCString name;
-    rv = m_newsFolder->GetAsciiName(getter_Copies(name));
+    rv = m_newsFolder->GetRawName(getter_Copies(name));
     if (NS_FAILED(rv)) return -1;
     if (!name) return -1;

     /* send group command to server */
     char outputBuffer[OUTPUT_BUFFER_SIZE];

     PR_snprintf(outputBuffer, OUTPUT_BUFFER_SIZE, "GROUP %.512s" CRLF, (const
char *)name);
     nsCOMPtr<nsIMsgMailNewsUrl> mailnewsurl = do_QueryInterface(m_runningURL);
@@ -3963,17 +3963,17 @@ PRInt32 nsNNTPProtocol::DisplayNewsRCRes
     NS_ASSERTION(NS_SUCCEEDED(rv),"DisplaySubscribedGroup() failed");
     if (NS_FAILED(rv)) status = -1;

     if (status < 0) return status;
   }
   else if (m_responseCode == MK_NNTP_RESPONSE_GROUP_NO_GROUP)
   {
     nsXPIDLCString name;
-    rv = m_newsFolder->GetAsciiName(getter_Copies(name));
+    rv = m_newsFolder->GetRawName(getter_Copies(name));

     if (NS_SUCCEEDED(rv)) {
       m_nntpServer->GroupNotFound(m_msgWindow, name.get(), PR_FALSE);
     }

     PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) NO_GROUP, so unset m_currentGroup",
this));
     m_currentGroup.Truncate();
   }
@@ -4394,17 +4394,17 @@ PRInt32 nsNNTPProtocol::XPATResponse(nsI

 PRInt32 nsNNTPProtocol::ListPrettyNames()
 {

   nsXPIDLCString group_name;
   char outputBuffer[OUTPUT_BUFFER_SIZE];
   PRInt32 status = 0; 

-  nsresult rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+  nsresult rv = m_newsFolder->GetRawName(getter_Copies(group_name));
   PR_snprintf(outputBuffer, 
     OUTPUT_BUFFER_SIZE, 
     "LIST PRETTYNAMES %.512s" CRLF,
     NS_SUCCEEDED(rv) ? (const char *) group_name : "");

   nsCOMPtr<nsIMsgMailNewsUrl> mailnewsurl = do_QueryInterface(m_runningURL);
   if (mailnewsurl)
     status = SendData(mailnewsurl, outputBuffer);
@@ -4471,17 +4471,17 @@ PRInt32 nsNNTPProtocol::ListPrettyNamesR
   PR_FREEIF(line);
   return 0;
 }

 PRInt32 nsNNTPProtocol::ListXActive()
 { 
   nsXPIDLCString group_name;
   nsresult rv;
-  rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+  rv = m_newsFolder->GetRawName(getter_Copies(group_name));
   if (NS_FAILED(rv)) return -1;

   PRInt32 status = 0;
   char outputBuffer[OUTPUT_BUFFER_SIZE];

   PR_snprintf(outputBuffer, 
     OUTPUT_BUFFER_SIZE, 
     "LIST XACTIVE %.512s" CRLF,
@@ -4624,17 +4624,17 @@ PRInt32 nsNNTPProtocol::SendListGroup()
   nsresult rv;
   char outputBuffer[OUTPUT_BUFFER_SIZE];
   PRInt32 status = 0; 

   NS_ASSERTION(m_newsFolder,"no newsFolder");
   if (!m_newsFolder) return -1;
   nsXPIDLCString newsgroupName;

-  rv = m_newsFolder->GetAsciiName(getter_Copies(newsgroupName));
+  rv = m_newsFolder->GetRawName(getter_Copies(newsgroupName));
   NS_ENSURE_SUCCESS(rv,rv);

   PR_snprintf(outputBuffer, 
     OUTPUT_BUFFER_SIZE, 
     "listgroup %.512s" CRLF,
     newsgroupName.get());

   m_articleList = do_CreateInstance(NS_NNTPARTICLELIST_CONTRACTID, &rv);
Only in mozilla/mailnews/news/src: nsNNTPProtocol.cpp~
diff -U 8 -p -r mailnews.old/news/src/nsNntpService.cpp
mozilla/mailnews/news/src/nsNntpService.cpp
--- mailnews.old/news/src/nsNntpService.cpp	2005-01-18 02:50:13.000000000
+0800
+++ mozilla/mailnews/news/src/nsNntpService.cpp 2005-04-13 13:56:02.000000000
+0800
@@ -650,18 +650,21 @@ nsNntpService::GetFolderFromUri(const ch
   // but the *name* of the newsgroup (we are calling ::GetChildNamed())
   // is unescaped.  see http://bugzilla.mozilla.org/show_bug.cgi?id=210089#c17
   // for more about this
   char *unescapedPath = PL_strdup(path.get() + 1); /* skip the leading slash
*/
   if (!unescapedPath)
     return NS_ERROR_OUT_OF_MEMORY;
   nsUnescape(unescapedPath);

+  nsAutoString newsgroupName;
+  //the encoding is UTF-8 after unescaped.
+  CopyUTF8toUTF16(unescapedPath,newsgroupName);
   nsCOMPtr<nsISupports> subFolder;
-  rv = rootFolder->GetChildNamed(NS_ConvertASCIItoUCS2(unescapedPath).get() ,
getter_AddRefs(subFolder));
+  rv = rootFolder->GetChildNamed(newsgroupName.get() ,
getter_AddRefs(subFolder));
   PL_strfree(unescapedPath);
   NS_ENSURE_SUCCESS(rv,rv);

   return CallQueryInterface(subFolder, aFolder);
 }

 NS_IMETHODIMP
 nsNntpService::CopyMessage(const char * aSrcMessageURI, nsIStreamListener *
aMailboxCopyHandler, PRBool moveMessage,
Comment on attachment 170507 [details] [diff] [review]
patch use utf8 in the newsrc/hostinfo file

diff -U 8 -p -r mailnews.old/base/prefs/resources/content/am-server.xul
mozilla/mailnews/base/prefs/resources/content/am-server.xul
--- mailnews.old/base/prefs/resources/content/am-server.xul	2005-03-04
02:08:45.000000000 +0800
+++ mozilla/mailnews/base/prefs/resources/content/am-server.xul 2005-04-13
13:20:05.000000000 +0800
@@ -47,16 +47,20 @@

 <page xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
       onload="parent.onPanelLoaded('am-server.xul');">

   <script type="application/x-javascript"
src="chrome://messenger/content/AccountManager.js"/>
   <script type="application/x-javascript"
src="chrome://messenger/content/am-server.js"/>
   <script type="application/x-javascript"
src="chrome://messenger/content/am-prefs.js"/>
   <script type="application/x-javascript"
src="chrome://messenger/content/amUtils.js"/>
+  <script type="application/x-javascript"><![CDATA[
+	var observerService =
Components.classes["@mozilla.org/observer-service;1"].getService(Components.int
erfaces.nsIObserverService);
+	observerService.notifyObservers(null, "charsetmenu-selected", "other");
+  ]]></script>
   <stringbundle id="bundle_messenger"
src="chrome://messenger/locale/messenger.properties"/>

   <broadcaster id="broadcaster_doBiff"/>
   <broadcaster id="broadcaster_notify"/>

   <label hidden="true" wsm_persist="true" id="server.type"/>

   <dialogheader title="&serverSettings.label;"/>
@@ -325,9 +329,22 @@
	    accesskey="&localPath.accesskey;"/>
     <hbox align="center">
       <textbox readonly="true" wsm_persist="true" flex="1"
id="server.localPath" datatype="nsIFileSpec"
		prefstring="mail.server.%serverkey%.directory"
class="uri-element"/>
       <button id="browseForLocalFolder" label="&browseFolder.label;"
filepickertitle="&localFolderPicker.label;"
	       accesskey="&browseFolder.accesskey;"
oncommand="BrowseForLocalFolders()"/>
     </hbox>
   </vbox>
+  <separator class="thin"/>
+   <hbox hidefor="imap,pop3,movemail" align="center" valign="middle"
iscontrolcontainer="true">
+     <label value="&serverDefaultCharset.label;" control="nntp.charset"/>
+     <menulist hidable="true" hidefor="imap,pop3,movemail" wsm_persist="true"
id="nntp.charset" 
+		  ref="NC:DecodersRoot" datasources="rdf:charset-menu"
+		  preftype="string"
prefstring="mail.server.%serverkey%.charset"> 
+	<template>
+	  <menupopup>
+	    <menuitem label="rdf:http://home.netscape.com/NC-rdf#Name"
value="..." uri="..."/>
+	  </menupopup>
+	</template>
+     </menulist>
+   </hbox> 
 </page>
Only in mozilla/mailnews/base/prefs/resources/content: am-server.xul~
diff -U 8 -p -r
mailnews.old/base/prefs/resources/locale/en-US/am-server-top.dtd
mozilla/mailnews/base/prefs/resources/locale/en-US/am-server-top.dtd
--- mailnews.old/base/prefs/resources/locale/en-US/am-server-top.dtd   
2005-02-09 01:01:47.000000000 +0800
+++ mozilla/mailnews/base/prefs/resources/locale/en-US/am-server-top.dtd       
2005-04-13 13:19:30.000000000 +0800
@@ -79,15 +79,16 @@
 <!ENTITY newsrcFilePath.label "newsrc file:">
 <!ENTITY newsrcFilePath.accesskey "f">
 <!ENTITY localPath.label "Local directory:">
 <!ENTITY localPath.accesskey "L">
 <!ENTITY abbreviate.label "Show newsgroup names in the Mail Folder pane as:">
 <!ENTITY abbreviateOn.label "Full names (For example,
'netscape.public.mozilla.mail-news')">
 <!ENTITY abbreviateOff.label "Abbreviate names (For example,
'n.p.m.mail-news')">
 <!ENTITY advancedButton.label "Advanced...">
+<!ENTITY serverDefaultCharset.label "Default Character Encoding:">
 <!ENTITY advancedButton.accesskey "v">
 <!ENTITY localFolderPicker.label "Select Local Directory">
 <!ENTITY browseFolder.label "Browse...">
 <!ENTITY browseFolder.accesskey "B">
 <!ENTITY newsrcPicker.label "Select newsrc File">
 <!ENTITY browseNewsrc.label "Browse...">
 <!ENTITY browseNewsrc.accesskey "e">
diff -U 8 -p -r mailnews.old/news/public/nsIMsgNewsFolder.idl
mozilla/mailnews/news/public/nsIMsgNewsFolder.idl
--- mailnews.old/news/public/nsIMsgNewsFolder.idl	2004-04-18
02:33:14.000000000 +0800
+++ mozilla/mailnews/news/public/nsIMsgNewsFolder.idl	2005-04-19
03:37:37.000000000 +0800
@@ -36,27 +36,28 @@
  * ***** END LICENSE BLOCK ***** */

 #include "nsISupports.idl"
 #include "nsIMsgFolder.idl"

 interface nsIMsgWindow;
 interface nsINntpIncomingServer;

-[scriptable, uuid(3716abe4-f6d4-11d2-86d5-004005263078)]
+[scriptable, uuid(02699533-7ba8-4e78-94dc-a20374fb2711)]
 interface nsIMsgNewsFolder : nsISupports {
   attribute string groupUsername;
   attribute string groupPassword;

-  /** the ascii name for the newsgroup.  
-   *  newsgroup names are always ascii, and this can help simply
-   *  the code, to prevent use from having to get the name (as a wstring)
-   *  and then converting it back to ascii.
+  readonly attribute AString unicodeName;
+  /** the raw name name for the newsgroup (server side name).	
+   *  newsgroup names are might not always ascii(bug#126453), and this raw
name name 
+   *  can help simply the code, to prevent use from having to get the unicode
name 
+   *  (as a wstring) and then converting it back to server side raw name.
    */
-  readonly attribute string asciiName;
+  readonly attribute string rawName;
   readonly attribute nsINntpIncomingServer nntpServer;
   attribute boolean saveArticleOffline;

   string getGroupPasswordWithUI(in wstring aPromptString, in wstring
aPromptTitle, in nsIMsgWindow aMsgWindow);
   string getGroupUsernameWithUI(in wstring aPromptString, in wstring
aPromptTitle, in nsIMsgWindow aMsgWindow);

   void forgetGroupUsername();
   void forgetGroupPassword();
Only in mozilla/mailnews/news/public: nsIMsgNewsFolder.idl~
diff -U 8 -p -r mailnews.old/news/public/nsINewsDownloadDialogArgs.idl
mozilla/mailnews/news/public/nsINewsDownloadDialogArgs.idl
--- mailnews.old/news/public/nsINewsDownloadDialogArgs.idl	2004-04-18
02:33:14.000000000 +0800
+++ mozilla/mailnews/news/public/nsINewsDownloadDialogArgs.idl	2005-04-13
13:22:00.000000000 +0800
@@ -33,19 +33,19 @@
  * and other provisions required by the GPL or the LGPL. If you do not delete
  * the provisions above, a recipient may use your version of this file under
  * the terms of any one of the MPL, the GPL or the LGPL.
  *
  * ***** END LICENSE BLOCK ***** */

 #include "nsISupports.idl"

-[scriptable, uuid(b5037f06-1dd1-11b2-bf61-fcab5b5837a7)]
+[scriptable, uuid(3634327c-392b-4686-adf5-576e6cef9196)]
 interface nsINewsDownloadDialogArgs: nsISupports {
-  attribute string groupName;
+  attribute AString groupName;
   attribute long articleCount;
   attribute string serverKey;
   attribute boolean hitOK;
   attribute boolean downloadAll;
 };

 %{ C++
 #define DOWNLOAD_HEADERS_URL "chrome://messenger/content/downloadheaders.xul"
diff -U 8 -p -r mailnews.old/news/public/nsINntpIncomingServer.idl
mozilla/mailnews/news/public/nsINntpIncomingServer.idl
--- mailnews.old/news/public/nsINntpIncomingServer.idl	2004-04-18
02:33:14.000000000 +0800
+++ mozilla/mailnews/news/public/nsINntpIncomingServer.idl	2005-04-13
13:22:54.000000000 +0800
@@ -61,16 +61,19 @@ interface nsINntpIncomingServer : nsISup
	attribute boolean markOldRead;

	/* abbreviate the newsgroup names in the folder pane? */
	attribute boolean abbreviate;

     /* do we use a single login per server or do we login per group */
     attribute boolean singleSignon;

+    /*  the server charset and it may be needed to display newsgroup folder
names correctly */
+    attribute ACString charset;
+
	/* the server keeps track of all the newsgroups we are subscribed to */
	void addNewsgroup(in string name);
	void removeNewsgroup(in string name);

     void writeNewsrcFile();

     attribute boolean newsrcHasChanged;

diff -U 8 -p -r mailnews.old/news/src/nsNewsDownloadDialogArgs.cpp
mozilla/mailnews/news/src/nsNewsDownloadDialogArgs.cpp
--- mailnews.old/news/src/nsNewsDownloadDialogArgs.cpp	2004-04-18
02:33:16.000000000 +0800
+++ mozilla/mailnews/news/src/nsNewsDownloadDialogArgs.cpp	2005-04-13
13:24:48.000000000 +0800
@@ -37,45 +37,41 @@
  * ***** END LICENSE BLOCK ***** */

 #include "nsNewsDownloadDialogArgs.h"
 #include "nsCRT.h"
 #include "nsReadableUtils.h"

 nsNewsDownloadDialogArgs::nsNewsDownloadDialogArgs()
 {
-    mGroupName = "";
     mArticleCount = 0;
     mServerKey = "";
     mHitOK = PR_FALSE;
     mDownloadAll = PR_FALSE;
 }

 nsNewsDownloadDialogArgs::~nsNewsDownloadDialogArgs()
 {
 }

 NS_IMPL_ISUPPORTS1(nsNewsDownloadDialogArgs, nsINewsDownloadDialogArgs)

-NS_IMETHODIMP nsNewsDownloadDialogArgs::GetGroupName(char * *aGroupName)
-{
-    NS_ENSURE_ARG_POINTER(aGroupName);
-
-    *aGroupName = ToNewCString(mGroupName);
-
-    return NS_OK;
-}
-NS_IMETHODIMP nsNewsDownloadDialogArgs::SetGroupName(const char * aGroupName)
-{
-    NS_ENSURE_ARG_POINTER(aGroupName);
-
-    mGroupName = aGroupName;
-
-    return NS_OK;
-}
+NS_IMETHODIMP nsNewsDownloadDialogArgs::GetGroupName(nsAString & aGroupName)
+ {
+    aGroupName = mGroupName;
+ 
+     return NS_OK;
+ }
+NS_IMETHODIMP nsNewsDownloadDialogArgs::SetGroupName(const nsAString &
aGroupName)
+ {
+ 
+     mGroupName = aGroupName;
+ 
+     return NS_OK;
+ }
 NS_IMETHODIMP nsNewsDownloadDialogArgs::GetArticleCount(PRInt32
*aArticleCount)
 {
     NS_ENSURE_ARG_POINTER(aArticleCount);

     *aArticleCount = mArticleCount;

     return NS_OK;
 }
diff -U 8 -p -r mailnews.old/news/src/nsNewsDownloadDialogArgs.h
mozilla/mailnews/news/src/nsNewsDownloadDialogArgs.h
--- mailnews.old/news/src/nsNewsDownloadDialogArgs.h	2004-04-18
02:33:16.000000000 +0800
+++ mozilla/mailnews/news/src/nsNewsDownloadDialogArgs.h	2005-04-13
13:25:12.000000000 +0800
@@ -47,16 +47,16 @@ class nsNewsDownloadDialogArgs : public 
 public:
   nsNewsDownloadDialogArgs();
   virtual ~nsNewsDownloadDialogArgs();

   NS_DECL_ISUPPORTS
   NS_DECL_NSINEWSDOWNLOADDIALOGARGS

 private:
-  nsCString mGroupName;
+  nsString mGroupName;
   PRInt32 mArticleCount;
   nsCString mServerKey;
   PRBool mHitOK;
   PRBool mDownloadAll;
 };

 #endif // nsNewsDownloadDialogArgs_h__
diff -U 8 -p -r mailnews.old/news/src/nsNewsFolder.cpp
mozilla/mailnews/news/src/nsNewsFolder.cpp
--- mailnews.old/news/src/nsNewsFolder.cpp	2005-02-15 08:13:35.000000000
+0800
+++ mozilla/mailnews/news/src/nsNewsFolder.cpp	2005-04-19 08:33:14.976769560
+0800
@@ -88,16 +88,18 @@

 #include "nsIInterfaceRequestor.h"
 #include "nsIInterfaceRequestorUtils.h"

 #include "nsReadableUtils.h"
 #include "nsNewsDownloader.h"
 #include "nsIStringBundle.h"
 #include "nsEscape.h"
+#include "nsMsgI18N.h"
+#include "nsNativeCharsetUtils.h"

 // we need this because of an egcs 1.0 (and possibly gcc) compiler bug
 // that doesn't allow you to call ::nsISupports::GetIID() inside of a class
 // that multiply inherits from nsISupports
 static NS_DEFINE_CID(kRDFServiceCID, NS_RDFSERVICE_CID);
 static NS_DEFINE_CID(kCNewsDB, NS_NEWSDB_CID);

 // ###tw  This really ought to be the most
@@ -112,33 +114,33 @@ static NS_DEFINE_CID(kCNewsDB, NS_NEWSDB



///////////////////////////////////////////////////////////////////////////////
/

 nsMsgNewsFolder::nsMsgNewsFolder(void) : nsMsgLineBuffer(nsnull, PR_FALSE),
      mExpungedBytes(0), mGettingNews(PR_FALSE),
     mInitialized(PR_FALSE), mOptionLines(""), mUnsubscribedNewsgroupLines(""), 
     m_downloadMessageForOfflineUse(PR_FALSE),
m_downloadingMultipleMessages(PR_FALSE), 
-    mReadSet(nsnull), mGroupUsername(nsnull), mGroupPassword(nsnull),
mAsciiName(nsnull)
+    mReadSet(nsnull), mGroupUsername(nsnull), mGroupPassword(nsnull),
mRawName(nsnull)
 {
   MOZ_COUNT_CTOR(nsNewsFolder); // double count these for now.
   /* we're parsing the newsrc file, and the line breaks are platform specific.
    * if MSG_LINEBREAK != CRLF, then we aren't looking for CRLF 
    */
   if (PL_strcmp(MSG_LINEBREAK, CRLF))
     SetLookingForCRLF(PR_FALSE);
 }

 nsMsgNewsFolder::~nsMsgNewsFolder(void)
 {
   MOZ_COUNT_DTOR(nsNewsFolder);
   delete mReadSet;
   PR_Free(mGroupUsername);
   PR_Free(mGroupPassword);
-  PR_Free(mAsciiName);
+  PR_Free(mRawName);
 }

 NS_IMPL_ADDREF_INHERITED(nsMsgNewsFolder, nsMsgDBFolder)
 NS_IMPL_RELEASE_INHERITED(nsMsgNewsFolder, nsMsgDBFolder)

 NS_IMETHODIMP nsMsgNewsFolder::QueryInterface(REFNSIID aIID, void**
aInstancePtr)
 {
   if (!aInstancePtr) return NS_ERROR_NULL_POINTER;
@@ -205,20 +207,24 @@ nsMsgNewsFolder::AddNewsgroup(const char
   nsCOMPtr <nsINntpIncomingServer> nntpServer;
   rv = GetNntpServer(getter_AddRefs(nntpServer));
   if (NS_FAILED(rv)) return rv;

   nsCAutoString uri(mURI);
   uri.Append('/');
   // URI should use UTF-8
   // (see RFC2396 Uniform Resource Identifiers (URI): Generic Syntax)
-  // since we are forcing it to be latin-1 (IS0-8859-1)
-  // we can just assign with conversion
+  nsCAutoString dataCharset;
+  rv = nntpServer->GetCharset(dataCharset);
+  if (NS_FAILED(rv)) return rv;
   nsAutoString newsgroupName;
-  newsgroupName.AssignWithConversion(name);
+  rv =
nsMsgI18NConvertToUnicode(dataCharset.get(),nsDependentCString(name),newsgroupN
ame);
+  if (NS_FAILED(rv)) {
+    CopyASCIItoUTF16(name, newsgroupName);
+  }

   nsCAutoString escapedName;
   rv = NS_MsgEscapeEncodeURLPath(newsgroupName, escapedName);
   if (NS_FAILED(rv)) return rv;

   rv = nntpServer->AddNewsgroup(escapedName.get());
   if (NS_FAILED(rv)) return rv;

@@ -511,32 +517,35 @@ nsMsgNewsFolder::GetMessages(nsIMsgWindo
 }

 NS_IMETHODIMP nsMsgNewsFolder::GetFolderURL(char **url)
 {
   NS_ENSURE_ARG_POINTER(url);

   nsXPIDLCString hostName;
   nsresult rv = GetHostname(getter_Copies(hostName));
-  nsXPIDLCString groupName;
-  rv = GetAsciiName(getter_Copies(groupName));
+  nsXPIDLString groupName;
+  rv = GetName(getter_Copies(groupName));
   NS_ENSURE_SUCCESS(rv, rv);

   nsCOMPtr<nsIMsgIncomingServer> server;
   rv = GetServer(getter_AddRefs(server));
   NS_ENSURE_SUCCESS(rv, rv);

   PRInt32 port;
   PRBool isSecure = PR_FALSE;
   rv = server->GetIsSecure(&isSecure);
   NS_ENSURE_SUCCESS(rv, rv);
   rv = server->GetPort(&port);
   NS_ENSURE_SUCCESS(rv, rv);
   const char *newsScheme = (isSecure) ? SNEWS_SCHEME : NEWS_SCHEME;
-  *url = PR_smprintf("%s//%s:%ld/%s", newsScheme, hostName.get(), port,
groupName.get());
+  nsXPIDLCString escapedName;
+  rv = NS_MsgEscapeEncodeURLPath(groupName, escapedName);
+  NS_ENSURE_SUCCESS(rv, rv);
+  *url = PR_smprintf("%s//%s:%ld/%s", newsScheme, hostName.get(), port,
escapedName.get());
   return NS_OK;

 }

 NS_IMETHODIMP nsMsgNewsFolder::SetNewsrcHasChanged(PRBool newsrcHasChanged)
 {
     nsresult rv;

@@ -549,18 +558,28 @@ NS_IMETHODIMP nsMsgNewsFolder::SetNewsrc

 NS_IMETHODIMP nsMsgNewsFolder::CreateSubfolder(const PRUnichar
*uninewsgroupname, nsIMsgWindow *msgWindow)
 {
   nsresult rv = NS_OK;

   NS_ENSURE_ARG_POINTER(uninewsgroupname);
   if (!*uninewsgroupname) return NS_ERROR_FAILURE;

-  nsCAutoString newsgroupname; 
-  newsgroupname.AssignWithConversion(uninewsgroupname);
+  nsCAutoString newsgroupname;
+  nsCOMPtr <nsINntpIncomingServer> nntpServer;
+  rv = GetNntpServer(getter_AddRefs(nntpServer));
+  if (NS_FAILED(rv)) return rv;
+  nsCAutoString dataCharset;
+  rv = nntpServer->GetCharset(dataCharset);
+  if (NS_FAILED(rv)) return rv;
+
+  rv = nsMsgI18NConvertFromUnicode(dataCharset.get(),
nsDependentString(uninewsgroupname), newsgroupname);
+  if (NS_FAILED(rv)) {
+    LossyCopyUTF16toASCII(uninewsgroupname,newsgroupname);
+  }

   nsFileSpec path;
   nsCOMPtr<nsIFileSpec> pathSpec;
   rv = GetPath(getter_AddRefs(pathSpec));
   if (NS_FAILED(rv)) return rv;

   rv = pathSpec->GetFileSpec(&path);

@@ -569,29 +588,32 @@ NS_IMETHODIMP nsMsgNewsFolder::CreateSub
   // Create an empty database for this mail folder, set its name from the user 
   nsCOMPtr<nsIMsgDatabase> newsDBFactory;
   nsCOMPtr <nsIMsgDatabase> newsDB;

   //Now we have a valid directory or we have returned.
   //Make sure the new folder name is valid

   // remember, some file systems (like mac) can't handle long file names
-  nsCAutoString hashedName = newsgroupname;
+  nsCAutoString fileName;
+  nsAutoString hashedName(uninewsgroupname);
   rv = NS_MsgHashIfNecessary(hashedName);
-  path += hashedName.get();
+  NS_CopyUnicodeToNative(hashedName, fileName);
+  path += fileName.get();

   //Now let's create the actual new folder
   rv = AddNewsgroup(newsgroupname.get(), "", getter_AddRefs(child));

   if (NS_SUCCEEDED(rv))
     SetNewsrcHasChanged(PR_TRUE); // subscribe UI does this - but maybe we got
here through auto-subscribe

-  if(NS_SUCCEEDED(rv) && child)
+  if(NS_SUCCEEDED(rv) && child){
+    child->SetCharset(dataCharset.get());
     NotifyItemAdded(child);
-
+  }
   return rv;
 }

 NS_IMETHODIMP nsMsgNewsFolder::Delete()
 {
   nsresult rv = GetDatabase(nsnull);

   if(NS_SUCCEEDED(rv)) 
@@ -1124,27 +1146,42 @@ nsMsgNewsFolder::HandleNewsrcLine(char* 
   //
   // So lines like this in a newsrc file should be ignored:
   // 3746EF3F.6080309@netscape.com:
   // 3746EF3F.6080309%40netscape.com:
   if (PL_strstr(line,"@") || PL_strstr(line,"%40")) 
     // skipping, it contains @ or %40
     subscribed = PR_FALSE;

+    nsCOMPtr<nsINntpIncomingServer> nntpServer;
+    rv = GetNntpServer(getter_AddRefs(nntpServer));
+    if (NS_FAILED(rv)) return rv;
+    //now we convert from UTF8 to the server charset encoding because the
newsrc file is always saved in UTF8
+    nsAutoString unicodeLine;
+    nsCAutoString rawLine;
+    CopyUTF8toUTF16(line,unicodeLine);
+
+    nsCAutoString dataCharset;
+    rv = nntpServer->GetCharset(dataCharset);
+    NS_ASSERTION(NS_SUCCEEDED(rv),"failed get sever charset");
+    rv = nsMsgI18NConvertFromUnicode(dataCharset.get(), unicodeLine, rawLine);
+    if (NS_FAILED(rv))
+	LossyCopyUTF16toASCII(unicodeLine,rawLine);
+
   if (subscribed) 
   {
     // we're subscribed, so add it
     nsCOMPtr <nsIMsgFolder> child;

-    rv = AddNewsgroup(line, setStr, getter_AddRefs(child));
+    rv = AddNewsgroup(rawLine.get(), setStr, getter_AddRefs(child));

     if (NS_FAILED(rv)) return -1;
   }
   else {
-    rv = RememberUnsubscribedGroup(line, setStr);
+    rv = RememberUnsubscribedGroup(rawLine.get(), setStr);
     if (NS_FAILED(rv)) return -1;
   }

   return 0;
 }


 nsresult
@@ -1503,20 +1540,22 @@ nsresult nsMsgNewsFolder::CreateBaseMess

 NS_IMETHODIMP
 nsMsgNewsFolder::GetNewsrcLine(char **newsrcLine)
 {
   nsresult rv;

   if (!newsrcLine) return NS_ERROR_NULL_POINTER;

+  nsXPIDLString uninewsgroupname;
   nsXPIDLCString newsgroupname;
-  rv = GetAsciiName(getter_Copies(newsgroupname));
+  rv = GetName(getter_Copies(uninewsgroupname));
   if (NS_FAILED(rv)) return rv;
-  
+  CopyUTF16toUTF8(uninewsgroupname, newsgroupname);
+
   nsCAutoString newsrcLineStr;
   newsrcLineStr = newsgroupname;
   newsrcLineStr += ':';

   if (mReadSet) {
     nsXPIDLCString setStr;
     mReadSet->Output(getter_Copies(setStr));
     if (NS_SUCCEEDED(rv)) 
@@ -1575,38 +1614,62 @@ nsMsgNewsFolder::GetOptionLines(char **o
 }

 NS_IMETHODIMP
 nsMsgNewsFolder::OnReadChanged(nsIDBChangeListener * aInstigator)
 {
     return SetNewsrcHasChanged(PR_TRUE);
 }

+NS_IMETHODIMP
+nsMsgNewsFolder::GetUnicodeName(nsAString & name)
+{
+  nsresult rv;
+  if(mUnicodeName.IsEmpty()){
+    nsXPIDLString newsgroupName;
+    rv = GetName(getter_Copies(newsgroupName));
+    NS_ENSURE_SUCCESS(rv,rv);
+    mUnicodeName = newsgroupName;
+  }
+  name = mUnicodeName;
+  return NS_OK;
+}

 NS_IMETHODIMP
-nsMsgNewsFolder::GetAsciiName(char **asciiName)
+nsMsgNewsFolder::GetRawName(char **rawName)
 {
   nsresult rv;
-  NS_ENSURE_ARG_POINTER(asciiName);
-  if (!mAsciiName) 
+  NS_ENSURE_ARG_POINTER(rawName);
+  if (!mRawName) 
   {
     nsXPIDLString name;
     rv = GetName(getter_Copies(name));
     NS_ENSURE_SUCCESS(rv,rv);

     // convert to ASCII
     nsCAutoString tmpStr;
-    tmpStr.AssignWithConversion(name);
+    nsCOMPtr <nsINntpIncomingServer> nntpServer;
+    rv = GetNntpServer(getter_AddRefs(nntpServer));
+    NS_ENSURE_SUCCESS(rv,rv);
+    
+    nsCAutoString dataCharset;
+    rv = nntpServer->GetCharset(dataCharset);
+    NS_ENSURE_SUCCESS(rv,rv);
+    rv = nsMsgI18NConvertFromUnicode(dataCharset.get(), name, tmpStr);
+
+    if (NS_FAILED(rv)) {
+      LossyCopyUTF16toASCII(name,tmpStr);
+    }

-    mAsciiName = nsCRT::strdup(tmpStr.get());
-    if (!mAsciiName) return NS_ERROR_OUT_OF_MEMORY;
+    mRawName = nsCRT::strdup(tmpStr.get());
+    if (!mRawName) return NS_ERROR_OUT_OF_MEMORY;
   }

-  *asciiName = nsCRT::strdup(mAsciiName);
-  if (!*asciiName) return NS_ERROR_OUT_OF_MEMORY;
+  *rawName = nsCRT::strdup(mRawName);
+  if (!*rawName) return NS_ERROR_OUT_OF_MEMORY;

   return NS_OK;
 }

 NS_IMETHODIMP
 nsMsgNewsFolder::GetNntpServer(nsINntpIncomingServer **result)
 {
     nsresult rv;
Only in mozilla/mailnews/news/src: nsNewsFolder.cpp~
diff -U 8 -p -r mailnews.old/news/src/nsNewsFolder.h
mozilla/mailnews/news/src/nsNewsFolder.h
--- mailnews.old/news/src/nsNewsFolder.h	2004-04-18 02:33:16.000000000
+0800
+++ mozilla/mailnews/news/src/nsNewsFolder.h	2005-04-19 08:33:47.599810104
+0800
@@ -154,18 +154,18 @@ protected:
   nsMsgKeySet *mReadSet; 

   nsCOMPtr<nsIFileSpec> mNewsrcFilePath; 

   // used for auth news
   char 		*mGroupUsername;
   char 	*mGroupPassword;

-  // the name of the newsgroup.
-  char    *mAsciiName;
+  char    *mRawName;
+  nsString mUnicodeName;
   PRInt32 mSortOrder;

 private:
     nsresult CreateNewsgroupUsernameUrlForSignon(const char *inUriStr, char
**result);
     nsresult CreateNewsgroupPasswordUrlForSignon(const char *inUriStr, char
**result);
     nsresult CreateNewsgroupUrlForSignon(const char *inUriStr, const char
*ref, char **result);

     nsCOMPtr <nsIFileSpec> mFilterFile;
Only in mozilla/mailnews/news/src: nsNewsFolder.h~
diff -U 8 -p -r mailnews.old/news/src/nsNntpIncomingServer.cpp
mozilla/mailnews/news/src/nsNntpIncomingServer.cpp
--- mailnews.old/news/src/nsNntpIncomingServer.cpp	2004-08-27
06:16:52.000000000 +0800
+++ mozilla/mailnews/news/src/nsNntpIncomingServer.cpp	2005-04-19
08:33:31.246296216 +0800
@@ -57,22 +57,25 @@
 #include "nsMsgUtils.h"
 #include "nsIPrompt.h"
 #include "nsIStringBundle.h"
 #include "nntpCore.h"
 #include "nsIWindowWatcher.h"
 #include "nsITreeColumns.h"
 #include "nsIDOMElement.h"
 #include "nsMsgFolderFlags.h"
+#include "nsMsgI18N.h"
+#include "nsIPrefLocalizedString.h"

 #define INVALID_VERSION	 0
 #define VALID_VERSION		 1
 #define NEW_NEWS_DIR_NAME	 "News"
 #define PREF_MAIL_NEWSRC_ROOT	 "mail.newsrc_root"
 #define PREF_MAIL_NEWSRC_ROOT_REL "mail.newsrc_root-rel"
+#define PREF_MAILNEWS_VIEW_DEFAULT_CHARSET "mailnews.view_default_charset"
 #define HOSTINFO_FILE_NAME	 "hostinfo.dat"

 #define NEWS_DELIMITER 	 '.'

 // this platform specific junk is so the newsrc filenames we create 
 // will resemble the migrated newsrc filenames.
 #if defined(XP_UNIX) || defined(XP_BEOS)
 #define NEWSRC_FILE_PREFIX "newsrc-"
@@ -304,16 +307,70 @@ nsresult nsNntpIncomingServer::SetupNews
     mNewsrcSaveTimer = do_CreateInstance("@mozilla.org/timer;1");
	mNewsrcSaveTimer->InitWithFuncCallback(OnNewsrcSaveTimer, (void*)this,
timeInMSUint32, 
					    nsITimer::TYPE_REPEATING_SLACK);

     return NS_OK;
 }

 NS_IMETHODIMP
+nsNntpIncomingServer::SetCharset(const nsACString & aCharset)
+{
+	nsresult rv;
+	rv = SetCharValue("charset", PromiseFlatCString(aCharset).get());
+	return rv;
+}
+
+NS_IMETHODIMP
+nsNntpIncomingServer::GetCharset(nsACString & aCharset)
+{
+	nsresult rv; 
+	nsXPIDLCString serverCharset;
+	//first we get the per-server settings mail.server.<serverkey>.charset
+	rv = GetCharValue("charset",getter_Copies(serverCharset));
+
+	//if the per-server setting is empty,we get the default charset from 
+	//mailnews.view_default_charset setting and set it as per-server
preference.
+	if(serverCharset.IsEmpty()){
+		nsXPIDLCString defaultCharset;
+		nsCOMPtr<nsIPrefBranch>
prefBranch(do_GetService(NS_PREFSERVICE_CONTRACTID, &rv));
+
+		if (NS_SUCCEEDED(rv)){
+			nsCOMPtr<nsIPrefLocalizedString> pls;
+			rv =
prefBranch->GetComplexValue(PREF_MAILNEWS_VIEW_DEFAULT_CHARSET,
+		       NS_GET_IID(nsIPrefLocalizedString),
getter_AddRefs(pls));
+			if (NS_SUCCEEDED(rv)) 
+			{
+				nsXPIDLString ucsval;
+				pls->ToString(getter_Copies(ucsval));
+				if (ucsval)
+				{
+					LossyCopyUTF16toASCII(ucsval,
defaultCharset);
+				}
+			}
+		}
+
+		if(defaultCharset.IsEmpty()){
+			serverCharset = "ISO-8859-1";
+		}else{
+			serverCharset = defaultCharset;
+		}
+
+#ifdef DEBUG_NEWS
+		printf("default charset for the server is %s\n", (const char
*)defaultCharset);
+#endif
+
+		SetCharset(serverCharset);
+	}
+
+	aCharset = serverCharset;
+	return NS_OK;
+}
+
+NS_IMETHODIMP
 nsNntpIncomingServer::WriteNewsrcFile()
 {
     nsresult rv;

     PRBool newsrcHasChanged;
     rv = GetNewsrcHasChanged(&newsrcHasChanged);
     if (NS_FAILED(rv)) return rv;

@@ -806,17 +863,17 @@ writeGroupToHostInfoFile(nsCString &aEle
     nsAutoString name;
     nsresult rv = NS_MsgDecodeUnescapeURLPath(aElement, name); 
     if (NS_FAILED(rv)) {
	 // stop, something is bad.
	 return PR_FALSE;
     }

     nsCAutoString nameOnDisk;
-    nameOnDisk.AssignWithConversion(name.get());
+    CopyUTF16toUTF8(name, nameOnDisk);

     // XXX todo ",,1,0,0" is a temporary hack, fix it
     *stream << nameOnDisk.get() << ",,1,0,0" << MSG_LINEBREAK;
     return PR_TRUE;
 }

 nsresult
 nsNntpIncomingServer::WriteHostInfoFile()
@@ -1128,19 +1185,41 @@ nsNntpIncomingServer::UpdateSubscribed()

 NS_IMETHODIMP
 nsNntpIncomingServer::AddTo(const char *aName, PRBool addAsSubscribed, PRBool
aSubscribable, PRBool changeIfExists)
 {
     nsresult rv = EnsureInner();
     NS_ENSURE_SUCCESS(rv,rv);

     nsAutoString newsgroupName;
-    newsgroupName.AssignWithConversion(aName);
+    char *escapedName = nsnull;
+
+    nsCOMPtr<nsIMsgFolder> rootFolder;
+    rv = GetRootFolder(getter_AddRefs(rootFolder));
+    NS_ENSURE_SUCCESS(rv,rv);
+    if (!rootFolder) return NS_ERROR_FAILURE;
+    nsCOMPtr <nsIMsgNewsFolder> newsFolder = do_QueryInterface(rootFolder,
&rv);
+    NS_ENSURE_SUCCESS(rv,rv);
+    if (!newsFolder) return NS_ERROR_FAILURE;
+    nsCOMPtr <nsINntpIncomingServer> nntpServer;
+    rv = newsFolder->GetNntpServer(getter_AddRefs(nntpServer));
+    NS_ENSURE_SUCCESS(rv,rv);
+    nsCAutoString dataCharset;
+    rv = nntpServer->GetCharset(dataCharset);
+    NS_ENSURE_SUCCESS(rv,rv);
+    rv =
nsMsgI18NConvertToUnicode(dataCharset.get(),nsDependentCString(aName),newsgroup
Name);
+    if (NS_SUCCEEDED(rv)) {
+	 escapedName = nsEscape(NS_ConvertUTF16toUTF8(newsgroupName).get(),
url_Path);
+     }
+
+    if (NS_FAILED(rv)) {
+      CopyASCIItoUTF16(aName, newsgroupName);
+      escapedName = nsEscape(aName, url_Path);
+    }

-    char *escapedName =
nsEscape(NS_ConvertUCS2toUTF8(newsgroupName.get()).get(), url_Path);
     if (!escapedName) return NS_ERROR_OUT_OF_MEMORY;

     rv = AddGroupOnServer(escapedName);
     NS_ENSURE_SUCCESS(rv,rv);

     rv = mInner->AddTo(escapedName,addAsSubscribed, aSubscribable,
changeIfExists);
     NS_ENSURE_SUCCESS(rv,rv);

@@ -1245,17 +1324,27 @@ nsNntpIncomingServer::HandleLine(char* l
	if (line[0] == '#' || line[0] == '\0') return 0;

	line[line_size] = 0;

	if (mHasSeenBeginGroups) {
		char *commaPos = PL_strchr(line,',');
		if (commaPos) *commaPos = 0;

-		nsresult rv = AddTo(line, PR_FALSE, PR_TRUE, PR_TRUE);
+		//now we convert from UTF8 to the server charset encoding
because the hostInfo.dat is always saved in UTF8
+	 nsAutoString unicodeLine;
+	 nsCAutoString encodedLine;
+	 CopyUTF8toUTF16(line,unicodeLine);
+	 nsCAutoString dataCharset;
+	 nsresult rv = GetCharset(dataCharset);
+	 NS_ASSERTION(NS_SUCCEEDED(rv),"failed to get sever charset");
+	 rv = nsMsgI18NConvertFromUnicode(dataCharset.get(), unicodeLine,
encodedLine);
+	 if (NS_FAILED(rv))
+	   LossyCopyUTF16toASCII(unicodeLine,encodedLine);
+		rv = AddTo(encodedLine.get(), PR_FALSE, PR_TRUE, PR_TRUE);
		NS_ASSERTION(NS_SUCCEEDED(rv),"failed to add line");
		if (NS_SUCCEEDED(rv)) {
	   // since we've seen one group, we can claim we've loaded the
hostinfo file
	   mHostInfoLoaded = PR_TRUE;
		}
	}
	else {
		if (nsCRT::strncmp(line,"begingroups", 11) == 0) {
Only in mozilla/mailnews/news/src: nsNntpIncomingServer.cpp~
diff -U 8 -p -r mailnews.old/news/src/nsNNTPNewsgroupList.cpp
mozilla/mailnews/news/src/nsNNTPNewsgroupList.cpp
--- mailnews.old/news/src/nsNNTPNewsgroupList.cpp	2005-02-01
05:53:40.000000000 +0800
+++ mozilla/mailnews/news/src/nsNNTPNewsgroupList.cpp	2005-04-13
13:55:22.000000000 +0800
@@ -319,21 +319,21 @@ nsNNTPNewsgroupList::GetRangeOfArtsToDow

		 nsCOMPtr<nsINewsDownloadDialogArgs> args =
do_CreateInstance("@mozilla.org/messenger/newsdownloaddialogargs;1", &rv);
		 if (NS_FAILED(rv)) return rv;
		 NS_ENSURE_SUCCESS(rv,rv);

		 rv = args->SetArticleCount(*last - *first + 1);
		 NS_ENSURE_SUCCESS(rv,rv);

-		 nsXPIDLCString groupName;
-		 rv = m_newsFolder->GetAsciiName(getter_Copies(groupName));
+		 nsXPIDLString groupName;
+		 rv = m_newsFolder->GetUnicodeName(groupName);
		 NS_ENSURE_SUCCESS(rv,rv);

-		 rv = args->SetGroupName((const char *)groupName);
+		 rv = args->SetGroupName(groupName);
		 NS_ENSURE_SUCCESS(rv,rv);

				// get the server key
				nsXPIDLCString serverKey;
				rv = server->GetKey(getter_Copies(serverKey));
		 NS_ENSURE_SUCCESS(rv,rv);

		 rv = args->SetServerKey((const char *)serverKey);
Only in mozilla/mailnews/news/src: nsNNTPNewsgroupList.cpp~
diff -U 8 -p -r mailnews.old/news/src/nsNNTPProtocol.cpp
mozilla/mailnews/news/src/nsNNTPProtocol.cpp
--- mailnews.old/news/src/nsNNTPProtocol.cpp	2005-02-16 11:18:46.000000000
+0800
+++ mozilla/mailnews/news/src/nsNNTPProtocol.cpp	2005-04-19
03:36:57.000000000 +0800
@@ -1934,17 +1934,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
	char *command=0;
	PRInt32 status = 0;

	if (m_typeWanted == ARTICLE_WANTED) {
		if (m_key != nsMsgKey_None) {
		    nsresult rv;
	     nsXPIDLCString newsgroupName;
	     if (m_newsFolder) {
-		 rv = m_newsFolder->GetAsciiName(getter_Copies(newsgroupName));
+		 rv = m_newsFolder->GetRawName(getter_Copies(newsgroupName));
		 NS_ENSURE_SUCCESS(rv,rv);
	     }

		    PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) current group = %s,
desired group = %s", this, m_currentGroup.get(), newsgroupName.get()));
	     // if the current group is the desired group, we can just issue
the ARTICLE command
	     // if not, we have to do a GROUP first
			if (!PL_strcmp(m_currentGroup.get(),
newsgroupName.get()))
			  m_nextState = NNTP_SEND_ARTICLE_NUMBER;
@@ -2045,17 +2045,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
	else if(m_typeWanted == GROUP_WANTED) 
     {
	 nsresult rv = NS_ERROR_NULL_POINTER;

	 NS_ASSERTION(m_newsFolder, "m_newsFolder is null, panic!");
	 if (!m_newsFolder) return -1;

	 nsXPIDLCString group_name;
-		rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+		rv = m_newsFolder->GetRawName(getter_Copies(group_name));
	 NS_ASSERTION(NS_SUCCEEDED(rv),"failed to get newsgroup name");
	 if (NS_FAILED(rv)) return -1;

	 m_firstArticle = 0;
	 m_lastArticle = 0;

	 NS_MsgSACopy(&command, "GROUP ");
	 NS_MsgSACat(&command, (const char *)group_name);
@@ -2090,17 +2090,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
	     PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) doing GROUP for XPAT", this));
	     nsXPIDLCString group_name;

			/* for XPAT, we have to GROUP into the group before
searching */
			if (!m_newsFolder) {
				NNTP_LOG_NOTE("m_newsFolder is null, panic!");
				return -1;
			}
-	     rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+	     rv = m_newsFolder->GetRawName(getter_Copies(group_name));
	     if (NS_FAILED(rv)) return -1;

			NS_MsgSACopy(&command, "GROUP ");
	     NS_MsgSACat (&command, group_name);

	     // force a GROUP next time
	     m_currentGroup.Truncate();
			m_nextState = NNTP_RESPONSE;
@@ -2201,17 +2201,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
   }
   else
   {
     nsresult rv = NS_OK;

     nsXPIDLCString group_name;
     NS_ASSERTION(m_newsFolder, "no newsFolder");
     if (m_newsFolder) {
-      rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+      rv = m_newsFolder->GetRawName(getter_Copies(group_name));
     }

     if (m_responseCode == MK_NNTP_RESPONSE_GROUP_NO_GROUP &&
       m_typeWanted == GROUP_WANTED) {
       PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) group (%s) not found, so unset
m_currentGroup",this,(const char *)group_name));
       m_currentGroup.Truncate();

       m_nntpServer->GroupNotFound(m_msgWindow, group_name.get(), PR_TRUE /*
opening */);
@@ -2316,17 +2316,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
 }

 PRInt32 nsNNTPProtocol::SendGroupForArticle()
 {
   nsresult rv;
   PRInt32 status = 0; 

   nsXPIDLCString groupname;
-  rv = m_newsFolder->GetAsciiName(getter_Copies(groupname));
+  rv = m_newsFolder->GetRawName(getter_Copies(groupname));
   NS_ASSERTION(NS_SUCCEEDED(rv) && groupname.get() && groupname.get()[0], "no
group name");

   char outputBuffer[OUTPUT_BUFFER_SIZE];

   PR_snprintf(outputBuffer, 
			OUTPUT_BUFFER_SIZE, 
			"GROUP %.512s" CRLF, 
			(const char *)groupname);
@@ -2348,17 +2348,17 @@ nsNNTPProtocol::SetCurrentGroup()
   nsresult rv;
   nsXPIDLCString groupname;
   NS_ASSERTION(m_newsFolder, "no news folder");
   if (!m_newsFolder) {
     m_currentGroup.Truncate();
     return NS_ERROR_UNEXPECTED;
   }

-  rv = m_newsFolder->GetAsciiName(getter_Copies(groupname));
+  rv = m_newsFolder->GetRawName(getter_Copies(groupname));
   NS_ASSERTION(NS_SUCCEEDED(rv) && groupname.get()[0], "no group name");
   PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) SetCurrentGroup to %s",this,(const char
*)groupname));
   m_currentGroup = groupname;
   return NS_OK;
 }

 PRInt32 nsNNTPProtocol::SendGroupForArticleResponse()
 {
@@ -3873,17 +3873,17 @@ PRInt32 nsNNTPProtocol::DisplayNewsRC()
     if (NS_FAILED(rv)) return -1;
     if (!currFolder) return -1;

     m_newsFolder = do_QueryInterface(currFolder, &rv);
     if (NS_FAILED(rv)) return -1;
     if (!m_newsFolder) return -1;

     nsXPIDLCString name;
-    rv = m_newsFolder->GetAsciiName(getter_Copies(name));
+    rv = m_newsFolder->GetRawName(getter_Copies(name));
     if (NS_FAILED(rv)) return -1;
     if (!name) return -1;

     /* send group command to server */
     char outputBuffer[OUTPUT_BUFFER_SIZE];

     PR_snprintf(outputBuffer, OUTPUT_BUFFER_SIZE, "GROUP %.512s" CRLF, (const
char *)name);
     nsCOMPtr<nsIMsgMailNewsUrl> mailnewsurl = do_QueryInterface(m_runningURL);
@@ -3963,17 +3963,17 @@ PRInt32 nsNNTPProtocol::DisplayNewsRCRes
     NS_ASSERTION(NS_SUCCEEDED(rv),"DisplaySubscribedGroup() failed");
     if (NS_FAILED(rv)) status = -1;

     if (status < 0) return status;
   }
   else if (m_responseCode == MK_NNTP_RESPONSE_GROUP_NO_GROUP)
   {
     nsXPIDLCString name;
-    rv = m_newsFolder->GetAsciiName(getter_Copies(name));
+    rv = m_newsFolder->GetRawName(getter_Copies(name));

     if (NS_SUCCEEDED(rv)) {
       m_nntpServer->GroupNotFound(m_msgWindow, name.get(), PR_FALSE);
     }

     PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) NO_GROUP, so unset m_currentGroup",
this));
     m_currentGroup.Truncate();
   }
@@ -4394,17 +4394,17 @@ PRInt32 nsNNTPProtocol::XPATResponse(nsI

 PRInt32 nsNNTPProtocol::ListPrettyNames()
 {

   nsXPIDLCString group_name;
   char outputBuffer[OUTPUT_BUFFER_SIZE];
   PRInt32 status = 0; 

-  nsresult rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+  nsresult rv = m_newsFolder->GetRawName(getter_Copies(group_name));
   PR_snprintf(outputBuffer, 
     OUTPUT_BUFFER_SIZE, 
     "LIST PRETTYNAMES %.512s" CRLF,
     NS_SUCCEEDED(rv) ? (const char *) group_name : "");

   nsCOMPtr<nsIMsgMailNewsUrl> mailnewsurl = do_QueryInterface(m_runningURL);
   if (mailnewsurl)
     status = SendData(mailnewsurl, outputBuffer);
@@ -4471,17 +4471,17 @@ PRInt32 nsNNTPProtocol::ListPrettyNamesR
   PR_FREEIF(line);
   return 0;
 }

 PRInt32 nsNNTPProtocol::ListXActive()
 { 
   nsXPIDLCString group_name;
   nsresult rv;
-  rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+  rv = m_newsFolder->GetRawName(getter_Copies(group_name));
   if (NS_FAILED(rv)) return -1;

   PRInt32 status = 0;
   char outputBuffer[OUTPUT_BUFFER_SIZE];

   PR_snprintf(outputBuffer, 
     OUTPUT_BUFFER_SIZE, 
     "LIST XACTIVE %.512s" CRLF,
@@ -4624,17 +4624,17 @@ PRInt32 nsNNTPProtocol::SendListGroup()
   nsresult rv;
   char outputBuffer[OUTPUT_BUFFER_SIZE];
   PRInt32 status = 0; 

   NS_ASSERTION(m_newsFolder,"no newsFolder");
   if (!m_newsFolder) return -1;
   nsXPIDLCString newsgroupName;

-  rv = m_newsFolder->GetAsciiName(getter_Copies(newsgroupName));
+  rv = m_newsFolder->GetRawName(getter_Copies(newsgroupName));
   NS_ENSURE_SUCCESS(rv,rv);

   PR_snprintf(outputBuffer, 
     OUTPUT_BUFFER_SIZE, 
     "listgroup %.512s" CRLF,
     newsgroupName.get());

   m_articleList = do_CreateInstance(NS_NNTPARTICLELIST_CONTRACTID, &rv);
Only in mozilla/mailnews/news/src: nsNNTPProtocol.cpp~
diff -U 8 -p -r mailnews.old/news/src/nsNntpService.cpp
mozilla/mailnews/news/src/nsNntpService.cpp
--- mailnews.old/news/src/nsNntpService.cpp	2005-01-18 02:50:13.000000000
+0800
+++ mozilla/mailnews/news/src/nsNntpService.cpp 2005-04-13 13:56:02.000000000
+0800
@@ -650,18 +650,21 @@ nsNntpService::GetFolderFromUri(const ch
   // but the *name* of the newsgroup (we are calling ::GetChildNamed())
   // is unescaped.  see http://bugzilla.mozilla.org/show_bug.cgi?id=210089#c17
   // for more about this
   char *unescapedPath = PL_strdup(path.get() + 1); /* skip the leading slash
*/
   if (!unescapedPath)
     return NS_ERROR_OUT_OF_MEMORY;
   nsUnescape(unescapedPath);

+  nsAutoString newsgroupName;
+  //the encoding is UTF-8 after unescaped.
+  CopyUTF8toUTF16(unescapedPath,newsgroupName);
   nsCOMPtr<nsISupports> subFolder;
-  rv = rootFolder->GetChildNamed(NS_ConvertASCIItoUCS2(unescapedPath).get() ,
getter_AddRefs(subFolder));
+  rv = rootFolder->GetChildNamed(newsgroupName.get() ,
getter_AddRefs(subFolder));
   PL_strfree(unescapedPath);
   NS_ENSURE_SUCCESS(rv,rv);

   return CallQueryInterface(subFolder, aFolder);
 }

 NS_IMETHODIMP
 nsNntpService::CopyMessage(const char * aSrcMessageURI, nsIStreamListener *
aMailboxCopyHandler, PRBool moveMessage,
Comment on attachment 170507 [details] [diff] [review]
patch use utf8 in the newsrc/hostinfo file

diff -U 8 -p -r mailnews.old/base/prefs/resources/content/am-server.xul
mozilla/mailnews/base/prefs/resources/content/am-server.xul
--- mailnews.old/base/prefs/resources/content/am-server.xul	2005-03-04
02:08:45.000000000 +0800
+++ mozilla/mailnews/base/prefs/resources/content/am-server.xul 2005-04-13
13:20:05.000000000 +0800
@@ -47,16 +47,20 @@

 <page xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
       onload="parent.onPanelLoaded('am-server.xul');">

   <script type="application/x-javascript"
src="chrome://messenger/content/AccountManager.js"/>
   <script type="application/x-javascript"
src="chrome://messenger/content/am-server.js"/>
   <script type="application/x-javascript"
src="chrome://messenger/content/am-prefs.js"/>
   <script type="application/x-javascript"
src="chrome://messenger/content/amUtils.js"/>
+  <script type="application/x-javascript"><![CDATA[
+	var observerService =
Components.classes["@mozilla.org/observer-service;1"].getService(Components.int
erfaces.nsIObserverService);
+	observerService.notifyObservers(null, "charsetmenu-selected", "other");
+  ]]></script>
   <stringbundle id="bundle_messenger"
src="chrome://messenger/locale/messenger.properties"/>

   <broadcaster id="broadcaster_doBiff"/>
   <broadcaster id="broadcaster_notify"/>

   <label hidden="true" wsm_persist="true" id="server.type"/>

   <dialogheader title="&serverSettings.label;"/>
@@ -325,9 +329,22 @@
	    accesskey="&localPath.accesskey;"/>
     <hbox align="center">
       <textbox readonly="true" wsm_persist="true" flex="1"
id="server.localPath" datatype="nsIFileSpec"
		prefstring="mail.server.%serverkey%.directory"
class="uri-element"/>
       <button id="browseForLocalFolder" label="&browseFolder.label;"
filepickertitle="&localFolderPicker.label;"
	       accesskey="&browseFolder.accesskey;"
oncommand="BrowseForLocalFolders()"/>
     </hbox>
   </vbox>
+  <separator class="thin"/>
+   <hbox hidefor="imap,pop3,movemail" align="center" valign="middle"
iscontrolcontainer="true">
+     <label value="&serverDefaultCharset.label;" control="nntp.charset"/>
+     <menulist hidable="true" hidefor="imap,pop3,movemail" wsm_persist="true"
id="nntp.charset" 
+		  ref="NC:DecodersRoot" datasources="rdf:charset-menu"
+		  preftype="string"
prefstring="mail.server.%serverkey%.charset"> 
+	<template>
+	  <menupopup>
+	    <menuitem label="rdf:http://home.netscape.com/NC-rdf#Name"
value="..." uri="..."/>
+	  </menupopup>
+	</template>
+     </menulist>
+   </hbox> 
 </page>
Only in mozilla/mailnews/base/prefs/resources/content: am-server.xul~
diff -U 8 -p -r
mailnews.old/base/prefs/resources/locale/en-US/am-server-top.dtd
mozilla/mailnews/base/prefs/resources/locale/en-US/am-server-top.dtd
--- mailnews.old/base/prefs/resources/locale/en-US/am-server-top.dtd   
2005-02-09 01:01:47.000000000 +0800
+++ mozilla/mailnews/base/prefs/resources/locale/en-US/am-server-top.dtd       
2005-04-13 13:19:30.000000000 +0800
@@ -79,15 +79,16 @@
 <!ENTITY newsrcFilePath.label "newsrc file:">
 <!ENTITY newsrcFilePath.accesskey "f">
 <!ENTITY localPath.label "Local directory:">
 <!ENTITY localPath.accesskey "L">
 <!ENTITY abbreviate.label "Show newsgroup names in the Mail Folder pane as:">
 <!ENTITY abbreviateOn.label "Full names (For example,
'netscape.public.mozilla.mail-news')">
 <!ENTITY abbreviateOff.label "Abbreviate names (For example,
'n.p.m.mail-news')">
 <!ENTITY advancedButton.label "Advanced...">
+<!ENTITY serverDefaultCharset.label "Default Character Encoding:">
 <!ENTITY advancedButton.accesskey "v">
 <!ENTITY localFolderPicker.label "Select Local Directory">
 <!ENTITY browseFolder.label "Browse...">
 <!ENTITY browseFolder.accesskey "B">
 <!ENTITY newsrcPicker.label "Select newsrc File">
 <!ENTITY browseNewsrc.label "Browse...">
 <!ENTITY browseNewsrc.accesskey "e">
diff -U 8 -p -r mailnews.old/news/public/nsIMsgNewsFolder.idl
mozilla/mailnews/news/public/nsIMsgNewsFolder.idl
--- mailnews.old/news/public/nsIMsgNewsFolder.idl	2004-04-18
02:33:14.000000000 +0800
+++ mozilla/mailnews/news/public/nsIMsgNewsFolder.idl	2005-04-19
03:37:37.000000000 +0800
@@ -36,27 +36,28 @@
  * ***** END LICENSE BLOCK ***** */

 #include "nsISupports.idl"
 #include "nsIMsgFolder.idl"

 interface nsIMsgWindow;
 interface nsINntpIncomingServer;

-[scriptable, uuid(3716abe4-f6d4-11d2-86d5-004005263078)]
+[scriptable, uuid(02699533-7ba8-4e78-94dc-a20374fb2711)]
 interface nsIMsgNewsFolder : nsISupports {
   attribute string groupUsername;
   attribute string groupPassword;

-  /** the ascii name for the newsgroup.  
-   *  newsgroup names are always ascii, and this can help simply
-   *  the code, to prevent use from having to get the name (as a wstring)
-   *  and then converting it back to ascii.
+  readonly attribute AString unicodeName;
+  /** the raw name name for the newsgroup (server side name).	
+   *  newsgroup names are might not always ascii(bug#126453), and this raw
name name 
+   *  can help simply the code, to prevent use from having to get the unicode
name 
+   *  (as a wstring) and then converting it back to server side raw name.
    */
-  readonly attribute string asciiName;
+  readonly attribute string rawName;
   readonly attribute nsINntpIncomingServer nntpServer;
   attribute boolean saveArticleOffline;

   string getGroupPasswordWithUI(in wstring aPromptString, in wstring
aPromptTitle, in nsIMsgWindow aMsgWindow);
   string getGroupUsernameWithUI(in wstring aPromptString, in wstring
aPromptTitle, in nsIMsgWindow aMsgWindow);

   void forgetGroupUsername();
   void forgetGroupPassword();
Only in mozilla/mailnews/news/public: nsIMsgNewsFolder.idl~
diff -U 8 -p -r mailnews.old/news/public/nsINewsDownloadDialogArgs.idl
mozilla/mailnews/news/public/nsINewsDownloadDialogArgs.idl
--- mailnews.old/news/public/nsINewsDownloadDialogArgs.idl	2004-04-18
02:33:14.000000000 +0800
+++ mozilla/mailnews/news/public/nsINewsDownloadDialogArgs.idl	2005-04-13
13:22:00.000000000 +0800
@@ -33,19 +33,19 @@
  * and other provisions required by the GPL or the LGPL. If you do not delete
  * the provisions above, a recipient may use your version of this file under
  * the terms of any one of the MPL, the GPL or the LGPL.
  *
  * ***** END LICENSE BLOCK ***** */

 #include "nsISupports.idl"

-[scriptable, uuid(b5037f06-1dd1-11b2-bf61-fcab5b5837a7)]
+[scriptable, uuid(3634327c-392b-4686-adf5-576e6cef9196)]
 interface nsINewsDownloadDialogArgs: nsISupports {
-  attribute string groupName;
+  attribute AString groupName;
   attribute long articleCount;
   attribute string serverKey;
   attribute boolean hitOK;
   attribute boolean downloadAll;
 };

 %{ C++
 #define DOWNLOAD_HEADERS_URL "chrome://messenger/content/downloadheaders.xul"
diff -U 8 -p -r mailnews.old/news/public/nsINntpIncomingServer.idl
mozilla/mailnews/news/public/nsINntpIncomingServer.idl
--- mailnews.old/news/public/nsINntpIncomingServer.idl	2004-04-18
02:33:14.000000000 +0800
+++ mozilla/mailnews/news/public/nsINntpIncomingServer.idl	2005-04-13
13:22:54.000000000 +0800
@@ -61,16 +61,19 @@ interface nsINntpIncomingServer : nsISup
	attribute boolean markOldRead;

	/* abbreviate the newsgroup names in the folder pane? */
	attribute boolean abbreviate;

     /* do we use a single login per server or do we login per group */
     attribute boolean singleSignon;

+    /*  the server charset and it may be needed to display newsgroup folder
names correctly */
+    attribute ACString charset;
+
	/* the server keeps track of all the newsgroups we are subscribed to */
	void addNewsgroup(in string name);
	void removeNewsgroup(in string name);

     void writeNewsrcFile();

     attribute boolean newsrcHasChanged;

diff -U 8 -p -r mailnews.old/news/src/nsNewsDownloadDialogArgs.cpp
mozilla/mailnews/news/src/nsNewsDownloadDialogArgs.cpp
--- mailnews.old/news/src/nsNewsDownloadDialogArgs.cpp	2004-04-18
02:33:16.000000000 +0800
+++ mozilla/mailnews/news/src/nsNewsDownloadDialogArgs.cpp	2005-04-13
13:24:48.000000000 +0800
@@ -37,45 +37,41 @@
  * ***** END LICENSE BLOCK ***** */

 #include "nsNewsDownloadDialogArgs.h"
 #include "nsCRT.h"
 #include "nsReadableUtils.h"

 nsNewsDownloadDialogArgs::nsNewsDownloadDialogArgs()
 {
-    mGroupName = "";
     mArticleCount = 0;
     mServerKey = "";
     mHitOK = PR_FALSE;
     mDownloadAll = PR_FALSE;
 }

 nsNewsDownloadDialogArgs::~nsNewsDownloadDialogArgs()
 {
 }

 NS_IMPL_ISUPPORTS1(nsNewsDownloadDialogArgs, nsINewsDownloadDialogArgs)

-NS_IMETHODIMP nsNewsDownloadDialogArgs::GetGroupName(char * *aGroupName)
-{
-    NS_ENSURE_ARG_POINTER(aGroupName);
-
-    *aGroupName = ToNewCString(mGroupName);
-
-    return NS_OK;
-}
-NS_IMETHODIMP nsNewsDownloadDialogArgs::SetGroupName(const char * aGroupName)
-{
-    NS_ENSURE_ARG_POINTER(aGroupName);
-
-    mGroupName = aGroupName;
-
-    return NS_OK;
-}
+NS_IMETHODIMP nsNewsDownloadDialogArgs::GetGroupName(nsAString & aGroupName)
+ {
+    aGroupName = mGroupName;
+ 
+     return NS_OK;
+ }
+NS_IMETHODIMP nsNewsDownloadDialogArgs::SetGroupName(const nsAString &
aGroupName)
+ {
+ 
+     mGroupName = aGroupName;
+ 
+     return NS_OK;
+ }
 NS_IMETHODIMP nsNewsDownloadDialogArgs::GetArticleCount(PRInt32
*aArticleCount)
 {
     NS_ENSURE_ARG_POINTER(aArticleCount);

     *aArticleCount = mArticleCount;

     return NS_OK;
 }
diff -U 8 -p -r mailnews.old/news/src/nsNewsDownloadDialogArgs.h
mozilla/mailnews/news/src/nsNewsDownloadDialogArgs.h
--- mailnews.old/news/src/nsNewsDownloadDialogArgs.h	2004-04-18
02:33:16.000000000 +0800
+++ mozilla/mailnews/news/src/nsNewsDownloadDialogArgs.h	2005-04-13
13:25:12.000000000 +0800
@@ -47,16 +47,16 @@ class nsNewsDownloadDialogArgs : public 
 public:
   nsNewsDownloadDialogArgs();
   virtual ~nsNewsDownloadDialogArgs();

   NS_DECL_ISUPPORTS
   NS_DECL_NSINEWSDOWNLOADDIALOGARGS

 private:
-  nsCString mGroupName;
+  nsString mGroupName;
   PRInt32 mArticleCount;
   nsCString mServerKey;
   PRBool mHitOK;
   PRBool mDownloadAll;
 };

 #endif // nsNewsDownloadDialogArgs_h__
diff -U 8 -p -r mailnews.old/news/src/nsNewsFolder.cpp
mozilla/mailnews/news/src/nsNewsFolder.cpp
--- mailnews.old/news/src/nsNewsFolder.cpp	2005-02-15 08:13:35.000000000
+0800
+++ mozilla/mailnews/news/src/nsNewsFolder.cpp	2005-04-19 08:33:14.976769560
+0800
@@ -88,16 +88,18 @@

 #include "nsIInterfaceRequestor.h"
 #include "nsIInterfaceRequestorUtils.h"

 #include "nsReadableUtils.h"
 #include "nsNewsDownloader.h"
 #include "nsIStringBundle.h"
 #include "nsEscape.h"
+#include "nsMsgI18N.h"
+#include "nsNativeCharsetUtils.h"

 // we need this because of an egcs 1.0 (and possibly gcc) compiler bug
 // that doesn't allow you to call ::nsISupports::GetIID() inside of a class
 // that multiply inherits from nsISupports
 static NS_DEFINE_CID(kRDFServiceCID, NS_RDFSERVICE_CID);
 static NS_DEFINE_CID(kCNewsDB, NS_NEWSDB_CID);

 // ###tw  This really ought to be the most
@@ -112,33 +114,33 @@ static NS_DEFINE_CID(kCNewsDB, NS_NEWSDB



///////////////////////////////////////////////////////////////////////////////
/

 nsMsgNewsFolder::nsMsgNewsFolder(void) : nsMsgLineBuffer(nsnull, PR_FALSE),
      mExpungedBytes(0), mGettingNews(PR_FALSE),
     mInitialized(PR_FALSE), mOptionLines(""), mUnsubscribedNewsgroupLines(""), 
     m_downloadMessageForOfflineUse(PR_FALSE),
m_downloadingMultipleMessages(PR_FALSE), 
-    mReadSet(nsnull), mGroupUsername(nsnull), mGroupPassword(nsnull),
mAsciiName(nsnull)
+    mReadSet(nsnull), mGroupUsername(nsnull), mGroupPassword(nsnull),
mRawName(nsnull)
 {
   MOZ_COUNT_CTOR(nsNewsFolder); // double count these for now.
   /* we're parsing the newsrc file, and the line breaks are platform specific.
    * if MSG_LINEBREAK != CRLF, then we aren't looking for CRLF 
    */
   if (PL_strcmp(MSG_LINEBREAK, CRLF))
     SetLookingForCRLF(PR_FALSE);
 }

 nsMsgNewsFolder::~nsMsgNewsFolder(void)
 {
   MOZ_COUNT_DTOR(nsNewsFolder);
   delete mReadSet;
   PR_Free(mGroupUsername);
   PR_Free(mGroupPassword);
-  PR_Free(mAsciiName);
+  PR_Free(mRawName);
 }

 NS_IMPL_ADDREF_INHERITED(nsMsgNewsFolder, nsMsgDBFolder)
 NS_IMPL_RELEASE_INHERITED(nsMsgNewsFolder, nsMsgDBFolder)

 NS_IMETHODIMP nsMsgNewsFolder::QueryInterface(REFNSIID aIID, void**
aInstancePtr)
 {
   if (!aInstancePtr) return NS_ERROR_NULL_POINTER;
@@ -205,20 +207,24 @@ nsMsgNewsFolder::AddNewsgroup(const char
   nsCOMPtr <nsINntpIncomingServer> nntpServer;
   rv = GetNntpServer(getter_AddRefs(nntpServer));
   if (NS_FAILED(rv)) return rv;

   nsCAutoString uri(mURI);
   uri.Append('/');
   // URI should use UTF-8
   // (see RFC2396 Uniform Resource Identifiers (URI): Generic Syntax)
-  // since we are forcing it to be latin-1 (IS0-8859-1)
-  // we can just assign with conversion
+  nsCAutoString dataCharset;
+  rv = nntpServer->GetCharset(dataCharset);
+  if (NS_FAILED(rv)) return rv;
   nsAutoString newsgroupName;
-  newsgroupName.AssignWithConversion(name);
+  rv =
nsMsgI18NConvertToUnicode(dataCharset.get(),nsDependentCString(name),newsgroupN
ame);
+  if (NS_FAILED(rv)) {
+    CopyASCIItoUTF16(name, newsgroupName);
+  }

   nsCAutoString escapedName;
   rv = NS_MsgEscapeEncodeURLPath(newsgroupName, escapedName);
   if (NS_FAILED(rv)) return rv;

   rv = nntpServer->AddNewsgroup(escapedName.get());
   if (NS_FAILED(rv)) return rv;

@@ -511,32 +517,35 @@ nsMsgNewsFolder::GetMessages(nsIMsgWindo
 }

 NS_IMETHODIMP nsMsgNewsFolder::GetFolderURL(char **url)
 {
   NS_ENSURE_ARG_POINTER(url);

   nsXPIDLCString hostName;
   nsresult rv = GetHostname(getter_Copies(hostName));
-  nsXPIDLCString groupName;
-  rv = GetAsciiName(getter_Copies(groupName));
+  nsXPIDLString groupName;
+  rv = GetName(getter_Copies(groupName));
   NS_ENSURE_SUCCESS(rv, rv);

   nsCOMPtr<nsIMsgIncomingServer> server;
   rv = GetServer(getter_AddRefs(server));
   NS_ENSURE_SUCCESS(rv, rv);

   PRInt32 port;
   PRBool isSecure = PR_FALSE;
   rv = server->GetIsSecure(&isSecure);
   NS_ENSURE_SUCCESS(rv, rv);
   rv = server->GetPort(&port);
   NS_ENSURE_SUCCESS(rv, rv);
   const char *newsScheme = (isSecure) ? SNEWS_SCHEME : NEWS_SCHEME;
-  *url = PR_smprintf("%s//%s:%ld/%s", newsScheme, hostName.get(), port,
groupName.get());
+  nsXPIDLCString escapedName;
+  rv = NS_MsgEscapeEncodeURLPath(groupName, escapedName);
+  NS_ENSURE_SUCCESS(rv, rv);
+  *url = PR_smprintf("%s//%s:%ld/%s", newsScheme, hostName.get(), port,
escapedName.get());
   return NS_OK;

 }

 NS_IMETHODIMP nsMsgNewsFolder::SetNewsrcHasChanged(PRBool newsrcHasChanged)
 {
     nsresult rv;

@@ -549,18 +558,28 @@ NS_IMETHODIMP nsMsgNewsFolder::SetNewsrc

 NS_IMETHODIMP nsMsgNewsFolder::CreateSubfolder(const PRUnichar
*uninewsgroupname, nsIMsgWindow *msgWindow)
 {
   nsresult rv = NS_OK;

   NS_ENSURE_ARG_POINTER(uninewsgroupname);
   if (!*uninewsgroupname) return NS_ERROR_FAILURE;

-  nsCAutoString newsgroupname; 
-  newsgroupname.AssignWithConversion(uninewsgroupname);
+  nsCAutoString newsgroupname;
+  nsCOMPtr <nsINntpIncomingServer> nntpServer;
+  rv = GetNntpServer(getter_AddRefs(nntpServer));
+  if (NS_FAILED(rv)) return rv;
+  nsCAutoString dataCharset;
+  rv = nntpServer->GetCharset(dataCharset);
+  if (NS_FAILED(rv)) return rv;
+
+  rv = nsMsgI18NConvertFromUnicode(dataCharset.get(),
nsDependentString(uninewsgroupname), newsgroupname);
+  if (NS_FAILED(rv)) {
+    LossyCopyUTF16toASCII(uninewsgroupname,newsgroupname);
+  }

   nsFileSpec path;
   nsCOMPtr<nsIFileSpec> pathSpec;
   rv = GetPath(getter_AddRefs(pathSpec));
   if (NS_FAILED(rv)) return rv;

   rv = pathSpec->GetFileSpec(&path);

@@ -569,29 +588,32 @@ NS_IMETHODIMP nsMsgNewsFolder::CreateSub
   // Create an empty database for this mail folder, set its name from the user 
   nsCOMPtr<nsIMsgDatabase> newsDBFactory;
   nsCOMPtr <nsIMsgDatabase> newsDB;

   //Now we have a valid directory or we have returned.
   //Make sure the new folder name is valid

   // remember, some file systems (like mac) can't handle long file names
-  nsCAutoString hashedName = newsgroupname;
+  nsCAutoString fileName;
+  nsAutoString hashedName(uninewsgroupname);
   rv = NS_MsgHashIfNecessary(hashedName);
-  path += hashedName.get();
+  NS_CopyUnicodeToNative(hashedName, fileName);
+  path += fileName.get();

   //Now let's create the actual new folder
   rv = AddNewsgroup(newsgroupname.get(), "", getter_AddRefs(child));

   if (NS_SUCCEEDED(rv))
     SetNewsrcHasChanged(PR_TRUE); // subscribe UI does this - but maybe we got
here through auto-subscribe

-  if(NS_SUCCEEDED(rv) && child)
+  if(NS_SUCCEEDED(rv) && child){
+    child->SetCharset(dataCharset.get());
     NotifyItemAdded(child);
-
+  }
   return rv;
 }

 NS_IMETHODIMP nsMsgNewsFolder::Delete()
 {
   nsresult rv = GetDatabase(nsnull);

   if(NS_SUCCEEDED(rv)) 
@@ -1124,27 +1146,42 @@ nsMsgNewsFolder::HandleNewsrcLine(char* 
   //
   // So lines like this in a newsrc file should be ignored:
   // 3746EF3F.6080309@netscape.com:
   // 3746EF3F.6080309%40netscape.com:
   if (PL_strstr(line,"@") || PL_strstr(line,"%40")) 
     // skipping, it contains @ or %40
     subscribed = PR_FALSE;

+    nsCOMPtr<nsINntpIncomingServer> nntpServer;
+    rv = GetNntpServer(getter_AddRefs(nntpServer));
+    if (NS_FAILED(rv)) return rv;
+    //now we convert from UTF8 to the server charset encoding because the
newsrc file is always saved in UTF8
+    nsAutoString unicodeLine;
+    nsCAutoString rawLine;
+    CopyUTF8toUTF16(line,unicodeLine);
+
+    nsCAutoString dataCharset;
+    rv = nntpServer->GetCharset(dataCharset);
+    NS_ASSERTION(NS_SUCCEEDED(rv),"failed get sever charset");
+    rv = nsMsgI18NConvertFromUnicode(dataCharset.get(), unicodeLine, rawLine);
+    if (NS_FAILED(rv))
+	LossyCopyUTF16toASCII(unicodeLine,rawLine);
+
   if (subscribed) 
   {
     // we're subscribed, so add it
     nsCOMPtr <nsIMsgFolder> child;

-    rv = AddNewsgroup(line, setStr, getter_AddRefs(child));
+    rv = AddNewsgroup(rawLine.get(), setStr, getter_AddRefs(child));

     if (NS_FAILED(rv)) return -1;
   }
   else {
-    rv = RememberUnsubscribedGroup(line, setStr);
+    rv = RememberUnsubscribedGroup(rawLine.get(), setStr);
     if (NS_FAILED(rv)) return -1;
   }

   return 0;
 }


 nsresult
@@ -1503,20 +1540,22 @@ nsresult nsMsgNewsFolder::CreateBaseMess

 NS_IMETHODIMP
 nsMsgNewsFolder::GetNewsrcLine(char **newsrcLine)
 {
   nsresult rv;

   if (!newsrcLine) return NS_ERROR_NULL_POINTER;

+  nsXPIDLString uninewsgroupname;
   nsXPIDLCString newsgroupname;
-  rv = GetAsciiName(getter_Copies(newsgroupname));
+  rv = GetName(getter_Copies(uninewsgroupname));
   if (NS_FAILED(rv)) return rv;
-  
+  CopyUTF16toUTF8(uninewsgroupname, newsgroupname);
+
   nsCAutoString newsrcLineStr;
   newsrcLineStr = newsgroupname;
   newsrcLineStr += ':';

   if (mReadSet) {
     nsXPIDLCString setStr;
     mReadSet->Output(getter_Copies(setStr));
     if (NS_SUCCEEDED(rv)) 
@@ -1575,38 +1614,62 @@ nsMsgNewsFolder::GetOptionLines(char **o
 }

 NS_IMETHODIMP
 nsMsgNewsFolder::OnReadChanged(nsIDBChangeListener * aInstigator)
 {
     return SetNewsrcHasChanged(PR_TRUE);
 }

+NS_IMETHODIMP
+nsMsgNewsFolder::GetUnicodeName(nsAString & name)
+{
+  nsresult rv;
+  if(mUnicodeName.IsEmpty()){
+    nsXPIDLString newsgroupName;
+    rv = GetName(getter_Copies(newsgroupName));
+    NS_ENSURE_SUCCESS(rv,rv);
+    mUnicodeName = newsgroupName;
+  }
+  name = mUnicodeName;
+  return NS_OK;
+}

 NS_IMETHODIMP
-nsMsgNewsFolder::GetAsciiName(char **asciiName)
+nsMsgNewsFolder::GetRawName(char **rawName)
 {
   nsresult rv;
-  NS_ENSURE_ARG_POINTER(asciiName);
-  if (!mAsciiName) 
+  NS_ENSURE_ARG_POINTER(rawName);
+  if (!mRawName) 
   {
     nsXPIDLString name;
     rv = GetName(getter_Copies(name));
     NS_ENSURE_SUCCESS(rv,rv);

     // convert to ASCII
     nsCAutoString tmpStr;
-    tmpStr.AssignWithConversion(name);
+    nsCOMPtr <nsINntpIncomingServer> nntpServer;
+    rv = GetNntpServer(getter_AddRefs(nntpServer));
+    NS_ENSURE_SUCCESS(rv,rv);
+    
+    nsCAutoString dataCharset;
+    rv = nntpServer->GetCharset(dataCharset);
+    NS_ENSURE_SUCCESS(rv,rv);
+    rv = nsMsgI18NConvertFromUnicode(dataCharset.get(), name, tmpStr);
+
+    if (NS_FAILED(rv)) {
+      LossyCopyUTF16toASCII(name,tmpStr);
+    }

-    mAsciiName = nsCRT::strdup(tmpStr.get());
-    if (!mAsciiName) return NS_ERROR_OUT_OF_MEMORY;
+    mRawName = nsCRT::strdup(tmpStr.get());
+    if (!mRawName) return NS_ERROR_OUT_OF_MEMORY;
   }

-  *asciiName = nsCRT::strdup(mAsciiName);
-  if (!*asciiName) return NS_ERROR_OUT_OF_MEMORY;
+  *rawName = nsCRT::strdup(mRawName);
+  if (!*rawName) return NS_ERROR_OUT_OF_MEMORY;

   return NS_OK;
 }

 NS_IMETHODIMP
 nsMsgNewsFolder::GetNntpServer(nsINntpIncomingServer **result)
 {
     nsresult rv;
Only in mozilla/mailnews/news/src: nsNewsFolder.cpp~
diff -U 8 -p -r mailnews.old/news/src/nsNewsFolder.h
mozilla/mailnews/news/src/nsNewsFolder.h
--- mailnews.old/news/src/nsNewsFolder.h	2004-04-18 02:33:16.000000000
+0800
+++ mozilla/mailnews/news/src/nsNewsFolder.h	2005-04-19 08:33:47.599810104
+0800
@@ -154,18 +154,18 @@ protected:
   nsMsgKeySet *mReadSet; 

   nsCOMPtr<nsIFileSpec> mNewsrcFilePath; 

   // used for auth news
   char 		*mGroupUsername;
   char 	*mGroupPassword;

-  // the name of the newsgroup.
-  char    *mAsciiName;
+  char    *mRawName;
+  nsString mUnicodeName;
   PRInt32 mSortOrder;

 private:
     nsresult CreateNewsgroupUsernameUrlForSignon(const char *inUriStr, char
**result);
     nsresult CreateNewsgroupPasswordUrlForSignon(const char *inUriStr, char
**result);
     nsresult CreateNewsgroupUrlForSignon(const char *inUriStr, const char
*ref, char **result);

     nsCOMPtr <nsIFileSpec> mFilterFile;
Only in mozilla/mailnews/news/src: nsNewsFolder.h~
diff -U 8 -p -r mailnews.old/news/src/nsNntpIncomingServer.cpp
mozilla/mailnews/news/src/nsNntpIncomingServer.cpp
--- mailnews.old/news/src/nsNntpIncomingServer.cpp	2004-08-27
06:16:52.000000000 +0800
+++ mozilla/mailnews/news/src/nsNntpIncomingServer.cpp	2005-04-19
08:33:31.246296216 +0800
@@ -57,22 +57,25 @@
 #include "nsMsgUtils.h"
 #include "nsIPrompt.h"
 #include "nsIStringBundle.h"
 #include "nntpCore.h"
 #include "nsIWindowWatcher.h"
 #include "nsITreeColumns.h"
 #include "nsIDOMElement.h"
 #include "nsMsgFolderFlags.h"
+#include "nsMsgI18N.h"
+#include "nsIPrefLocalizedString.h"

 #define INVALID_VERSION	 0
 #define VALID_VERSION		 1
 #define NEW_NEWS_DIR_NAME	 "News"
 #define PREF_MAIL_NEWSRC_ROOT	 "mail.newsrc_root"
 #define PREF_MAIL_NEWSRC_ROOT_REL "mail.newsrc_root-rel"
+#define PREF_MAILNEWS_VIEW_DEFAULT_CHARSET "mailnews.view_default_charset"
 #define HOSTINFO_FILE_NAME	 "hostinfo.dat"

 #define NEWS_DELIMITER 	 '.'

 // this platform specific junk is so the newsrc filenames we create 
 // will resemble the migrated newsrc filenames.
 #if defined(XP_UNIX) || defined(XP_BEOS)
 #define NEWSRC_FILE_PREFIX "newsrc-"
@@ -304,16 +307,70 @@ nsresult nsNntpIncomingServer::SetupNews
     mNewsrcSaveTimer = do_CreateInstance("@mozilla.org/timer;1");
	mNewsrcSaveTimer->InitWithFuncCallback(OnNewsrcSaveTimer, (void*)this,
timeInMSUint32, 
					    nsITimer::TYPE_REPEATING_SLACK);

     return NS_OK;
 }

 NS_IMETHODIMP
+nsNntpIncomingServer::SetCharset(const nsACString & aCharset)
+{
+	nsresult rv;
+	rv = SetCharValue("charset", PromiseFlatCString(aCharset).get());
+	return rv;
+}
+
+NS_IMETHODIMP
+nsNntpIncomingServer::GetCharset(nsACString & aCharset)
+{
+	nsresult rv; 
+	nsXPIDLCString serverCharset;
+	//first we get the per-server settings mail.server.<serverkey>.charset
+	rv = GetCharValue("charset",getter_Copies(serverCharset));
+
+	//if the per-server setting is empty,we get the default charset from 
+	//mailnews.view_default_charset setting and set it as per-server
preference.
+	if(serverCharset.IsEmpty()){
+		nsXPIDLCString defaultCharset;
+		nsCOMPtr<nsIPrefBranch>
prefBranch(do_GetService(NS_PREFSERVICE_CONTRACTID, &rv));
+
+		if (NS_SUCCEEDED(rv)){
+			nsCOMPtr<nsIPrefLocalizedString> pls;
+			rv =
prefBranch->GetComplexValue(PREF_MAILNEWS_VIEW_DEFAULT_CHARSET,
+		       NS_GET_IID(nsIPrefLocalizedString),
getter_AddRefs(pls));
+			if (NS_SUCCEEDED(rv)) 
+			{
+				nsXPIDLString ucsval;
+				pls->ToString(getter_Copies(ucsval));
+				if (ucsval)
+				{
+					LossyCopyUTF16toASCII(ucsval,
defaultCharset);
+				}
+			}
+		}
+
+		if(defaultCharset.IsEmpty()){
+			serverCharset = "ISO-8859-1";
+		}else{
+			serverCharset = defaultCharset;
+		}
+
+#ifdef DEBUG_NEWS
+		printf("default charset for the server is %s\n", (const char
*)defaultCharset);
+#endif
+
+		SetCharset(serverCharset);
+	}
+
+	aCharset = serverCharset;
+	return NS_OK;
+}
+
+NS_IMETHODIMP
 nsNntpIncomingServer::WriteNewsrcFile()
 {
     nsresult rv;

     PRBool newsrcHasChanged;
     rv = GetNewsrcHasChanged(&newsrcHasChanged);
     if (NS_FAILED(rv)) return rv;

@@ -806,17 +863,17 @@ writeGroupToHostInfoFile(nsCString &aEle
     nsAutoString name;
     nsresult rv = NS_MsgDecodeUnescapeURLPath(aElement, name); 
     if (NS_FAILED(rv)) {
	 // stop, something is bad.
	 return PR_FALSE;
     }

     nsCAutoString nameOnDisk;
-    nameOnDisk.AssignWithConversion(name.get());
+    CopyUTF16toUTF8(name, nameOnDisk);

     // XXX todo ",,1,0,0" is a temporary hack, fix it
     *stream << nameOnDisk.get() << ",,1,0,0" << MSG_LINEBREAK;
     return PR_TRUE;
 }

 nsresult
 nsNntpIncomingServer::WriteHostInfoFile()
@@ -1128,19 +1185,41 @@ nsNntpIncomingServer::UpdateSubscribed()

 NS_IMETHODIMP
 nsNntpIncomingServer::AddTo(const char *aName, PRBool addAsSubscribed, PRBool
aSubscribable, PRBool changeIfExists)
 {
     nsresult rv = EnsureInner();
     NS_ENSURE_SUCCESS(rv,rv);

     nsAutoString newsgroupName;
-    newsgroupName.AssignWithConversion(aName);
+    char *escapedName = nsnull;
+
+    nsCOMPtr<nsIMsgFolder> rootFolder;
+    rv = GetRootFolder(getter_AddRefs(rootFolder));
+    NS_ENSURE_SUCCESS(rv,rv);
+    if (!rootFolder) return NS_ERROR_FAILURE;
+    nsCOMPtr <nsIMsgNewsFolder> newsFolder = do_QueryInterface(rootFolder,
&rv);
+    NS_ENSURE_SUCCESS(rv,rv);
+    if (!newsFolder) return NS_ERROR_FAILURE;
+    nsCOMPtr <nsINntpIncomingServer> nntpServer;
+    rv = newsFolder->GetNntpServer(getter_AddRefs(nntpServer));
+    NS_ENSURE_SUCCESS(rv,rv);
+    nsCAutoString dataCharset;
+    rv = nntpServer->GetCharset(dataCharset);
+    NS_ENSURE_SUCCESS(rv,rv);
+    rv =
nsMsgI18NConvertToUnicode(dataCharset.get(),nsDependentCString(aName),newsgroup
Name);
+    if (NS_SUCCEEDED(rv)) {
+	 escapedName = nsEscape(NS_ConvertUTF16toUTF8(newsgroupName).get(),
url_Path);
+     }
+
+    if (NS_FAILED(rv)) {
+      CopyASCIItoUTF16(aName, newsgroupName);
+      escapedName = nsEscape(aName, url_Path);
+    }

-    char *escapedName =
nsEscape(NS_ConvertUCS2toUTF8(newsgroupName.get()).get(), url_Path);
     if (!escapedName) return NS_ERROR_OUT_OF_MEMORY;

     rv = AddGroupOnServer(escapedName);
     NS_ENSURE_SUCCESS(rv,rv);

     rv = mInner->AddTo(escapedName,addAsSubscribed, aSubscribable,
changeIfExists);
     NS_ENSURE_SUCCESS(rv,rv);

@@ -1245,17 +1324,27 @@ nsNntpIncomingServer::HandleLine(char* l
	if (line[0] == '#' || line[0] == '\0') return 0;

	line[line_size] = 0;

	if (mHasSeenBeginGroups) {
		char *commaPos = PL_strchr(line,',');
		if (commaPos) *commaPos = 0;

-		nsresult rv = AddTo(line, PR_FALSE, PR_TRUE, PR_TRUE);
+		//now we convert from UTF8 to the server charset encoding
because the hostInfo.dat is always saved in UTF8
+	 nsAutoString unicodeLine;
+	 nsCAutoString encodedLine;
+	 CopyUTF8toUTF16(line,unicodeLine);
+	 nsCAutoString dataCharset;
+	 nsresult rv = GetCharset(dataCharset);
+	 NS_ASSERTION(NS_SUCCEEDED(rv),"failed to get sever charset");
+	 rv = nsMsgI18NConvertFromUnicode(dataCharset.get(), unicodeLine,
encodedLine);
+	 if (NS_FAILED(rv))
+	   LossyCopyUTF16toASCII(unicodeLine,encodedLine);
+		rv = AddTo(encodedLine.get(), PR_FALSE, PR_TRUE, PR_TRUE);
		NS_ASSERTION(NS_SUCCEEDED(rv),"failed to add line");
		if (NS_SUCCEEDED(rv)) {
	   // since we've seen one group, we can claim we've loaded the
hostinfo file
	   mHostInfoLoaded = PR_TRUE;
		}
	}
	else {
		if (nsCRT::strncmp(line,"begingroups", 11) == 0) {
Only in mozilla/mailnews/news/src: nsNntpIncomingServer.cpp~
diff -U 8 -p -r mailnews.old/news/src/nsNNTPNewsgroupList.cpp
mozilla/mailnews/news/src/nsNNTPNewsgroupList.cpp
--- mailnews.old/news/src/nsNNTPNewsgroupList.cpp	2005-02-01
05:53:40.000000000 +0800
+++ mozilla/mailnews/news/src/nsNNTPNewsgroupList.cpp	2005-04-13
13:55:22.000000000 +0800
@@ -319,21 +319,21 @@ nsNNTPNewsgroupList::GetRangeOfArtsToDow

		 nsCOMPtr<nsINewsDownloadDialogArgs> args =
do_CreateInstance("@mozilla.org/messenger/newsdownloaddialogargs;1", &rv);
		 if (NS_FAILED(rv)) return rv;
		 NS_ENSURE_SUCCESS(rv,rv);

		 rv = args->SetArticleCount(*last - *first + 1);
		 NS_ENSURE_SUCCESS(rv,rv);

-		 nsXPIDLCString groupName;
-		 rv = m_newsFolder->GetAsciiName(getter_Copies(groupName));
+		 nsXPIDLString groupName;
+		 rv = m_newsFolder->GetUnicodeName(groupName);
		 NS_ENSURE_SUCCESS(rv,rv);

-		 rv = args->SetGroupName((const char *)groupName);
+		 rv = args->SetGroupName(groupName);
		 NS_ENSURE_SUCCESS(rv,rv);

				// get the server key
				nsXPIDLCString serverKey;
				rv = server->GetKey(getter_Copies(serverKey));
		 NS_ENSURE_SUCCESS(rv,rv);

		 rv = args->SetServerKey((const char *)serverKey);
Only in mozilla/mailnews/news/src: nsNNTPNewsgroupList.cpp~
diff -U 8 -p -r mailnews.old/news/src/nsNNTPProtocol.cpp
mozilla/mailnews/news/src/nsNNTPProtocol.cpp
--- mailnews.old/news/src/nsNNTPProtocol.cpp	2005-02-16 11:18:46.000000000
+0800
+++ mozilla/mailnews/news/src/nsNNTPProtocol.cpp	2005-04-19
03:36:57.000000000 +0800
@@ -1934,17 +1934,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
	char *command=0;
	PRInt32 status = 0;

	if (m_typeWanted == ARTICLE_WANTED) {
		if (m_key != nsMsgKey_None) {
		    nsresult rv;
	     nsXPIDLCString newsgroupName;
	     if (m_newsFolder) {
-		 rv = m_newsFolder->GetAsciiName(getter_Copies(newsgroupName));
+		 rv = m_newsFolder->GetRawName(getter_Copies(newsgroupName));
		 NS_ENSURE_SUCCESS(rv,rv);
	     }

		    PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) current group = %s,
desired group = %s", this, m_currentGroup.get(), newsgroupName.get()));
	     // if the current group is the desired group, we can just issue
the ARTICLE command
	     // if not, we have to do a GROUP first
			if (!PL_strcmp(m_currentGroup.get(),
newsgroupName.get()))
			  m_nextState = NNTP_SEND_ARTICLE_NUMBER;
@@ -2045,17 +2045,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
	else if(m_typeWanted == GROUP_WANTED) 
     {
	 nsresult rv = NS_ERROR_NULL_POINTER;

	 NS_ASSERTION(m_newsFolder, "m_newsFolder is null, panic!");
	 if (!m_newsFolder) return -1;

	 nsXPIDLCString group_name;
-		rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+		rv = m_newsFolder->GetRawName(getter_Copies(group_name));
	 NS_ASSERTION(NS_SUCCEEDED(rv),"failed to get newsgroup name");
	 if (NS_FAILED(rv)) return -1;

	 m_firstArticle = 0;
	 m_lastArticle = 0;

	 NS_MsgSACopy(&command, "GROUP ");
	 NS_MsgSACat(&command, (const char *)group_name);
@@ -2090,17 +2090,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
	     PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) doing GROUP for XPAT", this));
	     nsXPIDLCString group_name;

			/* for XPAT, we have to GROUP into the group before
searching */
			if (!m_newsFolder) {
				NNTP_LOG_NOTE("m_newsFolder is null, panic!");
				return -1;
			}
-	     rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+	     rv = m_newsFolder->GetRawName(getter_Copies(group_name));
	     if (NS_FAILED(rv)) return -1;

			NS_MsgSACopy(&command, "GROUP ");
	     NS_MsgSACat (&command, group_name);

	     // force a GROUP next time
	     m_currentGroup.Truncate();
			m_nextState = NNTP_RESPONSE;
@@ -2201,17 +2201,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
   }
   else
   {
     nsresult rv = NS_OK;

     nsXPIDLCString group_name;
     NS_ASSERTION(m_newsFolder, "no newsFolder");
     if (m_newsFolder) {
-      rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+      rv = m_newsFolder->GetRawName(getter_Copies(group_name));
     }

     if (m_responseCode == MK_NNTP_RESPONSE_GROUP_NO_GROUP &&
       m_typeWanted == GROUP_WANTED) {
       PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) group (%s) not found, so unset
m_currentGroup",this,(const char *)group_name));
       m_currentGroup.Truncate();

       m_nntpServer->GroupNotFound(m_msgWindow, group_name.get(), PR_TRUE /*
opening */);
@@ -2316,17 +2316,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
 }

 PRInt32 nsNNTPProtocol::SendGroupForArticle()
 {
   nsresult rv;
   PRInt32 status = 0; 

   nsXPIDLCString groupname;
-  rv = m_newsFolder->GetAsciiName(getter_Copies(groupname));
+  rv = m_newsFolder->GetRawName(getter_Copies(groupname));
   NS_ASSERTION(NS_SUCCEEDED(rv) && groupname.get() && groupname.get()[0], "no
group name");

   char outputBuffer[OUTPUT_BUFFER_SIZE];

   PR_snprintf(outputBuffer, 
			OUTPUT_BUFFER_SIZE, 
			"GROUP %.512s" CRLF, 
			(const char *)groupname);
@@ -2348,17 +2348,17 @@ nsNNTPProtocol::SetCurrentGroup()
   nsresult rv;
   nsXPIDLCString groupname;
   NS_ASSERTION(m_newsFolder, "no news folder");
   if (!m_newsFolder) {
     m_currentGroup.Truncate();
     return NS_ERROR_UNEXPECTED;
   }

-  rv = m_newsFolder->GetAsciiName(getter_Copies(groupname));
+  rv = m_newsFolder->GetRawName(getter_Copies(groupname));
   NS_ASSERTION(NS_SUCCEEDED(rv) && groupname.get()[0], "no group name");
   PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) SetCurrentGroup to %s",this,(const char
*)groupname));
   m_currentGroup = groupname;
   return NS_OK;
 }

 PRInt32 nsNNTPProtocol::SendGroupForArticleResponse()
 {
@@ -3873,17 +3873,17 @@ PRInt32 nsNNTPProtocol::DisplayNewsRC()
     if (NS_FAILED(rv)) return -1;
     if (!currFolder) return -1;

     m_newsFolder = do_QueryInterface(currFolder, &rv);
     if (NS_FAILED(rv)) return -1;
     if (!m_newsFolder) return -1;

     nsXPIDLCString name;
-    rv = m_newsFolder->GetAsciiName(getter_Copies(name));
+    rv = m_newsFolder->GetRawName(getter_Copies(name));
     if (NS_FAILED(rv)) return -1;
     if (!name) return -1;

     /* send group command to server */
     char outputBuffer[OUTPUT_BUFFER_SIZE];

     PR_snprintf(outputBuffer, OUTPUT_BUFFER_SIZE, "GROUP %.512s" CRLF, (const
char *)name);
     nsCOMPtr<nsIMsgMailNewsUrl> mailnewsurl = do_QueryInterface(m_runningURL);
@@ -3963,17 +3963,17 @@ PRInt32 nsNNTPProtocol::DisplayNewsRCRes
     NS_ASSERTION(NS_SUCCEEDED(rv),"DisplaySubscribedGroup() failed");
     if (NS_FAILED(rv)) status = -1;

     if (status < 0) return status;
   }
   else if (m_responseCode == MK_NNTP_RESPONSE_GROUP_NO_GROUP)
   {
     nsXPIDLCString name;
-    rv = m_newsFolder->GetAsciiName(getter_Copies(name));
+    rv = m_newsFolder->GetRawName(getter_Copies(name));

     if (NS_SUCCEEDED(rv)) {
       m_nntpServer->GroupNotFound(m_msgWindow, name.get(), PR_FALSE);
     }

     PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) NO_GROUP, so unset m_currentGroup",
this));
     m_currentGroup.Truncate();
   }
@@ -4394,17 +4394,17 @@ PRInt32 nsNNTPProtocol::XPATResponse(nsI

 PRInt32 nsNNTPProtocol::ListPrettyNames()
 {

   nsXPIDLCString group_name;
   char outputBuffer[OUTPUT_BUFFER_SIZE];
   PRInt32 status = 0; 

-  nsresult rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+  nsresult rv = m_newsFolder->GetRawName(getter_Copies(group_name));
   PR_snprintf(outputBuffer, 
     OUTPUT_BUFFER_SIZE, 
     "LIST PRETTYNAMES %.512s" CRLF,
     NS_SUCCEEDED(rv) ? (const char *) group_name : "");

   nsCOMPtr<nsIMsgMailNewsUrl> mailnewsurl = do_QueryInterface(m_runningURL);
   if (mailnewsurl)
     status = SendData(mailnewsurl, outputBuffer);
@@ -4471,17 +4471,17 @@ PRInt32 nsNNTPProtocol::ListPrettyNamesR
   PR_FREEIF(line);
   return 0;
 }

 PRInt32 nsNNTPProtocol::ListXActive()
 { 
   nsXPIDLCString group_name;
   nsresult rv;
-  rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
+  rv = m_newsFolder->GetRawName(getter_Copies(group_name));
   if (NS_FAILED(rv)) return -1;

   PRInt32 status = 0;
   char outputBuffer[OUTPUT_BUFFER_SIZE];

   PR_snprintf(outputBuffer, 
     OUTPUT_BUFFER_SIZE, 
     "LIST XACTIVE %.512s" CRLF,
@@ -4624,17 +4624,17 @@ PRInt32 nsNNTPProtocol::SendListGroup()
   nsresult rv;
   char outputBuffer[OUTPUT_BUFFER_SIZE];
   PRInt32 status = 0; 

   NS_ASSERTION(m_newsFolder,"no newsFolder");
   if (!m_newsFolder) return -1;
   nsXPIDLCString newsgroupName;

-  rv = m_newsFolder->GetAsciiName(getter_Copies(newsgroupName));
+  rv = m_newsFolder->GetRawName(getter_Copies(newsgroupName));
   NS_ENSURE_SUCCESS(rv,rv);

   PR_snprintf(outputBuffer, 
     OUTPUT_BUFFER_SIZE, 
     "listgroup %.512s" CRLF,
     newsgroupName.get());

   m_articleList = do_CreateInstance(NS_NNTPARTICLELIST_CONTRACTID, &rv);
Only in mozilla/mailnews/news/src: nsNNTPProtocol.cpp~
diff -U 8 -p -r mailnews.old/news/src/nsNntpService.cpp
mozilla/mailnews/news/src/nsNntpService.cpp
--- mailnews.old/news/src/nsNntpService.cpp	2005-01-18 02:50:13.000000000
+0800
+++ mozilla/mailnews/news/src/nsNntpService.cpp 2005-04-13 13:56:02.000000000
+0800
@@ -650,18 +650,21 @@ nsNntpService::GetFolderFromUri(const ch
   // but the *name* of the newsgroup (we are calling ::GetChildNamed())
   // is unescaped.  see http://bugzilla.mozilla.org/show_bug.cgi?id=210089#c17
   // for more about this
   char *unescapedPath = PL_strdup(path.get() + 1); /* skip the leading slash
*/
   if (!unescapedPath)
     return NS_ERROR_OUT_OF_MEMORY;
   nsUnescape(unescapedPath);

+  nsAutoString newsgroupName;
+  //the encoding is UTF-8 after unescaped.
+  CopyUTF8toUTF16(unescapedPath,newsgroupName);
   nsCOMPtr<nsISupports> subFolder;
-  rv = rootFolder->GetChildNamed(NS_ConvertASCIItoUCS2(unescapedPath).get() ,
getter_AddRefs(subFolder));
+  rv = rootFolder->GetChildNamed(newsgroupName.get() ,
getter_AddRefs(subFolder));
   PL_strfree(unescapedPath);
   NS_ENSURE_SUCCESS(rv,rv);

   return CallQueryInterface(subFolder, aFolder);
 }

 NS_IMETHODIMP
 nsNntpService::CopyMessage(const char * aSrcMessageURI, nsIStreamListener *
aMailboxCopyHandler, PRBool moveMessage,
changes:
encodedName changed to rawName.
the mailnews.view_default_charset becomes a complex value in the trunk,So using
getComplexValue to get the default charset.

Thank you very much for reviewing...

I don't know edit attachment will put changes to the comments ...
Really sorry for that..
Attachment #170507 - Attachment is obsolete: true
Attachment #181047 - Flags: review?(jshin1987)
Comment on attachment 181047 [details] [diff] [review]
patch use utf8 in the newsrc/hostinfo file

Sorry for the delay. Let's do one more iteration.

>+[scriptable, uuid(02699533-7ba8-4e78-94dc-a20374fb2711)]
> interface nsIMsgNewsFolder : nsISupports {
>   attribute string groupUsername;
>   attribute string groupPassword;

>+  readonly attribute AString unicodeName;
>+  /** the raw name name for the newsgroup (server side name).  
>+   *  newsgroup names are might not always ascii(bug#126453), and this raw name name 
>+   *  can help simply the code, to prevent use from having to get the unicode name 
>+   *  (as a wstring) and then converting it back to server side raw name.
>    */
>-  readonly attribute string asciiName;
>+  readonly attribute string rawName;

|rawName| isn't accessible from a script so that you should add [noscript].
Alternatively, it has to be changed to getter with an octet array. For now,
just adding [noscript] should suffice because it's not used in any script. As
for the comment, it needs some polishing. 

-------Cut-------Here----------------------
|rawName| is an 8-bit string to represent the name of a newsgroup used by a
news server. It's offered for the convenience of callers so that they don't
have to convert |unicodeName| to the server-side name when communicating with a
news server.  It's US-ASCII except for some 'stand-alone' Chinese news servers
that use GB2312 for newsgroup names violating RFC 1036. For those servers, it's
GB2312. However, it can be any other single and multibyte encoding in
principle. The encoding of this string is stored in |nsINntpIncomingServer|
because that's a server-wide property.
---------Cut----------Here--------------- 

Also, make it |ACString|


> nsMsgNewsFolder::~nsMsgNewsFolder(void)
> {
>   MOZ_COUNT_DTOR(nsNewsFolder);
>   delete mReadSet;
>   PR_Free(mGroupUsername);
>   PR_Free(mGroupPassword);
>-  PR_Free(mAsciiName);
>+  PR_Free(mRawName);
> }

 For mRawName, if you use nsCString, you can avoid alloc/free, can't you?


>+    nsCOMPtr<nsINntpIncomingServer> nntpServer;
>+    rv = GetNntpServer(getter_AddRefs(nntpServer));
>+    if (NS_FAILED(rv)) return rv;
>+    //now we convert from UTF8 to the server charset encoding because the newsrc file is always saved in UTF8
>+    nsAutoString unicodeLine;
>+    nsCAutoString encodedLine;
>+    CopyUTF8toUTF16(line,unicodeLine);
>+
>+    nsCAutoString dataCharset;
>+    rv = nntpServer->GetCharset(dataCharset);
>+    NS_ASSERTION(NS_SUCCEEDED(rv),"failed get sever charset");
>+    rv = nsMsgI18NConvertFromUnicode(dataCharset.get(), unicodeLine, encodedLine);


> nsMsgNewsFolder::GetNewsrcLine(char **newsrcLine)

>+  nsXPIDLString uninewsgroupname;
>   nsXPIDLCString newsgroupname;
>-  rv = GetAsciiName(getter_Copies(newsgroupname));
>+  rv = GetName(getter_Copies(uninewsgroupname));
>   if (NS_FAILED(rv)) return rv;

Perhaps, |newsgroupNameUtf16| (with intercap) would be better than
|uninewsgroupname|.




>+nsMsgNewsFolder::GetRawName(char **rawName)

>+  NS_ENSURE_ARG_POINTER(rawName);
>+  if (!mRawName) 

Using ns*CString instead of |char *|, you can see if it's empty and don't have
to do null-pointer check.



>+    mRawName = nsCRT::strdup(tmpStr.get());
>+    if (!mRawName) return NS_ERROR_OUT_OF_MEMORY;

This part is simplified, too.


>+  char    *mRawName;
    nsCString mRawName;

prefBranch(do_GetService(NS_PREFSERVICE_CONTRACTID, &rv));
>+
>+		if (NS_SUCCEEDED(rv)){
>+			nsCOMPtr<nsIPrefLocalizedString> pls;
>+			rv = prefBranch->GetComplexValue(PREF_MAILNEWS_VIEW_DEFAULT_CHARSET,
>+                      NS_GET_IID(nsIPrefLocalizedString), getter_AddRefs(pls));
>+			if (NS_SUCCEEDED(rv)) 
>+			{
>+				nsXPIDLString ucsval;
>+				pls->ToString(getter_Copies(ucsval));
>+				if (ucsval)
>+				{
>+					LossyCopyUTF16toASCII(ucsval, defaultCharset);
>+				}
>+			}
>+		}

Why don't you just use |NS_GetUnicharPreferenceWithDefault| defined at

http://lxr.mozilla.org/seamonkey/source/mailnews/base/util/nsMsgUtils.cpp#1059

>-    newsgroupName.AssignWithConversion(aName);
>+    char *escapedName = nsnull;
>+
>+    nsCOMPtr<nsIMsgFolder> rootFolder;
>+    rv = GetRootFolder(getter_AddRefs(rootFolder));
>+    NS_ENSURE_SUCCESS(rv,rv);
>+    if (!rootFolder) return NS_ERROR_FAILURE;
>+    nsCOMPtr <nsIMsgNewsFolder> newsFolder = do_QueryInterface(rootFolder, &rv);
>+    NS_ENSURE_SUCCESS(rv,rv);
>+    if (!newsFolder) return NS_ERROR_FAILURE;
>+    nsCOMPtr <nsINntpIncomingServer> nntpServer;
>+    rv = newsFolder->GetNntpServer(getter_AddRefs(nntpServer));
>+    NS_ENSURE_SUCCESS(rv,rv);
>+    nsCAutoString dataCharset;
>+    rv = nntpServer->GetCharset(dataCharset);

Why do you need all these (long and winding) indirections? Can't you just use
GetCharset here? This method (|AddTo|) belongs to |nsNntpIncomingServer| (an
implementation of |nsINntpIncomingServer| so that it should have |GetCharset|,
right?).


>@@ -1245,17 +1324,27 @@ nsNntpIncomingServer::HandleLine(char* 
>+		//now we convert from UTF8 to the server charset encoding because the hostInfo.dat is always saved in UTF8

This comment is over 80 columns. s/the server charset encoding/the server-side
character encoding/

>+        nsAutoString unicodeLine;
>+        nsCAutoString encodedLine;

Perhaps, 'utf16Line' and 'serverSideLine' are better.


>+                rv = m_newsFolder->GetRawName(getter_Copies(newsgroupName));
Attachment #181047 - Flags: review?(jshin1987) → review-
Attachment #170507 - Flags: review?(jshin1987)
One more thing. If you want mailnews.view_default_charset to be localizable, you
have to add it to either messenger.properties 
Attached patch patch according to comment#78 (obsolete) — Splinter Review
Thanks again for your comments and hints!
Attachment #181047 - Attachment is obsolete: true
Attachment #182461 - Flags: review?(jshin1987)
Comment on attachment 182461 [details] [diff] [review]
patch according to  comment#78

r=jshin thanks for the patch and sorry for taking so long.

> NS_IMETHODIMP nsMsgNewsFolder::CreateSubfolder(const PRUnichar *uninewsgroupname, nsIMsgWindow *msgWindow)
> {
>   nsresult rv = NS_OK;
>   
>   NS_ENSURE_ARG_POINTER(uninewsgroupname);
>   if (!*uninewsgroupname) return NS_ERROR_FAILURE;
>   
>-  nsCAutoString newsgroupname; 
>-  newsgroupname.AssignWithConversion(uninewsgroupname);
>+  nsCAutoString newsgroupname;
>+  nsCOMPtr <nsINntpIncomingServer> nntpServer;
>+  rv = GetNntpServer(getter_AddRefs(nntpServer));
>+  if (NS_FAILED(rv)) return rv;
>+  nsCAutoString dataCharset;
>+  rv = nntpServer->GetCharset(dataCharset);
>+  if (NS_FAILED(rv)) return rv;
>+
>+  rv = nsMsgI18NConvertFromUnicode(dataCharset.get(), nsDependentString(uninewsgroupname), newsgroupname);

Although it's not you who first used 'uninewsgroupname' here, would you change
it to newsgroupNameUtf16? Also, it'd be nice to replace |newsgroupname| with
|newsgroupName|. 


>+                rv = args->SetGroupName(groupName);
>                 NS_ENSURE_SUCCESS(rv,rv);
> 
> 				// get the server key
> 				nsXPIDLCString serverKey;
> 				rv = server->GetKey(getter_Copies(serverKey));
>                 NS_ENSURE_SUCCESS(rv,rv);

Again, it's not your code, but it'd be nice to fix up indentation of three
lines before |NS_ENSURE_SUCCESS|.


>+++ mozilla/mailnews/news/src/nsNNTPProtocol.cpp	2005-05-03 11:43:54.608116736 +0800
>@@ -1934,17 +1934,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
> 	char *command=0;
> 	PRInt32 status = 0;
> 
> 	if (m_typeWanted == ARTICLE_WANTED) {
> 		if (m_key != nsMsgKey_None) {
> 		    nsresult rv;
>             nsXPIDLCString newsgroupName;
>             if (m_newsFolder) {
>-                rv = m_newsFolder->GetAsciiName(getter_Copies(newsgroupName));
>+                rv = m_newsFolder->GetRawName(newsgroupName);
>                 NS_ENSURE_SUCCESS(rv,rv);
>             }
> 
> 		    PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) current group = %s, desired group = %s", this, m_currentGroup.get(), newsgroupName.get()));
>             // if the current group is the desired group, we can just issue the ARTICLE command
>             // if not, we have to do a GROUP first
> 			if (!PL_strcmp(m_currentGroup.get(), newsgroupName.get()))
> 			  m_nextState = NNTP_SEND_ARTICLE_NUMBER;

Phew, the indentation here is also horrible perhaps due to tabs. Can you fix
them all? 


>@@ -2045,17 +2045,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom

> 
>         nsXPIDLCString group_name;
>-		rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
>+		rv = m_newsFolder->GetRawName(group_name);
>         NS_ASSERTION(NS_SUCCEEDED(rv),"failed to get newsgroup name");

You're touching the line so that you'd better fix the indentation.

>@@ -2090,17 +2090,17 @@ PRInt32 nsNNTPProtocol::SendFirstNNTPCom
>             PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) doing GROUP for XPAT", this));
>             nsXPIDLCString group_name;
>             
> 			/* for XPAT, we have to GROUP into the group before searching */
> 			if (!m_newsFolder) {
> 				NNTP_LOG_NOTE("m_newsFolder is null, panic!");
> 				return -1;
> 			}
>-            rv = m_newsFolder->GetAsciiName(getter_Copies(group_name));
>+            rv = m_newsFolder->GetRawName(group_name);
>             if (NS_FAILED(rv)) return -1;
> 
> 			NS_MsgSACopy(&command, "GROUP ");
>             NS_MsgSACat (&command, group_name);
> 
>             // force a GROUP next time
>             m_currentGroup.Truncate();
> 			m_nextState = NNTP_RESPONSE;

I'm sorry to ask you to do more chores, but if you don't mind getting
cvs-blame, would you fix tabs here as well? 


>+  //the encoding is UTF-8 after unescaped.

   after unescaping
Attachment #182461 - Flags: superreview?(mscott)
Attachment #182461 - Flags: review?(jshin1987)
Attachment #182461 - Flags: review+
Attachment #182461 - Flags: superreview?(mscott)
Comment on attachment 182461 [details] [diff] [review]
patch according to  comment#78

can you post a new patch with jshin's comments and then re-request an sr from
me. thanks!
Attached patch patch according to comment#81 (obsolete) — Splinter Review
Sorry again,Yesterday I installed a leafnode to my linuxbox and did a full test
for chinese newsgroup names,I found some other related problems and did some
fixes.

changes:
changes according to comment#81

fix for newsgroupname are displayed as escaped string at compose window.
fix for cannot send mail to newsgroup because the groupname are escaped at the
compose window.
fix for not able to search in subscribe dialog window in case of chinese
newsgroup names(page subscribe.xul).
fix for not able to unsubscribe in subscribe dialog window in case of chinese
newsgroup names(exists before),which caused by the unsubscribe(name)
method,some place in .js file,called with escaped name as parameter,and
someplace called with unicode name as parameter,now only call with escaped name
(same as subscribe(name) ).
Attachment #182461 - Attachment is obsolete: true
Attachment #182761 - Flags: review?(jshin1987)
Attached patch purified patch (obsolete) — Splinter Review
hope this is the final version
now we handle newsgroupname internally as utf8 string only.
the conversion from serverSide name to internal utf8 newsgroup name only
happens when
communicate with news server.
Attachment #182761 - Attachment is obsolete: true
Attachment #183003 - Flags: review?(jshin1987)
Attachment #182461 - Flags: superreview?(mscott)
Comment on attachment 182761 [details] [diff] [review]
patch according to comment#81

i'll review the newest patch on Wednesday.
Attachment #182761 - Flags: review?(jshin1987)
*** Bug 151946 has been marked as a duplicate of this bug. ***
since thunderbird have some file duplicated in the base/content directory,
we need to add the same change for those files which we modifed in 
suite files.
Attachment #183003 - Attachment is obsolete: true
Attachment #184740 - Flags: review?(jshin1987)
Attachment #183003 - Flags: review?(jshin1987)
sorry for the delay and thanks for the update. I've gone about a half way
through. I'll make sure to finish it no later than Tuesday. Btw, can you give me
a couple of news servers with Chinese newsgroup names (in GB2312) I can access
freely? Is news.newstan.net one of them?
(In reply to comment #89)
Yes news.newsfan.net is one of them...

You can try the following servers,they are free to access: 
news://news.newsfan.net (IP: 210.12.195.2)
news://news.cnnb.net
news://zhnews.net (IP: 210.73.88.194)

Thanks again!
Fix this spacing before checkin:
   
-  attribute string  newsgroups;
+  attribute AString  newsgroups;
   attribute string  newshost;
   attribute string  newspostUrl;
   attribute string  followupTo;
(In reply to comment #91)
> Fix this spacing before checkin:
>    
> -  attribute string  newsgroups;
> +  attribute AString  newsgroups;
OK,Thanks!
Wen, I'm really sorry for my delay. I'll get back to this bug this weekend for sure.
we should get this in for b3 if at all possible.
Assignee: nhottanscp → jshin1987
Status: ASSIGNED → NEW
Flags: blocking1.8b3+
jshin, any update?
sorry again for the delay. I'll finish reviewing and testing today. there were
some hiccups in my builds on linux and mac..
It works fine on Linux (UTF-8 locale) and Mac OS X, but I just realized that it
wouldn't on Windows if the file system encoding can't cover characters used in
the names of newsgroups (e.g. Chinese newsgroup names on English Windows 9x).
Given this, it's a better idea to use UTF-7 or other ASCII-safe(and file system
safe name) for *msf and *sbd files. 

Sorry for the delay and not having pointed this out earlier. If you can make a
quick change, that would be great and I'll review as quickly as possible.


 
(In reply to comment #97)

> Given this, it's a better idea to use UTF-7 or other ASCII-safe(and file system
> safe name) for *msf and *sbd files. 

There are utility functions for modified UTF-7 in nsMsgI18n.h 
 
Comment on attachment 184740 [details] [diff] [review]
patch for both mozilla suite and thunderbird.

r=jshin with the following minor issues addressed. 
Sorry for the delay.

Let's put off the issue I raised yesterday for a while unless you can resolve
it quickly as I outlined. There'd be a migration issue, but not many users will
be affected by it.

 There are some functions that are always supposed to get 'UTF-8' |char *|
strings. Please, add a comment to that effect wherever that's the case.
When you add / modify lines longer than 80 columns, try to split them
into shorter lines (I noted some of them below, but not all of them).


> Index: mail/base/content/mailCommands.js
>   var subscribableServer =  server.QueryInterface(Components.interfaces.nsISubscribableServer);
> - subscribableServer.unsubscribe(folder.name);
> + subscribableServer.unsubscribe(encodeURI(folder.name));

  Perhaps, not in this bug, but later we want to change nsISubscribable
interface to make it more consistent.. Currently, it takes |wstring|
for |(un)subscribe|, but takes |string| for other methodes, which I guess
is why you just use url-escaping here.

> Index: mailnews/compose/src/nsMsgCompUtils.cpp
> +    // fixme:the newsgroups header must be encoded as the server-side character encoding
> +    // but this |charset| might be different from it.
> +    ENCODE_AND_PUSH("Newsgroups: ", PR_FALSE, newsgroupsHeaderVal.get(), charset, PR_FALSE);

 Actually, as long as RFC 2047 is used, it should be fine in theory/principle.
(modern standard-compliant newsreaders like TB wouldn't have any problem)
Given that, you may wanna tone down your comment ('must' is too strong.
'had better' may be about right)


> @@ -2182,22 +2182,22 @@ NS_IMETHODIMP QuotingOutputStreamListene
....
> -              compFields->SetNewsgroups(nsnull);
> +              compFields->SetNewsgroups(NS_LITERAL_STRING(""));

   Pls, use EmptyString() instead of NS_LITER...


> Index: mailnews/compose/src/nsMsgComposeService.cpp
> -          pMsgCompFields->SetNewsgroups(aNewsgroup);
> +          pMsgCompFields->SetNewsgroups(NS_ConvertUTF8toUTF16(aNewsgroup));

Perhaps, not here but later we may consider using |AUTF8String| for
|nsIMsgCompFields::newsgroup| so
that we can avoid the conversion. You have a dozen of them like this.

> Index: mailnews/news/public/nsIMsgNewsFolder.idl
> +  readonly attribute AString unicodeName;
> +  /**|rawName| is an 8-bit string to represent the name of a newsgroup used by a
> +   * news server. It's offered for the convenience of callers so that they don't
> +   * have to convert |unicodeName| to the server-side name when communicating with a

....... snip ...
> +   * because that's a server-wide property.

  Please, make this comment fit well within 80 columns


> Index: mailnews/news/public/nsINntpIncomingServer.idl
> +    /*  the server charset and it may be needed to display newsgroup folder names correctly */
> +    attribute ACString charset;

  Same here..

> Index: mailnews/news/src/nsNNTPProtocol.cpp
> -         PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) current group = %s, desired group = %s", this, m_currentGroup.get(), newsgroupName.get()));
> +            PR_LOG(NNTP,PR_LOG_ALWAYS,("(%p) current group = %s, desired group = %s", this, m_currentGroup.get(), newsgroupName.get()));

  Will you please use the string concatenation to split the above line
into two 'shorter-than-80 columns' lines?

> -         else
> -           m_nextState = NNTP_SEND_GROUP_FOR_ARTICLE;
> +            if (!PL_strcmp(m_currentGroup.get(), newsgroupName.get()))
> +              m_nextState = NNTP_SEND_ARTICLE_NUMBER;
> +            else
> +              m_nextState = NNTP_SEND_GROUP_FOR_ARTICLE;

  This file is rather messy in terms of indentation. It seems like
you used 4 column indentation earlier in the file. Will you do the same
here?

>      if (NS_SUCCEEDED(rv)) {
> -      m_nntpServer->GroupNotFound(m_msgWindow, name.get(), PR_FALSE);
> +      m_nntpServer->GroupNotFound(m_msgWindow, NS_ConvertUTF16toUTF8(name).get(), PR_FALSE);
>      }

   split the above line to two or three lines and get rid of braces.

   m_nntpServer->GroupNotFound(m_msgWindow,
			       NS_ConvertUTF16toUTF8(name).get(),
			       PR_FALSE);

> Index: mailnews/news/src/nsNewsDownloadDialogArgs.cpp
> +NS_IMETHODIMP nsNewsDownloadDialogArgs::GetGroupName(nsAString & aGroupName)
> + {
> +    aGroupName = mGroupName;
> +
> +     return NS_OK;

   nit: one column off here  for |return ....|

> Index: mailnews/news/src/nsNewsFolder.cpp
>
> -  nsXPIDLCString newsgroupname;
> -  rv = GetAsciiName(getter_Copies(newsgroupname));
> +  nsXPIDLString newsgroupNameUtf16;
> +  nsXPIDLCString newsgroupName;
> +  rv = GetName(getter_Copies(newsgroupNameUtf16));
>    if (NS_FAILED(rv)) return rv;
> -
> +  CopyUTF16toUTF8(newsgroupNameUtf16, newsgroupName);

  don't need |nsXPIDLCString newsgroupName|. Instead, just do :

     NS_ConvertUTF16toUTF8 newsgroupName(newsgroupNameUtf16);

> + if(serverCharset.IsEmpty()){
> +     nsXPIDLString defaultCharset;
> +     rv = NS_GetUnicharPreferenceWithDefault(nsnull,PREF_MAILNEWS_VIEW_DEFAULT_CHARSET,

 nit: split a long line

    rv = NS_GetUnicharPreferenceWithDefault(nsnull,
					    PREF_MAILNEWS_VIEW_DEFAULT_CHARSET,

					    NS_LITERAL_STRING("ISO-8859-1"),
					    defaultCharset);

> +#ifdef DEBUG_NEWS
> +     printf("default charset for the server is %s\n", (const char *)defaultCharset);

	printf("default charset for the server is %s\n",
	       (const char *)defaultCharset);

> @@ -806,17 +844,17 @@ writeGroupToHostInfoFile(nsCString &aEle
>      nsAutoString name;
>      nsresult rv = NS_MsgDecodeUnescapeURLPath(aElement, name);
>      if (NS_FAILED(rv)) {
>          // stop, something is bad.
>          return PR_FALSE;
>      }

>      nsCAutoString nameOnDisk;
> -    nameOnDisk.AssignWithConversion(name.get());
> +    CopyUTF16toUTF8(name, nameOnDisk);

       NS_ConvertUTF16toUTF8 nameInNewsrc(name);


>      // XXX todo ",,1,0,0" is a temporary hack, fix it
>      *stream << nameOnDisk.get() << ",,1,0,0" << MSG_LINEBREAK;

       *stream << nameInNewsrc.get() << ",,1,0,0" << MSG_LINEBREAK;

> + rv = nsMsgI18NConvertToUnicode(dataCharset.get(), nsDependentCString(aName), newsgroupName);

    rv = nsMsgI18NConvertToUnicode(dataCharset.get(),
				   nsDependentCString(aName),
				   newsgroupName);

> @@ -1204,24 +1248,19 @@ nsNntpIncomingServer::Unsubscribe(const

> -  rv = serverFolder->FindSubFolder(escapedName, getter_AddRefs(newsgroupFolder));
> +  rv = serverFolder->FindSubFolder(NS_LossyConvertUCS2toASCII(aUnicharName), getter_AddRefs(newsgroupFolder));

Please, add a comment as to why it's safe to use a lossy converter here
(because aUnicharName is already escaped ? )

     rv = serverFolder->FindSubFolder(
			NS_LossyConvertUTF16toASCII(aUnicharName),
			getter_AddRefs(newsgroupFolder));
Comment on attachment 184740 [details] [diff] [review]
patch for both mozilla suite and thunderbird.

forgot marking my 'r+'. asking David for sr. 

David, what do you think of putting off implementiong what I wrote in comment
#97 and comment #98?
Attachment #184740 - Flags: superreview?(bienvenu)
Attachment #184740 - Flags: review?(jshin1987)
Attachment #184740 - Flags: review+
If it's just English Windows 9x, i.e. win 95 and win 98, and it was broken
everywhere before, then this would be a big improvement. But, is there a clear
migration strategy? What data files are affected by this change? newsrc.dat?
hostinfo.dat? Is the format only changed if the user changes the server char set?

Also, have we made sure this works with imap folders with special characters?
Imap uses subscription as well, though the strings should all be imap mod-utf8,
iirc.

Comment on attachment 184740 [details] [diff] [review]
patch for both mozilla suite and thunderbird.

sr=bienvenu, as long as imap has been tested.
Attachment #184740 - Flags: superreview?(bienvenu) → superreview+
newsrc/hostdata files are not affected (they use UTF-8), but *msf and *sbd
filenames are affected. With this patch, they're stored in the local file system
encoding. On Mac OS X and modern Linux - by default -, it's UTF-8 so that
there's no problem. However, on other OS' where the file system encoding is not
Unicode-based (including all versions of Windows. I should have said this
although Windows 2k/XP will not have this problem once bug 162361 is fixed and
mailnews moves to nsI*File from nsIFileSpec), there's a problem with non-ASCII
newsgroup names (as long as Simp. Chinese windows with SC system locale is used,
there's no problem). To support those OS', using modified UTF-7 for *msf and
*sbd (as we do for IMAP) is the best, but I don't have time to figure that out
myself. If Shaohua can do that quickly, that'll be great.

As for the migration, that wouldn't be smooth for those who subscribe to
newsgroups with non-ASCII names (currently, only *stand-alone* Chinese
newsservers in China use them and I don't expect other news servers to do that).
They will have to unsubscribe and subscribe again, I guess because *msf and *sbd
file names will be changed. So, this may be a rather significant issue that can
block this patch from landing. If you believe this migration issue is too
important to let this patch land, we shoudln't check in this patch without
addressing the problem.

Another important problem: subscribe/unsubscribe with IMAP. Thanks for pointing
that out. As long as modifiued UTF-7 doesn't use characters subject to
url-escaping, that should be fine, but I want to make sure of that. Perhaps,
there are two better alternatives: 1) call encodeURI only for newsserver if it's
possible to tell the current server is a newsserver (is it possible now? if not,
we may add a method to nsISubscribableServer and a few constants) 2) even better
is not to use encodeURI at all (see the 4th paragraph of my comment #99)
Comment on attachment 184740 [details] [diff] [review]
patch for both mozilla suite and thunderbird.

is this patch ready to check in?
Attachment #184740 - Flags: approval1.8b3?
Attachment #184740 - Flags: approval-aviary1.1a2?
the patch needs a bit more testing to make sure it doesn't break imap subscribe 
(In reply to comment #105)
> the patch needs a bit more testing to make sure it doesn't break imap subscribe 

Characters like '%' and '^' can represent themselves in modified UTF-7, but
they'll be url-escaped by encodeURI so that the patch as it stands breaks imap
subscription to folders with names including such characters.
<quote : RFC 2060>
   In modified UTF-7, printable US-ASCII characters except for "&"
   represent themselves; that is, characters with octet values 0x20-0x25
   and 0x27-0x7e.  The character "&" (0x26) is represented by the two-
   octet sequence "&-".
</quote>

David, which way do you prefer?

  1) add a property ('enum type') to nsISubscribableServer so that we can query
whether it's a newsserver or not and use encodeURI only for newsservers.
  2) use modified-UTF-7 for *sbd and *msf files for news

1) is easier (for me) than 2), but 2) is a better long-term solution than 1).
Besides, if we go with 2), we won't have any migration problem except for
newsgroup names with '&' (hmm... is there any?) 

how much harder is 2)? I don't believe '&' is legal in newsgroup names...
At first (when I raised the issue for the first time), I thought it'd be rather
easy, but about an hour of poking around didn't lead me where changes need to be
made. Can you tell me where filenames for '*msf' and '*sbd' are 'constructed'
for newsgroups and where they're referred to if you know the answer off the top
of your head? 
mailnews/base/util/nsNewsSummarySpec.cpp is a good place to start...
please please please please please please don't use % in file or foldernames.
they cause serious issues for windows where % is the (env)variable marker.
(In reply to comment #110)
> please please please please please please don't use % in file or foldernames.
> they cause serious issues for windows where % is the (env)variable marker.

This patch doesn't use '%' in filenames. Nor will any future patch for this bug
as long as newsgroup names don't have '%' ('%' is not allowed for newsgroup
names afaik.). Note that '%' is not a part of Base64 alphabet (used in modified
UTF-7)


(In reply to comment #109)
> mailnews/base/util/nsNewsSummarySpec.cpp is a good place to start...

thanks !
Whiteboard: [cb] ready to land for 1.8b3?
Comment on attachment 184740 [details] [diff] [review]
patch for both mozilla suite and thunderbird.

withdrawing my sr and obsoleting patch, so we can deal with the imap issue.
Attachment #184740 - Attachment is obsolete: true
Attachment #184740 - Flags: superreview+ → superreview-
Attachment #184740 - Flags: approval1.8b3?
Attachment #184740 - Flags: approval-aviary1.1a2?
I think this is the simplest way to use utf7 for .msf .dat file names,but 
it impact not only news,but local/imap also.
Dear all,
  I have no access to any imap server with non-ascii folders.Could any one help
to test the unsubscribe function with this kind of imap server?
that's a very strange place to put that code, and I don't want to change the
.msf file names for anything but news, if we can help it.
(In reply to comment #115)
> that's a very strange place to put that code, and I don't want to change the
> .msf file names for anything but news, if we can help it.

I tried override this ParseURI method in nsNewsFolder.cpp by declare a same
method in nsNewsFolder.h,and add this convert2mutf7 line,but it seems not
working(the method in nsNewsFolder is never called...).
Everytime this method was called,it will update the mPath,which is the key piont
of the .msf .dat file names.
Could you please point me out a way?
Thanks!
(In reply to comment #115)
> that's a very strange place to put that code, and I don't want to change the
> .msf file names for anything but news, if we can help it.

OK,I figured it out.
What I need do is make the parseURI method a virtual one in nsMsgDBFolder.h
and rewrite it on nsNewsFolder.h&nsNewsFolder.cpp.
tested and works fine.
I will upload the full patch after I can sync with the mozilla.org's cvs server.
(In reply to comment #112)
> (From update of attachment 184740 [details] [diff] [review] [edit])
> withdrawing my sr and obsoleting patch, so we can deal with the imap issue.
> 

I installed a cyrus imap server and tested subscribe/unsubscribe with chinese 
names  folders ,works fine.
(In reply to comment #118)

> > withdrawing my sr and obsoleting patch, so we can deal with the imap issue.
> > 
> 
> I installed a cyrus imap server and tested subscribe/unsubscribe with chinese 
> names  folders ,works fine.

Did you get rid of |encodeURI| in subscribe/unsubscribe? See comment #106. 
closing down for 1.8b3, let's try and get this in for 1.8b4
Flags: blocking1.8b4+
Flags: blocking1.8b3-
Flags: blocking1.8b3+
Whiteboard: [cb] ready to land for 1.8b3?
(In reply to comment #119)
> Did you get rid of |encodeURI| in subscribe/unsubscribe? See comment #106. 
> 

I tried to create a folder contains % or ^,thunderbird just closed the "create
folder" dialog 
and nothing happens.
tried evolution also,it says that % or ^ is invalid.
what's the problem?
(In reply to comment #121)
> (In reply to comment #119)
> > Did you get rid of |encodeURI| in subscribe/unsubscribe? See comment #106. 
> > 
> 
> I tried to create a folder contains % or ^,thunderbird just closed the "create
> folder" dialog 
> and nothing happens.
> tried evolution also,it says that % or ^ is invalid.
> what's the problem?

 At least '^' is allowed and it's Evolution's fault to claim that '^' is
invalid. TB does allow you to create a new IMAP folder with '^' in its name. 

I think nsISubscribale needs to be changed  as I mentioned in comment #99. It's
quite a lot of job, but it can be done in an ad-hoc fashion (with an
understanding that it'll be revisited before long. perhaps after 1.1branch is
cut?) so that this bug is fixed properly. I'm making that change now.
 
Attachment #187989 - Flags: review?(jshin1987)
Comment on attachment 187989 [details] [diff] [review]
full patch to use Modified UTF7 for .msf and .dat file names.


>-	subscribableServer.unsubscribe(folder.name);
>+	subscribableServer.unsubscribe(encodeURI(folder.name));

David and I already wrote that this would break IMAP.
Attachment #187989 - Flags: review?(jshin1987)
I'll try to upload a new patch combining Shaohua's patch with what I wrote in
comment #123 soonish.

Status: NEW → ASSIGNED
(In reply to comment #123)
>  At least '^' is allowed and it's Evolution's fault to claim that '^' is
> invalid. TB does allow you to create a new IMAP folder with '^' in its name. 
> 
> I think nsISubscribale needs to be changed  as I mentioned in comment #99. It's
> quite a lot of job, but it can be done in an ad-hoc fashion (with an
> understanding that it'll be revisited before long. perhaps after 1.1branch is
> cut?) so that this bug is fixed properly. I'm making that change now.
>  
I tried other charactors also,not only % ^ are forbidden,also
 ! @ # $ & * ( ) are forbidden in the folder name (TB just close the "New
Folder" Dailog and nothing happens).
Could you please try to verify that?


(In reply to comment #124)
> (From update of attachment 187989 [details] [diff] [review] [edit])
> 
> >-	subscribableServer.unsubscribe(folder.name);
> >+	subscribableServer.unsubscribe(encodeURI(folder.name));
> 
> David and I already wrote that this would break IMAP.
> 

If those charactors are not allowed,then ,this code will not break IMAP,right?
See http://www.faqs.org/rfcs/rfc3501.html (RFC 3501)
I did what I wrote I would in comment #99. It turned out to have quite an
extensive ripple effect across mailnews/. I can produce imap folders with '^'
and other special characters (but not '%'...) as well as non-ASCII names.
Subscribed group names are now stored in UTF-8 without escaping, but there are
still some places escaped groupnames are used. I wanted to get rid of them,
too, but there	is a 'bytewise inflation' of UTF-8 to UTF-16 (with ASCIItoUTF16
or AssignWithConversion) which I can't pinpoint
(In reply to comment #115)
> that's a very strange place to put that code, and I don't want to change the
> .msf file names for anything but news, if we can help it.

My patch also touches parseURI, but changes msf filenames only for news/snews (I
just added to parseURL in nsMsgDBFolder a check as to whether a scheme is
news/snews/nntp without making it virtual and overriding it). 

Btw, nsNewsSummarySpec is only used when deleting msf files for unsubscribed
groups so that changing it doesn't seem to work. 

Comment on attachment 188182 [details] [diff] [review]
Shaohua's patch plus mine

David, can you take a look? The size of this patch is a bout twice as large as
before, but that's primarily because I replaced |string| in two interface files
with |AUTF8String| or |AString|.

Btw, two chunks in nsMsgCompose.cpp are not parts of this patch and I got rid
of them in my tree.
Attachment #188182 - Flags: review?(bienvenu)
Jungshik, it basically looks good. Thx for doing this. There are a few places
where you've just commented out old lines of code - those can just be removed, I
think.

Also,
+  [noscript] readonly attribute ACString rawName;

this doesn't have to be noscript, does it?

+    // fixme: the newsgroups header had better be decoded using the server-side
+    // character encoding,but this |charset| might be different from it.

is this draft fixme going to be a problem for these GB132 servers?



+            // to handle non-ASCII newsgroup names, we store them internally
+            // as escaped. decode and unescape the newsgroup name so we'll
+            // display a proper name.
+
             nsAutoString unescapedName;
             rv = NS_MsgDecodeUnescapeURLPath(group, unescapedName);
+            //NS_ConvertUTF8toUTF16 unescapedName(group);

is this part finished, or is NS_MsgDecodeUnescapeURLPath doing this conversion
now, so the commented out line can be removed?

a -uw diff would probably be easier - the news code is filled with tabs and bad
indentation, so thanks for cleaning that up.


remove this, and the other dumps: 
+  dump("ReverseStateFromNode: name=" + name);

Do you want me to try this patch?
(In reply to comment #131)
> Jungshik, it basically looks good. Thx for doing this. There are a few places
> where you've just commented out old lines of code - those can just be removed, I
> think.

Yes, I've just done that in my tree and I'll upload a new '-uw' patch for you.

 
> +  [noscript] readonly attribute ACString rawName;
> 
> this doesn't have to be noscript, does it?

  If we want to allow it in scripts, we have to add a very prominent warning
that what one get in JS is a JS string made up of 'byte-extended' characters
(i.e. if a string is '0xac 0xb0' in GB2312, what you get in JS is 'U+00AC'
'U+00B0' instead of a single Unicode character corresponding to '0xac 0xb0' in
GB2312). This can be reverted back to the proper JS string using
nsIScriptableUnicodeConverter, but that's a long and winding alternative to a
lot simpler |newsFolder.unicodeName| :-)
 
> +    // fixme: the newsgroups header had better be decoded using the server-side
> +    // character encoding,but this |charset| might be different from it.
> 
> is this draft fixme going to be a problem for these GB132 servers?

If someone wants to post to a newsgroup on one of these servers (with GB2312
used on their ends) in an char. encoding other than GB2312-compatible ones,
it'll be a problem. I guess that'll be very rare and we can get away with it for
now. 


>              rv = NS_MsgDecodeUnescapeURLPath(group, unescapedName);
> +            //NS_ConvertUTF8toUTF16 unescapedName(group);
> 
> is this part finished, or is NS_MsgDecodeUnescapeURLPath doing this conversion
> now, so the commented out line can be removed?

 Yes, I'll remove it.


> remove this, and the other dumps: 
> +  dump("ReverseStateFromNode: name=" + name);

I'll remove dump's as well.
 
> Do you want me to try this patch?

Yes, that'll be nice. I've done some tests and haven't found a problem, but it's
not thorough. 

I got rid of commented out lines and dump() statements in JS per David's
comment. attachment 188182 [details] [diff] [review] should be used to patch (for testing) while this is
for an easier reading (-uw patch)
(In reply to comment #133)
> Created an attachment (id=188215) [edit]
mailnews/base/public/nsISubscribableServer.idl
     // for searching
-    void setSearchValue(in string searchValue);
+    void setSearchValue(in AString searchValue);
should it be AUTF8String here also? does it works with non ascii name search here?
AString and AUTF8String are both intl-safe. (as opposed to ACString and string)
re the noscript, it sounds like you have an excellent reason for making it
[noscript], so that's fine. You could add a brief comment to the idl for that.

I'll try the patch tomorrow and then put my r= on it.
Comment on attachment 188215 [details] [diff] [review]
Shaohua's patch plus mine  (-uw patch) with dump/commented line removed

sorry for all the nits; rebuilding is taking a long time :-) some nits:

+      if (NS_SUCCEEDED(url->GetScheme(scheme))) {
+	 isNewsFolder = scheme.EqualsLiteral("news") ||  

can you either use the prevailing braces style in this file,
if (a)
{
  ..
}

or no braces at all.

similarly, in nsMsgUtils.cpp, the prevalent braces style isn't K&R.

re this change, are we still doing case-insensitive search, or did we drop
that? It's unclear to me why the same server would return the same newsgroup
with difference cases. Actually, looks like we're using
nsCaseInsensitiveStringComparator() down the road, so we're OK?

-    // force the search string to be lower case
-    // so that we can do case insensitive searching
-    ToLowerCase(mSearchValue);

+nsMsgNewsFolder::GetUnicodeName(nsAString & name)
+{
+  nsresult rv;
+  nsXPIDLString newsgroupName;
+  rv = GetName(getter_Copies(newsgroupName));

here, you can move the decl of rv to where you first use it. And tiny nit,
instead of using NS_ENSURE_SUCCESS(rv, rv), you could do

if (NS_SUCCEEDED(rv))
  name = newsgroupName;
return rv;

-  uri.Append(escapedName.get());
+  uri.Append(escapedName);
+  //uri.Append(name);

missed a commented out line here.

rev the uuid on nsINntpIncomingServer.idl


there are a number of places where you've changed code that wasn't using the
interface parameter name standards - while you're in there, you could change
the arg names. e.g., setAsSubscribed(const nsACString &path) should be
setAsSubscribed(const nsACString &aPath);

The only nit that needs to be fixed is rev'ing the uuid on
nsINntpIncomingServer.idl
Attachment #188215 - Flags: review+
Comment on attachment 188215 [details] [diff] [review]
Shaohua's patch plus mine  (-uw patch) with dump/commented line removed

asking for sr.
I addressed David's nits except for the parameter renaming. I changed some of
them to be compliant to our naming convention, but not all of them.
Attachment #188215 - Flags: superreview?(mscott)
I did some quick tests with imap subscribe of folders with non-alphanumeric
characters and it worked fine.
David, do you think we should hold off on this patch until Alpha 2 gets out the
door?
I think we might as well go for it so this patch can get testing. The imap risk
is actually pretty low. And the non-chinese newsgroup risk is probably pretty
low as well.
Comment on attachment 188182 [details] [diff] [review]
Shaohua's patch plus mine

clear r? flag because David already reviewed the -uw patch
Attachment #188182 - Flags: review?(bienvenu)
Comment on attachment 188215 [details] [diff] [review]
Shaohua's patch plus mine  (-uw patch) with dump/commented line removed

Now that the trunk is open for beta work, now is a good time for us to get this
in. 

Instead of putting this inline with the XUL file we should instead just put it
inside of am-server.js, preferrably inside of either the onInit method or the
onPreInit.

Please make that change before checking in. Thanks!
Attachment #188215 - Flags: superreview?(mscott) → superreview+
Comment on attachment 188215 [details] [diff] [review]
Shaohua's patch plus mine  (-uw patch) with dump/commented line removed

thanks for r/sr. 
I made a change mentioned by mscott in my tree.  (I put it in onPreInit() )

This blocks both 1.8b4 and 1.1alpha and both David and Scott want this to be
tested by landing it. My test result indiciates that it works fine, but letting
others test it will help fix any possible regression.
Attachment #188215 - Flags: approval1.8b4?
Attachment #188215 - Flags: approval1.8b4? → approval1.8b4+
BTW: A bustage fix was needed for this patch, a change to a Makefile was missing
(are you normally building with enable-static?). Timeless probably checked in
the correct for this (it's still building), see
http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&subdir=mozilla/mailnews/news/src&command=DIFF_FRAMESET&file=Makefile.in&rev1=1.52&rev2=1.53&root=/cvsroot
for the change.
thanks for the note. For debug builds, I don't use 'enable-static'. I'll be more
careful with this. I'll mark this as fixed when windows build goes through with
timeless' patch.

Whiteboard: [landed, needs testing]
jshin, after I updated my build from the 19th to todays build (the 25th), I'm
getting problems with my virtual folders.

Virtual folders with spaces in them like:

Review Requests

are showing up twice in my folder pane. Once as:
Review Requests
and once as
Review%20Requests

related?
Found a bug of the patch...
  In Thunderbird 1.0+ zh-CN builds,we have set mailnews.view_default_charset to
GB2312,but when I added a new news server,the default charset for this server is
set to: chrome://messenger/locale/messenger.properties but not GB2312,I checked
mailnews.js
/mailnews/mailnews.js, line 215 -- pref("mailnews.view_default_charset",
"chrome://messenger/locale/messenger.properties");
  Is this the reason or we should change NS_GetUnicharPreferenceWithDefault(...)
in nsNntpIncomingServer::GetCharset(...) to
NS_GetLocalizedUnicharPreferenceWithDefault ?
(In reply to comment #148)
> jshin, after I updated my build from the 19th to todays build (the 25th), I'm
> getting problems with my virtual folders.
> 
> Virtual folders with spaces in them like:

> are showing up twice in my folder pane. Once as:
> Review Requests
> and once as
> Review%20Requests
> 
> related?

I thought it might be, but a cursory look didn't lead to any suspicious spot.
Moreover, my debug builds (one I made right after landing the patch and the
other I've just built with the trunk source) don't have such a problem with a
virtual folder whose name has a space. I diffed my tree against the cvs tree and
I couldn't find anything likely to cause this difference between you and me. 
Aha... I made a virtual folder with a build with this patch included. I'll make
a virtual folder with an older build and run a recent build after that.

re: comment #149
yeah, we should use NS_GetLocalizedUnicharPreferenceWithDefault
file a new bug, upload a patch and i'll quickly review it.



(In reply to comment #150)

> yeah, we should use NS_GetLocalizedUnicharPreferenceWithDefault
> file a new bug, upload a patch and i'll quickly review it.
> 

filed a new Bug 302275 .
Scott, I've just tried 2005-07-25 nightly, but couldn't reproduce the problem
you experienced. I made a virtual folder from a search result (with TB 1.0.2).
Is there any other way? 
Status: ASSIGNED → RESOLVED
Closed: 19 years ago
Resolution: --- → FIXED
*** Bug 147823 has been marked as a duplicate of this bug. ***
Hi shin,
  It seems that there is a regression on this bug in TB1.5.04.
  Please have a look on the screenshot.The tree display is broken.
Thanks
Holy
Dear all, This regression happens on 2.0 develop branch also.should we reopen this bug?
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: