260034 - Cannot send mail if temporary directory contains non-ascii characters

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[Met - Martin Hassman]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; cs-CZ; rv:1.7.3) Gecko/20040910
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; cs-CZ; rv:1.7.3) Gecko/20040910

If temporary directory contains non-ascii characters, Thunderbird cannot send
mails, because it cannot create temporary nsmail.* files.

Observer in TB0.8.
Observed only in Windows XP Professional CZ
Non-observed in Windows 98 CZ and Windows 2000 CZ.

This is bad, because a lot of beginer users in Czech have these characters in
the path (e.g. in login name).

Reproducible: Always
Steps to Reproduce:
1. Set you TMP and TEMP variables to path containing non-ascii characters
2. open Thunderbird
3. create and send mail

Actual Results:  
Appeares alert:

"Sending of message failed
Unable to open the temporary file
... path nsmail.html..... Check your 'Temporary Directory' settings.

In the alert box there is displayed complete path to the file, but in place of
non-ascii characters it containes bad characters (screenshot is comming).


Important note:

This bug was presented also in the Mozilla 1.7.3 (and older), but it is not
presented in the Mozilla 1.8a3. I have searched bugzilla, but wasn't abel to
find appropriate Mozilla bug.

Please can you fix it in the Thunderbird too. It realy affects many beginner
user which have non-asci characters in their login names.

Comment 1

[Met - Martin Hassman]

Attachment: screenshot of temp variable setting

First screenshot

Comment 2

[Met - Martin Hassman]

Attachment: Alert in TB0.8 CZ

Other screenshot

Comment 3

[Met - Martin Hassman]

Attachment: Alert in M1.7.3 (english version ;-)

Other screenshot.

Comment 4

[gregor]

I'm using TB0.8 on a german W2K Professional and have the same problem.
My temporary directory contains an "ü" (=u Umlaut, 0x81) character which is
mapped to the "¼" (one quarter, 0xAC) character.

Comment 5

[u60234]

Confirmed with branch build 2004-10-24 on WinXP with username "Björn".

Comment 6

[Scott MacGregor]

taking for 0.9.

I'm pretty sure this was fixed by some part of jshin's patch for Bug #229032 the
question is which part. Let's start with the changes to nsMsgSend.cpp.

I need some volunteers who can test a possible fix in the current 0.8 nightlies.
Any takers? Martin? If so, then I'll check in a potential fix.

Comment 7

[Scott MacGregor]

Attachment: possible fix

Comment 8

[Scott MacGregor]

I just checked in this potential fix. If one of you can see if this does indeed
fix it in the 10/26 builds that come out tomorrow I would really appreciate it!

Comment 9

[Met - Martin Hassman]

>Any takers? Martin? If so, then I'll check in a potential fix.

I can test it.

Cannot found 10/26 builds in the
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/ now they are still
building? (I have 26.10 8.30h now)

Comment 10

[u60234]

Sorry, I just tried todays build 2004-10-26-07, and the bug is still there.

Mozilla 1.8a1 (2004-05-20) have this bug, but 1.8a2 (2004-07-14) do not. I will
try to narrow down this window by testing some builds from archive.mozilla.org. 

Comment 11

[u60234]

Mozilla trunk build 2004-06-12-08 has the bug, 2004-06-25-08 is fixed.

I can't narrow it down any further than that, can't find any trunk builds to
test between those dates.

Comment 12

[Met - Martin Hassman]

>Cannot found 10/26 builds in the

OK (they are building it the bad time for me in Europe 8-().
I have just tested
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-0.9/thunderbird-win32.zip
"26-Oct-2004 11:41". But there is no improvement. Same error message.

Comment 13

[Met - Martin Hassman]

Attachment: Screenshot from the Filemon

If it can help, here is screenshot from the Filemon program - log of failed dir
readings.

Comment 14

[Jungshik Shin]

reporter, what's the default system locale of your Windows 2k/XP? If the default
system code page doesn't support letters in question (e.g. if your system locale
is for Western Europe instead of Czech), Mozilla can't access (create. read,
delete, etc) files with those characters. We've been working on it to fix it,
but I don't have enough time to fix it and Brodie is on vacation. (see bug
162361 and bug 260034).

If you still have a problem after switching the default system locale to Czech
(which I doubt is the case), then we definitely have to fix it for 1.0 release.

Comment 15

[Met - Martin Hassman]

(In reply to comment #14)
> reporter, what's the default system locale of your Windows 2k/XP?

Cannot check it now (but *very probably* Windows XP *Czech* Edition with the
*Czech* locale).

But take into account - latest Mozilla (Mozilla 1.8a3 and greater) has not this
problem! The problem was fixed there. We only cannot found the critical patch. 

Comment 16

[Jungshik Shin]

Thanks for the reply. I somehow missed that you wrote 'it's not present in 1.8a3'.

As Scott wrote, I am likely to have fixed it rather recently, but I'm not sure
with which patch I did. I'll try to find it out.

Comment 17

[Jungshik Shin]

attachment 163375 [details] [diff] [review] just replaces nsMsgGetNativePathString with
NS_CopyNativeToUnicode so that it can't help. 

According to comment #0, it only happens on Win XP but not on Win 2k/9x. I
couldn't reproduce it on Windows 2003 serve, either. (I don't have Win XP
unfortunately) My check-in's in July and August didn't turn up anything related. 

Can you set your TEMP and TMP varibale to something very simple such as 'C:\Ą'
(read this with View | Encoding set to UTF-8. Please, do the same for additional
comments here and in other bugs at bugzilla.mozilla.org)? 'Ą' is U+0104 (Latin
Capital Letter A with Ogonek) whose code point in ISO-8859-2 is 0xA1 (decimal
161). Then, try to send an email and post the screenshot of an alert you get. 
Thanks.

Comment 18

[Jungshik Shin]

(In reply to comment #4) (this is in UTF-8)
> I'm using TB0.8 on a german W2K Professional and have the same problem.

Contrary to comment #0, you're observing this on Win 2k. Hmm...

> My temporary directory contains an "ü" (=u Umlaut, 0x81) character which is
> mapped to the "¼" (one quarter, 0xAC) character.

u umlaut is 0xFC in ISO-8859-1/Windows-1252 and U+00FC in Unicode. 'one quarter'
is 0xBC in ISO-8859-1/Windows-1252 and U+00BC in Unicode. Probably, you got the
code point wrong but got characters right. I believe u umlaut was turned to two
characters '¬' and '¼'(U+00AC NOT SIGN U+00BC One quarter), but you missed the
first character. If that's the case, it means somewhere 'evil'(!) 
AssignWithConversion (or CopyASCIItoUTF16) is applied to a UTF-8 string. Another
possibility is that 'Native to UTF-16' converter is applied to a UTF-8 string

Mike and Wada, can you help me with this bug? If you can test with TEMP/TMP set
to a single character and see what happens, that'll be nice. I can't reproduce
it on Win2003 server, but others have problems on Win2k or WinXP. 

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #18)

Jshin, which temp directry does Mozilla use, TEMP or TMP?
My "SET" on DOS command prompt(Win-2K) say both are set.
> TEMP=C:\DOCUME~1\Wada\LOCALS~1\Temp
> TMP=C:\DOCUME~1\Wada\LOCALS~1\Temp

Comment 20

[Jungshik Shin]

You can change both TEMP and TMP in the control panel | System | Advanced |
Environment variables. I'm not sure which one Mozilla uses. Why don't you set
them both to an identical (and simple) value and see what happens? For instance,
setting them to 'C:\DOCUME~1\Wada\LOCALS~1\Temp\<a single Japanese character>'
and testing would help. (or 'C:\<a single Japanese char>')
 

Comment 21

[WADA:World Anti-bad-Duping Agency]

Attachment: Test result when TEMP/TMP = C:\<a single Japanese char> (Text of Charset=UTF-8, no BOM)

[Tested with]
  - Japanese MS Windows 2000
  - Mozilla trunk nightly 2004102906
[Test condition]
 (1) Windows Login User ID = Single Japanese Character (0x835C in Shift_JIS)
 (2) TEMP/TMP is set to C:\<Single Japanese Character> (0x835C in Shift_JIS)
 (3) Mozilla Profile Name = Another Single Japanese Char (0x837C in Shift_JIS)
[ Test result ]
 (1) Mail compose & send
     No problem
 (2) Mail download (POP3)
     No problem
 (3) Mail save as EML
     No problem
 (4) Download by Browser
     No problem
     Thunderbird 1.7.3 was successufully downloaded from mozilla.org

See attachment for detail.
(Note : attachment is text/plain data with charset=UTF-8, no BOM)

Comment 22

[WADA:World Anti-bad-Duping Agency]

Jshin, problem on composed character?
See http://www.unicode.org/reports/tr15/

Comment 23

[Jungshik Shin]

(In reply to comment #22)
Thank you for your thorough test. 
Met-Martin, can you conduct a similar test (comment #17. it's in UTF-8)? You
wrote that you don't have a problem on Win2k but has a problem on Win XP so that
testing on Win XP is necessary.

> Jshin, problem on composed character?
> See http://www.unicode.org/reports/tr15/

No, it can't be. Windows file systems(both NTFS and FAT32) use NFC (usually. I'm
not sure if they really normalize to NFC given non-NFC strings). Mozilla uses
NFC (most of cases). On Mac OS X, NFD is used so that I took care of conversion
between NFC and NFD.

Comment 24

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #23)
> No, it can't be.

Is it true when Cyrillic on MS Windows XP?
It maybe different from Win-2K or former.
MS says that unicode is now used instead of codepage on Win XP although codepage
is supported for backword compatibility.
See "Microsoft Windows XP - Unicode and Code Pages"
(Top site in google search  with "windows xp unicode Cyrillic")
http://www.microsoft.com/resources/documentation/Windows/XP/all/reskit/en-us/Default.asp?url=/resources/documentation/Windows/XP/all/reskit/en-us/prff_mul_zwlh.asp

Someone says Cyrillic is super set of all language's characters in Unicode... :-)

Comment 25

[WADA:World Anti-bad-Duping Agency]

Met - Martin Hassman, as Jshin says in comment #23, test on MS win XP and CZ
locale is required.
Please try following test cases.
At leaset (Case-1-2) is required since you say "temp directry name problem".

(Case-1) Czech Windows user login ID (your current ID)
  (Case-1-1)  With localized "TEMP" directry name
              (This is your case)
  (Case-1-2)  With ASCII only "TEMP" directry name
              - Create directry of "C:\T1" and "C:\T2"
              - Go Control Panel/System
                - Change environment variables
                  "TEMP" = "C:\T1" , "TMP" = "C:\T2"
              - Logoff/re-login to avoid unwanted results
              - Start Mozilla
                - send mail

(Case-2) ASCII only Windows user login ID
         - Create new user with ASCII only characters
         - Logoff/login with this new ID
  (Case-2-1)  With localized "TEMP" directry name
              - Start Mozilla
                - Create profile, account, then send mail
  (Case-2-2)  With ASCII only "TEMP" directry name
              - Create directry of "C:\T1" and "C:\T2"
              - Go Control Panel/System
                - Change environment variables
                  "TEMP" = "C:\T1" , "TMP" = "C:\T2"
              - Logoff/re-login to avoid unwanted results
              - Start Mozilla
                - send mail

Comment 26

[Jungshik Shin]

(In reply to comment #24)
> (In reply to comment #23)
> > No, it can't be.
> 
> Is it true when Cyrillic on MS Windows XP?

Czech doesn't use Cyrillic letters. It uses Latin letters. Anyway, that's
irrelevant. 

> It maybe different from Win-2K or former.
> MS says that unicode is now used instead of codepage on Win XP although codepage
> is supported for backword compatibility.

Win NT4, 2k, and XP are all the same in that they're based on Unicode ('W' APIs)
while Win 95/98/ME are based on old 'A' APIs. Note that Win XP/2k/NT4 are
totally different from Win 9x/ME although they look similar on the surface.
Using Unicode does NOT mean that it uses NFD (Normalization Form D). Mac OS X
uses NFD, but Win 32 W APIs use NFC (Normalization Form C) as I wrote earlier. 

Comment 27

[Scott MacGregor]

I'll leave this on the radar for one more day but we probably won't hold 0.9 for
it if we can't come up with a fix real soon. Sounds like we need more feedback
from Met if I'm reading the comments jshin and mwada have been making recently. 

Comment 28

[Jungshik Shin]

Comment on attachment 164118 [details]
Test result when TEMP/TMP = C:\<a single Japanese char> (Text of Charset=UTF-8, no BOM)

Scott, you're right that we need more feeback from Met-Martin or Gregor. We
haven't been able to reproduce this bug on Win 2k/Win2003 server.

Comment 29

[Met - Martin Hassman]

>Scott, you're right that we need more feeback from Met-Martin or Gregor

I am sorry for the delay, I am not every day connected to the internet.

Windows XP Czech Edition with Czech Locale

Tested version:
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2004-11-01-14-0.9/thunderbird-win32.zip
01-Nov-2004 17:12

Doesn't work (same problem):
C:\Ü 
C:\Ë
C:\Ě

Does work:
C:\É

This confirms my suspicion from attachment 163543 [details] There are only some
problematic character.

From some tests here short list

Problematic chars: ÜËĚš

Correct chars: Éáú

Looking at the terminology at 
http://www.mozilla.org/quality/browser/bft/bft_browser_entities.html

So for me now seems, that only characters with some diacritic marks are
problematic. Chars like é (letters with acute) are OK, but chars like ë (letters
with diaeresis) or ě (cannot find this mark on the page) are problematic.

Comment 30

[Jungshik Shin]

Thanks for testing. It's very strange that only some letters are problematic. 
Can you make a screenshot like attachment 159194 [details] for a couple of characters that
triggers this bug? 

Comment 31

[Scott MacGregor]

no holding up 0.9 for this now. will leave open since it sounds like some good
discussion is going on.

Comment 32

[WADA:World Anti-bad-Duping Agency]

Attachment: Screen Shot of "Save as" Error, DIR in Codepage 932,850,437

Problem was probably recreated on Japanese MS Win-2K.
Mozilla Browser failed to "Save Image As" to C:\X\Ü directry.

I created C:\X\Ü directry (first 'does'nt work' example in Comment
#29) by next procedure.
 (1) Select text of Ü on Mozilla, then CTRL+C(copy to clip board)
     (Mozilla auto-char detection says this page is Windows-1252)
 (2) On MS DOS Command Prompt
      (2-1) C:
      (2-2) MD X (To avoid other problems - GIMP for Win crash etc.)
      (2-3) CD X
      (2-4) MD <Paste above test from Edit menu>
      (2-5) Enter -> Directry is created
     See screen shots in attachment for DIR listing in Codepage 932,850 and
437.	  
 (3) "Save Image As" on Mozilla (2004110104,Win-2K)
     Save error occured. (See screen shot in attachment)

Comment 33

[WADA:World Anti-bad-Duping Agency]

Additional information.
Thunderbird 11/01 build could handle mail folder name of Ü normaly, but created
file set of e13a6a03/e13a6a03.msf.
This indicates that Thunderbird considered one of the folder name characters as
illegal file name character.

Comment 34

[Jungshik Shin]

Wada, your latest comments are not related to this bug. You're just seeing bug
162361 because two characters in question are not covered by Shift_JIS. 

Comment 35

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #34)
> Wada, your latest comments are not related to this bug. You're just seeing bug
> 162361 because two characters in question are not covered by Shift_JIS. 

Sorry, I missed your Comment #17.
I could now create folders of Eacute(&#201), Uuml(&#220) etc. by changing
character coding to UTF-8 before copy&paste.
And Mozilla's "Save Image As" failed to save in C:\X\<U Umlaut> directry too.
Does it mean recreation test on Japanese MS Win is impossible?
Or trying of set TMP=C:\<U Umlaut> on Japanese MS Win is meanigfull?

Comment 36

[Jungshik Shin]

(In reply to comment #35)

> And Mozilla's "Save Image As" failed to save in C:\X\<U Umlaut> directry too.

Gee, that's bug 162361 as I wrote before. The character (U Umlaut) is not
covered by Shift_JIS (your default system locale is Japanese and the default
system code page is 932 - SJIS)

> Does it mean recreation test on Japanese MS Win is impossible?
> Or trying of set TMP=C:\<U Umlaut> on Japanese MS Win is meanigfull?

You can change the default system locale to Czech in the control panel -
regional & language setting. On Windows XP, it's labelled as the code page to
use for non-Unicode-based programs. On Win2k, it's called 'default system
locale' or something like that. You have to reboot after the change. 

Comment 37

[Jan Matousek]

Same problem, Win XP Czech SP2, build 2004-11-02.

On my system mozilla use TMP env. var, and dont carry about TEMP.

I found funny workaround how to set TMP to dir containing these problematic
characters: set the TMP var to the 8.3 equivalent, on my system the 8.3
equivalent doesnt contain ugly characters :) Mabye this help somebody until it
will be properly fixed.

T:\>dir /X
<DIR>   PERNLU~1     pψνernμluouθkύkωςϊpμlοαbelkισdy

T:\>set TMP=T:\pψνernμluouθkύkωςϊpμlοαbelkισdy
T:\>"C:\Program Files\Mozilla Thunderbird\thunderbird.exe"

mail cannot be sent, the error ocurs

T:\>set TMP=T:\PERNLU~1
T:\>"C:\Program Files\Mozilla Thunderbird\thunderbird.exe"

this is ok, mail is sent

Comment 38

[Met - Martin Hassman]

Attachment: Alerts with problematic chars

Here is screenshots.

3 alerts - in every alert you can see written bold character = the problematic
charecters in the patch as it should look.

Comment 39

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #38)

I tested "TEMP=C:\X\<U Umlaut>" and "TMP=C:\X\<U Umlaut>" on Japanese MS
Win-2K(Japanes locale) and I found that "C:\X\U" was created when download by
Mozilla browser but was not created by Mozilla mail&news.
This is possibly related to this bug. 

Test result is as follows.
 (0) (Setups)
    (0-1)Change character coding of this bug's page to UTF-8
    (0-2)Copy <U Umlaut> in this page to clip board(CTRL+C)
    (0-3)On MS DOS prompt, make directry of <U Umlaut> using paste of Edit menu
    (0-4)Set TEMP/TMP environment variable to "C:\X\<U Umlaut>"
    (0-5)Logoff/re-login MS Win user ID
         - Locale = Japanese
         - MS Win login ID = 0x835C in shift JIS   
         - Mozilla Profile Name = 0x837C in shift JIS   
 (1) (Case-1)
    (1-1) Delete C:\X\U if exists
    (1-2) Start Mozilla browser
          - No operations exept statrting mail&news
          - C:\X\U does not exist
    (1-3) Start Mozilla mail&news from browser window
          - C:\X\U does not exist
    (1-4) Compose a mail and try to send it
          - Send failed with error message on "C:\X\U\nsmail.html"
          - Save to Drfts also failed with error for on directry
    (1-5) Terminate Mozilla(all of browser and mail&news)
          - C:\X\U does not exist
 (2) (Case-2)
    (2-1) Start Mozilla browser
    (2-2) Download using download manager
          - Download Thunderbird 0.8 from mozilla.org to D:\temp
            => C:\X\U was created
    (2-3) Start Mozilla mail&news from browser window
    (2-4) Compose a mail and try to send it
            => Successful

Diferrence between Case-1 and Case-2 is existence of "C:\X\U" only.
("C:\X\<U Umlaut>" always exists)

Error at step (1-4) seems to be similar to attachment 164396 [details] in Comment #38,
although it may be different form original case of attachment 159194 [details] in Comment
#3, in which special characters are not converted or changed.

Another test result.
Thunderbird(11/1 build) and Mozilla mail&news(11/01 build) created files of "U"
and "U.msf" when I created a folder named <U Umlaut> on Japanese MS Windows.
  
From above test result, nexts can be guessed ;
 - Mozilla uses converted string when <U Umlaut> is included in "TEMP/TMP"
   settings on Japanese MS Windows.
 - Mozilla browser or OS(as a result of API call by Mozilla browser) creates
   "C:\X\U" when "C:\X\<U Umlaut>" is specified in TEMP/TMP
   on Japanese MS Windows.
 - But Mozilla mail&news does not create it or does not force creation of it.

Jshin, is this only another case of Bug 162361?

If my result is applicable to this bug, next 2 procedures can be workarounds. 
 (A)Manual cration of directry(s) displayed in attachment 164396 [details] in Comment #38
 (B)Use download manager of Mozilla browser before mail sending on Thunderbird
Met - Martin Hassman, can you try these procedure?

Comment 40

[Jan Matousek]

Attachment: all problematic characters from Czech alphabet

I tested it on dir named by standard Czech sentence with all non-ascii
charactars, in lower and upper case. This characters are problematic (posted in
UTF-8)

š -> z
ž -> ~
ť -> }
Ř -> ¸
Ě -> ¬
Ž -> n
Č -> ¨
Ň -> ˛

And those reported by Met are problematic too.

Ü -> Ľ
Ë -> «

Notice that case does matter.

WADA and comment 39:
Yes, when I manually create directory with all problematic characters replaced
by their "equivalents", then I can send mail.

But my Mozilla 1.7.3 has no problem with downloading, and use whatever TMP i
gave it (but not the mail, there the problem is). Should 1.7.3 be affected by
your "download error"?

Comment 41

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #40)

> But my Mozilla 1.7.3 has no problem with downloading, and use whatever TMP
> i gave it (but not the mail, there the problem is). 
> Should 1.7.3 be affected by your "download error"?

My download problem of Mozilla(browser,"Save As") was problem when I tried to
save into "C:\X\<U Umlaut>" etc.
This is bug 162361 as Jshin says in comment #34 and comment #36.
Mozilla(browser,11/01 trunk build) on Japanese MS Windows does not have download
problem even when <U Umlaut> is specified in TEMP and TMP environment variables
as I described in coment #39.
 (See step (2-2), saving to D:\temp, not into C:\X\<U Umlaut>, and successuful)

Comment 42

[Jan Matousek]

>> 
>> If my result is applicable to this bug, next 2 procedures can be workarounds. 
>>  (A)Manual cration of directry(s) displayed in attachment 164396 [details] in Comment #38
>>  (B)Use download manager of Mozilla browser before mail sending on Thunderbird
>> Met - Martin Hassman, can you try these procedure?

> My download problem of Mozilla(browser,"Save As") was problem when I tried to
> save into "C:\X\<U Umlaut>" etc.
> This is bug 162361 as Jshin says in comment #34 and comment #36.
> 

So then the B isn't workaround on Win XP Czech. But I find one difference
between 1.7.3 branch and seamonkey in lxr in 

http://lxr.mozilla.org/mozilla1.7/source/xpcom/io/SpecialSystemDirectory.cpp#261

and

http://lxr.mozilla.org/seamonkey/source/xpcom/io/SpecialSystemDirectory.cpp#276

Now I'm compiling mozilla 1.7.3 with removed CharUpper call, in two ours I will
know if this is the problem.

Comment 43

[Jan Matousek]

(In reply to comment #42)
> Now I'm compiling mozilla 1.7.3 with removed CharUpper call, in two ours I will
> know if this is the problem.
> 

It isn't so easy. even it is the only difference in lxr, the CharUpper is not
affecting this bug..

Comment 44

[Jan Matousek]

Attachment: 1.8a4 affected too

But there are less problematic characters, from Czech alphabet only 
Ů, Ú, Ň (UTF-8). Strange..

Comment 45

[WADA:World Anti-bad-Duping Agency]

"Sending failure problem" itself is not a problem only on special character.
This bug occurs on any ASCII characters if directry specified on TEMP or TMP
envirronment variables does not exist.
Try next;
 (1) MD C:\ABC (To make test easy)
 (2) Set TEMP and TMP to C:\ABC\DEF
 (3) Delete C:\ABC\DEF if exists
 (4) Start Mozilla Browser 
     - C:\ABC\DEF does not exist
 (5) Start Mozilla Mail&News
     - C:\ABC\DEF does not exist
 (6) Compose a mail and try to save it to Draft
     => Error occurs on C:\ABC\DEF  
 (7) Download a file and save it into a local directry of ASCII only name.
     For example, download Thunderbird 0.8 and save it in to D:\DOWNLOAD
     - C:\ABC\DEF is created
       (This indicates download manager creates TEMP or TMP if not exists)
 (8) Compose a mail and try to save it into Draft again
     => Saving to Drafts is sucessfull

I think main cause of this bug is lack of TEMP and/or TMP directry creation
logic in Mail&News(also in Thunderbird).
This "TEMP or TMP directry not found" condition is forced by Mozilla himself if
special character is used in TEMP/TMP setting, because Mozilla uses "eqivalent"
character.
And because TEMP or TMP directry is created by download manager on download if
not exist, this bug usually does not occur on Mozilla Mail&News.

Comment 46

[Jungshik Shin]

Jan and Met-Martin, when you tested, had you created a directory pointed to by
'TEMP/TMP' environment variable before testing (i.e. before launching TB or
mozilla)? If you hadn't, this bug has little to do with 'non-ASCII' characters.

I'm sitting in front of a Win XP box with the locale set to Czech and I couldn't
reproduce the bug with TB 0.9.  

Comment 47

[Jungshik Shin]

(In reply to comment #46)
> Jan and Met-Martin, when you tested, had you created a directory pointed to by
> 'TEMP/TMP' environment variable before testing (i.e. before launching TB or
> mozilla)? If you hadn't, this bug has little to do with 'non-ASCII' characters.

When I set TMP/TEMP to a yet-to-be-made directory (an unexisting directory), I
got the alert box as shown here a few times. So, this bug has nothing to do with
whether characters are ASCII or not. This bug should be either resolved invalid
or changed to 'RFE' (request for enhancement) : 'if the directory pointed to by
TEMP/TMP doesn't exist, create one instead of complaining'. I'm not sure whether
creating one automatically is wise, 

Comment 48

[Met - Martin Hassman]

>had you created a directory pointed to by 'TEMP/TMP' environment variable
before testing

Of course!

Open console, set both variables, start thunderbird from the console with the
prepared environment, close TB, set new TMP and TEMP, open TB etc. (from the
Filemon I have checked that the TB is really trying to write to my directory).

If it can calm you, this problem was at first discovered by users who have
non-ASCII characters in their login names (it is not rare for beginner users) so
they have TMP/TEMP set everytime to the problematic path.

Our little game with change TMP, start TB, close TB, change TMP, start TB, close
TB.... is only simulation of the problem of (not-only) Czech beginner users.

Comment 49

[Met - Martin Hassman]

> Our little game with change TMP, start TB, close TB, change TMP, start TB, close
> TB.... is only simulation of the problem of (not-only) Czech beginner users.

Ooosp, I have forgotten *create directory*, change TMP, start TB..... and
sometimes test it also with the Mozilla 1.8 which is working. So I am sure it is
not enhancement problem with non-created directory, but it is a real bug.

Comment 50

[WADA:World Anti-bad-Duping Agency]

Why is severity still critical?
This bug can not be critical any more, I think.

Comment 51

[Jan Matousek]

Attachment: debug output from retrieving temporary file

This is output from program Debug View. It shows where is the problem with ours
characters. The first 24 lines is from test run with TMP directory with no
problematic charcters. Then I set TMP to directory witch some diacrititcs (this
directory had allready exist). Problematic is call to MakeUpperCase func in
xpcom/obsolete/nsSpecialSystemDirectory.cpp. This function does not properly
make upper case charcters with diacritics.

Comment 52

[Jan Matousek]

Attachment: shows where was the calls to OutputDebugString function

Comment 53

[Jan Matousek]

Attachment: just removed MakeUpperCase func

this small patch solves this problem.

But it does not solves the problematic MakeUpperCase function. Every call of
MakeUpperCase function could malform path. I think the MakeUpperCase is really
useless, it only makes problems.

Comment 54

[Met - Martin Hassman]

Shin> Please, can you look at it? With the Jan we think, we have found the
problem. (Jan have tested it by compiling TB with some debuging messages.)

We think that the problem is in the MakeUpperCase() which harms the path string.
You can see the log in the attachment 165031 [details]

nsSpecialSystemDirectory.cpp

         case OS_TemporaryDirectory:
 #if defined (XP_WIN)
         {
             char path[_MAX_PATH];
[1]             DWORD len = GetTempPath(_MAX_PATH, path);
[2]            *this = MakeUpperCase(path);
         }

On the line [1] is correct string returned by the WinAPI GetTempPath(), but on
the line [2] after MakeUpperCase() the string is incorrect.

Looking at the code of MakeUpperCase() the first branch with
(!gGlobalDBCSEnabledOS) seems correct for me (the is probably reason why it is
working on the Win98) but next branch seems suspicious for me. Is _toupper() for
Unicode right solution? What about towupper()?

If the problem is here, I cannot imagine, why it is working on the some WinXP
and on some other it is not working.
We made the poll in our Czech forum and from the reports really seems that on
*some Czech WinXP* this problem happened and on some it didn't happened. We are
not able to find some similar characteristics like service pack etc.

Any idea?

Comment 55

[Jungshik Shin]

Thanks for the investigation and sorry for my premature conclusion. Indeed,
MakeUpperCase is broken. Actually, on the surface (assuming |_toupper| does the
right thing), it's not broken for Latin letters (single byte code pages) but
certainly broken for double byte code pages. In attachment 165031 [details], |_toupper|
(Windows API) converts lowercase letters for which there's no uppercase letters
to 'random'(?) characters. 

Darin, do you know the rationale behind 'MakeUpperCase'? It may affect bug
255344 as well.

Comment 56

[Jungshik Shin]

islower() depends on LC_CTYPE (set by setlocale()) even on Windows [1]. On
Windows, I don't think we set call setlocale(). 

'Ě' (0xcc in Windows-1250) is converted to '¬' (0xac in Windows-1250) by
_toupper() which appears to subtract 0x20 blindly regardless of  LC_CTYPE
setting. It only works if isascii(c) AND either islower(c) or isupper(c) are
true.[2] Obviously, MakeUpperCase misuses it. 

Either we have to rewrite MakeUpperCase() (convert to UTF-16 and use iswlower()
and towupper() and then convert back to the native encoding) or get rid of it.


[1]
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore98/html/_crt_islower.2c_.iswlower.asp

[2]
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore98/html/_crt_islower.2c_.iswlower.asp

Comment 57

[Jungshik Shin]

Attachment: patch (rewrite of MakeUpperCase)

Assuming there's a good reason to call MakeUpperCase(), I rewrote it to work
with non-ASCII Latin letters. Russian and Greek users must have a similar
problem as well. 
Jan, can you apply it and test on your Czech Windows with 'MakeUpperCase' call
retained (i.e. without attachment 165034 [details] [diff] [review] applied)? My patch may needs some
tweaking to get compiled (I haven't compiled it because I'm on Linux at the
moment. I need a VMware ;-))

Comment 58

[Jan Matousek]

jshin, it can't compile. and i'm no C++ hacker (yet :) ), so i can't fix it.

m:/mozilla/xpcom/obsolete/nsSpecialSystemDirectory.cpp: In function `char*
   MakeUpperCase(char*)':
m:/mozilla/xpcom/obsolete/nsSpecialSystemDirectory.cpp:184: error: no match for
   'operator<' in 'start < end'
../../dist/include/string/nsTAString.h:489: error: candidates are: PRBool
   operator<(const nsAString&, const nsAString&)
../../dist/include/string/nsTAString.h:489: error:                 PRBool
   operator<(const nsACString&, const nsACString&)
m:/mozilla/xpcom/obsolete/nsSpecialSystemDirectory.cpp: In member function `
   void
   nsSpecialSystemDirectory::operator=(nsSpecialSystemDirectory::SystemDirectories)
   ':
m:/mozilla/xpcom/obsolete/nsSpecialSystemDirectory.cpp:503: warning: unused
   variable `DWORD len'
make[1]: *** [nsSpecialSystemDirectory.o] Error 1
make[1]: Leaving directory `/cygdrive/m/obj-win32/xpcom/obsolete'
make: *** [all] Error 2

Comment 59

[Jungshik Shin]

Ooops. Sorry. |while (start < end)| should be |while (start != end)|. Can you
try again replacing '<' with '!=' in while-statement? Thanks.

Comment 60

[Jan Matousek]

Attachment: final fix?

ok, now I can compile it and it correctly use whatever TMP I gave it.

The (debug) build will be on http://plathel.iglu.cz/mozilla/builds/ for next
week if someone want test it.

Comment 61

[Jungshik Shin]

Thanks for testing and building. can you ask someone to test it on Win 95/98/ME?
According to MSDN, 'towupper' is available on Win 9x/ME as well as on NT4/2k/XP,
but I want to be sure.

Comment 62

[Christian :Biesinger (don't email me, ping me on IRC)]

What about NT 3.51?

Comment 63

[Met - Martin Hassman]

I have tested now Jan's build on Czech WinXP (which didn't work before) and on
the Czech Win2000 (which worked before). On the both it is OK.

Tonight I can test it on the Czech Win98 too.

Comment 64

[Met - Martin Hassman]

Attachment: screenshot TB is tricky 8-(

I have two news! Good and bad 8-)

Good news: I have tested Jan build on the Win98 and it still should send mails.

Bad news: I was too carefull with the investigation and I have found some new
aspect, which we didn't consider before:

Shortly: TB *is not* able to find the TEMP dir on the Win98 but it still can
send mails. I am sorry to say, but it invalidates some of our tests made
before.

In the attachment is Filemon log: TB is not able to create nsmail.eml in the
TEMP dir (with non-ASCII characters) but use the *current dir* instead. There
*is still* the MakeUppercase() bug, but mail is send. Our test failed. Strange.
I have same results with the TB0.8, TB0.9 and with the Plathel buidls. So the
problem with MakeUppercase() was and still is presented on the Win98 (as the
oposite of WinXP, where this patch *IS* an improvement). Maybe this can explain
why you, Shin, was not able to reproduce this bug before.

As the result *on the Win98 was the problem presented before and is still*.
This can affect every functionality using MakeUppercase().

We need new tests: Jan and Shin please try not to check only if the mail was
send, but set you SMTP server to some non-existent value and send mail. During
the SMTP error alert (do not close it!!!) look in the TEMP dir (or current dir
or even dir with the thunderbird.exe) for the nsmail.eml. The dir with the this
temporary file is used. TB is sometimes tricky and when cannot find TMP dir it
try to use another.

Comment 65

[Jungshik Shin]

Hmm. according to MSDN, towupper() should work on Win 9x/ME, but apparently it
doesn't.  Met-Martin, can you repeat what you did to obtain attachment 165310 [details] on
Win 2k/XP and upload a screenshot? BTW, it'd be also nice if you can type the
value of 'TEMP' and 'TMP' you used (in UTF-8) as a comment. I can read them off
from the screenshot, but I need to look up the Unicode table to type them before
converting to Windows-1250. 


Darin and Doug, do you know why we use MakeUpperCase()? Before fixing it, I like
to know whether it's really necessary.

Comment 66

[Met - Martin Hassman]

Attachment: Screenshot on the Win2000

Screenshot on the Win2000 with TB0.8 and with the Jan's compilation (both are
working, but compare they are different in the 1st character)
Strange is that in the Filemon log are small characters ?

And the PATH in the UTF-8: C:\+ěščřžýáíéúů

Comment 67

[Jungshik Shin]

Hmm, that's odd. It seems like MakeUpperCase just returns what's passed (without
changing cases). It should only happen when NS_CopyNativeToUnicode fails, but it
can't fail if you run this on Czech Win XP(with the system locale encoding set
to Windows 1250). Can you do me another favor? Can you type here the mangled
path in UTF-8 (not the value of TEMP/TMP) in attachment 165310 [details]?  Thanks !

Comment 68

[Met - Martin Hassman]

Attachment: Uppercased path on Win98

OK, new tests on the Czech Win98. Please forget attachment 165310 [details] and sorry for
my mistake 8-( - writing path in the Win98 console is affected by the DOS
codepage (850 codepage for me - I am not strong in DOS codepages). When set by
editing the bat file in the Win-1250 encoding, everythig is OK (for TB0.8 and
Jan's compilation too) and the path is correctly UPPERCASED (see screenshot).

So I think your patch is working on the Win98. Question is still why it doesn't
uppercases on the Win2000.

Comment 69

[Jan Matousek]

I have one more checked in filemon the case of path, which Thunderbird opens. It
is for example (captured by filemon):

T:\PříšERNěžKUťOUčKýKůňúPěLďáBELSKéóDYPŘÍŠERNĚŽLUŤOUČKÝKŮŇÚPĚLĎÁBELSKÉÓDY\nscopy.tmp

the directory is named:

příšerněžkuťoučkýkůňúpělďábelskéódyPŘÍŠERNĚŽLUŤOUČKÝKŮŇÚPĚLĎÁBELSKÉÓDY

That means that none of chars with some diacritics are not in upper case. I have
overlooked this because in my font chars with diacritics look simililar in both
upper and lower case. The MakeUpperCase still don't work perfectly, but at least
the path is correct, if not uppercased. I don't think that the case of path can
be problem. 

Still my opinion is that the best solution is to completely remove
MakeUpperCase. It is straighforward and definitive.. after all troubles with
this bug. And if it is nessesary in some cases to have TMP in upper case, why
not set it upper case, instead of convertiong it from program?

Comment 70

[Scott MacGregor]

jshin, should we take your final fix for thunderbird 1.0 or is this still a work
in progress? We're trying to wrap up the bug list this week. 

Comment 71

[Scott MacGregor]

moving off the list pending feedback from jshin.

Comment 72

[Stefan Borggraefe]

*** Bug 271903 has been marked as a duplicate of this bug. ***

Comment 73

[Scott MacGregor]

I actually checked this change into the aviary 1.0 branch as it does improve the
situation, it just doesn't fix it for all scenarious if I'm reading the last few
comments correctly.

Can someone volunteer to verify this change in the 11/29 builds tomorrow? Jan/Met?
Thanks!

Comment 74

[Met - Martin Hassman]

> Can someone volunteer to verify this change in the 11/29 builds tomorrow? 

Tested latest-trunk/thunderbird-win32.zip 29-Nov-2004 23:02

- WinXP CZ, progress (now some other non-ASCII characters are working too) but 
- Win2000 CZ, regression. It worked before, now there is problem with some
characters.

Comment 75

[Adam Hauner]

Requesting blocking, as this is not still fixed in TB on W2K.

Comment 76

[OstGote!]

*** Bug 246794 has been marked as a duplicate of this bug. ***

Comment 77

[OstGote!]

*** Bug 248957 has been marked as a duplicate of this bug. ***

Comment 78

[Met - Martin Hassman]

version 1.0RC1 (20041201) working on WinXP CZ and Win2000 for Czech characters

(My comment 74 was about latest-trunk which has been probably broken.)

Tested path: c:\Příliš žluťoučký kůň úpěl ďábelské ódy

Note: Non-ASCII characters aren't still uppercased, but because this patch is
windows-only - it is working correctly.

Report on Win98 CZ will come evening.

Remove blocking.

Comment 79

[Met - Martin Hassman]

>Report on Win98 CZ will come evening.

TB 1.0RC on Win98 CZ - no problem 8-)

Good progress.

Comment 80

[Wayne Mery (:wsmwk)]

*** Bug 269689 has been marked as a duplicate of this bug. ***

Comment 81

[Victor Luchits]

Do we really need to call MakeUpperCase here? It doesn't make sense to me since
windows filesystems (both FAT and NTFS) are case-insensitive. It just adds a
little overhead and causes troubles. Correct me if I'm wrong. Scott, David?

Comment 82

[Jungshik Shin]

Attachment: patch update

this was already checked into aviary-1.0.

Comment 83

[Darin Fisher]

Comment on attachment 184216 [details] [diff] [review]
patch update

>Index: xpcom/obsolete/nsSpecialSystemDirectory.cpp

>+  nsresult rv = NS_CopyNativeToUnicode(nsDependentCString(aPath), widePath);
>+  NS_ASSERTION(NS_SUCCEEDED(rv),  "failed to convert a path to Unicode");
>+  if (NS_FAILED(rv))
>+      return aPath;

nit: use NS_ERROR instead...

    if (NS_FAILED(rv)) {
	NS_ERROR("failed to convert a path to Unicode");
	return aPath;
    }


>+  nsString::iterator start, end;
>+  widePath.BeginWriting(start);
>+  widePath.EndWriting(end);

FWIW, you can also write:

    PRUnichar *start = widePath.BeginWriting();
    PRUnichar *end = widePath.EndWriting();


>+  PRUint32 len = strlen(aPath);

nit: when you wrap aPath with nsDependentCString up above, you are
implicitly calling strlen.  You could re-use that object to avoid
calling strlen twice.  e.g.:

  nsDependentCString path(aPath);
  ...
  PRUint32 len = path.Length();


>+  NS_ASSERTION(newCPath.Length() <= len, 
>+               "uppercased string is longer than original");

If this is true, then shouldn't we return an error?  It would seem
to depend on the behavior of WideCharToMultiByte, which is beyond
our control.

Comment 84

[Doug Turner (:dougt)]

Comment on attachment 184216 [details] [diff] [review]
patch update

clearing request per darin's -

Comment 85

[Jungshik Shin]

(In reply to comment #83)

> >+  NS_ASSERTION(newCPath.Length() <= len, 
> >+               "uppercased string is longer than original");
> 
> If this is true, then shouldn't we return an error?  It would seem
> to depend on the behavior of WideCharToMultiByte, which is beyond
> our control.

Actually, what I had in mind was that uppercasing may change the length.
However, that would never heppen because 1) towupper works on an individual
character and returns a single character 2) towupper doesn't change any
character outside US-ASCII (I don't know if it's a Windows bug). I've tested it
in a standalone console program and it was also reported in comment #69.
Nonetheless, we need to apply this patch because the current code has a serious
problem with non-ASCII characters (MBCS or not). 

Comment 86

[Jungshik Shin]

Attachment: patch addressing darin's comment

I addressed darin's concerns. As I wrote in the previous comment, the length
will never change so that I just got rid of the check and used strcpy.

Comment 87

[Darin Fisher]

Comment on attachment 184579 [details] [diff] [review]
patch addressing darin's comment

I'm still a bit paranoid when using strcpy.  The cost of checking that
length of aPath == length of newCPath seems tolerable.	You can minimize the
cost by doing:

nsDependentCString path(aPath);

and then compare:

path.Length() to newCPath.Length() before calling strcpy.

sr=darin but i'd still recommend this check, even if only as a debug
NS_ASSERTION.

Comment 88

[Jungshik Shin]

Thanks for sr.

(In reply to comment #87)
> (From update of attachment 184579 [details] [diff] [review] [edit])
> I'm still a bit paranoid when using strcpy.  The cost of checking that
> length of aPath == length of newCPath seems tolerable.	You can minimize the
> cost by doing:

Yup, I noted that in your previous comment..

> sr=darin but i'd still recommend this check, even if only as a debug
> NS_ASSERTION.

  I'll add that. If I go a step further, you prefer to return |nsnull| instead
of filling up the buffer with an incomplete string (as I did in the previous
patch), don't you?  

Comment 89

[Doug Turner (:dougt)]

Comment on attachment 184579 [details] [diff] [review]
patch addressing darin's comment

when will mail move to nsLocalFile where these problems are fixed?

Comment 90

[Jungshik Shin]

Comment on attachment 184579 [details] [diff] [review]
patch addressing darin's comment

thanks for r/sr.
asking for a1.1a1.
A patch virtually identical to this one (with |strcpy|) has been in the 1.0
branch for a while without a known regression. Given this,
I'll just add NS_ASSSERTION just in case without additional error handling
before landing on the trunk.

Comment 91

[Scott MacGregor]

Comment on attachment 184579 [details] [diff] [review]
patch addressing darin's comment

this needs to wait for 1.1a2 since 1.1a1 is *hopefully* done.

Comment 92

[Mike Shaver (:shaver emeritus)]

Comment on attachment 184579 [details] [diff] [review]
patch addressing darin's comment

a=shaver

Comment 93

[Jungshik Shin]

Sorry this was landed a month ago. Marking as resolved. If it doesn't solve the
problem completely, please reopen it.