180849 - Mail loss in import of NC4 mail when 0x5C(\) is used as 2nd byte of muti-byte character in folder name.

Status: RESOLVED FIXED

(MailNews Core :: Internationalization, defect, P1)

Description

[WADA:World Anti-bad-Duping Agency]

User-Agent:       Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.3a) Gecko/20021118
Build Identifier: Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.3a) Gecko/20021115

On 2002111508-trunc/Win-Me, import of NC4.x mail was not done correctly when
folder name contains illegal file name character as 2nd byte of multi-byte
character.

Final fix of bug 117385 has improved handling of mail folders with folder name
which has an illegal file name character in 2nd byte, such as 0x835C(5c="\"),
0x837C(7C="|").
According to change by above fix, mail import from NC 4.x should be modified.

For example, after the fix for bug 117385, Mozilla creates 3db4216 ,
3db4216.msf, 3db4216.sbd for a new folder of 0x837C.
0x837C is Shift_JIS code of "po" of Japanese Katakana and 2nd byte is 0x7C("|").
For folder name 0x837C, NC4.x creates files/directory named 0x837C , 0x837C.snm
, 0x837C.sbd.
But Mozilla creates 0x837C , 3db4216.msf, 3db4216.sbd when mail import from NC4.x.
Due to inproper file name, 0x837C, Mozilla can not access to mbox file and mails
are lost.

After renaming of 0x837C mbox file to 3db4216, Mozilla can process 0x837C folder
successufully.
This indicates that lack of renaming logic for mbox file is the cause of this
problem.

Renaming of mbox file is sufficient to resolve this problem.
However, if all Japanese muti-byte characters are judged as legal file name
character, this problem will not occur because file-name and folder-name become
identical.
All Japanese characters can be used as file name even when 2nd byte in Shift_JIS
is an illegal file name character.
With this big change, bug 180546 will be automatically resolved for Japanese
folder name case.

For folder name with ASCII special characters, import itself is done properly
since NC4.x creates file name of valid character only. 
For example, when folder name is "\"(0x5C), NC4.x created files("_" , "_.snm")
and a directory("_.sbd").
Mozilla can import them as "_" folder.
This cause difficulty in distinguish of original folders.
But this is not a critical issue because special ASCII characters are usually
used under combination with alphabets.


Reproducible: Always

Steps to Reproduce:
(1) Create folder named 0x837C(in Shift_JIS) on Netscape 4.x, copy a mail,
create sub folder
 -> 0x837C , 0x837C.snm , 0x837C.sbd are created
(2) Import NC4 mail on Mozilla
 -> Import completes without error
 -> 0x837C , 3db4216.msf , 3db4216.sbd are created
 -> Folder name is 0x837C and proper but mails are lost because 0x837C is
inproper on Mozlla
 -> Sub-folder in 0x837C folder can be viewed because 3db4216.sbd is proper file
name
(3) Rename 0x837C file to 3db4216 and restart Mozilla
 -> Mozilla can access the folder properly

Comment 1

[u69748]

I can reproduce this bug.

Comment 2

[Hideo Oshima]

Can anyone confirm this?

Comment 3

[Hiro]

It is reported by the mozillaZine Japanese.
http://ryuzi.dyndns.org/mozillazine/html/modules/newbb/viewtopic.php?topic_id=1108&forum=5#forumpost3113

Windows2000 SP4 + Thunderbird 0.7.2

Comment 4

[WADA:World Anti-bad-Duping Agency]

(Note: Since Japanese character relateed problem, I write in UTF-8.)
(      See this bug with View/Character Encodign/UTF-8,please.    )

As bug 266027 has been fixed, I tested again with
 -  Mozilla Suite latest-trunk 2004111606(Win-2K)
 -  Thunderbird latest-0.9 2004/11/16 build

(Case-1) Folder name of 'ポ', 'po' in KATAKANA, 0x837C(7C='|') in Shift_JIS
         => Successufully imported
(Case-2) Folder name of 'ソ', 'so' in KATAKANA, 0x835C(5C='\') in Shift_JIS
         Real file name on NS 4 is 'ソ_' and 'ソ_.snm'.
         (NS 4 adds '_' to make last '\' escape character in string handling)     
         => Import failed
  (Case-2-a) Mozilla case
       (2-a-1) Import failed and error dialog panal was didplayed,
                but no error message is diaplayed, blank message instead.
       (2-a-2) Next file/directry was created.
               'ソ_.msf' (file)
               'ソ_'     (directry!)
               'ソ_.sbd' (directry)
       (2-a-3) Next files/directries was NOT created.
               'ソ_' (file) '
       (2-a-4) Move/Rname/Delete was impossible
  (Case-2-b) Mozilla case
       (2-b-1) Import failed and error dialog panal was didplayed,
               with saying "Import failed, check disk space".
       (2-a-2) Next file/directry was created.
               'ソ_.msf' (file)
               'ソ_.sbd' (directry)
       (2-a-3) Next files/directries was NOT created.
               'ソ_' (file) '
       (2-a-4) Move/Rname/Delete was impossible

Folder creation of 'ソ_' has no problem on Mozilla/Thunderbird. 
So this is problem when import only.
By fix for bug 264071, many Japanese character in folder name have been
resolved, but there is still a small hole.

I can understand Thunderbird result, creation failure of folder file of 'ソ_'.
But I can not imagine why and how Mozilla created directry of 'ソ_'.
Jshin, why directry was created instead of file?

Comment 5

[WADA:World Anti-bad-Duping Agency]

Change summary since 0x7C(|) problem did not occur on latest nightlies.

Comment 6

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Attachment: Patch rv1.0

Comment 7

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Attachment: Patch rv1.0 (-u8pw)

Comment 8

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

This is simple bug.
In ImportComm4xMailImpl::ImportMailbox,
http://lxr.mozilla.org/mozilla/source/mailnews/import/comm4x/src/nsComm4xMailImport.cpp#273

|pDestination->GetParent(getter_AddRefs(parent))| is failed, but the result is NS_OK. Because nsFileSpecWin uses strrchr for finding the path separator. We should use _mbsrchr.

# Maybe, |GetLeafName| and |SetLeafName| are having similar problem...

Comment 9

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

> # Maybe, |GetLeafName| and |SetLeafName| are having similar problem...

These methods may not have same problem. Because they may use |GetLastSeparator| that is using |_mbsrchr|.

Comment 10

[Daniel Veditz [:dveditz]]

blocking 1.8.0.2 denied, not a regression, no trunk-baked patch.

Comment 11

[Jungshik Shin]

Comment on attachment 211028 [details] [diff] [review]
Patch rv1.0

r=jshin

Comment 12

[Doug Turner (:dougt)]

Comment on attachment 211028 [details] [diff] [review]
Patch rv1.0

The whitespace changes do not match the style of tihs file.

This needs to be fixed bofore you check in.

Comment 13

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

checked-in to trunk.

# Doug said, the patch's white space doesn't have a problem.

Comment 14

[Scott MacGregor]

This needs landed on the 1.8.1 branch too if you can. Otherwise I can do it for you Masayuki. 

Comment 15

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Yeah, we don't have any regression report by this on Trunk.
Checked-in to 1.8 branch too.

Comment 16

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 211028 [details] [diff] [review]
Patch rv1.0

Let's go to 1.8.0.2. This patch's risk is low.

Comment 17

[Ronny Perinke]

patch leads to build error using MS VC++ 2008 (VC8)

---------
make[3]: Entering directory `/cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/firefox_vc8/xpcom/obsolete'
nsFileSpec.cpp
Building deps for /cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp
/cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/mozilla/build/cygwin-wrapper cl -FonsFileSpec.obj -c  -DMOZILLA_INTERNAL_API -D_IMPL_NS_GFX -D_IMPL_NS_MSG_BASE -D_IMPL_NS_WIDGET  -DOSTYPE=\"WINNT5.1\" -DOSARCH=\"WINNT\" -DBUILD_ID=0000000000 -D_IMPL_NS_COM_OBSOLETE -I.. -I/cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/../io  -I../../dist/include/xpcom -I../../dist/include/string -I../../dist/include/libreg -I../../dist/include/xpcom_obsolete -I../../dist/include -I../../dist/include/nspr    -I../../dist/sdk/include       -TP -nologo -W3 -Gy -FdnsFileSpec.pdb  -DNDEBUG -DTRIMMED -O2 -GA -GF -GL -Gs -GT -arch:SSE -MD            -DX_DISPLAY_MISSING=1 -DMOZILLA_VERSION=\"1.8\" -DMOZILLA_VERSION_U=1.8 -DHAVE_SNPRINTF=1 -D_WINDOWS=1 -D_WIN32=1 -DWIN32=1 -DXP_WIN=1 -DXP_WIN32=1 -DHW_THREADS=1 -DWINVER=0x400 -D_WIN32_WINNT=0x400 -DSTDC_HEADERS=1 -DWIN32_LEAN_AND_MEAN=1 -DNO_X11=1 -D_X86_=1 -DD_INO=d_ino -DMOZ_ENABLE_CANVAS=1 -DMOZ_DEFAULT_TOOLKIT=\"windows\" -DMOZ_PHOENIX=1 -DMOZ_BUILD_APP=browser -DMOZ_XUL_APP=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DOJI=1 -DIBMBIDI=1 -DMOZ_VIEW_SOURCE=1 -DMOZ_XPINSTALL=1 -DMOZ_JSLOADER=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_XTF=1 -DMOZ_MATHML=1 -DMOZ_SVG=1 -DMOZ_SVG_RENDERER_CAIRO=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_LOGGING=1 -DMOZ_USER_DIR=\"Mozilla\" -DHAVE_UINT64_T=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DMOZ_MORK=1 -DMOZ_DLL_SUFFIX=\".dll\" -DJS_THREADSAFE=1 -DMOZILLA_1_8_BRANCH=1 -DMOZILLA_LOCALE_VERSION=\"1.8b5\" -DMOZILLA_REGION_VERSION=\"1.8b5\" -DMOZILLA_SKIN_VERSION=\"1.8\"  -D_MOZILLA_CONFIG_H_ -DMOZILLA_CLIENT /cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp
nsFileSpec.cpp
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(175) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(198) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(199) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(373) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(379) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(461) : warning C4996: 'strcpy' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(73) : see declaration of 'strcpy'
        Message: 'This function or variable may be unsafe. Consider using strcpy_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(462) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:\building_mozilla\source\branch_1_5_x\mozilla\xpcom\obsolete\nsFileSpecWin.cpp(242) : warning C4244: '=' : conversion from 'time_t' to 'nsFileSpec::TimeStamp', possible loss of data
l:\building_mozilla\source\branch_1_5_x\mozilla\xpcom\obsolete\nsFileSpecWin.cpp(420) : error C2440: 'initializing' : cannot convert from 'const unsigned char *' to 'unsigned char *'
        Conversion loses qualifiers
l:\building_mozilla\source\branch_1_5_x\mozilla\xpcom\obsolete\nsFileSpecWin.cpp(681) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
make[3]: *** [nsFileSpec.obj] Error 2
make[3]: Leaving directory `/cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/firefox_vc8/xpcom/obsolete'
make[2]: *** [tier_2] Error 2
make[2]: Leaving directory `/cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/firefox_vc8'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/firefox_vc8'
make: *** [build] Error 2

Comment 18

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Attachment: Patch rv1.1

Sorry. I fogot the VC8's problem. This patch is used in Trunk.

Comment 19

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

fixed the bustage on 1.8 branch too.
http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=MOZILLA_1_8_BRANCH&branchtype=match&dir=mozilla%2Fxpcom%2Fobsolete&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2006-02-18&maxdate=2006-02-19&cvsroot=%2Fcvsroot

The patch was created by pavlov.
http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=HEAD&branchtype=match&dir=mozilla%2Fxpcom%2Fobsolete&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2006-02-10+12%3A00%3A00&maxdate=2006-02-10+13%3A00%3A00&cvsroot=%2Fcvsroot

Comment 20

[Ronny Perinke]

thanks, works again :)

Comment 21

[Daniel Veditz [:dveditz]]

Comment on attachment 212317 [details] [diff] [review]
Patch rv1.1

Please combine this with the bustage fixes for approval on the 1.8.0 branch -- this one won't work alone.

Comment 22

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Comment on attachment 212317 [details] [diff] [review]
Patch rv1.1

Daniel:

This patch already fixed the bustage.

Comment 23

[Daniel Veditz [:dveditz]]

Comment on attachment 212317 [details] [diff] [review]
Patch rv1.1

approved for 1.8.0 branch, a=dveditz for drivers

Comment 24

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

checked-in to 1.8.0 branch.

Comment 25

[Samuel Sidler (old account; do not CC)]

Masayuki, can you please verify this on the 1.8 branch as well? I'm not clear on how to verify it. Thank you.