Mail loss in import of NC4 mail when 0x5C(\) is used as 2nd byte of muti-byte character in folder name.

RESOLVED FIXED in mozilla1.8.1

Status

MailNews Core
Internationalization
P1
major
RESOLVED FIXED
15 years ago
9 years ago

People

(Reporter: World, Assigned: masayuki)

Tracking

(Blocks: 1 bug, 5 keywords)

Trunk
mozilla1.8.1
x86
Windows 2000
dataloss, fixed1.8.1, intl, jp-critical, verified1.8.0.2
Bug Flags:
blocking-thunderbird2 +
blocking1.8.0.2 -

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 2 obsolete attachments)

(Reporter)

Description

15 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.3a) Gecko/20021118
Build Identifier: Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.3a) Gecko/20021115

On 2002111508-trunc/Win-Me, import of NC4.x mail was not done correctly when
folder name contains illegal file name character as 2nd byte of multi-byte
character.

Final fix of bug 117385 has improved handling of mail folders with folder name
which has an illegal file name character in 2nd byte, such as 0x835C(5c="\"),
0x837C(7C="|").
According to change by above fix, mail import from NC 4.x should be modified.

For example, after the fix for bug 117385, Mozilla creates 3db4216 ,
3db4216.msf, 3db4216.sbd for a new folder of 0x837C.
0x837C is Shift_JIS code of "po" of Japanese Katakana and 2nd byte is 0x7C("|").
For folder name 0x837C, NC4.x creates files/directory named 0x837C , 0x837C.snm
, 0x837C.sbd.
But Mozilla creates 0x837C , 3db4216.msf, 3db4216.sbd when mail import from NC4.x.
Due to inproper file name, 0x837C, Mozilla can not access to mbox file and mails
are lost.

After renaming of 0x837C mbox file to 3db4216, Mozilla can process 0x837C folder
successufully.
This indicates that lack of renaming logic for mbox file is the cause of this
problem.

Renaming of mbox file is sufficient to resolve this problem.
However, if all Japanese muti-byte characters are judged as legal file name
character, this problem will not occur because file-name and folder-name become
identical.
All Japanese characters can be used as file name even when 2nd byte in Shift_JIS
is an illegal file name character.
With this big change, bug 180546 will be automatically resolved for Japanese
folder name case.

For folder name with ASCII special characters, import itself is done properly
since NC4.x creates file name of valid character only. 
For example, when folder name is "\"(0x5C), NC4.x created files("_" , "_.snm")
and a directory("_.sbd").
Mozilla can import them as "_" folder.
This cause difficulty in distinguish of original folders.
But this is not a critical issue because special ASCII characters are usually
used under combination with alphabets.


Reproducible: Always

Steps to Reproduce:
(1) Create folder named 0x837C(in Shift_JIS) on Netscape 4.x, copy a mail,
create sub folder
 -> 0x837C , 0x837C.snm , 0x837C.sbd are created
(2) Import NC4 mail on Mozilla
 -> Import completes without error
 -> 0x837C , 3db4216.msf , 3db4216.sbd are created
 -> Folder name is 0x837C and proper but mails are lost because 0x837C is
inproper on Mozlla
 -> Sub-folder in 0x837C folder can be viewed because 3db4216.sbd is proper file
name
(3) Rename 0x837C file to 3db4216 and restart Mozilla
 -> Mozilla can access the folder properly
(Reporter)

Updated

15 years ago
Severity: critical → major

Comment 1

15 years ago
I can reproduce this bug.

Comment 2

13 years ago
Can anyone confirm this?

Comment 3

13 years ago
It is reported by the mozillaZine Japanese.
http://ryuzi.dyndns.org/mozillazine/html/modules/newbb/viewtopic.php?topic_id=1108&forum=5#forumpost3113

Windows2000 SP4 + Thunderbird 0.7.2
(Reporter)

Comment 4

13 years ago
(Note: Since Japanese character relateed problem, I write in UTF-8.)
(      See this bug with View/Character Encodign/UTF-8,please.    )

As bug 266027 has been fixed, I tested again with
 -  Mozilla Suite latest-trunk 2004111606(Win-2K)
 -  Thunderbird latest-0.9 2004/11/16 build

(Case-1) Folder name of 'ポ', 'po' in KATAKANA, 0x837C(7C='|') in Shift_JIS
         => Successufully imported
(Case-2) Folder name of 'ソ', 'so' in KATAKANA, 0x835C(5C='\') in Shift_JIS
         Real file name on NS 4 is 'ソ_' and 'ソ_.snm'.
         (NS 4 adds '_' to make last '\' escape character in string handling)     
         => Import failed
  (Case-2-a) Mozilla case
       (2-a-1) Import failed and error dialog panal was didplayed,
                but no error message is diaplayed, blank message instead.
       (2-a-2) Next file/directry was created.
               'ソ_.msf' (file)
               'ソ_'     (directry!)
               'ソ_.sbd' (directry)
       (2-a-3) Next files/directries was NOT created.
               'ソ_' (file) '
       (2-a-4) Move/Rname/Delete was impossible
  (Case-2-b) Mozilla case
       (2-b-1) Import failed and error dialog panal was didplayed,
               with saying "Import failed, check disk space".
       (2-a-2) Next file/directry was created.
               'ソ_.msf' (file)
               'ソ_.sbd' (directry)
       (2-a-3) Next files/directries was NOT created.
               'ソ_' (file) '
       (2-a-4) Move/Rname/Delete was impossible

Folder creation of 'ソ_' has no problem on Mozilla/Thunderbird. 
So this is problem when import only.
By fix for bug 264071, many Japanese character in folder name have been
resolved, but there is still a small hole.

I can understand Thunderbird result, creation failure of folder file of 'ソ_'.
But I can not imagine why and how Mozilla created directry of 'ソ_'.
Jshin, why directry was created instead of file?
Component: Import → Internationalization
OS: Windows 98 → Windows 2000
(Reporter)

Updated

13 years ago
Blocks: 124287
(Reporter)

Comment 5

13 years ago
Change summary since 0x7C(|) problem did not occur on latest nightlies.
Summary: Mail loss in import of NC4 mail when 0x5C(\) or 0x7C(|) is used as 2nd byte of muti-byte character in folder name. → Mail loss in import of NC4 mail when 0x5C(\) is used as 2nd byte of muti-byte character in folder name.
Product: MailNews → Core
(Assignee)

Updated

12 years ago
Assignee: cavin → smontagu
QA Contact: nbaca
(Assignee)

Updated

12 years ago
Status: UNCONFIRMED → NEW
Ever confirmed: true
Created attachment 211028 [details] [diff] [review]
Patch rv1.0
Assignee: smontagu → masayuki
Status: NEW → ASSIGNED
Attachment #211028 - Flags: review?(jshin1987)
(Assignee)

Updated

12 years ago
Flags: blocking1.8.0.2?
Flags: blocking-thunderbird2?
Priority: -- → P1
Target Milestone: --- → mozilla1.8.1
Created attachment 211029 [details] [diff] [review]
Patch rv1.0 (-u8pw)
This is simple bug.
In ImportComm4xMailImpl::ImportMailbox,
http://lxr.mozilla.org/mozilla/source/mailnews/import/comm4x/src/nsComm4xMailImport.cpp#273

|pDestination->GetParent(getter_AddRefs(parent))| is failed, but the result is NS_OK. Because nsFileSpecWin uses strrchr for finding the path separator. We should use _mbsrchr.

# Maybe, |GetLeafName| and |SetLeafName| are having similar problem...

Updated

12 years ago
Flags: blocking-thunderbird2? → blocking-thunderbird2+
> # Maybe, |GetLeafName| and |SetLeafName| are having similar problem...

These methods may not have same problem. Because they may use |GetLastSeparator| that is using |_mbsrchr|.
blocking 1.8.0.2 denied, not a regression, no trunk-baked patch.
Flags: blocking1.8.0.2? → blocking1.8.0.2-

Comment 11

12 years ago
Comment on attachment 211028 [details] [diff] [review]
Patch rv1.0

r=jshin
Attachment #211028 - Flags: review?(jshin1987) → review+
(Assignee)

Updated

12 years ago
Attachment #211028 - Flags: superreview?(dougt)
(Assignee)

Updated

12 years ago
Keywords: intl, jp-critical
(Assignee)

Updated

12 years ago
Keywords: dataloss
(Assignee)

Updated

12 years ago
Attachment #211028 - Flags: branch-1.8.1?(dougt)

Comment 12

12 years ago
Comment on attachment 211028 [details] [diff] [review]
Patch rv1.0

The whitespace changes do not match the style of tihs file.

This needs to be fixed bofore you check in.
Attachment #211028 - Flags: superreview?(dougt)
Attachment #211028 - Flags: superreview+
Attachment #211028 - Flags: branch-1.8.1?(dougt)
Attachment #211028 - Flags: branch-1.8.1+
checked-in to trunk.

# Doug said, the patch's white space doesn't have a problem.
Status: ASSIGNED → RESOLVED
Last Resolved: 12 years ago
Resolution: --- → FIXED

Comment 14

12 years ago
This needs landed on the 1.8.1 branch too if you can. Otherwise I can do it for you Masayuki. 
Yeah, we don't have any regression report by this on Trunk.
Checked-in to 1.8 branch too.
Keywords: fixed1.8.1
Comment on attachment 211028 [details] [diff] [review]
Patch rv1.0

Let's go to 1.8.0.2. This patch's risk is low.
Attachment #211028 - Flags: approval1.8.0.2?

Comment 17

12 years ago
patch leads to build error using MS VC++ 2008 (VC8)

---------
make[3]: Entering directory `/cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/firefox_vc8/xpcom/obsolete'
nsFileSpec.cpp
Building deps for /cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp
/cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/mozilla/build/cygwin-wrapper cl -FonsFileSpec.obj -c  -DMOZILLA_INTERNAL_API -D_IMPL_NS_GFX -D_IMPL_NS_MSG_BASE -D_IMPL_NS_WIDGET  -DOSTYPE=\"WINNT5.1\" -DOSARCH=\"WINNT\" -DBUILD_ID=0000000000 -D_IMPL_NS_COM_OBSOLETE -I.. -I/cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/../io  -I../../dist/include/xpcom -I../../dist/include/string -I../../dist/include/libreg -I../../dist/include/xpcom_obsolete -I../../dist/include -I../../dist/include/nspr    -I../../dist/sdk/include       -TP -nologo -W3 -Gy -FdnsFileSpec.pdb  -DNDEBUG -DTRIMMED -O2 -GA -GF -GL -Gs -GT -arch:SSE -MD            -DX_DISPLAY_MISSING=1 -DMOZILLA_VERSION=\"1.8\" -DMOZILLA_VERSION_U=1.8 -DHAVE_SNPRINTF=1 -D_WINDOWS=1 -D_WIN32=1 -DWIN32=1 -DXP_WIN=1 -DXP_WIN32=1 -DHW_THREADS=1 -DWINVER=0x400 -D_WIN32_WINNT=0x400 -DSTDC_HEADERS=1 -DWIN32_LEAN_AND_MEAN=1 -DNO_X11=1 -D_X86_=1 -DD_INO=d_ino -DMOZ_ENABLE_CANVAS=1 -DMOZ_DEFAULT_TOOLKIT=\"windows\" -DMOZ_PHOENIX=1 -DMOZ_BUILD_APP=browser -DMOZ_XUL_APP=1 -DMOZ_DISTRIBUTION_ID=\"org.mozilla\" -DOJI=1 -DIBMBIDI=1 -DMOZ_VIEW_SOURCE=1 -DMOZ_XPINSTALL=1 -DMOZ_JSLOADER=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DMOZ_XTF=1 -DMOZ_MATHML=1 -DMOZ_SVG=1 -DMOZ_SVG_RENDERER_CAIRO=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_LOGGING=1 -DMOZ_USER_DIR=\"Mozilla\" -DHAVE_UINT64_T=1 -DMOZ_XUL=1 -DMOZ_PROFILELOCKING=1 -DMOZ_MORK=1 -DMOZ_DLL_SUFFIX=\".dll\" -DJS_THREADSAFE=1 -DMOZILLA_1_8_BRANCH=1 -DMOZILLA_LOCALE_VERSION=\"1.8b5\" -DMOZILLA_REGION_VERSION=\"1.8b5\" -DMOZILLA_SKIN_VERSION=\"1.8\"  -D_MOZILLA_CONFIG_H_ -DMOZILLA_CLIENT /cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp
nsFileSpec.cpp
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(175) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(198) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(199) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(373) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(379) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(461) : warning C4996: 'strcpy' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(73) : see declaration of 'strcpy'
        Message: 'This function or variable may be unsafe. Consider using strcpy_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:/building_Mozilla/source/BRANCH_1_5_x/mozilla/xpcom/obsolete/nsFileSpec.cpp(462) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
l:\building_mozilla\source\branch_1_5_x\mozilla\xpcom\obsolete\nsFileSpecWin.cpp(242) : warning C4244: '=' : conversion from 'time_t' to 'nsFileSpec::TimeStamp', possible loss of data
l:\building_mozilla\source\branch_1_5_x\mozilla\xpcom\obsolete\nsFileSpecWin.cpp(420) : error C2440: 'initializing' : cannot convert from 'const unsigned char *' to 'unsigned char *'
        Conversion loses qualifiers
l:\building_mozilla\source\branch_1_5_x\mozilla\xpcom\obsolete\nsFileSpecWin.cpp(681) : warning C4996: 'strcat' was declared deprecated
        D:\Programme\Microsoft Visual Studio 8\VC\include\string.h(78) : see declaration of 'strcat'
        Message: 'This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_DEPRECATE. See online help for details.'
make[3]: *** [nsFileSpec.obj] Error 2
make[3]: Leaving directory `/cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/firefox_vc8/xpcom/obsolete'
make[2]: *** [tier_2] Error 2
make[2]: Leaving directory `/cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/firefox_vc8'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/cygdrive/l/building_Mozilla/source/BRANCH_1_5_x/firefox_vc8'
make: *** [build] Error 2
Created attachment 212317 [details] [diff] [review]
Patch rv1.1

Sorry. I fogot the VC8's problem. This patch is used in Trunk.
Attachment #211028 - Attachment is obsolete: true
Attachment #211029 - Attachment is obsolete: true
Attachment #212317 - Flags: superreview+
Attachment #212317 - Flags: review+
Attachment #212317 - Flags: approval1.8.0.2?
Attachment #212317 - Flags: approval-branch-1.8.1+
Attachment #211028 - Flags: approval1.8.0.2?
fixed the bustage on 1.8 branch too.
http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=MOZILLA_1_8_BRANCH&branchtype=match&dir=mozilla%2Fxpcom%2Fobsolete&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2006-02-18&maxdate=2006-02-19&cvsroot=%2Fcvsroot

The patch was created by pavlov.
http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=HEAD&branchtype=match&dir=mozilla%2Fxpcom%2Fobsolete&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2006-02-10+12%3A00%3A00&maxdate=2006-02-10+13%3A00%3A00&cvsroot=%2Fcvsroot

Comment 20

12 years ago
thanks, works again :)
Comment on attachment 212317 [details] [diff] [review]
Patch rv1.1

Please combine this with the bustage fixes for approval on the 1.8.0 branch -- this one won't work alone.
Attachment #212317 - Flags: approval1.8.0.2? → approval1.8.0.2-
Comment on attachment 212317 [details] [diff] [review]
Patch rv1.1

Daniel:

This patch already fixed the bustage.
Attachment #212317 - Flags: approval1.8.0.2- → approval1.8.0.2?
Comment on attachment 212317 [details] [diff] [review]
Patch rv1.1

approved for 1.8.0 branch, a=dveditz for drivers
Attachment #212317 - Flags: approval1.8.0.2? → approval1.8.0.2+
checked-in to 1.8.0 branch.
Keywords: fixed1.8.0.2
(Assignee)

Updated

12 years ago
Keywords: fixed1.8.0.2 → verified1.8.0.2
Masayuki, can you please verify this on the 1.8 branch as well? I'm not clear on how to verify it. Thank you.
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.