242579 - Multiple mail copy/move hangs when drive of mail directry is different from one for profile directry (bug 242970 also occurs). This causes hang of Junk move/Junk purge and multiple mail delete.

Status: VERIFIED FIXED

(MailNews Core :: Backend, defect)

Description

[WADA:World Anti-bad-Duping Agency]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8a) Gecko/20040504
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8a) Gecko/20040504

"Copy to"/"Move to" on more than two mails hangs.

Reproducible: Always
Steps to Reproduce:
1. Select 3 mails at mail list pane of POP3 account
2. Execute "Copy to" or "Move to" to another folder from context menu
Actual Results:  
(1) Select 3 mails at mail list pane of POP3 account
(2) Execute "Copy to" or "Move to" to another folder from context menu
    ("Copy to" is safer for testing)
(3) Mail count of target folder does not increase ( = N ).
    Timer mark was dislayed for the folder.
    This makes the target folder unaccessible.
(4) Terminate Mozilla
    Mail data of first mail was found in mail folder file
    but no data for second nor third one.
(5) Restart Mozilla
    Mail count of the folder became N+2.
(6) Click the folder
    Mail count of the folder was changed to N+1.
   (1 was reduced from count just after restart)
    First one in selected three mails was accessible in the target foldr. 

Expected Results:  
Copy or move of mails completes successufully.

This causes failure of automatic Junk move on manual marking as Junk, if
multiple mails are marked as Junk at same time.
This also causes failure of automatic Junk move by Junk filter, if multiple
mails are marked as Junk on automatic mail downloading.
And accessing to Junk folder(for example, move mail to Junk folder, comact Junk
folder) becomes impossible.
This situation could not be recovered by Mail&News only restart.
I experienced Junk mail move failure on builds after 4/30.

Comment 1

[WADA:World Anti-bad-Duping Agency]

This problem also occured on 2004050510-trunk build(Win32,ZIP).

In addition to hang in Junk Mail move to Junk folder, hang in Junk Mail Purge
occured.
If multiple mails are moved from Junk to Trash by Junk Purge, move hangs.
This causes Trash folder busy.
So deletion of mail also became impossible.
 

Comment 2

[David :Bienvenu]

this works in my 05/01 build - I'll try a newer trunk build.

Comment 3

[David :Bienvenu]

my trunk build works fine, copying multiple local messages between folders...

Comment 4

[WADA:World Anti-bad-Duping Agency]

David, thanks for your test. 
I experienced this bug's problem on 2004/05/01 build too.
As problem does not occur on your system, problem depends on environment.

My mail directry is placed on D: drive and problem occurs on any account
including Local Folders.
So I tested with new pop3 mail account - mail directry is placed in "C:\Document
and Settings" with 2004/05/05-trunk build.
Result is : No problem occured on new account even when multiple copy/move.

My guess is as follows;

My mail directry is placed on D: drive and drive size is 4GB(Win-2K,SP4,NTFS).
One of my activities before problem is mail directry backup(copy to C:) & delete
& defrag & restore(copy from C:).
Because my D: drive is 4GB large and defragged and more than 2GB is used before
restore, mail directry was placed abeve 2GB by my back-up&delete&defrag&restore.
Since 1.7RC1 release build and latest 1.7 brach build does not have problem,
trunk builds have something wrong in handling of directry(and/or file) placed
above 2GB.

Can someone comfirm this?

Comment 5

[David :Bienvenu]

my drive is 70GB and I don't have any problems. I think there's a pretty good
chance my mail directory is not in the first 2GB. Am I misunderstanding something?

Comment 6

[WADA:World Anti-bad-Duping Agency]

David, do you place your mail directry in Windows's personal directry such as
"C:\Document Settings\<UserName>\...\Application Data\Mozilla..."?
If yes, pointer to directry block chain is usually below 2GB because this
directry is created at MS Windows installation.
Even not, if fragmentation exists on the drive and space is available below 2GB,
pointer to directry block chain becomes below 2GB.

In my environment, since I repeated defrag, free spaces was gathered again and
again, and almost free space in first half of drive was padded by files before
restore of my mail directry.
This makes pointer to directry block chain above 2GB.

If high level directry/file I/O request is used, this will not usually cause
application problem.
But for mail, which is Unix Mbox file, low level directry/file I/O request is
often used for performance reason.
(Not sequential access. Random access=read/write to specific address is usualy used)
If this is true, physical pointer value of directry block chain sometimes
affects application program.

I can not imagine any other reason why problem occurs on
"D:\MAIL-NEWS\Mail\Local Folders" directry but does not occur "C:\Documents and
Settings \<UserName> \Application Data \Mozilla \Profiles \<Profile>
\<random>.slt \Mail\test.pop3.server" directry in my system.

These are the reasons why I suspect "above 2GB" problem.

Are there any possibility of such situation on mail directry/file I/O?

Comment 7

[WADA:World Anti-bad-Duping Agency]

[Possible scenario]

(A) When single mail copy/move
  (A-1) Open file
  (A-2) Write mail data after end of extention of the file
  (A-3) Close file
(B) When multiple mail copy/move
  (A-1) Open file
  (A-2) Write first mail data after end of extention of the file
  (A-3) Calculate next block address to write
  (A-4) Write second mail data next to first data
    |
  (A-X) Close file
Since block address is obtained by file open, there is no problem in single mail
or first mail writing.
But multiple copy/move, block address other than first mail is calculated.
If this calculation step has some flaw, problem may occur on second mail writing.

[Real result]

When multiple move/copy hang, first copy/move mail was always written in mail
filder file, but second mail was always not written.
This indicates hang at (A-3) or (A-4).

Comment 8

[David :Bienvenu]

unless I'm misunderstanding you, you're saying the file system is
broken...Mozilla just opens the file and writes to it, and doesn't care where it
is on the disk. We seek around in the file, but those are relative offsets,
relative to the beginning of the file.

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #8)
 
> you're saying the file system is broken...

No. File system itself is not broken.
(a) Receiving multiple mails from POP server, copy/move single mail,
    compact folder, emptty trash has no problem.
    Problem is only when multiple mail copy/move and hang always occurs
    just after writing first mail to target mail folder file.
(c) When I delete ".msf", Mozilla recreates ".msf" successfully
    and no problem in mail folder access.
(d) Problem occurs only with trunk builds.
    No problem with Mozilla 1.7RC1(Release version) and 1.7 branch nightly.

> Mozilla just opens the file and writes to it, and doesn't care where it is on
the disk. 
> We seek around in the file, but those are relative offsets, relative to the
beginning of the file.

I tested with local directory of "D:\test.pop3.server", which is copied from
Mozilla's profile directry to my D: drive, and problem was re-created.

Funny. Bug of MS Windows or BIOS problem?
Since my HDD is 10GB drive and have two partitions, last half of D: drive became
above 8GB.
# Always "SPEC" of MS ;-)
# If this is MS DOS or Win 3.1 or 95, I suspect him at first every time :-)
# But this is Win 2K... However possbility is not so low...

Or GRE related?
I have installer build of 1.7RC1 release version and ZIP version of 1.7 nightly,
but ZIP version only for trunk builds.

Anyway, since no problem on 1.7, could you check difference between 1.7 branch
and trunk around mail copy/move?
I'll try installer build of trunk nightly.
Thanks, David.

Comment 10

[WADA:World Anti-bad-Duping Agency]

Problem began on 2004-04-29 trunk build(Win32,ZIP build).
No problem on 4/28,4/27,4/24,4/21 trunk nightly builds(Win32,ZIP build).

Comment 11

[WADA:World Anti-bad-Duping Agency]

I could isolate the problem.
It was not "2GB"/"8GB" problem nor bug of MS Win.
It was problem on drive other than C:

[Test Results]

[Case-1. Server Settings - local folder display]

  [Case-1-1. 1.7 RC1]
    (1) For D:\test.pop3.server
        When prefs.js specifies correct path for D:\test.pop3.server,
            user_pref("mail.server.serverN.directory", "D:\\test.pop3.server");
        local directory of Server settings becomes
        D:\test.pop3.server  (No problem)
    (2) For C:\test.pop3.server 
        When prefs.js specifies correct path for C:\test2.pop3.server,
            user_pref("mail.server.serverN.directory", "C:\\test2.pop3.server");
        local directory of Server settings becomes
        C:\test2.pop3.server (No problem)

  [Case-1-2. 2004-04-29 Build] 
    (1) For D:\test.pop3.server
        Even though prefs.js specifies correct path for D:\test.pop3.server,
            user_pref("mail.server.serverN.directory", "D:\\test.pop3.server");
        local directory of Server settings was changed to
        \\.\D:\test.pop3.server
        (Not saved in prefs.js if Server Settings is canceled)
    (2) For C:\test.pop3.server 
        When prefs.js specifies correct path for C:\test2.pop3.server,
            user_pref("mail.server.serverN.directory", "C:\\test2.pop3.server");
        local directory os Server settings was not changed.
        C::\test2.pop3.server
         (Preceding \\.\ is not added when C: drive)

[Case-2. When mail.server.serverN.directory = <Drive>:\\xxx.yyy.zzz format]

  [Case-2-1. 1.7 RC1]
    (1) For D:\test.pop3.server
       (1-A) Single copy : OK
       (1-B) Multi  copy : OK
    (2) For C:\test.pop3.server 
       (2-A) Single copy : OK
       (2-B) Multi  copy : OK

  [Case-2-2. 2004-04-29 Build] 
    (1) For D:\test.pop3.server
       (1-A) Single copy : OK
       (1-B) Multi  copy : HANG <= This is the problem
    (2) For C:\test.pop3.server 
       (2-A) Single copy : OK
       (2-B) Multi  copy : OK

[Case-3. When mail.server.serverN.directory = "\\\\.\\D:\\test.pop3.server]

  [Case-3-1. 1.7 RC1]
    (1) For \\.\D:\test.pop3.server
       (1-A) Single copy : OK
       (1-B) Multi  copy : HANG (Same situation as this bug)

  [Case-3-2. 2004-04-29 Build] 
    (1) For \\.\D:\test.pop3.server
       (1-A) Single copy : OK
       (1-B) Multi  copy : HANG  (Same situation as this bug)

[Summary]

 (Summary-1)
   Case-3 says that ;
     When local directry is specified as \\\\.\\D:\\test.pop3.server,
       - single copy is not affected
                on both 1.7RC1 and 2004-04-29 trunk.
       - multi copy is affected
                on both 1.7RC1 and 2004-04-29 trunk. (This bug)
 (Summary-2)
   Case-1 says that ;
     If local directry is placed on drive other than C:
         (D:\test.pop3.server, D:\\test.pop3.server in prefs.js)
       2004-04-29 thinks it as \\.\D:\test.pop3.server
       (This is the cause of the problem).
     However, if local directry is placed on C:
       2004-04-29 thinks it as C:\test2.pop3.server (not changed)

Case-2 & Case-3 indicate that \\.\D:\test.pop3.server
is used at least in multile copy of 2004-04-29 build.
Mozilla can do nothing until Windows returns time out error.
This causes long hang because time out takes long time.
This is the main cause of this bug, I believe.

Since this change is not performed for directry on C: drive,
problem did not occur when mail directry was placed on C: drive.

Change in [Case-1] is introduced by fix for Bug 241708, I think.
> 2004-04-29 01:16 mozilla/ xpcom/ io/ nsLocalFileWin.cpp 	1.118
> Bug 241708 nsIFile support for unc paths is almost entirely broken
> Handle local computer (\\.) allowing enumeration of local drives,
> and access to local computer from local drives via |parent|.
> The parent of local computer is null.

Comment 12

[WADA:World Anti-bad-Duping Agency]

David, when local directry is changed to \\.\D:\test.pop3.server(internaly or
externaly), why all other folder access doesn't wait but multiple copy/move
waits time out?
I do not know meaning of "\\.\D:\..." format but it sounds like another local
resource representation on MS Windows.
If true, it seems that this is the reason why single copy/move does not affected
even if local directry setting is changed to "\\.\D:\test.pop3.server"
externaly(prefs.js).
Or this means that only multiple copy/move uses local directry setting properly?

Comment 13

[WADA:World Anti-bad-Duping Agency]

I've opened Bug 242970 for local directry change other than C: problem.

Comment 14

[David :Bienvenu]

since 1.7, we've specified local directories with profile-relative paths, so
instead of using the server "directory" pref, we use the server "directory-rel"
pref. Did you edit your prefs.js by hand? You should just use the UI for setting
the directory, and it should do the right thing...

Comment 15

[WADA:World Anti-bad-Duping Agency]

In [Case-3], both .directry and .derectry-rel are changed in prefs.js, because I
saved Server settings on 2004-04-29 build instead of prefs.js editing.
Case-3 says that same problem of this bug occurs on any Mozilla, if the
converted format of file path is already written in prefs.js.
This means second write in multiple copy/move uses ".directry" but others use
".directry-rel", or vice versa?

Comment 16

[David :Bienvenu]

I'm not seeing a hang, but I do see problems - the underlying problem is that
the uri we're creating for these messages is not correct because of the d:\ in
the path. I believe this used to work OK, so something must have changed in uri
parsing/escaping...

Comment 17

[WADA:World Anti-bad-Duping Agency]

"Hang" on my environment is long wait for time out, I think.
I use Net BIOS over TCP and LAN client to share files among Notebook(Win
2K,Mozilla runs), Desktop(MS Win 2K, OS/2 and Samba on Linux), but other clients
will never be concurrently active, because other OSes are installed on one
Desktop machine.

Using converted "[ProfD]../../../../../../../..//./D:/test.pop3.server" format
seems to force network resource access, and seems to cause long wait until
timeout if other servers are not active.

Why directry-rel was NOT changed to this format when C: ?
Why directry-rel was changed to this format when other than C: ?
Why long wait occurs only when multiple mail copy/move if directry-rel is
converted to this invalid format?

*******************************************************
Following is Bug 242970 Comment #2.
*******************************************************

2004-04-29 build changed both directry & directory-rel.

[D: drive] 

(Before restart)
user_pref("mail.server.server4.directory", "D:\\test.pop3.server");
user_pref("mail.server.server4.directory-rel",
"[ProfD]../../../../../../../../D:/test.pop3.server");

(Local directory didplay) \\.\D:\test.pop3.server

(After Server settings save)
user_pref("mail.server.server4.directory", "\\\\.\\D:\\test.pop3.server");
user_pref("mail.server.server4.directory-rel",
"[ProfD]../../../../../../../..//./D:/test.pop3.server");

[C: drive]

(Before restart)
user_pref("mail.server.server11.directory", "C:\\test2.pop3.server");
user_pref("mail.server.server11.directory-rel",
"[ProfD]../../../../../../../test2.pop3.server");

(Local directory didplay) C:\test2.pop3.server (Unchaged)

(After Server settings save)
.directry & .directory-rel were not changed if C: drive

Comment 18

[WADA:World Anti-bad-Duping Agency]

dir "\\.\D:\test.pop3.server" returned next result.
> Drive \\.\D: Volume Label EDICUBE-D (<= This is label of my local D: )
>  Volume Serial ACF3-18EC
>  \\.\D:\test.pop3.server
>   -- file list of directry --
And dir "\\\\.\\D:\\test.pop3.server" returned network error.
> dir "\\\\.\\D:\\test.pop3.server"
>  Netwok Path not found.
(I stopped workstaion by NET STOP WORKSTATION and inhibited NETBIOS access by
filrewall setting for test.)

My guess is as follows;
Since single mail copy/move has no problem even when
"\\\\.\\D:\\test.pop3.server" is written as .directry in prefs.js, this is
translated to proper format (escaping of \ by \\ is properly handled =>
\\.\D:test.pop3.server) when single mail copy/move.
But "\\\\.\\D:\\test.pop3.server" is used without this escaping handling on
.directry value in second(or after) mail write process of mutiple mail copy/move.

Comment 19

[WADA:World Anti-bad-Duping Agency]

Same problem occured on local directry(mail directry) on C: drive, when I
defined Mozilla's profile directry on D: drive.
Problem always occurs when mail directry is defined on different drive from one
for profile.

Workarounds :
(A) Move mail directry(local directry) to the drive on which profile is defined.
(B) Move profile directry to the drive on which mail directry(local directry) is
defined.

Additional Symptoms :
Since target folder becomes busy and unaccessible, many symptoms can be observed
on target folder.
(1) No mail list is displayed after hang.
(2) Compact folder can not be initiated.
(3) Additional move/copy to the target folder is impossible.
These symptoms are often observed on Trash folder, because automatic Junk Purge
is usualy multiple mail move to trash folder and users often delete(=move to
trash folder) multiple mails at a time.

Comment 20

[David :Bienvenu]

should be fixed by backing out part of bug 241708 - please try tomorrow's build,
thx.

Comment 21

[David :Bienvenu]

resolving fixed.

Comment 22

[WADA:World Anti-bad-Duping Agency]

Thanks David, but 1.8a1 release build(Zip)/Win-2K (build at 2004-05-09-20 21:44)
still have Bug 242970 problem, and, needless to say, problem of this bug.
This bug should be set blocker of 1.8(or prior releases), I believe.

Comment 23

[David :Bienvenu]

I'm not sure that build has my fix - I'm not sure my checkin made the 1.8a
branch, since I'm not sure when the branch tag was made...can you try a trunk
build before re-opening this?

Comment 24

[WADA:World Anti-bad-Duping Agency]

Sorry, problem was fixed by trunk build of 2004052108(Win32,Zip).
Your patch is perfect. Thanks again, David.
I change back status of this bug "FIXED".

But, David, I can not believe timeless have completely understood exact reasons
why problems occured.
Could you watch Bug 241708, please.

Comment 25

[Dick Miller]

*** Bug 244327 has been marked as a duplicate of this bug. ***