624806 - Repeated rebuild of msf summary files beginning in version 3. profile on Samba network share (CIFS)

Status: UNCONFIRMED

(MailNews Core :: Database, defect)

Description

[xelalex]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.04 (lucid) Firefox/3.6.13
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7

Setup:
I have one account defined. The folders and files for this one and for the Local Folders reside on a Samba share hosted on an Ubuntu 8.04 server. Thunderbird runs on an Ubuntu 10.04 box. 

Observations:
1) Whenever I select a message folder for the first time after opening Thunderbird, the summary file for that folder gets rebuild, which is fairly time consuming for larger folders. 

2) Occasionally, strange folders named "cifs{4 digit hex number}" are created underneath the mail account folder, e.g. cifs32e1, cifs70d3. 

3) The sorting order set for a folder is not remembered. 


Reproducible: Always

Steps to Reproduce:
1. Set up Thunderbird with messages stored on Samba share. 
2. Close Thunderbird
3. Open Thunderbird
4. Select message folders
5. back to step 2.
Actual Results:  
In step 4, summaries for selected folders get rebuilt.


To resolve observation 1), I already tried the following, without any success:

a) Delete all msf files
b) Compact folders
c) Re-create mail account in new account, move all messages from Inbox of old to Inbox of new account, then delete old account.

Comment 1

[xelalex]

Attachment: Sample of three "CIFS..." folders

Comment 2

[WADA:World Anti-bad-Duping Agency]

> 2) Occasionally, strange folders named "cifs{4 digit hex number}" are created
underneath the mail account folder, e.g. cifs32e1, cifs70d3. 

Upon restart, Tb automatically detects unkown/new mail folder of XXX if file named XXX exists in Tb's mail directory.
Tb doesn't generates file named "CIFS...". As file is named "CIFS...", it looks generated by CIFS software you are using and it seems work file.
Do you set Tb's mail directry as work directory or current directory of CIFS software you are using?

This part is INVALID. User shouldn't keep user file under Tb's mail directory.

> 1) Whenever I select a message folder for the first time after opening
Thunderbird,
> the summary file for that folder gets rebuild, (snip)

Tb checks file size and timestamp of of <mail_folder_name> and Tb compares it with saved data in <mail_folder_name>.msf file upon mail folder open. If inconsistncy is found(outdated .msf codition), Tb automatically executes rebuild-index inrernally.
Is file size/timestamp returned from CIFS server/client to Tb consistent for Tb?

Comment 3

[Wayne Mery (:wsmwk)]

xelalex, 
1. "Do you set Tb's mail directry as work directory or current directory of CIFS
software you are using?"
2. "Is file size/timestamp returned from CIFS server/client to Tb consistent for
Tb?"

Comment 4

[xelalex]

Wada-san, Wayne, 

sorry for the long pause. There havbe been many things going on recently. 

> 1. "Do you set Tb's mail directry as work directory or current directory
> of CIFS software you are using?"

"CIFS software" in my case mean that I mount a Samba share from my file server on my desktop with OS means. This is controlled via /etc/fstab. So there is no work directory I set. No other software creates anything underneath the mail folder. 


> 2. "Is file size/timestamp returned from CIFS server/client to Tb consistent
> for Tb?"

I can't say whether it's consistent for Tb. At least all other software I'm using that is working on shares of my file server has no problems (Nautilus, VLC, OpenOffice, ...).

Comment 5

[Wayne Mery (:wsmwk)]

Also, did you use version 2, and see this problem in version 2?

Comment 6

[xelalex]

Never had this in version 2.

Comment 7

[Wayne Mery (:wsmwk)]

had you ever used 3.0?
or any 3.1.x earlier than version 3.1.7?
and seen the problem in those earlier releases?

Comment 8

[xelalex]

I switched to Evolution for my mail client, and to my suprise, cifsXXXX files also started to appear there. After some more digging around, I found this: 

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/478649

There seems to be a problem with CIFS/Samba in recent Linux kernels. So you can close this incident as invalid.

Comment 9

[Wayne Mery (:wsmwk)]

> There seems to be a problem with CIFS/Samba in recent Linux kernels.

so how *precisely* does this fit in with comment 0 and indexes being rebuilt?? https://bugs.launchpad.net/ubuntu/+source/linux/+bug/478649 doesn't seem to give technical information about what happens and why.

Comment 10

[xelalex]

Somewhat more technical information is here: 

http://samba.2283325.n4.nabble.com/Problems-with-rename-to-open-file-and-open-unlink-create-td2521316.html

I suspect that the CIFS problems affect TB's consistency check. With the mail folder residing on a local drive, everything works ok. When moving it to the CIFS share, problems start. Unless TB does things differently based on the mail folder location, i.e. whether it's mounted, symlinked, or whatnot (which would be surprising), CIFS must be causing the problems. 

I will try using an NFS share next.

Comment 11

[crapo2]

I have seen all of the problems described on this thread. However, I don't understand how the "it's all samba's fault" explains that settings like sort order are not remembered. 

Is there a method of mounting a network drive that is known to work with Thunderbird?

Comment 12

[Ruud]

Using TB6.0 on Windows XP SP3, TB profile on H: (Samba share: FreeBSD 7.3 with Samba 3.5.5), POP3 accounts, mail filters move selection of incoming mail to specific Local (sub-)Folders. Known bug: mail comes in, gets moved, mail folders do NOT show new unread mail (mail filters do NOT mark moved mail as read). Also new mail is NOT visible in folder view (so user doesn't even see the new mail :-( ). Workaround: delete all *.msf files and let TB rebuild them: mail visible in folder view AND shown as unread again.... Getting seriously annoyed here. Willing to use vfs_full_audit on Samba share and tcpdump on server to log activity, but how to get debug info from TB ?

Comment 13

[Wayne Mery (:wsmwk)]

If you never had the problem in version 2, and if this problem is highly reproducible, then I suspect the quickest path to success is to find the regression range of the problem. Meaning, test version 3.0. If 3.0 fails then test some of the development builds leading up to version 3.0.  If 3.0 works, then test some of the development builds leading up to version 3.1.

Unless you no longer seen the problem when using version 9 or newer.

Comment 14

[John]

Mounting using cifs with the "noserverino" made this problem go away for me. This may not be a Thunderbird issue but rather just an issue with how shares are mounted. Note that my samba server is Windows server.

Some details: I am running Thunderbird 9.0.1 on Ubuntu 11.10 connecting to a samba share on a Windows server. Thunderbird was rebuilding summary files pretty much every time I clicked on a folder. There was also the problem with the global search where if I clicked on a search result, the new tab that would open would be empty. I would have to click on the folder where the email of the search result was stored and have Thunderbird rebuild the summary file and then click the result for it to show up. (I checked the file system and it turns out that the summary file was being deleted but not rebuilt after clicking a email in the global search result.)

Here's what I did to get it to work:
1. Mount the share permanently
(see https://help.ubuntu.com/community/MountWindowsSharesPermanently)
Open the file:
sudo gedit /etc/fstab

Add this to the end of the file, changing the parameters as needed (i.e server, directory, and share_name). Follow the directions in the MountWindowsSharesPermanetly Guide to setup the .smbcredentials file.
//server/directory /media/share_name cifs rw,nounix,noserverino,credentials=/root/.smbcredentials,iocharset=utf8,nobrl,file_mode=0777,dir_mode=0777,uid=1000,gid=1000 0 0

noserverino does this: "Client generates inode numbers (rather than using the actual one from the server) by default." More cifs options and details can be found here: http://www.samba.org/samba/docs/man/manpages-3/mount.cifs.8.html. For other users, these other options may be useful (especially _netdev):

//server/directory /media/share_name cifs defaults,rw,noexec,nosuid,nounix,noserverino,credentials=/root/.smbcredentials,iocharset=utf8,nobrl,_netdev,file_mode=0777,dir_mode=0777,uid=1000,gid=1000 0 0

2. Update the unmount order to prevent CIFS from hanging during shutdown.
According to https://help.ubuntu.com/community/MountWindowsSharesPermanently, the following commands should be run:
sudo update-rc.d -f umountnfs.sh remove
sudo update-rc.d umountnfs.sh stop 15 0 6 .

This may or may not work for everyone.

3. Update grub.
In my case, Ubuntu would hang on unmounting these shares. Making a change to grub seemed to fix this. More info here: http://ubuntuforums.org/showthread.php?t=1741668&page=7. Just add the option "reboot=pci" to GRUB_CMDLINE_LINUX in /etc/default/grub:

sudo gedit /etc/default/grub

Look for:
GRUB_CMDLINE_LINUX=""

and change it to:
GRUB_CMDLINE_LINUX="reboot=pci"

save, exit and update grub:
sudo update-grub

After doing all of this, Thunderbird no longer rebuilt the summary files. The global search works as expected and will always show results.

Comment 15

[Matthew Jurgens]

Nice to see John has found a workaround for a specific combination. This does not help Samba on Linux with Windows client users like myself. I have described my version of the problem in detail at http://getsatisfaction.com/mozilla_messaging/topics/tb3_repeatedly_rebuilding_summary_files_for_many_folders

I got the problem as soon as going to 3.0 and it is still with me at v9

Comment 16

[Wayne Mery (:wsmwk)]

Leaving this bug open for further comments, but we seem to have hit a wall? Except for Matthew's example ...

(In reply to Matthew Jurgens from comment #15)
> Nice to see John has found a workaround for a specific combination. This
> does not help Samba on Linux with Windows client users like myself. I have
> described my version of the problem in detail at
> http://getsatisfaction.com/mozilla_messaging/topics/
> tb3_repeatedly_rebuilding_summary_files_for_many_folders
> 
> I got the problem as soon as going to 3.0 and it is still with me at v9

Matthew, could file a new bug for your issue, citing this bug number and also your getsatisfaction link?  Then please cite the bug number here. Thanks


FWIW, list of bugs mentioning SMB, CIFS, Samba

[1] created roughly since v3 beta - (22) https://bugzilla.mozilla.org/buglist.cgi?bug_id=546287%2C564794%2C536873%2C588207%2C539389%2C581707%2C623448%2C545126%2C566289%2C653856%2C630605%2C576850%2C668952%2C527381%2C624653%2C723497%2C527331%2C702127%2C624806%2C737874%2C749983%2C752866;bug_id_type=anyexact;list_id=3323873;query_format=advanced

[2] touched in last 5 years - (78) https://bugzilla.mozilla.org/buglist.cgi?bug_id=353308%2C357757%2C334632%2C215390%2C172387%2C201394%2C238357%2C123451%2C242579%2C112072%2C55814%2C85613%2C180871%2C255066%2C98538%2C124229%2C136049%2C167700%2C216439%2C248439%2C250961%2C283490%2C142196%2C390992%2C151486%2C264879%2C126146%2C169145%2C102910%2C441886%2C364695%2C419856%2C206247%2C172907%2C117577%2C68877%2C414350%2C484340%2C494273%2C502555%2C497627%2C220179%2C546287%2C189875%2C564794%2C536873%2C343907%2C588207%2C539389%2C480557%2C581707%2C623448%2C143451%2C42538%2C545126%2C566289%2C653856%2C630605%2C62480%2C241220%2C576850%2C668952%2C319670%2C442908%2C527381%2C232047%2C624653%2C723497%2C380275%2C527331%2C702127%2C624806%2C386306%2C435986%2C378425%2C737874%2C749983%2C752866;bug_id_type=anyexact;list_id=3323844;query_format=advanced

Comment 17

[Matthew Jurgens]

Open bug 905576 as requested
https://bugzilla.mozilla.org/show_bug.cgi?id=905576

Comment 19

[Eric Valette]

I disagree for the duplicates: I do have lost of data because of buggy error handling!

Comment 20

[Wayne Mery (:wsmwk)]

(In reply to Eric Valette from comment #19)
> I disagree for the duplicates: I do have lost of data because of buggy error
> handling!

Your slowness issue is covered in other bugs.  Your own testing demonstrated that your other problems are likely covered by this bug and others like it.  If your issue doesn't get resolved by the eventual fixing of these bugs then we can reopen yours.

Comment 21

[Eric Valette]

This bug is tree years old so I guess I have to be patient. I'm considering moving to kmail as soon as an ews implementation for exhange calendar is available.

Comment 22

[Wayne Mery (:wsmwk)]

Matthew, Eric, Ignaz,
Does this issue still reproduce for you? Any on Windows?

Comment 23

[Matthew Jurgens]

I switched to "file per message" as soon as it was made available in the beta versions. That resolved it for me.

Comment 24

[Ignaz Forster]

I'm sorry, I can't test this any more: I changed my employer and don't have any Samba setup available for testing...

Comment 25

[Eric Valette]

No chnage for me but I'm stuck on 60.9 for various reasons...