Closed Bug 428147 Opened 16 years ago Closed 14 years ago

Using same thunderbird directory in windows and linux is now broken

Categories

(MailNews Core :: Database, defect)

x86
Linux
defect
Not set
major

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 462665

People

(Reporter: jd1008, Assigned: Bienvenu)

Details

User-Agent:       Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.13) Gecko/20080325 Fedora/2.0.0.13-1.fc7 Firefox/2.0.0.13
Build Identifier: 1.5.0.14, 2.0.0.6, 2.0.0.12

On Linux (I have dual boot),
/sda1 is mounted read/write after installing:
ntfs-3g-1.1120-1.fc7
ntfsprogs-1.13.1-8.fc7
fuse-kmdl-2.6.23.15-80.fc7-2.7.3-8_9.fc7
fuse-libs-2.7.3-8_9.fc7
ntfsprogs-gnomevfs-1.13.1-8.fc7
fuse-2.7.3-8_9.fc7

I set up a sym link in my home dir
.thunderbird -> /sda1/Documents and Settings/Application Data/Thunderbird

I fire up thunderbird in. and enter my email account
address (gmail account) and download my mew messages.
Clicking on each new message, it is either empty, or
it contains a fragment of a 2 y/o message. Always
same fragment.
Sometimes, same message shows multiple times in the Inbox,
and each instance is identical to the other instances of
same message.

Booting back into windows, the msf gets re-populated,
and the messages which were downloaded under the linux version
of t-bird, are all there and they are OK.

Since I cannot afford to have my email strewn all over creation,
you need to make the format of the Inbox and Inbox.msf
identical for Windows and Linux.

If you have any ideas how I can get around this problem,
and still use same directory for thunderbird in windows and
linux, please let me know.

Best regards,

JD


Reproducible: Always

Steps to Reproduce:
1. On linux, mount your windows partition read/write
2. in your home dir, create a symlink .thunderbird which
   points to /mountPoint/Documents and Settings/Application Data/Thunderbird
3. fire up thunderbird, and login to your email account.
4. Send a few emails to yourself.
5. download the new email messages.
6. try to read them.
Actual Results:  
Empty message bodies,
Duplicated Inbox entries for noew messages
sometimes message bodies are fragments of
an old message body. When this is the case,
always the same fragment appears as the
body of the message(s).


Expected Results:  
Normal and good message in Inbox

I tried it with 1.5.0.14, 2.0.0.6, 2.0.0.12
and all exhibit the same behavior.
It works for me.  When I follow your steps 1 - 6, I can read the messages normally.  I do not see any difference in the file formats.

To investigate further, you could rename your Inbox, then download some test messages into a new Inbox.  If the problem persists, you could attach the files in this bug report to illustrate how the formats differ.
Status: UNCONFIRMED → RESOLVED
Closed: 16 years ago
Resolution: --- → WORKSFORME
Bug is still not resolved because you did not have an Inbox
folder as large as mine - mine is 2.6GB big.

I saved my Inbox, and created an empty Inbox, and sent
myself new messages of 1-4  KB in size. 
No problems at all.

Then I took another small folder containing about 100 messages,
and catenated it repeatedly to a new file, until the new
file size was just a hair under 2.6GB.

Then I fired up Firefox and used the web to send myself
a few messages.

Then clicked on "Get Mail" in Firefox, and the messages 
came in. But when I clicked on them, I was being shown
the the body of some other message - actually, seemed
like a combination of other messages,
none of which were the new message.

However, if I "tail" the Inbox for about 500 lines, I can
see the message looks just fine (headers and body). In other words, ntfs-3g
did not screw up at all. The message is intact and in the folder.

It is indeed Thunderbird that is screwing up!!


So, the BUG is still there.
Status: RESOLVED → UNCONFIRMED
Resolution: WORKSFORME → ---
(In reply to comment #2)
> Then I took another small folder containing about 100 messages,
> and catenated it repeatedly to a new file, until the new
> file size was just a hair under 2.6GB.

Catenated on what OS? With line mode? Or with binary mode?
On what OS "Rebuild Index"(.msf recreation) was done? 

One of next can occur, depends on last data in file, when Tb's mail box file(==Unix Mbox file) is catenate by utility or command.
  (a) non_CRLF/non_LF + "From - ..."
  (b) any +    [CRLF] + "From - ..."
  (c) non_CRLF + [LF] + "From - ..."
  (d) [CRLF]   + [LF] + "From - ..."
When (a), on both OS, "From -" can not be unix mbox separator line.
When (b), on both OS, "From -" is treated as unix mbox separator line by Tb.
When (c)/(d), on Win, "From -" is probably not unix mbox separator line for Tb.
Sorry but I don't know about (c)/(d) on Linux.

Please note that mail folder file of Tb(==Unix Mbox file) is never be simply a text file, even though it can be read/written in text mode. And, Tb accesses the Unix Mbox file in binary mode, so manual breaking of line break character in Unix Mbox file can easily cause problem.
If you need to test with big mbox, do next procedure.
 (1) Copy a unix mbox file as a mail folder file of Tb (say folde-A)
 (2) Create folder-B for big mbox on Tb
 (3) Copy all mails in folder-A to folder-B, using Thunderbird
 (4) Repeat (3) till required size
Hello???
I dont agree at all.
Mail headers are STANDARD!!

On both windows AND Linux, the message header that
start a new message is (for example)
From - Thu Jan 26 16:49:09 2008

I see this is the case for both Linux and Windows.

Catenation is done via the cat command.
The cat command has no knowledge of line mode.
It just reads bytes and writes bytes regardless of file contents.
The tty driver, which interprets what constitutes a line,
has nothing to do with the cat command.

You asked on What OS? My bug report stated: "On Linux (I have dual boot)," ...etc

Please let the experts of the internals of TB address this issue.

Regards,

JD


(In reply to comment #4)
> You asked on What OS? My bug report stated: "On Linux (I have dual boot),"  ...etc

Where is your description about "catenation is executed under what OS and how the catenation is done"?
Since you say "catenate", "cat command on Linux" becomes top of guesses. But you say "dual boot", so "copy (/B) file+file+...file filex on MS Win" case is to be ruled out first. This is the reason why I asked "how catenated on what OS".

> Mail headers are STANDARD!!

I never talked about "Mail header". Where did I refer to "Mail header"? 
I refered to line-break character(last bytes of of the catenated file) and "From - ..." line(first line in the catenated file) in Unix Mbox file only, didn't I?

> Please let the experts of the internals of TB address this issue.

I'm never be an expert of the internal of Thunderbird. But it's required to clarify what is going on in what environment in what situation, before asking "experts of the internal of Thunderbird" for analysis of problem on you. At least, (a) evidence that no user error is involved, (b) evidence that culprit other than Tb's flaw in code doesn't exist, and (c) base that the phenomenon was caused by Tb's flaw in code, are required to ask for help to "experts of the internal of Thunderbird".   

> Catenation is done via the cat command.
> The cat command has no knowledge of line mode.

"cat command on Linux" seems to execute binary copy.
What is last 2-bytes of the catenated file? [CRLF](0x0D0A)? Or other?
(In reply to comment #2)
Other questions.

> Then clicked on "Get Mail" in Firefox, and the messages came in. 

What do you mean by "Get mail in Firefox"? What phenomenon is "the messages came in" on Firefox? What relation to this bug does Firefox have?
Confirmation of "new mail for test" via Web Mail Interface by Firefox?

> But when I clicked on them, I was being shown the the body of some other message
>  - actually, seemed like a combination of other messages, none of which were the new message.

Phenomenon on Firefox? Or Thunderbird?
If Thunderbird, after download of the new mails? Or before download?
If thunderbird, was "catenate" executed while Tb was active? Or after shutdown of Tb?

> However, if I "tail" the Inbox for about 500 lines, I can see the message looks just fine (headers and body).

Was the "tail" is executed while Tb is active? Or after shutdown of Tb?
Oh sorry, I misunderstood "tail" command on Linux. It was to view last 500 lines in file. (I misunderstood as "give tail to a file", "append data")

> the new file size was just a hair under 2.6GB.

Wraparound of offset is suspectable.
(32bit but signed initially, and changed to unsigned to support up to 4GB file)

"Order Received" column of message list pane is the offset value when local mail folder. (UID when IMAP, article number when NNTP)
What is displayed in "Order Received" column of newest mail?
What happens when sorted by "Order Received" column? 

Does problem occur even after explicit "Rebuild Index" on Linux?
("Rebuild Index" via folder properties)
Please note that previous question is to rule out "rebuild index is not done after file content change" case.

Another question to clarify your environment.
> gmail
POP3? Gmail IMAP? (probably POP3, because you tested with local file)
I also suspect the same problem. Like I stated previously, the new messages
are either displayed as empty or contain a fragment of a message near the start of the Inbox file.

Re: Order of messages: Always displayed by ASCENDING date, so that latest messages are at bottom.

Re: rebuild of Inbox.msf : Yes I did it every time - and it takes quiet a long time.

Re: account type: It is a pop3 gmail account.

Also, to add insult to injury, the Fedora distro build of TB 2.0.0.12 (i386) rpm 
just exits with status 1. It never starts up;  The x86_64 build rpm, starts up,
but nothing works. Clicking on GetMail is a no-op. Makes me wonder why the behavior of TB from the distro is so hosed?
Some other questions to clarify environment and situation.

> User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); (snip)
> Build Identifier: 1.5.0.14, 2.0.0.6, 2.0.0.12

64-bit version Tb? Or 32-bit version Tb on 64-bit Linux?
(Probably 64-bit Tb, because you say "The x86_64 build rpm")
When problem started to occur? After upgrade of OS or Tb or something else?
Or since initial use of 64-bit Tb on 64-bit Fedora?

> Booting back into windows, the msf gets re-populated,

What phenomenon do you mean by "re-populated"?
It sounds to imply "not get re-populated when Linux". Does it mean next?
  "Automatic/internal "Rebuild Index" during restart of Tb is ;
   - invoked when Tb on MS Win uses ".msf" which was updated by Tb on Linux
   - not invoked when Tb on Linux uses ".msf" which was updated by Tb on MS Win

> Re: Order of messages: Always displayed by ASCENDING date, so that latest
messages are at bottom.

Oh, not "Order of messages". "Order Received" column.
Click right most button at column header row of message list pane, and display "Order Received" column (==offset value when local mail folder).
What value is displayed in "Order Received" column for the mail at bottom?
  Positive/greater than 2GB? Or positive/small value? Or negative value?

Because you don't say "mail list pane display(Subject: etc.) for the mail at bottom is corrupted), "Rebuild Index" looks to handle file greater than 2GB properly. So, I think suspect is logic to obtain mail data based on offset value.
FYI.
Following is list of bugs of : open, "2GB" in summary, OS=Linux.
Possibly file handler's bug when Linux(Tb is a victim).
> Bug 278738 file:// directory not shown containing a file larger than 2GB
> Bug 384592 Files over 2GB are shown as zero-size in file open dialog
> Bug 383446 upload of huge (~2GB) files through file input fails silently
I dont think TB is a victim.
I think some TB modules which is responsible for openning
the mail folders is not compiled with full LARGEFILE support.

Now, getting back to you question about the Received Order,

All messages after the message with the Received order of 2143617487
have negative numbers.

Now for the wraparound problem, all new messages have a
Received Order of Zero (0)

So, what is the msf file used for? Is it used for lseek'ing to
the start of the message? If so, the program that is building
the msf file at the start of TB, must be using a signed integer,
causing the mail client to lseek to 0 (Zero) for new messages,
thus displaying the wrong message body.

Are any seasoned TB developers reading this? Please PLEASE respond !
(In reply to comment #11)
> Are any seasoned TB developers reading this? Please PLEASE respond !
To JD(bug opener):
Be quiet, please. Please don't shout.
Before asking "seasoned TB developers" for help, get back up of ".msf" file please, in order that developers can see ".msf" content.
 (1) Back up of ".msf" after "Rebuild Index" on MS Win
 (2) Back up of ".msf" after "Rebuild Index" on Linux

Changing to Product=Core. I don't know which issue, mail db or file handler, so set Component=MailNews:Database initially. Please change to appropriate one.
Assignee: nobody → bienvenu
Component: General → MailNews: Database
Product: Thunderbird → Core
QA Contact: general → database
I would appreciate it if you would just shut
up your comments and leave this bug I have submitted
to those who are knowledgeable and capable.
So, go and do something else, OK?
This is about as polite as I can respond to your
dumb comment.


(In reply to comment #13)
> I would appreciate it if you would just shut
> up your comments and leave this bug I have submitted
> to those who are knowledgeable and capable.

I, and I think the majority of seasoned bugzilla experts would agree, would appreciate it if you would read the Bugzilla Etiquette page before making further comments: https://bugzilla.mozilla.org/page.cgi?id=etiquette.html.

Several questions:
1. Are you using the /same/ versions on both Windows and Linux partitions?
2. Do you still see this on Trunk TB?
3. Does this problem exist only for files >=2GiB in size? If so, then this is merely a (known) problem with large file sizes. The workaround is simple: don't use 2GiB+ mboxes.

(In reply to comment #11)
> So, what is the msf file used for? Is it used for lseek'ing to
> the start of the message?

Finally, the .msf is the message summary file, which caches simple information about messages (file offsets, subject, author, date, threading information, etc.) to avoid opening and reading an mbox file.

> Are any seasoned TB developers reading this? Please PLEASE respond !

I am not exactly a "seasoned" TB developer, but that doesn't disqualify me from helping.
Answers to your questions:

1. Yes I am using TB 2.0.0.12
2. I am sorry, I am not familiar with "Trunk" TB. Do you mean
   building building it myself from CVS? If so, no. I used the
   Link http://www.mozilla.com/en-US/thunderbird/ which automatically
   selects the latest TB for my platform.
3. Yes.

Thank you for the tip re; TGB Inbox'es.
I find it strange that numerous other programs on 32 bit Linux
have no problem handling 8GB and larger files without problems.

Presently, I have started using Evolution, which avoids the whole size
issue and creates a seperate file for each message. I am sure I am going
to run into other issues with evolution, such as security...etc.
Guess I will have to wait and see.

Best regards,

JD
Product: Core → MailNews Core
JD are you willing to test beta of version 3?
 http://en-us.www.mozillamessaging.com/en-US/thunderbird/early_releases/downloads/
Whiteboard: closeme 2009-06-01
I tried the URL
http://en-us.www.mozillamessaging.com/en-US/thunderbird/early_releases/downloads/

it leads to:
http://gateway.2wire.net/xslt?PAGE=HURL07

Server not found      

Firefox can't find the server at gateway.2wire.net.

    *   Check the address for typing errors such as
          ww.example.com instead of
          www.example.com

    *   If you are unable to load any pages, check your computer's network
          connection.

    *   If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the Web.

-----------------------------------------------------------------------------

Anyway, I have completely gotten around the problem by splitting my Inbox into
2 files: Inbox and Inbox.0 (making sure I broke it at message boundary).

I think that the inability of tbird to handle files bigger than 2 or 4 GB is extremely lame!!! Linux has been addressing gigantic files for a long time!
tbird is still living in the past!
I would not blame it on some external factor like some libs of some third party package. I think it is a flaw and it has to be fixed.
(In reply to comment #18)
> (In reply to comment #17)
> > I tried the URL
> > http://en-us.www.mozillamessaging.com/en-US/thunderbird/early_releases/downloads/
> > 
> > it leads to:
> > http://gateway.2wire.net/xslt?PAGE=HURL07
> > 
> > Server not found      
> 
> gozer, might this be another bad mirror?  

Yes, but sounds like bouncer is not noticing these are not there. Might want to poke MoCo IT about this.
I finally got my network back and I downloaded the t bird3.0 beta 2

I crashed and popped up a death banner saying:

We're Sorry

Thunderbird had a problem and crashed.

To help us diagnose and fix the problem, you can send us a crash report.
(In reply to comment #19)
> (In reply to comment #18)
> > (In reply to comment #17)
> > > I tried the URL
> > > http://en-us.www.mozillamessaging.com/en-US/thunderbird/early_releases/downloads/
> > > 
> > > it leads to:
> > > http://gateway.2wire.net/xslt?PAGE=HURL07
> > > 
> > > Server not found      
> > 
> > gozer, might this be another bad mirror?  
> 
> Yes, but sounds like bouncer is not noticing these are not there. Might want to
> poke MoCo IT about this.

Or rather, your DSL modem is being silly.

From IRC:
justdave: gateway.2wire.net is the default internal hostname on 2Wire DSL modems.

So the original mozillamessaging.com page was probably cached, but the download links were not, and the DSL modem tried to redirect you somewhere.
The DSL modem (at&t uverse) had decided to disable all http outgoing requests because of a bug in the firmware of the modem, which is made by 2wire.com.
So, let's forget about this glitch. Neither the URL nor the cached page had
anything to do with it. The damned thing (modem) gave me the same error page
after I went back and tried other URL's (which I should have done the first
time, and not had filed the error about the URL. Sorry, all!
Status: UNCONFIRMED → RESOLVED
Closed: 16 years ago15 years ago
Resolution: --- → INVALID
Whiteboard: closeme 2009-06-01
I insist that bug is valid as far as inbox file size is concerned.
This bug has to be addressed. It is a lame thing that TB cannot handle
inbox folder, or any folder, larger than 2GB. TB must be rebuilt so that
any variables dealing with message index numbers or file offsets or sizes
use 64 bit data types FOR 64 bit platforms.
The 64bit Linux distros which create the TB rpm for x86_64 platforms,
do not bother to fix the internal design of TB and make it a truly 64 bit
app (again, as far as folder file size, message index numbers...etc are
concerned).
This has to be fixed by the TB internals team.
32 bit limitations on any app is not only lame, it is also inexcusable with
todays cpu's and compilers.
Status: RESOLVED → VERIFIED
Resolution: INVALID → WONTFIX
my bad. I closed this based on the most recent comments, not the original issue. Which, I will assert, if you want to see addressed you will want show some respect to m-wada.

m-wada, there is a reasonable dupe here, no?
Status: VERIFIED → UNCONFIRMED
Resolution: WONTFIX → ---
Whiteboard: dupme
still needs to be duped iirc
If I understand this issue correctly, the root problem is that the file limit for Thunderbird on Linux is 2GiB. This is what bug 462665 will fix.
Status: UNCONFIRMED → RESOLVED
Closed: 15 years ago14 years ago
Resolution: --- → DUPLICATE
the 2gb limit on liux/mac has been fixed, so that it's on par with the 4gb limit on windows. See  Bug 538512
Folks, even a 4GB limit is lame! It means that TB is using 32 bit unsiged data type for message offsets into the inbox.
I am currently using FC12 32 bit (i686), and firefox-3.5.8-1.fc12.i686
I recombined my Inbox's 3 fragments into a single Inbox
of 3138023630 bytes, and started firefox.
No problems. All messages seem to be there and are perfectly readable.
However, I do not know what will happen when I exceed 4GB size.
Personally, I do not want go through another fiasco as I did when
I first submitted this bug.
I will shutdown firefox, delete my big Inbox, and restore my last small Inbox of 1.5GB.
(In reply to comment #28)
> Folks, even a 4GB limit is lame! 

Hence bug 462665
Cleanup *dupeme* whiteboard flag from bugs that are marked as Resolved
Duplicate!
Whiteboard: dupme
You need to log in before you can comment on or make changes to this bug.