Closed Bug 1911076 Opened 2 months ago Closed 1 month ago

[32bit only] Lost thousands of e-mails after Nebula upgrade, large pop mboxes truncated to ~4 GB - tb128/32-bit can't handle compacting of mbox files of more than 4GB

Categories

(Thunderbird :: Folder and Message Lists, defect, P1)

Thunderbird 128

Tracking

(thunderbird_esr115 unaffected, thunderbird_esr128+ fixed, thunderbird129 wontfix, thunderbird130+ fixed)

RESOLVED FIXED
131 Branch
Tracking Status
thunderbird_esr115 --- unaffected
thunderbird_esr128 + fixed
thunderbird129 --- wontfix
thunderbird130 + fixed

People

(Reporter: it, Assigned: mkmelin)

References

(Blocks 1 open bug, Regression)

Details

(Keywords: dataloss, regression)

Attachments

(1 file, 1 obsolete file)

Steps to reproduce:

2 days ago our over 20 PC's with Thunderbirds upgrades from 115.13 to 128 (Help - > About Menu). Windows 10 Pro.

Actual results:

On few PC's we have lost thousands of e-mails. Example (from 06.2021 to 07.2024).
Repair / deleting .msf file doesn't work.
In many cases "Inbox" files files have been reduced from 11-12 GB to 4-5 GB.
We tried opening damaged profiles with older version (115.13), but e-mails are disappeared.

Missing from Inbox?
imap accounts or pop?
On what type of disk storage?
What other folders did you check for the missing messages?

Flags: needinfo?(it)
Keywords: dataloss

E-mails missing from Inbox.
Pop3, disk storage SSD. In all cases big Thunderbird profiles - over 50 GB.
We checked all folders (Archives and other folders made by users) - for now e-mails are gone only from "Inbox".
We even opened "Inbox" file via Notepad++ and searched content - mails are gone.

We have backup Thunderbird profiles from 19.07.2024, and we made a comparison.

On first PC we lost e-mails from 06.2021 to 31.07.2024. Original Inbox file was bigger than 11 GB and after upgrade was reduced to ~ 4GB.
On second PC we lost e-mails from 02.2022 to 31.07.2024. Original Inbox file was bigger than 9,2 GB and after upgrade was reduced to ~ 4GB.
On third PC we lost e-mails from 03.07.2024 to 31.07.2024. Inbox was ~3,5 GB, we lost ~200 MB.

Additional, yesterday we deleted "popstate.dat" file to download missing e-mails from last two weeks.
Yesterday, these e-mails were fine, we could read it, but today these e-mails are only visible on list, but when we click on email subjects - no content and no attachments.

Flags: needinfo?(it)
Summary: Lost thousands of e-mails after Nebula upgrade → Lost thousands of e-mails after Nebula upgrade, large mboxes truncated to ~4 GB

Thanks for these great details.

Do you have automatic compact enabled in settings? If you do, please disable for now, just in case.

Severity: -- → S1
Flags: needinfo?(gds)
Flags: needinfo?(benc)
Flags: needinfo?(alessandro)
Priority: -- → P1
Summary: Lost thousands of e-mails after Nebula upgrade, large mboxes truncated to ~4 GB → Lost thousands of e-mails after Nebula upgrade, large pop mboxes truncated to ~4 GB

Reporter "it", Can you provide a few more config details?
For you typical POP3 server settings in TB how are these set, i.e., checked/unchecked and values for "X":

  • Check for new messages at startup
  • Check for new messages every X minutes
  • Automatically download new messages
  • Fetch headers only
  • Leave messages on server
    • For at most X days
    • Until I delete them
  • Message storage type (mbox or maildir?)

If leaving messages on server, about how many messages are on the server?

Wayne, is the fix for this bug in 128: bug 1875633

Flags: needinfo?(gds)

Wayne, is the fix for this bug in 128: bug 1875633

According to push log it isn't: https://hg.mozilla.org/releases/comm-esr128/pushloghtml?startdate=6+months+ago&enddate=now
Anyhow, I'm not sure that having this in 128 would fix what is reported here.

This landed when TB was at version 125: https://hg.mozilla.org/comm-central/rev/2e171d15d953. Of course it's in 128, see here:
https://hg.mozilla.org/releases/comm-esr128/rev/2e171d15d953

Blocks: tb128found

(In reply to it from comment #2)

Additional, yesterday we deleted "popstate.dat" file to download missing e-mails from last two weeks.
Yesterday, these e-mails were fine, we could read it, but today these e-mails are only visible on list, but when we click on email subjects - no content and no attachments.

DO you have streaming backups overnight, or a scheduled antivirus scan, perhaps. While the initial report is about the update, you are saying here the data was restored and again went missing "overnight".

Was Thunderbird running overnight?

See Also: → 1890230

It'd also be nice to know if filter moves were involved. POP3 filtering for new messages occurs during the message parsing (in nsParseNewMailState).
(I'll be mostly traveling for the next week, but I'll keep peeking in here).

Flags: needinfo?(benc)

I won't be able to get to it for a few days, but the first test I'd try is:

  1. generate a >4GB mbox file and copy it into a folder (https://github.com/bcampbell/fakemail is what I use for this - you can have a folder of big files which it can random attach to messages, so you could get to 4BG quickly).
  2. run TB and wait for it to parse the newly-added local subfolder
  3. delete a message or two (but make sure you don't go below the 4GB mark)
  4. compact the folder
  5. see if it got wrongly truncated to 4GB

This'll determine if it's a pop3 problem or just a compaction issue.

Ben, I used fakemail to create a 9G mailbox (mbox) in a pop3 account. My Local Folders are Maildir so wouldn't be a good test. Anyhow, I don't see any problems doing a manual compact after deleting a few messages. The file sizes just goes down a little bit and doesn't drop to 4G or less and all message seem accessible. I tested with latest 128.0.1esr on linux from archive site.

Not sure if the reporter had any non-Inbox folders that were greater than 4G.

I was thinking maybe you have a "fake" imap or pop3 server that could be used for testing with LOTS of large emails like are created by fakemail? I have some on dovecot and cyrus locally but total size is only around 1G on each POP3 account. Not seeming any issues with these accounts.

Another test worth doing would be to use fakemail to generate a large mailbox in 115, and then force the 128 upgrade.
I have a suspicion that the migration code to clean up overblown nstmp files caused by failed compaction might be the culprit (bug 1878541).
I can try this on Monday if no one gets to it before me.

Flags: needinfo?(alessandro)

Ben,

I was thinking maybe you have a "fake" imap or pop3 server that could be used for testing with LOTS of large emails like are created by fakemail? I have some on dovecot and cyrus locally but total size is only around 1G on each POP3 account. Not seeing any issues with these accounts.

I just copied the "fakemail" messages to my cyrus imap inbox which causes them to look like "new" message in the matching cyrus POP3 inbox. So they are downloading into cyrus POP3 now, but still takes a while to download 9G even locally. So will look at results tomorrow.

(In reply to Alessandro Castellani [:aleca] from comment #11)

Another test worth doing would be to use fakemail to generate a large mailbox in 115, and then force the 128 upgrade.
I have a suspicion that the migration code to clean up overblown nstmp files caused by failed compaction might be the culprit (bug 1878541).
I can try this on Monday if no one gets to it before me.

I don't think the nstmp-killer will be the culprit. It's very specific about only targeting files named nstmp (and nstmp-1 etc).

(In reply to gene smith from comment #10)

Ben, I used fakemail to create a 9G mailbox (mbox) in a pop3 account. My Local Folders are Maildir so wouldn't be a good test. Anyhow, I don't see any problems doing a manual compact after deleting a few messages. The file sizes just goes down a little bit and doesn't drop to 4G or less and all message seem accessible. I tested with latest 128.0.1esr on linux from archive site.

That lines up with my findings so far.
I just set up a new profile and copied a 8.5GB mbox into the "Local Folders/" before starting TB. I then deleted a few messages, compacted, and ran down the whole message list, randomly picking messages to view, and it all worked fine. And the file size on disk after the compact was as I'd expect: just under 8.5GB.
Also on Linux.

Not sure if the reporter had any non-Inbox folders that were greater than 4G.

I was thinking maybe you have a "fake" imap or pop3 server that could be used for testing with LOTS of large emails like are created by fakemail? I have some on dovecot and cyrus locally but total size is only around 1G on each POP3 account. Not seeming any issues with these accounts.

I just settled on running a local dovecot, and using dovecot-lda to load fakemail messages into it from the commandline -
fakemail -a $ATTACHMENTS -n $COUNT -f eml -o $TMPDIR, then loop over the resulting files feeding them into dovecot-lda.

Works for both IMAP and POP3 testing, but I haven't had time to do that here (I'm on a debug build, which is sloooow).

My bet is currently on pop3 folder delivery screwing things up (on the folder side of things, rather than the protocol side). Maybe just any pop3 delivery, or quite likely the filter code (which does some very screwy stuff in nsParseNewMailState to move matching messages).
Possibly just on windows? But at a guess I'd say this is cross-platform borkage.

My next steps would be:

  1. set up a >4GB mbox for a pop3 folder (could probably just install a fake mbox file directly there, bypassing server delivery? not sure)
  2. have a filter set up to move incoming mail to another folder
  3. cause a matching email to be delivered
  4. see if the resulting filter hit causes the mbox to be drastically truncated.
    Try again where the filter rule destination folder is >4GB.
    Try again with both folders >4GB.

I'll be on planes for the next couple of days, but will try and check in here again when I can!

(In reply to Ben Campbell from comment #13)

I don't think the nstmp-killer will be the culprit. It's very specific about only targeting files named nstmp (and nstmp-1 etc).

Bug 1878541 hasn't been backported to 128 yet.

(In reply to it from comment #0)

2 days ago our over 20 PC's with Thunderbirds upgrades from 115.13 to 128 (Help - > About Menu). Windows 10 Pro.

32-bit or 64-bit Thunderbird installations?

32-bit or 64-bit Thunderbird installations?

I was thinking from the beginning in terms of 32 vs 64 bit windows but since they say win10 pro and it worked before with more than 4 GB files I discounted it. But maybe they did install 32 bit TB? I installed 32 bit 128esr and my 9 GB Inbox is now truncated to 4 GB:

-rw-------  1 gene gene 4309043608 Aug  4 14:07 Inbox

Also, my bottom half of messages come up blank and only see the headers.
Edit: However, they say there are some files truncated to 5G so not sure this is why.

Did the steps suggested by Ben in comment 14. Didn't see any problems or file truncation with huge mbox files (>= 8 GiB) with 64bit 128.0.1 on linux.

More users report problems.

The problem occurs on both Windows 10 and 11 Pro. I don't know if the problem occurs on Linux, because my Thunderbird (Fedora 40) is still on version 115.13.

Our TB profiles are large. On most computers 30 -70 GB (one PC ~ 160 GB, all mails from 2008 to 2024).
Some users create folders to organize their messages. On most PC, we use message filtering for new messages, but on first PC where 3 years of emails disappeared there was no filter.
Archive files were not reduced in size, some of them were larger than 4 GB.

The problem does not occur immediately after upgrade, but after few next launches of Thunderbird (2-3 times).
Maybe 128-version compaction function can't handle "Inbox" files larger than a few GB and shrinks them?

We use the 32-bit version of TB because the 64-bit version does not work with our ERP program (32bit).

We don't have streaming backups overnight, or a scheduled antivirus scan. We use ESET antivirus, but it has never deleted more than 1 email. Thunderbird did not work at night. We use PC's only during office hours (standard 8 hours work-day).
Emails downloaded from pop3 server, after deleting the "popstate" disappeared after the next launch of Thunderbird.

On some computers the content of emails cannot be seen. The messages are in the list, but when we click on them, the content is not visible.

Where it is possible, I am restoring the July 19th backup and going back to version 115.13, and disabling auto-upgrade in settings.
In TB version 115 the problems does not occur.

It appears that 128/32-bit can't handle compacting of mbox files of more than 4GB. I tried running 115.13.0/32-bit and deleted a few messages in 8GB mbox folder and then compacted them and the size just went down a small amount. Tried this on several large mbox folders with no problem. Then installed 128.0.1/32-bit again and repeated the experiment and the foldes are truncated to 4GB with half the messages inaccessible.

I wasn't sure if 32-bit TB is supposed to be able to handle mbox files sizes greater than 4GB. But this says it should be able to: https://kb.mozillazine.org/Limits_-_Thunderbird

From comment 19:

Archive files were not reduced in size, some of them were larger than 4 GB.

I suspect these are OK since probably nothing has been added or deleted from these non-Inbox "archive" folders.

Emails downloaded from pop3 server, after deleting the "popstate" disappeared after the next launch of Thunderbird.

Is it correct that you have TB set so that only the messages for the last 14 days remain on server? I have all message left on server (8-9MB)(8-9GB) now and when I delete popstate, Inbox and Inbox.msf and do "Get messages" with 32 bit 115 or 128, I see a lockup or crash. Seems to be working OK re-downloading with 128/64-bit.

We use the 32-bit version of TB because the 64-bit version does not work with our ERP program (32bit).

I don't understand why the ERP program cares about TB. I would think they are independent of each other? Or maybe you are also running a 32-bit Windows?

(In reply to gene smith from comment #21)

I don't understand why the ERP program cares about TB. I would think they are independent of each other?

If the the 32bit ERP program sends messages via MAPI, TB also needs to be 32bit currently (see bug 1879717).

@ comment 4:
Check for new messages at startup X
Check for new messages every X minutes 10
Automatically download new messages 10
Fetch headers only
Leave messages on server X
For at most X days 14
Until I delete them X
Message storage type (mbox or maildir?) MBOX

@ comment 21:
We have 64 bit Win 10/11 Pro on all PC's. Our ERP system is 32-bit.
We use "Comarch ERP XL". This is a Polish ERP program, 32 bit, originally written in Clarion programming language.

The program has a function that opens the email program with a new, empty email with an attachment generated from the program (tables, lists, invoices, etc.).
For this feature to work, PDF Creator (version 1.7.3) and email client (Outlook or TB 32-bit) are required.
It has been working for many years. This functionality does not work with the 64-bit version of TB.

@ comment 22:
Yes it's true. For this functionality to work, Thunderbird must be linked to the "mapimail" feature in Windows.

Confirming based on Gene's comment 20

Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary: Lost thousands of e-mails after Nebula upgrade, large pop mboxes truncated to ~4 GB → [32bit only] Lost thousands of e-mails after Nebula upgrade, large pop mboxes truncated to ~4 GB - tb128/32-bit can't handle compacting of mbox files of more than 4GB

Presumably fall-out from the compact rewrite (bug 1890448)

Keywords: regression
Regressed by: 1890448

This patch simulates what 32bit would see. (Just testing it on linux 64 bit now.)
On win32, size_t will be up to 4GB, while on win64 it would be up to 16TB, IIUC.

With the patch applied, I copy a 4.9G folder into a local test account.
Then I delete some message, and compact. All good, on the surface.
Then I again delete a message, and compact. The resulting folder is capped at ~ 4.1G.
I don't quite understand why it doesn't happen the first time, but likely it would be due to a wrong offset getting stored in the db.

In the process of reproducing this, at least with the patch applied I got a (so far) reproducible case showing bug 1890230 on a local folder.
I just copy the 4.9G file to a local folder, then let it rebuild then .msf and always the same message gets corrupted.

So at the moment it would appear to more be regression mainly from bug 1719121, possibly in combination bug 1890448.

Regressed by: 1719121
Assignee: nobody → mkmelin+mozilla
Status: NEW → ASSIGNED

Gene, would you be able to use a build from the try run and confirm that the problem is fixed with the STR?

Flags: needinfo?(gds)

(In reply to Alessandro Castellani [:aleca] from comment #33)

Gene, would you be able to use a build from the try run and confirm that the problem is fixed with the STR?

I think Magnus didn't include linux-32bit in the try run. Been trying to get a 32-bit linux build using c-c try server but had some problems, not sure why. Anyhow, finally got a successful build here:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=7b3e79b8e05fc88837349038488273c0ec227b49&selectedTaskRun=LCRAGuPrS0OgQQ0RJpBpgg.0

Testing it right now using local pop3 server with about 17 GB in Inbox. Right now it's writing to Inbox and have stored about 15 GB at this time (which has always worked fine). The problem occurs only after a message is deleted from Inbox and then when it is compacted, while writing to "temp" file Inbox-1, it reaches a max size of 4GB and then gets renamed to Inbox and remains at 4 GB causing missing messages.

I'll report back on how it behaves doing the compact ASAP.

Flags: needinfo?(gds)

It's working great doing the compact with Magnus' last patch on linux with a 32-bit try build based on c-c tip. The Inbox-1 "temp" file grows to the full 17 GB and then gets renamed to Inbox like it should.
I should probably boot up win11 and test it there too this evening (unless you think testing on linux is enough).

Testing on Windows would be great since that platform is usually the most brittle when it comes to moving and generating large files.
I'll test on Win11 Pro with TB 32 bit as well in a few hours.
Thank you so much for your speedy help!

I would like to help.
But, does anyone know how to build 32-bit version of C-C TB under 64-bit linux?
Being able to build a 32-bit version under 64-bit linux and run it through mochitest and xpcshelltest without an issue related to 32 vs 64 bit issues would be a plus.

( I am tempted to build SANITIZED version of C-C TB even this is because UBSAN check includes many cast that lose accuracy, overflow, etc. Looking at comment 31 and comment 32 makes me feel that the compiler may not be able to catch such issues with only static analysis only. Maybe clever use of very strict warning might help, but I doubt if we can catch them all at compile time.
32-bit version sanitized version may not be possible though because C-C TB is already big enough and added code size may not fit in the 32bit address space with all the runtime libraries.

(In reply to Alessandro Castellani [:aleca] from comment #38)

This should be an installable 32bit Windows build: https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=f159d889f9d0c2d382f5eae236998fb8483043b4&selectedTaskRun=Gfv-5mceS3qYhBhouhXCGA.0

Thank you. Unfortunately, I do not run xpcshell test, mochitest, etc. under windows now.
(I develop patches under linux 64 bit)
I see there is no linux 32 bits version created and tested on tryserver.
Wait! |linux opt| seems to be the linux 32-bit version builld (!) It is NOT TESTED, but binary is created, I think.
Now I have verified that the compilation uses -m32 flag to compile 32-bit version.
(Not sure if the host linux is 32 bit or 64 bit. It seems that the build uses 32-bit OS for 32-bit compilation because "live_backing.log" contains |checking for 64-bit OS... no|).
So I can create a patch and submit to tryrserver and fetch this linux opt 32-bit version and try to test locally, I think.
(Maybe I need a host of 32-bit compatibility libraries...)

Thank you for the pointer.
I will try to run 32-bit linux opt version under my linux 64-bit hopefully later today.

Tested the 32 Windows build on Windows 11 Pro.
Here's the result:

  • Created a local folder named foo with 20k messages and various size attachments. Total folder size 24.2GB
  • Deleted 65 messages containing various attachments.
  • Triggered folder compaction

Result:

  • Compaction took ~20 seconds
  • The file foo-1 was temporarily created
  • Compaction finished, the foo-1 file was removed
  • The local folder foo still has all the messages.
  • Total folder size 24.1GB

This works as expected for me!

Wait! |linux opt| seems to be the linux 32-bit version builld (!)

Yes, somehow I didn't see that. I must have been looking at Magnus' first try in comment 30. For some reason it doesn't list the 32 bit linux opt but the "try" in comment 32 does.

Anyhow, went ahead and tested on win11 pro using a real POP3 account. First verified that 32-bit 128.1.0esr has the bug. It does, same problem: on compact even after just deleting one message, Inbox-1 goes to a max size of 4GB and gets no bigger and becomes Inbox. Upper message UIDs are not accessible. Then downloaded Magnus' try build for Windows 2012, deleted Inbox, Inbox.msf and popstate and re-downloaded the 9 GB Inbox and delete a couple small messages and then compacted. Inbox-1 (the temp file) goes to the full 9 GB size and then becomes Inbox. Inbox.msf and popstate look good and all messages are accessible.

Note: I tested this on linux and windows with POP3 accounts configured to always leave messages on the server even when they are deleted locally. The reporter kept only the latest 14 days on the server so they will see data loss.

Attachment #9418097 - Attachment is obsolete: true

I was able to test on win32 now, and it seems to work like it should.

Target Milestone: --- → 131 Branch

Hello!

Managed to reproduce the affected behaviour on 128.1.0esr(20240730200333) using Windows 11.

Confirming this issue is fixed on the try build from comment 32 with Windows 11, Windows 10, Ubuntu 22 and Ubuntu 24.

Using the STR from comment 40, here's what I did in order to verify:

  • Create a local folder with 16k messages (different attachment types and sizes) . Total folder size 14.6GB

  • Deleted 40 emails (with and without attachments)

  • Proceeded to folder compaction

  • Compaction process took around ~15 seconds

  • The local folder has no data loss after compaction

  • Local folder compaction successfully reduced the size to 14.4GB

It is also worth mentioning that under Magnus guidance , the delete+compact part was done twice in order to make sure that the issue is no longer present.

Pushed by brendan@thunderbird.net:
https://hg.mozilla.org/comm-central/rev/ff8aac8ed7e2
ensure MboxMsgInputStream msgOffset is 64bit, so 32bit installations can handle files > 4GB. r=aleca,babolivier

Status: ASSIGNED → RESOLVED
Closed: 1 month ago
Resolution: --- → FIXED

Comment on attachment 9418102 [details]
Bug 1911076 - ensure MboxMsgInputStream msgOffset is 64bit, so 32bit installations can handle files > 4GB. r=benc!,#thunderbird-reviewers

[Approval Request Comment]
Regression caused by (bug #): primarily bug 1719121
User impact if declined: dataloss
Testing completed (on c-c, etc.): c-c
Risk to taking this patch (and alternatives if risky): safe

Attachment #9418102 - Flags: approval-comm-esr128?
Attachment #9418102 - Flags: approval-comm-beta?

Comment on attachment 9418102 [details]
Bug 1911076 - ensure MboxMsgInputStream msgOffset is 64bit, so 32bit installations can handle files > 4GB. r=benc!,#thunderbird-reviewers

[Triage Comment]
Approved for beta
Approved for esr128

Attachment #9418102 - Flags: approval-comm-esr128?
Attachment #9418102 - Flags: approval-comm-esr128+
Attachment #9418102 - Flags: approval-comm-beta?
Attachment #9418102 - Flags: approval-comm-beta+
Duplicate of this bug: 1912408
Duplicate of this bug: 1912314

Yesterday I tested TB 130 b2 (both 32 & 64 version). Profile ~ 36GB. The inbox file was not reduced, but there is still a bug with "blank emails". bug 1912314

In version 128 (32 and 64) some emails were showing up in the list, but when we opening they were "empty".
After updating TB 128.1 to 130 b2, these "blank, empty" emails are not showing up at all (they are not on the email list, but they are still exist in Inbox file).

After uninstalling TB 130 b2 and installing TB 129 emails, they are back in the list, but they are "blank, empty".

Duplicate of this bug: 1911827
Duplicate of this bug: 1911393
Duplicate of this bug: 1912232
Duplicate of this bug: 1914420

I am just curious. Around the time the 32bit issue was reported, 32bit version test under windows did not seem run on try-comm-cebtral, but it definitely runs now.
Was it just an oversight or some temporal glitches that ketp 32-bit test to run on try-comm-central?
Running 32-bit test is a good thing to be sure.

In comment 39, I offered to check on 32-bit linux version test, but I gave up.
I have used Debian GNU/Linux for a long time, and when I tried to run 32-bit version, I realized it is not easy to do so under 64-bit linux under which I develop patches.
It requires me either to install so called mutli-architecture version or use chroot and run the program under the chroot-ed 32-bit userland environment. Both were difficult to do since my linux parttitions were now close to overflowing.
The explanation behind 32-bit userland support under 64-bit Debian kernel is explained in the following article.
https://www.theregister.com/2023/12/19/debian_to_drop_x86_32/

Other popular distro such as Ubuntu does not offer 32-bit version any more.
So I suspect it would be difficult to test 32-bit linux version on try-comm-central, too.
Hopefully testing the 32-bit windows version would catch the 32-bit vs 64-bit issue in C/C++ code like this bugzilla.

Is there a particular test for this program? I think there is a tes to see if compact works for a folder over 4GB.
I think that will be good enough, but if there is a special test added for this bugzilla, that will give us a good night sleep.

I am sure those who got bitten by this bug and lost a large chunk of e-mails, and do not have any programming skill to understand the issue are likely to leave TB forever. Tough.
About a dozen years ago, I lost half my Inbox when there was a bug in compact code that does not check for the return code of file write (ENOSPC, there was no space in filesystem anymore), but luckily I had a backup, and only lost a few day's worth of e-mails which I remembered did not contain any important messages, AND I could see there were bunch of coding errors (in the era when 32-bit vs 64-bit issues pop up) and I could fix them and have stuck to TB until now.

See Also: → 1915166
No longer duplicate of this bug: 1912408
See Also: → 1912408
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: