Closed Bug 487375 Opened 15 years ago Closed 2 months ago

Severe performance issues on USB stick while deleting or receiving mail

Categories

(Thunderbird :: Mail Window Front End, defect)

Product:
Thunderbird
Component:
Mail Window Front End
Platform:
x86
Windows XP
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: kristo, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: perf, Whiteboard: [needs performance profile]) 

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8
Build Identifier: All current builds for 4 weeks

When Thunderbird is installed USB sticks, it is currently almost not bearable to receive larger amounts of mail or deleting several mails at once.

Even creating a new mail results in several seconds waiting time if you haven't got the fastest USB stick.

I cannot imagine that the database access can be so painfully slow. Sqlite itself is smooth and fast even on USB which I 've tested myself.

Reproducible: Always



Expected Results:  
I really think that using thunderbird on a slower media than HDDs shouldn't result in such slow reactions.
Severity: normal → major
Version: unspecified → Trunk
are you talking about pop3 mail? Are you receiving lots of small messages or several large messages? Is it the quantity or the count of messages that seems to cause the slowdown?

I suspect creating a new message is more tied to the time it takes to create the compose window. Is opening it the second time faster?
Hi David,

yes, currently I'm only using pop3 accounts. All my mail is constantly sorted
out and deleted, so only a handful of mails remain in my inbox.

Mails are rather small and I'm receiving only few mails because of our 
rather restricted server spam filters. 
(4 pop3 accounts. about 5 to 10 per account/day - only 1 or 2 per day
is stored permanently)

Creating the compose window when not using USB is blazing fast on my PCs.
The same goes for deleting mails.

As soon as I use my stick (Kingston HyperX - very fast) deleting becomes
almost unbearable. TB needs several seconds to delete on small mail.
This has also been tested with 3 other different USB sticks.
you haven't turned on global search, have you? Options | Advanced | general, enable global search and indexer?
Hey David!

Of course I had. Just turned it off!

As a result deleting feels a lot better now.
Not really blazing fast, but a lot better.

Displaying mails still feels like it takes an eternity, 
but maybe someone will begin to tune those routines
sometime in the future.

Thanks
Status: UNCONFIRMED → RESOLVED
Closed: 15 years ago
Resolution: --- → WORKSFORME
the file i/o aspects of displaying messages is pretty straightforward - we seek to the start of the message in the file and read msg size bytes. Marking the message read involves seeking to near the start of the message and writing 4 bytes, and doing a sync. And of course, updating the .msf file to note that the message is read, which involves seeking to the end and writing a few dozen bytes or so.

I doubt there's a lot of tuning to be done there - I wonder if it's the actual display that's slower. We do a bit of disk i/o the first message you click on to load your address book, I think, so we can do clever display of the from and to fields.
Gloda deletion speed issues in general (not specifically related to flash memory) are already on the radar as bug 4798008.  As you say, the addressbook issue should really only effect the first message displayed, since after that, the entire addressbook will be in-memory (via Mork).  Adding hyc to the CC of this bug, as he's done some thinking about flash-related performance tradeoffs, and may have more thoughts here.
I actually meant bug 479008.
Not sure what else I can add. As I've noted in bug 484119, the type of operations David lists in comment #5 are precisely the type that flash memory is worst at. I.e., sectors cannot be overwritten; "in-place changes" are performed by copying the data to new/empty blocks. If there are no blank blocks available for immediate use, some old sectors need to be coalesced, some block erased, and then all of the data written back out. It can be extremely time-consuming depending on the age and free capacity of the flash device. The only way to minimize this is to remove the syncs, and allow the filesystem to buffer all of these changes. This way if multiple rewrites are done in the same block, the remapping/coalescing only has to occur once for that block, instead of once for each individual write to that block.

Syncing after each write will kill most flash devices in far shorter time than otherwise; you need to let the filesystem cache the writes so they can be aggregated into larger I/Os.
Kristo, is your USB formatted as FAT or NTFS?   bug 303268 comment 4 observes that formatting the USB as ntfs will help.

http://tinyurl.com/d4o9lz lists other USB performance bugs.  Some of these bugs observe that non-mozilla products perform better with USB.
Kristo. I am reopening. This should be further scrutinized because a) turning gloda off is a workaround, not a long term solution for all users, b) it's not yet clear bug 479008 will be a magic bullet.

Let's at least wait on bug 479008.

As for USB in theory ...

unscientific study - I tested a small old USB FAT32 vs NTFS. NTFS was ~30% faster in copying files to it.  (10 sec vs 14 sec).

References:
excellent + links to HDTUNE speed testing tool - http://www.uwe-sieber.de/usbstick_e.html
good basics - http://www.irongeek.com/i.php?page=security/usb-flash-thumb-drive-ntfs-filesystem
historical info - http://bink.nu/news/slow-file-transfer-to-usb-stick-issue-acknowlegded.aspx

from http://www.testfreaks.com/blog/information/usb-flash-drive-comparison-part-2-fat32-vs-ntfs-vs-exfat/
"exFAT is essentially developed for flash-based removable media, so its file allocation scheme considers the wear-leveling issues found on flash drives.

NTFS clearly supports files > 4GB as well, but exFAT is the better choice for ReadyBoost because:
(1) It has lower overhead
(2) It’s meant for removable drives whereas NTFS is not
(3) It has better semantics for disk verification across power transitions such that you are more likely to maintain the ReadyBoost cache contents across standby/hibernate."
Status: RESOLVED → UNCONFIRMED
Depends on: 484119, 479008
Keywords: perf
Resolution: WORKSFORME → ---
Whiteboard: [investigate USB perf][recheck after dep bugs]
Hi Wayne,

Usually I format drives with NTFS as I did with this one.
I did test FAT32 but peformance was much slower with Kingston HyperX
Stick than with NTFS. Strange, because Kingston told me differently on
their website. I guess this as a mistake. NFTS is defintely faster now.

My other "MailClient" is a portable version of LotusNotes8. That's why I 
know the stick is fast enough for almost anything. 

I'll keep an eye on any changes you make with an impact on USB peformance
and let you know about the effect.

Cheers
Kristo
see bug 303268 comment 13 (which has been reopened) which implies USB performs unacceptably on mozilla+windows.  So it would be useful to know if Mac and/or Linux show the same poor performance.
Wayne, currently I'm using WindowsXP only. I'm afraid I cannot help with MAC or Linux, but I will forward this request to my collegue who is using Linux.
(In reply to comment #0)
> User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.8)
> Gecko/2009032609 Firefox/3.0.8
> Build Identifier: All current builds for 4 weeks

I use Thunderbird 1.5.0.4 on my Sandisk Cruzer USB U3 8Gb, FAT32 format.  It is not clear to me if we are talking about the same products.
1- Clearing 12 e-mails from the Inbox (all highlighted) takes less than half a second, but you can feel the delay.
2- Clicking on "Write" button opens the composition window instantly.

So I take it that you are not using the U3 version of the Thunderbird, or are you?
Parkhideh, did you have "indexing" turned on?
Nope. Dont't like U3.
Currently I'm using a Kingston HyperX, which loads apps like GIMP in no time, but still TB seems to be very slow compared to using it from a local drive.
(In reply to comment #15)
> Parkhideh, did you have "indexing" turned on?
Wayne, What is indexing in Thuderbird?
       If you mean OS indexing or quick find type of cataloging; I have all turned off including virus checkers.  Otherwise a good part of the hour is gone in their scanning my USB flash.


> Comment #17 from Kristo:
> Currently I'm using a Kingston HyperX, which loads apps like GIMP in no time, but still TB seems to be very slow compared to using it from a local drive.

Some numbers for you Kristo:
First time load from USB 2.0 flash;
- GMIP takes 18 seconds to load.
- Thunderbird 2.0.0.21 takes 15 seconds to load (Inbox appearing); however, the USB is still loading data in the background for another 12 seconds.  Total data size of my e-mail is 880 Mbytes, around 28000 mail messages.
- Thunderbird 1.5.0.4 from Sandisk U3 Cruzer takes half the time.

Second load, closing applications (GMIP and Thunderbird) but not PortableApps:
- GMIP takes 3 seconds to load.
- Thunderbird 2.0.0.21 takes 3 seconds to load (Inbox appearing); however, the USB is still loading data in the background for another 4 seconds.

- Deleting 200 e-mails from Trash takes 5 seconds.
- Deleting 200 e-mails from Inbox takes 10 seconds for the screen to repaint but the actual work on USB flash takes another 25 seconds, this to complete the transfer from Inbox to Trash.

I hope this helps; it is clear that the bottleneck is the USB flash device.  Same tests when PortableApps when loaded on the hard disk takes less than 5 seconds.
(In reply to comment #18)
> (In reply to comment #15)
> > Parkhideh, did you have "indexing" turned on?
> Wayne, What is indexing in Thuderbird?
>        If you mean OS indexing or quick find type of cataloging; I have all
> turned off including virus checkers.  Otherwise a good part of the hour is gone
> in their scanning my USB flash.

either:
* indexing for windows search - in nightlies and betas it's in options>advanced>general>Allow Windows Search to search messages
* gloda indexing - options>advanced>general> Enable GLobal Search
(In reply to comment #19)
> (In reply to comment #18)
> * indexing for windows search - in nightlies and betas it's in
> options>advanced>general>Allow Windows Search to search messages
note this is in version 2 and 3

> * gloda indexing - options>advanced>general> Enable GLobal Search
this is only version 3

I wonder how bad portable is with version 3 and Bug 539389.
(Kristo might be gone.  I haven't tested)
Not sure if this belongs here; but it might be related or helpful.  While running T-bird mobile (latest update as of two days ago; not sure of version but I think it was 14) on a USB stick I began having problems with deleting files.  I was cleaning out old mail and selecting large groups of mails that were sorted by 'sender' and then hitting delete key.  It was taking long time to delete so I would open a new mail and read it while letting delete 'run'.  I noticed that mail that I thought was deleted kept showing up in inbox.  I also was offered 'compact' choice so clicked yes on that.   Eventually shut down computer; seemed like a normal shutdown; but windows7 Home will not reboot or restore.  Laptop is Toshiba Satellite C655D, USB stick is 16G Staples (very small physical size).
(In reply to John from comment #22)
Without implying that you should crash your OS again :) , it is useful to know how the delay in USB write and delete is affecting the work.  Next time you use the USB in the manner reported; could you wait until the blinking USB light is OFF or fully ON as the idle case maybe.  If you do not have an LED on the USB, try its icon and see if you can Eject it via OS. 
What I like to know is did you shut down the system in the middle of USB activity?
Parkhideh,    Too long ago to remember for sure.  There was no activity led on the flash drive; so I guess I am not sure about shutting down during USB activity.  I would have assumed that the OS (WIN 7) would not let that happen; but maybe it would? I was not in the habit of ejecting the USB; it had a small sillouette and I just left it plugged in permanently unless I needed the slot for something else on a temporary basis. So I shutdown via the OS and left the USB stick plugged in. I am using the USB stick infrequently now (but it does appear to be functioning) because until a few days ago that laptop was dead. I was very reluctant to reformat the drive as I was trying to salvage the data.  I bought a new h/d and reinstalled to out of box condition.
John, does this mean that you will repeat the test in comment 22?  :)
Parkhideh,  
Not sure it would do any good.  Apparently the problem was caused by a corrupted OS due to a faulty h/d. I have installed new h/d and new OS.
 
See bug Bug 787197  for additional details.
John,
I will make my comments here in the hope of closing this bug.

1) You have already indicated that the crash had nothing to do with the portable application.

2) In general using a flash and demanding various flash memory activity is not good.  So compacting is the worse thing you can do on a flash memory.  Leave it un-compacted.  (Actually defragmentation is the worst)

3) If for any reason you want to compact or cleanup any file from Flash; the best approach is to transfer it to hard disk, do the work, then erase the USB files and copy back.  The trick for portable applications, in case you do not know, is to copy the whole structure of your USB (in this case folder ThunderbirdPortable which has ThunderbirdPortable.exe and three folders APP  DATA  OTHER) to hard disk.  Then start the exec from there and compact.  Return the whole structure back to USB.
4) The comment in bug 787197 regarding e-mail being on hard disk should not be correct.  The whole goal of "portable application" is that you use the USB flash on someones system, no interference and no traces left.  Otherwise, later everyone at "Coffee Net" can undelete your e-mails and defeat the purpose.  :)
reply to #27
Parkhideh,  I concur with your summary of my problem in para. 1 

regarding para. 2 &3) Thank you for the help and advice; I never gave much thought to the differences between h/d and flash when using the portable Thunderbird on a USB 'stick'; but I suppose with flash not suffering the same access time and 'wear' penalties of an h/d 'seek', it really does not make much sense to use the compact function of Thunderbird on a flash memory.  I wonder why it even 'pops up' in the portable ap.; it is inviting uninformed or distracted users to get themselves into trouble.

para. 4) Yes, it seems that some of the responders to my problem did not understand the concept of 'portable ap'; but it may have been caused by my absent mindedly calling it a 'mobile ap'.

Thank you again for coming to my assistance; sorry I could not replicate the scenario for you.
I'll test this USB scenario while working on bug 558528.
Depends on: 558528
(In reply to :aceman from comment #29)
> I'll test this USB scenario while working on bug 558528.

While I do not comprehend the issue in bug 558528, I will advise you not to mix it with a FLASH memory device.  Now, how is that for the worth of a free advice?  :)  

Incorrectly some assume that a flash is a slow read or write device.  Hence they attribute the issues to its I/O.  Well, relatively speaking your normal hard disk running at 5400 rpm with 8 Mb cache is a slow device.  I use a Seagate 7200 rpm with 16 Mb cache and 4 Gb of front end semi-conductor memory just for starters, this to put I/O issues out of the way.  :)

The problem with flash memory is that it has three states.  Read / Write / Erase.  The culprit is in the Erase latency.  Erase has a very long delay and you never know when it kicks in.  Unless you know the chip used, you cannot determine how many bytes are erased in an erase action.  So it is near impossible to duplicate a problem unless you know your starting point.  John probably has his 16 Gb flash with lots of empty space; however, they may be flagged "deleted" and not erased yet.  So he will then see different response times based on the flash condition.  When a problem is to be duplicated, I caution you not use a flash device unless you can erase it every time.

Now, can some one in the know tell me what else is left to hash on this bug report?  Can we close it?
Bug 558528 is about the size of the chunk TB writes out in any file operation. The size currently is 4KiB in many situations. AFAIK the erase block on flash devices is 512KiB or more. It may happen that after each 4KB write (update of a file) the flash must do erase of the full block. So for (re)writing a file of 512Kib (e.g. a new email appended to the mailbox file) it must erase the same block 128 times. What about writing the file out in 512KB blocks? We just reduced erase cycles to 1 (or 2 if crossing block boundary). So is that a wrong assumption that bug 558528 may help this case?
My comment #22 really should be withdrawn from this thread as subsequent to my posting it there has been a reasonable determination that the whole cause of the problem was due to a corrupted OS which most likely had nothing to do with Portable Thunderbird running on a USB stick.  In any event, I cannot reproduce the scenario as a new h/d and OS have been installed.  (see comment 28)
(In reply to John from comment #22)
> Not sure if this belongs here; but it might be related or helpful.  While
> running T-bird mobile (latest update as of two days ago; not sure of version
> but I think it was 14) on a USB stick I began having problems with deleting
> files.  I was cleaning out old mail and selecting large groups of mails that
> were sorted by 'sender' and then hitting delete key.  It was taking long
> time to delete so I would open a new mail and read it while letting delete
> 'run'.  I noticed that mail that I thought was deleted kept showing up in
> inbox.  I also was offered 'compact' choice so clicked yes on that.  
> Eventually shut down computer; seemed like a normal shutdown; but windows7
> Home will not reboot or restore.  Laptop is Toshiba Satellite C655D, USB
> stick is 16G Staples (very small physical size).
(In reply to :aceman from comment #31)
Correct you are that 4Kb is too small.  The USB flashes are pushing 16K and 32K byte boundaries for their smallest file size.
If you tell me the names of one or two Thunderbird files that will be affected, then I can look at their size and growth to see if buffer size will have any effect beyond Bug 558528.

I searched the internet and some USB flash manufacturers' web sites to see if they spell out how "append" to a file electrically works.  Unfortunately no luck.  So without testing I cannot say if an append operation is a data block added or is it a rewrite, or a mixture.
(In reply to :aceman (away till Aug 11) from comment #29)
> I'll test this USB scenario while working on bug 558528.

aceman were you able to test this?
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(acelists)
Not yet, but I am still planning to.
I realize that for Pop3 usage, TB downloads the message from the server using
32KB buffer (using |recvfrom| system call),
*BUT*,
TB writes the message to Inbox *line by line* using a |write| system call
each time. That is the output is performed without buffering.
For example, a large attachment of JPEG or PDF ended up
of calling many smallish |write| system calls.

I suspect that USB memory stick works like SSD in terms of update of an internal block:
if it is asked to update only 75 octets, say, it needs to locate the memory block
where the data is being written to.
Then the whole data block is read into an intermediate RAM area, and
the region to be updated is filled with new data in this RAM area,
THEN the whole intermediate RAM buffer/block is written back to EEPROM block
(reprogrammed). That is, not  only 75 octets, but whatever the minimum internal block size needs to be reprogrammed: it takes time 1-50 msec range.

Depending on how clever the OS driver is handling such smallish |write|,
and how clever the EEPROM controller software,
we may end up with reprogramming EEPROM as many as the # of |write| calls in the worst case.
And the worst case is very performance poor.
If the attachment comes in MIME attachment and
in TB's case, it sends MIME as consisting of lilnes of 75 octet including ENDOFLINE,
then 3MB of such lines will cause more than 40K |write| calls. Even if one reprogramming
takes 10 m sec, 40K reprogramming is 400 seconds!

I am trying to make the write to Inbox from pop3 server a buffered output operation, but
no success so far. There seems to be some thread race issues somewhere.

TIA
(In reply to ISHIKAWA, Chiaki from comment #36)
> I realize that for Pop3 usage, TB downloads the message from the server using
> 32KB buffer (using |recvfrom| system call),
> *BUT*,
> TB writes the message to Inbox *line by line* using a |write| system call
> each time. That is the output is performed without buffering.
> For example, a large attachment of JPEG or PDF ended up
> of calling many smallish |write| system calls.

This is only significant if the filedescriptor has O_SYNC (or O_DSYNC) set on it, otherwise the OS is still buffering each write. And if the descriptor *does* have O_SYNC set, the simpler fix is to leave it turned off and just do an fsync at the end of the message. Nothing more complicated than this is needed.
(In reply to Howard Chu from comment #37)
> (In reply to ISHIKAWA, Chiaki from comment #36)
> > I realize that for Pop3 usage, TB downloads the message from the server using
> > 32KB buffer (using |recvfrom| system call),
> > *BUT*,
> > TB writes the message to Inbox *line by line* using a |write| system call
> > each time. That is the output is performed without buffering.
> > For example, a large attachment of JPEG or PDF ended up
> > of calling many smallish |write| system calls.
> 
> This is only significant if the filedescriptor has O_SYNC (or O_DSYNC) set
> on it, otherwise the OS is still buffering each write. And if the descriptor
> *does* have O_SYNC set, the simpler fix is to leave it turned off and just
> do an fsync at the end of the message. Nothing more complicated than this is
> needed.

Thank you for the clarification.

I should check then in what mode Inbox is opened,
and also where fsync is called. (I have an issue of fsync called too often and,  
in some places, it can be replaced by fdatasync under linux at least.
I am surprised to find that fsync is always called in mozilla code by
defining a macro 
#define fdatasync fsync
Reported in
https://lists.mozilla.org/pipermail/dev-platform/2014-May/004762.html
but maybe I should file a bugzilla entry.
)

Unfortunately, the smallish |write| translates immediately to small network packet, though, if the write is to remotely mounted file system (NFS, CIFS, etc.). There is no internal buffering on the local side. (Well, we can tweak some setting, but
usually, by default, one |write| means immediate sending of packets to write data to the remote server. Unnecessarily calling  |write| often is certainly a performance killer. 

But obviously with good OS filesystem code, we can reduce the impact on 
SSD, USB write.

Thank you again.
(In reply to ISHIKAWA, Chiaki from comment #38)
> I should check then in what mode Inbox is opened,
> and also where fsync is called. (I have an issue of fsync called too often
> and,  
> in some places, it can be replaced by fdatasync under linux at least.
> I am surprised to find that fsync is always called in mozilla code by
> defining a macro 
> #define fdatasync fsync
> Reported in
> https://lists.mozilla.org/pipermail/dev-platform/2014-May/004762.html
> but maybe I should file a bugzilla entry.
> )

Be careful there. fdatasync is broken on Linux ext3/ext4 for a large number of Linux revisions.
https://lkml.org/lkml/2012/9/3/83

My searches thru the LKML archive patch lists tell me that this *should* be fixed as of 2.6.32.60, 2.6.34.15, 3.2.30, 3.5.4, and 3.6.0+ but a fair number of those older revisions are still in active use.

> Unfortunately, the smallish |write| translates immediately to small network
> packet, though, if the write is to remotely mounted file system (NFS, CIFS,
> etc.). There is no internal buffering on the local side.

Ugh.

> (Well, we can tweak
> some setting, but
> usually, by default, one |write| means immediate sending of packets to write
> data to the remote server. Unnecessarily calling  |write| often is certainly
> a performance killer. 

Agreed.

> But obviously with good OS filesystem code, we can reduce the impact on 
> SSD, USB write.
> 
> Thank you again.
Regarding fdatasync vs fsync issue, I filed
Bug 1120444 - Uset fdatasync properly instead of fsync where appropriate 

TIA
Depends on: 1120444
Flags: needinfo?(acelists)
Blocks: 484119
No longer depends on: 484119
Kristo, do you still see this issue?
Flags: needinfo?(kristo)
Hello Wayne Mery, not sure if I can help, but I was surfing why my TB is slow and found this bug as most recent connected to USB issues.

I run Lubuntu 16.04.3 LTS 32bit on 3.0 USB Stick with TB installed.
"Repair" in folder properties of Inbox brought some performance as well as compressing it, but I still have reproducable waiting time on different action like every minute.

As I am not a professional programmer I have an affinity, so I offer my help, that you can send me instructions for debuggin / analyzing my system. Best Regards
Severity: major → normal
Severity: normal → S3

(In reply to chukawahwah from comment #43)

I offer my help, that you can send me instructions for debuggin / analyzing my system.

Thanks. A performance profile would help.
https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance

Flags: needinfo?(kristo) → needinfo?(damosaki)
Whiteboard: [investigate USB perf][recheck after dep bugs] → [closeme 2024-03-01][needs performance profile]

Wisewiz, does this reproduce for you?

Flags: needinfo?(danawilson001)

Resolved per whiteboard

Status: NEW → RESOLVED
Closed: 15 years ago → 2 months ago
Flags: needinfo?(danawilson001)
Flags: needinfo?(damosaki)
Resolution: --- → INCOMPLETE
Whiteboard: [closeme 2024-03-01][needs performance profile] → [needs performance profile]
You need to log in before you can comment on or make changes to this bug.