1358471 - Keep messages in current view in RAM rather than disk to have instant visualization, because slow in retrieving messages for preview pane with HWA disabled

Status: RESOLVED WONTFIX

(Thunderbird :: Message Reader UI, defect)

Description

[Anselmo Canfora]

User Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0
Build ID: 20170413192749

Steps to reproduce:

This is an idea/feature request.
TB is painfully slow in retrieving messages for preview pane while switching between them in list pane, even putting TB in offline mode (I guess in that case it fetches messages from disk) on a SSD makes no difference.
Is it possible to keep pre-render of messages currently visible in list pane (to update when I scroll/filter) in RAM in order to have *instant* visualisation when I select them?

What I do? Try to visualise a message


Actual results:

Message is being displayed with 1/2 seconds delay, it is not *instant* like should be.


Expected results:

Message visualisation in preview pane or new tab should be instant.

Comment 1

[Wayne Mery (:wsmwk)]

Your solution is to find what is causing the performance issue, not in loading messages into memory, which is impractical.

https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems can help get you started. If after going through those steps a bug is found, please file a new bug report.  If you need help going through that process, please file a support request at https://support.mozilla.org/products/thunderbird

Comment 2

[Anselmo Canfora]

Sir, I have absolutely no problems with my RAM, which is being used at its 50%, the problem is that TB is dog slow on an 4 cores / 8 threads i7 with 8GByte RAM and a Samsung 840 Pro SSD. It could be nice from TB support to try to raise this issue to developers instead of closing bugs with irrelevant remarks like yours seems to me.
Have a nice day.

Comment 3

[Wayne Mery (:wsmwk)]

Look, we're volunteers here and trying to be both nice, and helpful.  Your best hope for timely resolution are the items I mentioned.  I am fully aware a performance issues (heck, I wrote most of that article), and even aware that people on super fast machines with SSD sometimes have performance issues.  

As for devs, I am fully qualified to speak on developers behalf on this point - we wouldn't do what you suggested as far as caching because not everyone has gobs of memory to do as you suggest. In addition, in a properly functioning system the slowness you describe doesn't happen (with rare exception - and these are documented in the wiki)

Comment 4

[Jorg K (CEST = GMT+2)]

Hmm, I have a humble AMD A10-5700 from 2013 (4 y/o) and an SSD worse than yours, but the message store is on a hard disk.

It takes a while to open a large folder, but once open, the message visualisation is instant, certainly not with 500 ms delay.

Are we talking about local folders or IMAP folders here? And if IMAP, they are synchronised for offline use, right? Otherwise of course the message will be fetched from the server when you click it. After that it's cached in memory.

Comment 5

[Anselmo Canfora]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #3)
> Look, we're volunteers here and trying to be both nice, and helpful.  Your
> best hope for timely resolution are the items I mentioned.  I am fully aware
> a performance issues (heck, I wrote most of that article), and even aware
> that people on super fast machines with SSD sometimes have performance
> issues.  
> 
> As for devs, I am fully qualified to speak on developers behalf on this
> point - we wouldn't do what you suggested as far as caching because not
> everyone has gobs of memory to do as you suggest. In addition, in a properly
> functioning system the slowness you describe doesn't happen (with rare
> exception - and these are documented in the wiki)

If you are a volunteer please accept my apologies for being somewhat rude.
I have no idea why TB is so slow, the only item on your checklist I was missing was unsubscribing "All Mail" folder for a Gmail account (it didn't solve the issue anyway).

Comment 6

[Anselmo Canfora]

(In reply to Jorg K (GMT+2) from comment #4)
> Hmm, I have a humble AMD A10-5700 from 2013 (4 y/o) and an SSD worse than
> yours, but the message store is on a hard disk.
> 
> It takes a while to open a large folder, but once open, the message
> visualisation is instant, certainly not with 500 ms delay.
> 
> Are we talking about local folders or IMAP folders here? And if IMAP, they
> are synchronised for offline use, right?

double yes, IMAP folders marked for offline use

> Otherwise of course the message
> will be fetched from the server when you click it. After that it's cached in
> memory.

I tried also to put TB in offline mode, the delay in visualisation is almost the same, I cannot tell it apart from online mode.
The delay is the same on local folders, so seems definitely something to do with disk I/O

Comment 7

[Jorg K (CEST = GMT+2)]

Well, IMAP folders marked for offline use are as good as local folders. Sorry, I can't help you any further, the message visualisation it pretty quick on local folders for me, even with my old hardware. There was a 500 ms delay going through a non-synchronised IMAP folder.

Does your IMAP store use mbox or maildir storage? I'm using mbox. For maildir, every message is stored in its own file so it could be slower to request this file from the OS, instead of perhaps OS-caching (large portions of) the mbox find and then just seeking in it.

You'd have to drive the usual performance tools at it.

Comment 8

[Jorg K (CEST = GMT+2)]

Typo: instead of perhaps OS-caching (large portions of) the mbox *file* and then just seeking in it.

Comment 9

[Anselmo Canfora]

(In reply to Jorg K (GMT+2) from comment #7)
> Well, IMAP folders marked for offline use are as good as local folders.
> Sorry, I can't help you any further, the message visualisation it pretty
> quick on local folders for me, even with my old hardware. There was a 500 ms
> delay going through a non-synchronised IMAP folder.
> 
> Does your IMAP store use mbox or maildir storage?

MBOX, I would like to try Maildir, but it is not recommended due to pending bugs.

> I'm using mbox. For
> maildir, every message is stored in its own file so it could be slower to
> request this file from the OS, instead of perhaps OS-caching (large portions
> of) the mbox find and then just seeking in it.
> 
> You'd have to drive the usual performance tools at it.

Comment 10

[Wayne Mery (:wsmwk)]

we are all volunteers.  Except jorg :)

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #1)
> Your solution is to find what is causing the performance issue, not in
> loading messages into memory, which is impractical.
> 
> https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems 

If you went through and tested *all* numbered items at that link, please note the last line following item #24 which has a link about providing more details.

In regard to "use numbers - for example cpu percentage, amount of memory, number of folders and messages, etc." I'd be particularly interested in how much memory Thunderbird process is using, number of accounts by account type, number of messages in the folders involved and any other items you can quantify

Comment 11

[Anselmo Canfora]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #10)
> we are all volunteers.  Except jorg :)
> 
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #1)
> > Your solution is to find what is causing the performance issue, not in
> > loading messages into memory, which is impractical.
> > 
> > https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems 
> 
> If you went through and tested *all* numbered items at that link, please
> note the last line following item #24 which has a link about providing more
> details.

cannot find it

> In regard to "use numbers - for example cpu percentage, amount of memory,
> number of folders and messages, etc." I'd be particularly interested in how
> much memory Thunderbird process is using, number of accounts by account
> type, number of messages in the folders involved and any other items you can
> quantify

I have three imap accounts, one of them is a Gmail one. The Gmail has 17k mails in his AllMail, the only local folder has 24k mails in it. I would say that I have less than 50k mails in total.
My TB profile folder is on a SSD distinct from the system one, and is 3.44 GiB big. The OS here is Windows 8.1 with indexing enabled only on start folder (200 elements), Defender real time check disabled, furthermore TB process and profile explicitly put in Defender exclusion lists for times when I enable Defender realtime check. TB Process takes less than 140 MiB of RAM and less than 1% of CPU when idle, with spikes to 7-8% when changing folder. panacea.dat is about 70 KiB, global-messages-db.sqlite is about 147 MiB, I have the following add-ons installed:

- GlodaQuilla
- ImportExportTools (never used though)
- Password Exporter
- Remove Duplicate Messages
- Saved Password Editor
- StartupMaster

any clue?

Comment 12

[Jorg K (CEST = GMT+2)]

Not really, you'd have to use some Windows performance tools to figure out what happens when you click on the next locally stored message. Clicking or advancing with the cursor keys should be pretty much instantaneous.

Comment 13

[Anselmo Canfora]

(In reply to Jorg K (GMT+2) from comment #12)
> Not really, you'd have to use some Windows performance tools to figure out
> what happens when you click on the next locally stored message. Clicking or
> advancing with the cursor keys should be pretty much instantaneous.

Any advice about the tool to use or what parameters to watch? I would guess Sysinternals Process Explorer is sufficient, but no idea where to look. The only strange thing I can see is CPU spike from 0.1% to to 6-9% when I switch message in message list. Checked several things all system wide, even SSD firmware updates, disabled Windows Search completely, TB seems just a little bit more reactive but forget about instant message display, it takes about 1 second when I switch message, no way.

Comment 14

[Anselmo Canfora]

Attachment: sshot-2017-04-29-[14-47-22].png

Comment 15

[Anselmo Canfora]

To whom is interested, this:

https://www.faqforge.com/windows/how-to-speedup-slow-mozilla-thunderbird-email-client-on-windows-7/

really helped, layers.acceleration.disabled = true improved performances a lot.

Comment 16

[Wayne Mery (:wsmwk)]

(In reply to Anselmo Canfora from comment #15)
> To whom is interested, this:
> 
> https://www.faqforge.com/windows/how-to-speedup-slow-mozilla-thunderbird-
> email-client-on-windows-7/
> 
> really helped, layers.acceleration.disabled = true improved performances a
> lot.

Anselmo, Thanks for this info. Please copy the video section from Help > Troubleshooting and paste it here in the bug report

Comment 17

[Anselmo Canfora]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #16)
> (In reply to Anselmo Canfora from comment #15)
> > To whom is interested, this:
> > 
> > https://www.faqforge.com/windows/how-to-speedup-slow-mozilla-thunderbird-
> > email-client-on-windows-7/
> > 
> > really helped, layers.acceleration.disabled = true improved performances a
> > lot.
> 
> Anselmo, Thanks for this info. Please copy the video section from Help >
> Troubleshooting and paste it here in the bug report



  Application Basics

    Name: Thunderbird
    Version: 52.3.0
    User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0
    Profile Folder: Open Folder

              (Local drive)
    Application Build ID: 20170815040357
    Enabled Plugins: about:plugins
    Build Configuration: about:buildconfig
    Memory Use: about:memory
    Profiles: about:profiles

  Crash Reports
    https://crash-stats.mozilla.com/report/index/bp-86efc65f-1298-4d65-a492-b53462151106 (06/11/2015)

  Extensions
    GlodaQuilla, 0.3.3, true, [e-mail]
    ImportExportTools, 3.2.4.1, true, {3ed8cc52-86fc-4613-9026-c1ef969da4c3}
    Password Exporter, 1.3.4, true, {B17C1C5A-04B1-11DB-9804-B622A1EF5492}
    Remove Duplicate Messages, 0.1.14, true, {12345678-1234-1234-1234-123456789abc}
    Saved Password Editor, 2.10.3, true, [e-mail]
    StartupMaster, 1.6.3, true, {506d044e-41fa-4cc8-9dc6-9ff70e96eebf}

  Important Modified Preferences

    Name: Value

      accessibility.lastLoadDate: 1502794089
      accessibility.typeaheadfind.flashBar: 0
      browser.cache.disk.capacity: 358400
      browser.cache.disk.filesystem_reported: 1
      browser.cache.disk.smart_size_cached_value: 358400
      browser.cache.disk.smart_size.first_run: false
      browser.cache.disk.smart_size.use_old_max: false
      dom.apps.reset-permissions: true
      extensions.lastAppVersion: 52.3.0
      font.internaluseonly.changed: false
      font.name.monospace.el: Consolas
      font.name.monospace.tr: Consolas
      font.name.monospace.x-baltic: Consolas
      font.name.monospace.x-central-euro: Consolas
      font.name.monospace.x-cyrillic: Consolas
      font.name.monospace.x-unicode: Consolas
      font.name.monospace.x-western: Consolas
      font.name.sans-serif.el: Calibri
      font.name.sans-serif.tr: Calibri
      font.name.sans-serif.x-baltic: Calibri
      font.name.sans-serif.x-central-euro: Calibri
      font.name.sans-serif.x-cyrillic: Calibri
      font.name.sans-serif.x-unicode: Calibri
      font.name.sans-serif.x-western: Calibri
      font.name.serif.el: Cambria
      font.name.serif.tr: Cambria
      font.name.serif.x-baltic: Cambria
      font.name.serif.x-central-euro: Cambria
      font.name.serif.x-cyrillic: Cambria
      font.name.serif.x-unicode: Cambria
      font.name.serif.x-western: Cambria
      font.size.fixed.el: 14
      font.size.fixed.tr: 14
      font.size.fixed.x-baltic: 14
      font.size.fixed.x-central-euro: 14
      font.size.fixed.x-cyrillic: 14
      font.size.fixed.x-unicode: 14
      font.size.fixed.x-western: 14
      font.size.variable.el: 17
      font.size.variable.tr: 17
      font.size.variable.x-baltic: 17
      font.size.variable.x-central-euro: 17
      font.size.variable.x-cyrillic: 17
      font.size.variable.x-unicode: 17
      font.size.variable.x-western: 17
      gfx.direct2d.disabled: false
      mail.openMessageBehavior.version: 1
      mail.winsearch.firstRunDone: true
      mail.winsearch.global_reindex_time: 1359924072
      mailnews.database.global.datastore.id: 50582eea-7d0a-43c3-a762-c36decaa92b
      mailnews.database.global.views.conversation.columns: {"threadCol":{"visible":true,"ordinal":"1"},"flaggedCol":{"visible":true,"ordinal":"3"},"attachmentCol":{"visible":false…
      mailnews.database.global.views.global.columns: {"threadCol":{"visible":true,"ordinal":"1"},"flaggedCol":{"visible":true,"ordinal":"3"},"attachmentCol":{"visible":false…
      media.gmp.storage.version.observed: 1
      network.cookie.prefsMigrated: true
      network.predictor.cleaned-up: true
      places.database.lastMaintenance: 1506071740
      places.history.expiration.transient_current_max_pages: 122334
      plugin.importedState: true
      plugin.state.npadobeaamdetect: 0
      plugin.state.npccplugin: 0
      plugin.state.npccpluginex: 0
      plugin.state.npffwloplugin: 0
      plugin.state.npgoogleupdate: 0
      plugin.state.nppdf: 0
      plugin.state.nptcplugin: 0
      privacy.donottrackheader.enabled: true
      privacy.sanitize.timeSpan: 0
      security.sandbox.content.tempDirSuffix: {a4ff3643-e0a5-4e9e-aae6-f9b30842ec38}

  Graphics

      GPU #1
      Description: Intel(R) HD Graphics 4000
      Vendor ID: 0x8086
      Device ID: 0x0166
      RAM: Unknown
      Drivers: igdumdim64 igd10iumd64 igd10iumd64 igdumdim32 igd10iumd32 igd10iumd32
      Driver Version: 10.18.10.3958
      Driver Date: 9-30-2014

      GPU #2
      Description: NVIDIA GeForce GT 630M
      Vendor ID: 0x10de
      Device ID: 0x0de9
      RAM: 2048
      Drivers: nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um
      Driver Version: 10.18.13.5382
      Driver Date: 8-7-2015

      Features
      Direct2D: false
      DirectWrite: true (6.3.9600.18696)
      ClearType Parameters: Gamma: 2,2 Pixel Structure: RGB ClearType Level: 50 Enhanced Contrast: 100
      WebGL Renderer: Google Inc. -- ANGLE (Intel(R) HD Graphics 4000 Direct3D9Ex vs_3_0 ps_3_0) -- OpenGL ES 2.0 (ANGLE 2.1.0.2a250c8a0e15)

      AzureCanvasBackend: skia
      AzureCanvasAccelerated: 0
      AzureFallbackCanvasBackend: cairo
      AzureContentBackend: skia

  JavaScript

  Incremental GC: 1

  Accessibility

    Activated: 0
    Prevent Accessibility: 0

  Library Versions

      Expected minimum version
      Version in use

      NSPR
      4.13.1
      4.13.1

      NSS
      3.28.5
      3.28.5

      NSS Util
      3.28.5
      3.28.5

      NSS SSL
      3.28.5
      3.28.5

      NSS S/MIME
      3.28.5
      3.28.5