Closed Bug 1192578 Opened 9 years ago Closed 6 years ago

Very slow start profile located on NAS drive and Lightning active but no calendars defined. 2 minute increase in startup

Categories

(Calendar :: General, defect)

Lightning 4.0.1.2
x86_64
Windows
defect
Not set
major

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1202613

People

(Reporter: palle.skyttegaard, Unassigned)

Details

(Keywords: perf)

User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0
Build ID: 20150806001005

Steps to reproduce:

After updating to latest version where Lightning was automatically included, the load time for my profiles which are located on a NAS drive (average real life tranf. speed 6-10mb/s) increased to an unacceptable level (TB .exe are in local disk).    


Actual results:

With Lightning extension active, the profile load time is around 2½ minutes, without about 20 sec.

By the way, TB is generally very slow in accessing data on NAS drives, including read/write of messages. The low level read/write routines may have an implementation unsuitable for network data access. 


Expected results:

Only a minor extra delay for loading an extension was expected
OS: Unspecified → Windows
Hardware: Unspecified → x86_64
Severity: normal → major
Keywords: perf
What calendars do you have (local, ics, caldav,google provider, wcap, ...) and is offline support enabled for those? How many events are therein and do you have (many) recurring events? Which other addons do you have enabled?

If you start with the same profile on local HDD, what startup time are you expiriencing?
Component: Untriaged → General
Product: Thunderbird → Calendar
Version: 38 → Lightning 4.0.1.2
The calendar is empty, has never been used and is not sync'ed with any online calendar. The load time is therefore measured just after a fresh install and I guess it is load time of components that is time consuming. Further adons: only a couple of spelling checkers.
Long, long time since I had any profiles on local disks - oh, have one on an Ubuntu laptop - load times (as I remember it) is less than 5 sec. (also Windows)
It is an access implementation issue which isn't noticeable when accessing local HDD, but I have been annoyed by the slow TB to NAS access time for many years, and for the Lightning addon the slowdown is a show stopper for people with profiles on NAS.
First, can you please provide all the messages you get displayed in the console (ctrl+shift+j) after startup until everything is up and running?

If there's nothing obvious in the log it will be neccessary to dig a little bit deaper following [1] and check with the perf tool [2].

What is the filesystem on the partion of the NAS where the profile is stored? On your Windows system you're on NTFS, I guess?

[1] https://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging/Thunderbird
[2] https://developer.mozilla.org/en-US/docs/Tools/Performance
The NAS drive is using EXT3 file system and running RAID1, local disk is as assumed NTFS.

I get the following messages:
--start--
Could not read chrome manifest 'file:///C:/Program%20Files%20(x86)/Mozilla%20Thunderbird/chrome.manifest'.
Could not read chrome manifest 'file:///N:/Data/Thunderbird%20mail/Profiles/Palle/extensions/en-GB@dictionaries.addons.mozilla.org/chrome.manifest'.
Could not read chrome manifest 'file:///N:/Data/Thunderbird%20mail/Profiles/Palle/extensions/sv@dictionaries.addons.mozilla.org/chrome.manifest'.
Could not read chrome manifest 'file:///C:/Program%20Files%20(x86)/Mozilla%20Thunderbird/extensions/%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D/chrome.manifest'.
Time: 18-08-2015 22:24:29

Advarsel: Brugen af Mutation Events er forældet. Brug i stedet MutationObserver.
Kildefil: chrome://calendar/content/widgets/calendar-widgets.xml 
Linje: 496
--end--
Last message line translated:
Warning: Use of Mutation Events is obsolete. Use instead MutationsObserver. Source file: chrome://calendar/content/widgets/calendar-widgets.xml 
Line: 496
Drive N: above is the NAS drive. The two spelling extension with manifest reading errors seems to be working properly anyway.
Hi,

can you check the size of the Lightning extension directory in your profile? It's in extensions and has the name {e2fda1a4-762b-4020-b5ad-a41df1933103}. Lightning tends to install some 2000 files in 550 directories in the user's profile. Not only does this slow down logon/logoff with a roaming profile to a crawl, Thunderbird also processes all of these files on startup, slowing down startup as well. The amount of files is due to all the languages being installed it seems.
Well, this is not the case in my profile, that folder contains 91 files in 6 folders.
I've deactivated Ligthting due to the slow startup, maybe I should try to active it again to check if the problem persist.
Okay, so you have the version of Lightning distributed with Thunderbird. What we see is that when the user chooses to update Lightning to the latest version, the size of {e2fda1a4-762b-4020-b5ad-a41df1933103} increases to 2044 files in 561 folders. I guess I'd better file a separate bug for this behaviour.
FYI, a good collection of NAS performance related bugs http://mzl.la/1Nnedlc  
http://mzl.la/1NncWdY includes closed bug reports
Noticed that Lightning now is at version 4.02
New test shows startup time have improved since last test. With Lightning deactivated, the TB main window appears after 6-8 sec. and with Lightning activated it takes 18-20 sec. - that's much better than before, but still too slow. Same HW infrastructure used.
Wayne, the link you provided show several cases where users experience slow I/O performance when reading/copying/moving/deleting messages stored on NAS - I do believe that the read/write implementation in TB may be unsuitable for access through a network.
I have been using Lightning for many years now ...
A few days ago I deleted older events (2009, 2010), and it seems that since I did it Thunderbird is very very very slow to start:
- Lightning deactivated: ~10 / 12 seconds
- Lightning activated: almost 3 minutes today to get the main Thunderbird window displayed, and 20 seconds more to get the reminders window displayed !!

I use Mageia Linux with Thunderbird 38.4.0 and Lightning 4.0.4.

Pierre
Hello, after Thunderbird started bundling Lightning, many of our users have had trouble with Thunderbird being very slow to start. We also use a network share for storing the Thunderbird profile.

After looking into it, I have a theory on why this is a problem. Looking into the extensions folder, I can see that the Lightning extension folder contains over 2000 files. It looks like when Lightning is installed for the first time, or when it is being updated, it has to move around all these files in the user's Thunderbird profile. And even though they are small files, it can still be painfully slow when done on a network drive because of the sheer amount of files.

I assume there's no easy way to reduce the number of files used by this extension...

The best solution I can think of now is to add a command-line option to not include Lightning when deploying Thunderbird, is that possible?
Can some of the posters who felt the slow start trace the read/write system calls
during thunderbird slowly starts?
Specifically, can someone find out the size of buffers used for read/write?

I have a theory that TB may not be reading buffered read (or write) at all
thus causing a large amount of read system calls (or write system calls)
thus slowing down the startup very much.

Too many small read/write system calls is bad enough for local hard disk files, but
considering the overhead of network stack processing, many small read/write system calls
against remote file systems can mean a crawling speed even for a fast CPU machine.

Actually, I think saving of attachment is done by small write if I am not mistaken,
thus causing a large amount of time even locally. I shudder to think about it.

TIA
As far as I can see the Lightning package that is shipped with Thunderbird is not that large. Looking at the content of "Thunderbird Setup 38.4.0.exe" Lightning consists of just ~90 files because most files are packed inside chrome.jar. The problem seems to be the Lightning package that is distributed via addons.mozilla.org. This pacakge consists of ~2000 files and no files are packed together.
I tested now on an older version of Thunderbird (3.8.30) on a new clean Thunderbird profile, and it does indeed look like the problem occurs when Thunderbird tries to update the Lightning add-on to a newer version. The Lightning add-on that comes with the Thunderbird installation is only 91 files, but the update is ~2000 files.

However, it appears that the Lightning add-on on Mozilla's add-on repository has no longer been updated since November 30 2015, and it is now an older version (Lightning 4.0.4.1) than the one shipped with Thunderbird 38.5.1 (which is Lightning 4.0.5.1).

Does this mean Mozilla will no longer update the Lightning add-on in their add-ons repository? Because that would probably prevent this issue.
Aaaaand nope, the addon on the repository was updated again, and already a user is complaining about Thunderbird not working because it takes 10-15 minutes to update the addon (when the profile is on a network drive) and to the users it just looks like Thunderbird is not starting up at all.

Is it possible to release the addon in the repository with only ~100 combined files (like the one included with Thunderbird) instead of the thousands of files it currently has?

Or at the very least show some kind of dialog window when it is updating the addons after a restart?
susi_s0rglos, who has a roaming environment, filed bug 1202613.
Summary: Very slow start with profile located on NAS drive and Lightning active → Very slow start profile located on NAS drive and Lightning active but no calendars defined. 2 minute increase in startup
p.skyttegaard,

Do you still see a problem when using version 60?
Flags: needinfo?(p.skyttegaard)
Whiteboard: [closeme 2018-10-15]
Don't know, stopped using Lightning due to the problem as no solution seemed to be coming up soon
Flags: needinfo?(p.skyttegaard)
Aha, addons are no longer unpacked in Thunderbird 60.0? That *does* sound like it should solve the issue.

I had changed from having the Lightning addon installed on the user profile, to having it centrally installed in Program Files, so we haven't had the issue either at our workplace either.

But I tested it now on my own computer:

- Removed the centrally installed Lightning addon from Program Files and tried to install Lightning again manually in Thunderbird 54, this immediately made Thunderbird unresponsive because it was unpacking thousands of files to my network drive.

- Force closed Thunderbird and removed the addon files from extensions/staged in my TB profile

- Upgraded Thunderbird to version 60.0

- Lightning didn't show up at first, had to use the configuration editor tip from https://support.mozilla.org/en-US/kb/calendar-updates-issues-thunderbird to make it reappear I think, probably because I removed the addon earlier and Thunderbird *remembered it* (RIP Telltale)

- Restarted Thunderbird, it started up quickly and Lightning seems to be installed.

- Had a look at the extensions folder in my Thunderbird profile on my network drive, it does seem like the Lightning addon is installed as one file now as advertised.

So at least in this one quick test, it does seem to me like TB 60.0 has finally fixed this particular issue.
Thanks for the test, and updating the bug.
Status: UNCONFIRMED → RESOLVED
Closed: 6 years ago
Resolution: --- → DUPLICATE
Whiteboard: [closeme 2018-10-15]
You need to log in before you can comment on or make changes to this bug.