Closed Bug 576017 Opened 14 years ago Closed 12 years ago

lightning hangs thunderbird regularly only when caching is enabled

Categories

(Calendar :: General, defect)

Product:
Calendar
Component:
General
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: spotter, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: perf, Whiteboard: [no more comments please]) 

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100423 Ubuntu/10.04 (CK-IBM) (CK-IBM) Firefox/3.6.3
Build Identifier: 2010625031746

it seems that when lightning tries to check my network calendars for updates / refreshes them, it hangs thunderbird so that it can't be used for a decent amount of time.

I have 7 calendars.

1 local (rw)
1 via google (rw) via their caldav support

5 ro 

2 of them via google
2 of them via http ics file (facebook exports vs fbcal)
1 of them via http ics file (debian debconf cal)

nothing out of the ordinary I think, just very annoying how all of thunderbird hangs while it appears the update is going on.

Reproducible: Always
I'm going to guess that this is related to the sqlite issue that affected firefox.

why?

#1) when its hanging, my disk IO is going crazy on my laptop (hard drive light is pegged)

#2) when run under strace, I see lots of sqlite activity (well work on sqlite file descriptors) with lots of calls to fdatasync() where even strace pauses.
further proof, just like setting

"toolkit.storage.synchronous" to 0 back in the day for firefox fixed the issue with firefox, it fixes the issue with thunderbird as well.

But it will obviously make my thunderbird very unreliable in regards to data integrity.
I'll further add, if one doesn't cache calendars, one doesn't have this problem at all.
Severity: normal → major
Keywords: perf
Blocks: 441710
fallen, do you need a stacktrace for this?
Summary: lightning hangs thunderbird regularly → lightning hangs thunderbird regularly only when caching is enabled
(In reply to comment #2)
> further proof, just like setting
> 
> "toolkit.storage.synchronous" to 0 back in the day for firefox fixed the issue
> with firefox, it fixes the issue with thunderbird as well.
> 
> But it will obviously make my thunderbird very unreliable in regards to data
> integrity.

I'd think bug 501689 is what's needed.

If not, which of these storage/networking bug#s should this be dependent on?  https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=anywordssubstr&field0-0-0=short_desc&type0-0-1=anywordssubstr&field0-0-1=component&type1-0-1=allwordssubstr&resolution=---&classification=Client%20Software&classification=Components&query_format=advanced&longdesc=cach%20async&value0-0-1=cache%20storag&type0-0-0=anywordssubstr&field1-0-0=short_desc&longdesc_type=allwordssubstr&field1-0-1=keywords
Depends on: 501689
interesting. The actual sql queries are pretty fast and thus shoulnd't be hanging the main thread. Maybe my upcoming new patch to bug 494788, which will use js generators to make the listener calls quasi-asynchronous will fix this bug.
Depends on: 494788
Shaya, now that bug 494788 is fixed, is your issue resolved when using Thunderbird 5 using lightning 1.0b4 or b5, or TB6+1.0b5? 

(In reply to Philipp Kewisch [:Fallen] from comment #6)
> interesting. The actual sql queries are pretty fast and thus shoulnd't be
> hanging the main thread. Maybe my upcoming new patch to bug 494788, which
> will use js generators to make the listener calls quasi-asynchronous will
> fix this bug.
I see this happening in TB5.0.2 with L1.0b5 -- I subscribed to a read-only remote caldav calendar with *LOTS* of events in it, and with caching enabled I see TB creating, opening, closing and deleting $profile\calendar-data\cache.sqlite-journal repeatedly, it looks like once per event!  I managed to open the file in emacs once while it was still open; the file contained an awful lot of ^@ characters, some email addresses matching people in the calendar, and then several CREATE TABLE commands at the tail of the file.

Turning off caching immediately returned the responsiveness of TB, because it wasn't doing bizarrely frequent disk IO.
I've tried the cache recently due to prompting by Wayne, and it seemed to behave fine for me, but didnt stress it out.
I still have this problem with Thunderbird 8.0 and Lightning 1.0 for 2 Google calendars and 3 intranet ics calendars under Windows 7 x64.

Thunderbird completely hangs for several minutes with heavy disk io.

Maybe this output from Procmon might help:

13:52:43,0550274	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\cache.sqlite-journal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0553435	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\cache.sqlite-wal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0602981	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\local.sqlite-journal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0606159	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\local.sqlite-wal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0627554	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\cache.sqlite-journal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0630633	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\cache.sqlite-wal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0638045	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\cache.sqlite-journal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0641424	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\cache.sqlite-wal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0649110	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\cache.sqlite-journal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0652096	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\cache.sqlite-wal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0659243	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\cache.sqlite-journal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0662220	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\cache.sqlite-wal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0670278	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\cache.sqlite-journal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0673246	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\cache.sqlite-wal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0682656	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\local.sqlite-journal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
13:52:43,0685761	thunderbird.exe	2668	CreateFile	C:\Users\fred\AppData\Roaming\Thunderbird\Profiles\2x5hjklp.default\calendar-data\local.sqlite-wal	NAME NOT FOUND	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a



The actual contents of cache are:

02.12.2011  13:44         1.277.952 cache.sqlite
14.11.2011  09:40            30.720 local.sqlite


I deleted these to get them regenerated, but this did not help.
I want to add, that just before the hang, there is an alarm popup window and alarm sound without any content that quickly disappears, despite that there are no alarms set for any event in all of my calendars.

The hang occures directly after thunderbird startup and from then on: occasionally but at least once within 30min.

Very annoying :(
The hang can be triggerd by initiating "reload external calendars"

Then, a sqlite journal file appears in cache folder whose size quickly fluctuates between 0 and 200kb. 

When the hang is over, the journal file disappears.
I don't get the alarm popup window like Fred does, but the journal file behaviour is the same. It happens every single time that external calendars are reloaded, which also explains the regular hangs.

Two things need to be fixed. First, the disk i/o really should not be this bad. Second, UI responsiveness should not depend on data being refreshed.

The hang time also grows at least lineary with the amount of external calendars used. It only happens with cache enabled.
For those of you who haven't noted this yet, what kind of calendar are you using? Does setting the pref mentioned above (temporarily) work around the issue?
I am strictly using remote calendars on a CalDAV server (Horde). toolkit.storage.synchronous does not exist in my about:config by default; adding a new Boolean with that name that is set to "false" does not seem to have any effect at all.
I use roundcube calendar and google remote ics on my Thunderbird and have the same effect.
I deactivated the cache and the "temporary freeze" disappeared but Tb is still a few slower during some secs.

I think I will unable caching on the computers of customers without SSD but, anyway, I hope the issue will be fixed one day.

Courage!
Just a hint: SSD or not doesn't matter. I have an SSD in my laptop and the problem still happens the same way (with the same delay) as on my desktop with an HDD.
I am seeing this (I think) on OS/2, as well, so this may not be limited to Linux (I'll test against one of my Linux systems to see if the behavior is consistent).

Presently, I am using:

Mozilla/5.0 (OS/2; Warp 4.5; rv:9.0.1) Gecko/20111221 Firefox/9.0.1 SeaMonkey/2.6

with Lightning 1.1pre and GData Provider 0.9

Creating & setting toolkit.storage.synchronous to false has no effect. Enabling caching for my Google calendar results in the CPU jumping to 75-99% for several seconds, constant disk I/O, and SeaMonkey becoming unresponsive (in one instance, enabling caching for three additional holiday calendars - read-only - resulted in an application crash). This is also seen when a calendar is set for caching, caching is disabled, and then re-enabled (the calendar cache does not change in size, but apparently is rebuilt). At the height of the activity, I see:

 2-15-12  11:48       5,734,400      0  cache.sqlite
 2-15-12  11:48         328,272      0  cache.sqlite-journal

Once the cache is rebuilt, the journal file disappears, and I am left with the db file of the same size as above.

This thread's priority needs to be lowered, I think.
(In reply to Shaya Potter from comment #9)
> I've tried the cache recently due to prompting by Wayne, and it seemed to
> behave fine for me, but didnt stress it out.

WFM per OP - OP feels her/his issue is solved.

So other existing issues might be pushed into one off these bugs https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=substring;keywords=perf;keywords_type=allwords;list_id=2845058;field0-0-0=short_desc;type0-0-1=substring;field0-0-1=keywords;type1-0-1=allwordssubstr;resolution=---;classification=Client%20Software;classification=Components;query_format=advanced;type0-0-0=anywordssubstr;field1-0-0=short_desc;product=Calendar;field1-0-1=short_desc or a new bug.

You might first want to try Tbird version 12 (currently beta) and calendar 1.4, which has fixed Bug 710351 - Full sync providers like ICS very slow when refreshing
Status: UNCONFIRMED → RESOLVED
Closed: 12 years ago
Resolution: --- → WORKSFORME
Another data point: I moved to TB12 and there are no more hangs. Life has gotten so much better!
I'm using Thunderbird 12.0.1 with Lightning 1.4 and Provider for Google Calendar 0.9. I have one main large Google calendar I've been using for months (and for years before that as a local file), and a second Google calendar that has very few entries (several weeks blocked for travel). I also have two additional remote read-only holiday ICS calendars. This configuration worked just fine for months.

Now, within the last couple of releases, suddenly everything freezes up every few seconds so that I can't effectively use the software. In the Multiweek view, for instance, the first "next" arrow click for displaying the next week works fine. Another click and the whole thing hangs for several seconds.

Even if I uncheck all my calendars, this still occurs. It is hardly usable. Please undo whatever was did in the past couple of releases of Thunderbird/Lightning.
Update: when I disable all calendars and give it a while to settle down, I have no hangs. Then I re-enable my main Google calendar only, and click "next week" a couple of times. There is a huge red spike from Thunderbird on ProcessExplorer one of my cores that dies back down quickly. But then on another core Thunderbird kicks in with a huge green spike that stays for several seconds until Thunderbird becomes unfrozen.
So do I open a new bug for this, or reopen this one?
Sorry, Garret, but unless I missed it, I didn't see you mention whether you had caching enabled or not. Perhaps review the other bugs in Wayne's comment #19 and see if something there is more specific to your problem. If not, then I guess we could re-open this one (I'm waiting on a new Lightning build for OS/2 to see if this has actually been fixed for me).
I do *not* have caching enabled, and as far as I know, I never have.
And ultimately if Shaya continues to work OK (comment 9) then the bug should stay closed anyway.
Thanks, Wayne.

Garret, as your situation does not involve caching, then it is either something new or something related to one of the other links previously mentioned. As Wayne said, this bug should remain closed.
OK, thanks. I've opened Bug 753297.
Using Ubuntu 12.10 with thunderbird 17.0. the problem has seriously deteriorated. When I set one calender to have offline support, I experience several freezes in a row, each of about a minute and more. Looking in the calender-data in my thunderbird profile folder, I see the file cache-sqlite-journal being created (after which a period of freezing occurs) and deleted. This caching as currently implemented is clearly not working as it should. This bug should therefore be reopened.
FYI: my cache.sqlite is 2.3 MB in size.
I can confirm the bug for Ubuntu 12.04 with Thunderbird 17.0.2 and Lightning 1.9. As soon as I enable caching on one of my network ics calendars, I get the described hanging & hdd activity behavior. Please reopen this bug!
This bug is still there, yet this bug report is marked as "RESOLVED WORKSFORME". Please reopen this bug or mark is as a duplicate. It is NOT solved.
(In reply to ftack from comment #31)
> This bug is still there, yet this bug report is marked as "RESOLVED
> WORKSFORME". Please reopen this bug or mark is as a duplicate. It is NOT
> solved.

please see comment 26. If you still see this problem please file a new bug and post the number here
Has anybody found a fix or workaround for this bug? For me, Thunderbird hangs immediately after startup.
The same happens to me using a fresh installation of Ubuntu 12.04.3 (LTS).
The workaround is to disable caching as first suggested:
1. Pull out the network cable os disable the wifi device;
2. Start Thunderbird/Lightning;
3. Open the Lightning tab;
4. Right-click your calendar and click "properties";
5. Uncheck "Offline Support".
No longer depends on: 501689
Still painfully affects my Lightning too (2.6.4 on Thunderbird 24.2.0).
Every hour, when my CalDav Google calendar syncs the application freezes for several minutes (while spinning the laptop disk continuously).
Probably due to large calendar size (a few thousands of appointments).
I would be fine with limiting the sync time window (e.g. 1 month into the past) but there is not such option I know of in Lightning.
Asynchronous sync would also be very nice, since now I cannot do anything (not even read an e-mail) during the sync period.
affects me too. the strange thinh: all of a sudden started on my pc, while my laptop (with same gmail+google calendar in thunderbird) keeps working fine with offline support (maybe because a solid state disk/fresher install/ubuntu instead of kubuntu?)
This affects me.  I am on Thunderbird 31.1.2.
If you read my comments above, you'll see that I had this problem but that it went away when I turned off the search bar.

But suddenly the Provider for Google Calendar plugin https://addons.mozilla.org/en-US/thunderbird/addon/provider-for-google-calendar/ (used with Lightning) updated to version 1.x, and it uses some sort of new token system with Google and now pulled all its calendars directly from Google. (In earlier versions of the provider I had to add each calendar manually and provide the specific link given by Google.) I have six Google calendars and one task list.

Now the performance is unbearable!! When I am on the Multiweek view of Lightning, each time I hit "next" to go to the next week, the calendar blanks out and it takes about five seconds for my events to appear again. This is unusable!! Please, how can I bring back the quick response I used to have from these same calendars?

I don't see any option for caching, neither in the individual calendars, nor in Lightning settings, nor anywhere in Thunderbird's settings.

(Note that because of the changes in this extension, one has to remove all the manual calendars, remove the ThunderBirthday extension, remove the stored passwords, restart Thunderbird, and then add the calendars back. See the comments and complaints on the add-on's home page.)
It's the birthdays. If you use Provider for Google Calendar 1.0.1 to add your Google contact birthdays as a calendar to Lightning, it causes about a 5 second delay every time you even change a tab.
I upgraded to the new google calendar, delisted the Birthdays calendar (which I never added to Lightning in any case), and now I find that Thunderbird reliably hangs within seconds after going online whenever Provider 1.02 is enabled. Hope someone in a position to fix this takes a look at this highly reproducible problem ...
Further comments are wasted here - no one pays attention to closed bug reports.
Reporter no longer sees this problem, so the bug report was closed.

Please file a new bug report with precise, reproducible steps to reproduce.
Whiteboard: [no more comments please]
You need to log in before you can comment on or make changes to this bug.