410874 - ICS file is deleted from Webdav server in some cases

Status: RESOLVED INCOMPLETE

(Calendar :: Provider: ICS/WebDAV, defect)

Description

[Mészáros Gyula]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b2) Gecko/2007121016 Firefox/3.0b2
Build Identifier: Lightning 0.7 build 2007102303

I created a new event in the family calendar and I decided to move it to my own calendar. Lightning (i.e. Thunderbird 2.0.0.9 (20070801)) is freezed. The effect: the event remained in the family calendar and my own calendar file has gone.
The webdav server (apache 2.2.4) logged the following error messages
access_log:
8:58:28 ... "PUT /webdav/xxxx.ics HTTP/1.1" 500 1069
error_log:
8:58:53...[error] [client ... ] Could not get next bucket brigade [500, #0]
Both the webdav server and the workstation runs openSUSE 10.3 with all the latest patches.

Reproducible: Couldn't Reproduce

Steps to Reproduce:
1.Use webdav ics calendars.
2.Create a new event in a calendar.
3.Try to move it to the other calendar (by opening it and changing the "calendar" drop down box).
Actual Results:  
In this case:
- the event remained in the first calendar
- the second calendar file is deleted from the webdav server


Expected Results:  
- the event is moved to the second calendar
- but at lest the second calendar is not deleted

See, I can accept that it is a fault in the communication, in apache, or even the whole webdav thing is bad, but at least some protocols need to be invented to prevent deleting calendar files (using temporary files and renaming (if it's possible with webdav), keep some local copies from which the calendar can be restored after such a glitch, or anything else). Calendar deletion has happened to me second time already in a half year.
The whole calendar thing is a big showstopper in enterprise acceptance of open source desktops and Mozilla made remarkable steps forward, but despite the fact Lightning is only version 0.7 it still mustn't loose data in any circumstances.

Comment 1

[Mészáros Gyula]

It can be important: as far as I remember after changing the calendar of the event and clicking to the save button I immediately tried to quit Thunderbird then it (or he, or she?) got frozen.
So addition to steps to reproduce:
4. Quit Thunderbird immediately

Yes, I found backup calendars in the BackupData folder of Thunderbird profile, and I copied back the latest one to the server manually but it is not a seamless method an ordinary end user can deal with.

Comment 2

[Sebastian Schwieger]

I did have similar problems more than once. I think I did kill the thunderbird process after it became unresponsive. Afterwards the calendar was gone. It would be nice to offer the user to use the backup calendar if things have gone wrong. It's however not clear how to detect that...

Comment 3

[Mészáros Gyula]

If Lightning (or Sunbird) can not use a calendar there may well be an option to restore it from the latest backup. However, there can be cases when the calendar was deleted from another workstation so that workstation contains the recent backup, consequently the process shouldn't be full automatic.

How about this use case:

Actors:
-------
user, Lightning (or Sunbird), webdav server

The main success scenario:
--------------------------
start
1. The user opens Lightning.
2. Lightning tries to open a calendar from webdav server.
3.a. The calendar opens.
end

Exceptional cases:
------------------

3.b. There is a problem reported back from webdav server.
4. Lightning displays an error dialog to the user that there is a problem with the calendar with the choices
- to open the calendar read-only
- to try to fix the calendar from the latest backup, and it also displays the date and time of the last backup.
5.a. The user chooses to fix the calendar.
6. Lightning sends the calendar to the webdav server.
7. Go to 2.

5.b. The user chooses to open the calendar read-only.
6.b. Lightning opens the calendar read-only.
end

If it helps I can also create a model in Rational Software Architect with Use-case and Activity Diagram.

Comment 4

[Daniel Boelzle [:dbo]]

Unless we have a reproducible scenario, this bug hardly helps us. It's not clear to me whether your server is causing your dataloss.

Comment 5

[Mészáros Gyula]

Yes, I know. However, some two-phase commit-like protocol may help to overcome the situation. What about the use case I wrote in Comment #3?
My server is https://www.mgyhardsoft.hu which is an Apache httpd 2.2.8 coming with openSUSE 11.0 x86_64 with the mod_webdav extension. HTH.

Comment 6

[john cass]

Im surprised more people arent seeing this bug, I have a completely vanilla apache webdav server holding my ICS files and to see one just disappear is terrible, particularly since I am trying to promote usage of this tool within my group.  Its happened to me at least twice in the last month.
Luckily thanks to this thread I found a local recent backup  in ~/.mozilla/sunbird/... to restore
But this should just not happen.  
I will try to pin down causes and post again if possible...

Comment 7

[Mészáros Gyula]

Yesterday I realized that the family calendar has been truncated and it could not be opened anymore. If it helps and anyone tries to deal with this bug I can upload the file fraction.

As I wrote in my first comment this bug simply cannot happen if Mozilla wants to play any role in the enterprise. IMHO.

Gyula Mészáros
former CIO, CIB Leasing Group, Budapest, Hungary
former Head of Development, Postbank and Savings Bank, Budapest, Hungary

Comment 8

[fklucke]

Same problem here: After a hungup of Thunderbird 2.0.0.18 / Lightning 0.9 my ics on the remote server (apache 2.2.3 with module webdav) was gone. Maybe the following four logfile-lines (and especially the statuscodes and -in one case- missing username) may help: 


> [MY_IP] - [MY_USERNAME] [09/Dec/2008:11:48:18 +0100] "GET [MY.ICS] HTTP/1.1" 200 147907 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.8.1.18) Gecko/20081105 Lightning/0.9 Thunderbird/2.0.0.18"
> [MY_IP] - [MY_USERNAME] [09/Dec/2008:11:48:18 +0100] "PUT [MY.ICS] HTTP/1.1" 500 1146 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.8.1.18) Gecko/20081105 Lightning/0.9 Thunderbird/2.0.0.18"
> [MY_IP] - - [09/Dec/2008:14:50:37 +0100] "GET [MY.ICS] HTTP/1.1" 401 4587 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.8.1.18) Gecko/20081105 Lightning/0.9 Thunderbird/2.0.0.18"
> [MY_IP] - [MY_USERNAME] [09/Dec/2008:14:50:37 +0100] "GET [MY.ICS] HTTP/1.1" 404 842 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.8.1.18) Gecko/20081105 Lightning/0.9 Thunderbird/2.0.0.18"

The crash of Thunderbird has been at round about 14:45: I gave it 5 minutes until I killed the process since I already missed my server-ics at earlier days. 

Regards, 

Florian

Comment 9

[fklucke]

Sorry, I forgot the error.log-part: 

>[Tue Dec 09 11:53:19 2008] [error] [client MY_IP] Could not get next bucket brigade  [500, #0]
>[Tue Dec 09 14:50:37 2008] [error] [client MY_IP] File does not exist: [MY_ICS]

Regards, 

Florian

Comment 10

[fklucke]

Once again: Confirming an alert and opening an imap-folder on the same server as the ics resulted in a crash of Ligthning and a loss of the ics on the webdav. 

Logs: 

>[MY_IP] - [MY_USERNAME] [10/Dec/2008:12:38:28 +0100] "GET MYCAL.ics HTTP/1.1" 200 144466 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.8.1.18) Gecko/20081105 Lightning/0.9 Thunderbird/2.0.0.18"
>[MY_IP] - [MY_USERNAME] [10/Dec/2008:12:39:02 +0100] "GET MYCAL.ics HTTP/1.1" 200 147955 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.8.1.18) Gecko/20081105 Lightning/0.9 Thunderbird/2.0.0.18"
>[MY_IP] - [MY_USERNAME] [10/Dec/2008:12:39:03 +0100] "PUT MYCAL.ics HTTP/1.1" 500 0 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.8.1.18) Gecko/20081105 Lightning/0.9 Thunderbird/2.0.0.18"

and

>[Wed Dec 10 12:40:56 2008] [error] [client [MY_IP]] (104)Connection reset by peer: Could not get next bucket brigade  [500, #0]

Regards, 

Florian

Comment 11

[Lutz]

I can still confirm this:

/var/log/apache2/access.log:
>IP - USER [09/Mar/2009:10:46:34 +0100] "GET FILE.ics HTTP/1.1" 200 66619 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.19) Gecko/20081209 Lightning/0.9 Thunderbird/2.0.0.19"
>IP - USER [09/Mar/2009:10:46:35 +0100] "PUT FILE.ics HTTP/1.1" 500 763 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.19) Gecko/20081209 Lightning/0.9 Thunderbird/2.0.0.19"

/var/log/apache2/error.log:
>[Mon Mar 09 10:51:37 2009] [error] [client IP] Could not get next bucket brigade  [500, #0]
>[Mon Mar 09 11:04:05 2009] [error] [client IP] File does not exist: /PATH/FILE.ics

The server is running with Debian stable, all distribution packages. DAV-related Apache2 modules: dav_fs, dav, dav_lock. Configuration settings:

/etc/apache2/mods-enabled/dav_fs.conf:
>DAVLockDB /var/lock/apache2/DAVLock

/etc/apache2/sites-enabled/secure:
>  <Directory PATH>
>      AuthUserFile /dev/null
>      AuthBasicAuthoritative off 
>
>      <IfModule mod_auth_pgsql.c>
>          # ... [user authentication]
>      </IfModule>
>      Dav On
>      ForceType text/plain
>      DavMinTimeout 300
>      AllowOverride AuthConfig
>      
>      AuthType Basic
>      AuthName "internal calendar"
>      <Limit GET POST PUT>
>          require valid-user
>      </Limit>
>      
>      <Files .htacess>
>          order deny,allow
>          deny from all
>          allow from 127.0.0.1
>      </Files>
>  </Directory>
>

Interestingly enough, it is mainly one person who has this problem frequently, though I can reproduce the scenario of calendar loss when attempting to move an event from one calendar to another. To me, it seems that the calendar deletion is a local problem, as the PUT command sends fewer/no data, after having retrieved a valid file.

A personal complaint: The problem has been around for years, even before the initial report for this bug. In my team (~5 people), it occurs at an average of - optimistically speaking - 2-4 times a month. I have installed several backup mechanisms, so I can re-create the WebDAV files rather quickly from latest copies. Still, assuming it takes me about 10 minutes every time, makes about half an hour a month, makes 6 hours a year, only in dealing with this problem. Not counting the numerous hours I have tried to track down the error...

Comment 12

[Bas van den Bosch]

Googling for "Could not get next bucket brigade" seams to show it's either a client-side issue "connection reset by pear" or a server-side issue with mod-webdav. Either way, this problem doesn't look like Lightnings fault, which (in reply to comment 6) would explain why not many people are seeing this. At
http://www.archivum.info/[e-mail]/2006-11/msg00769.html
there seems to be a workaround, putting "EnabledSendfile Off" in the server-config. Also, make sure you have the latest mod-dav packages.

Comment 13

[Wayne Mery (:wsmwk)]

Pete, bvdbos, do you still see this issue?

The reporter writes "Since then I haven’t use Thunderbird with calendar, so I can’t share any more details. Maybe the ticket can be closed."

Comment 14

[Phoenix]

Resolved per whiteboard