471456 - Investigate errors and disconnections from users in live chat

Status: RESOLVED FIXED

(support.mozilla.org Graveyard :: Chat, defect, P1)

Description

[Matthew Middleton (:zzxc)]

Many users in live chat are being disconnected early, either with a generic "disconnected" error or with a Java uncaught exception error page.  We need to figure out why, see if any JSP errors are being logged, and fix the bug(s).

Comment 1

[Tom Ellins]

Can I suggest just for now we put a messege on the live chat page to let users know we are having a few problems and they may be disconected and we are working on fixing this?

Tom

Comment 2

[David Tenser [:djst]]

Laura, do you know who is the best person to ask for Java server error logs or similar that can give us a better clue?

Tom: as far as I understand, we don't really know how many users are affected today. Before we know that, I'd rather not acknowledge a problem and say we're working on it.

Comment 3

[Laura Thomson :laura]

Not sure, but ccing server-ops/mrz to find out who can help us with logs.

Comment 4

[Matthew Middleton (:zzxc)]

Errors from the webchat client should be logged in webchat-error.log.  This
should be in the "logs" directory in /opt/openfire.  This file can't be
accessed through the admin panel.

Comment 5

[Laura Thomson :laura]

Matthew, should we open another ticket to add that functionality?  Seems kind of useful.

Comment 6

[Matthew Middleton (:zzxc)]

(In reply to comment #5)
> Matthew, should we open another ticket to add that functionality?  Seems kind
> of useful.

This file should probably be exposed in the fastpath admin panel, that can be another bug.  For now, the quickest way would be for someone to grab this file from production and attach it to this bug.

Comment 7

[Jeremy Orem [:oremj]]

Attachment: openfire error logs

Comment 8

[Matthew Middleton (:zzxc)]

The file attached is the error.log, not the webchat-error.log.  The contents of /opt/openfire/logs should be:

zzxc@zzxc:~/sumo/webchat/openfire/target/openfire/logs$ ls
debug.log  error.log  info.log  stats.txt  warn.log  webchat-error.log  webchat-error.log.1  webchat-error.log.1.lck  webchat-error.log.lck

The errors in webchat-error.log are specific to the webchat plugin.

Comment 9

[Tom Ellins]

Hi 

just to let you no I have had 3 disconects tonight so far

Comment 10

[Jeremy Orem [:oremj]]

Attachment: webchat error logs

Comment 11

[Matthew Middleton (:zzxc)]

There are a number of uncaught exceptions in the webchat log, but it's not clear what is causing all of them, or if they correspond to all disconnections.  To get more information, I enabled logging of every XMPP packet.  These are saved in /opt/openfire/logs/jive.audit*.log; these files may be quite large.

To compare this data, I need a tarball of everything in /opt/openfire/logs.  (This could be uploaded to my account on khan, or to sm-chat01)

Comment 12

[Matthew Middleton (:zzxc)]

Is there an update on getting the rest of the server log data from comment 11?  I still haven't been able to reproduce this on staging, so I need to look at the full production logs to try to see what is happening when users are disconnected.

Comment 13

[Laura Thomson :laura]

Hey Jeremy, assigning to you so we can get these logs ASAP.

Comment 14

[Jeremy Orem [:oremj]]

Logs @ khan:/home/zzxc/openfire_logs.tar.gz

Comment 15

[Laura Thomson :laura]

We can tag 0.8.1 without this because it's in a separate codebase, but it would be good to have it resolved soon.

Comment 17

[Matthew Middleton (:zzxc)]

Attachment: Disable caching and force Connection: close (Checked in)

When investigating more of these reports last week, I found that some users are disconnecting every time they use live chat until they clear their browser cache.  I got one user to paste all the errors in the Error Console after being disconnected, but there were no javascript errors from Fastpath.  This leads me to think that servlet might be serving empty or corrupted files which are then cached.

Based on the primary symptom - no server messages being delivered - it seems that the ajax library (DWR) is not functioning.  A portion of DWR javascript (eg. https://chat-support.mozilla.com:9091/webchat/dwr/engine.js) being served empty, for example, could explain this behavior.

So, to prevent this type of problem, this patch forces Connection: close on all HTTP requests to /webchat/*, as well as disables all caching in that folder.  This will hopefully solve any blank page errors users are experiencing, and in any case will prevent such errors from being cached.

Comment 18

[Matthew Middleton (:zzxc)]

Comment on attachment 362647 [details] [diff] [review]
Disable caching and force Connection: close (Checked in)

Checked in r23001

Comment 19

[David Tenser [:djst]]

Why isn't this marked as fixed?

Comment 20

[Matthew Middleton (:zzxc)]

->fixed

I'll file followup bugs for any more work that needs to be done here.