470283 - AUS sending connection timeouts

Status: RESOLVED WORKSFORME

(mozilla.org Graveyard :: Server Operations, task)

Description

[Samuel Sidler (old account; do not CC)]

Some users trying to update to Firefox 2.0.0.19 were seeing "connection timed out" errors in the update window for AUS.

Now, Juan is seeing the same thing when testing Thunderbird 2.0.0.18 -> 2.0.0.19 on the betatest channel.

This is the URL that is erroring out for Thunderbird: https://aus2.mozilla.org/update/3/Thunderbird/2.0.0.18/2008110519/WINNT_x86-msvc/en-US/betatest/Windows_NT%206.0/default/default/update.xml?force=1

The problem seems to be intermittent (it works for Al, but not for Juan), but it's definitely affecting releasing. We're holding back on the Thunderbird beta for the moment to get this problem looked at.

Comment 1

[Aravind Gottipati [:aravind]]

I am unable to reproduce it, but I will continue looking

Comment 2

[Jeremy Orem [:oremj]]

Is aus2 timing out or stage-old.mozilla.org?

Comment 3

[juan becerra [:juanb]]

We get the AUS error dialog regardless of whether there is an update available or not. For example, I checked for updates on the latest version of Thunderbird, which should not have any updates, but I saw the error. So it seems that AUS2 is timing out.

Comment 4

[Jeremy Orem [:oremj]]

We are unable to reproduce.  Juan are you getting the error every time?  Can you get us a tcpdump (tcpdump -e your_interface -s 0 -w aus2_error) of the problem occuring and a traceroute to aus2.mozilla.org?

Comment 5

[Jeremy Orem [:oremj]]

that should be tcpdump -i not -e

Comment 6

[juan becerra [:juanb]]

Attachment: output filtered for tcp using wireshark

While testing Thunderbird updates in betatest, on a Vista machine.

Comment 7

[Jeremy Orem [:oremj]]

You got a timeout on attachment 353712 [details]?  It looks pretty similar to my request:

  1   0.000000  10.2.74.124 -> 63.245.209.49 TCP 55699 > https [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=3161445787 TSER=0 WS=7
  2   0.000685 63.245.209.49 -> 10.2.74.124  TCP https > 55699 [SYN, ACK] Seq=0 Ack=1 Win=8190 Len=0 MSS=1380
  3   0.000711  10.2.74.124 -> 63.245.209.49 TCP 55699 > https [ACK] Seq=1 Ack=1 Win=5840 Len=0
  4   0.035764  10.2.74.124 -> 63.245.209.49 SSLv2 Client Hello
  5   0.036050 63.245.209.49 -> 10.2.74.124  TLSv1 Server Hello, Certificate, Server Hello Done
  6   0.036075  10.2.74.124 -> 63.245.209.49 TCP 55699 > https [ACK] Seq=118 Ack=836 Win=6680 Len=0
  7   0.037348  10.2.74.124 -> 63.245.209.49 TLSv1 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
  8   0.037585 63.245.209.49 -> 10.2.74.124  TCP https > 55699 [ACK] Seq=836 Ack=300 Win=40659 Len=0
  9   0.038341 63.245.209.49 -> 10.2.74.124  TLSv1 Change Cipher Spec, Encrypted Handshake Message
 10   0.038509  10.2.74.124 -> 63.245.209.49 TLSv1 Application Data
 11   0.038776 63.245.209.49 -> 10.2.74.124  TCP https > 55699 [ACK] Seq=879 Ack=601 Win=40358 Len=0
 12   0.039195 63.245.209.49 -> 10.2.74.124  TLSv1 Application Data
 13   0.039400  10.2.74.124 -> 63.245.209.49 TLSv1 Encrypted Alert
 14   0.039694 63.245.209.49 -> 10.2.74.124  TCP https > 55699 [ACK] Seq=1964 Ack=624 Win=40335 Len=0
 15   0.040586  10.2.74.124 -> 63.245.209.49 TCP 55699 > https [FIN, ACK] Seq=624 Ack=1964 Win=8680 Len=0
 16   0.040836 63.245.209.49 -> 10.2.74.124  TCP https > 55699 [FIN, ACK] Seq=1964 Ack=625 Win=40334 Len=0
 17   0.040855  10.2.74.124 -> 63.245.209.49 TCP 55699 > https [ACK] Seq=625 Ack=1965 Win=8680 Len=0

Comment 8

[juan becerra [:juanb]]

On my Vista vm I don't see this problem, and after removing and reinstalling Thunderbird on the hardware where I saw this problem, I am now getting updates pretty much every time...

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

I'm also seeing delays or timeouts on requests
 http://download.mozilla.org/?product=firefox-2.0.0.20-complete&os=linux&lang=de

Don't know if that's related but a big deal for Fx 2.0.0.20. Is the cluster of app servers OK ?

Comment 10

[Jeremy Orem [:oremj]]

(In reply to comment #9)
> I'm also seeing delays or timeouts on requests
> 
> http://download.mozilla.org/?product=firefox-2.0.0.20-complete&os=linux&lang=de
> 
> Don't know if that's related but a big deal for Fx 2.0.0.20. Is the cluster of
> app servers OK ?

Are you seeing delays getting the 302 or delays once you get the 302?

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #10)
> Are you seeing delays getting the 302 or delays once you get the 302?

Getting the 302.

Since IT can't reproduce this, nor can host-tracker.com except for a few responses taking a 2-3 seconds, are we going to WFM ?

Comment 12

[Jeremy Orem [:oremj]]

Let me know if you start hearing more complaints and can get some tcpdumps of the problem occurring or a way to reproduce.