1008413 - Bouncer.prod and Mozilla.org production/staging jobs (run from within the QA lab) see frequent 500s/503s

Status: RESOLVED WORKSFORME

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: consoleText.txt

Jake tried to help us a bit, earlier today, with this, but we're still seeing it, and I want to make sure it's tracked -- for my team and the project (Mozilla.org).

tl;dr: our bouncer.prod and the mozilla.com/.org production and staging 

<jakem> Service Protection Class banned-download: Too many concurrent connections from 63.245.221.36, dropped - 2 time(s)
<jakem> at 09/May/2014:11:09:28 -0700

http://selenium.qa.mtv2.mozilla.com:8080/view/Bouncer/job/bouncer.prod/38758/console

Comment 1

[:Atoll]

Moving this to the webops queue for Jake.

Comment 2

[Jake Maul [:jakem]]

I added that IP to the allowed (un-throttled) list for bouncer, but not for mozilla.org... didn't realize you were getting similar problems for that, too.

I'll get that now. Will take a bit longer b/c it's in both datacenters.

Comment 3

[Jake Maul [:jakem]]

Done as of now. Let me know if it gets any better. Dropping prio to prevent paging.

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: mozilla-dot-org.png

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Jake Maul [:jakem] from comment #3)
> Done as of now. Let me know if it gets any better. Dropping prio to prevent
> paging.

Thx, Jake.  I think we can call this fixed!

Comment 6

[Robert Chira [:RobertC]]

Should this be fixed on bouncer.prod? We are seeing similar failuresin the Jenkins test runs.

http://selenium.qa.mtv2.mozilla.com:8080/view/All%20not%20B2G/job/bouncer.prod/39156/console

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

Jake, mind looking into the corresponding log entries for http://selenium.qa.mtv2.mozilla.com:8080/view/Bouncer/job/bouncer.prod/39186/console and friends?  Thanks!

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

Switching to :jabba; can you help us figure out why tests run from our VLAN are experiencing 500s/ISEs from Bouncer, while the ones run from https://ci.mozilla.org/job/bouncer.prod/ don't?  Thanks!

Comment 9

[Philippe M. Chiasson (:gozer)]

Looking at bouncer logs from around that time shows some connection failures to MySQL
L

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: bouncer-good.png

Hopefully I don't jinx us, but it seems to have been fixed between:

Success > Console Output  #39517 	May 14, 2014 12:43:28 PM	 
Failed > Console Output  #39516 	May 14, 2014 11:30:48 AM

Comment 11

[Stephen Donner [:stephend] Not actively reading bugmail]

Not sure if bug 1010383 or some other, related work, fixed this, so I'll mark it WFM now.