1091707 - Frequent FTP/proxxy timeouts across all trees

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ryan VanderMeulen [:RyanVM]]

https://treeherder.mozilla.org/ui/logviewer.html#?job_id=561077&repo=mozilla-central
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=734953&repo=b2g-inbound

Etc. All trees closed.

Comment 1

[Chris Cooper [:coop] (he/him)]

I see an obvious network blip this morning at 9:50am PT, which corresponds well to this bug. It seems to have affected just about everything:

use1:

http://netops2.private.scl3.mozilla.com/smokeping/sm.cgi?displaymode=n;start=2014-10-30%2007:32;end=now;target=Datacenters.RELENG-SCL3.nagios1-releng-use1~admin1b.private.releng.scl3.mozilla.com

scl3:

http://netops2.private.scl3.mozilla.com/smokeping/sm.cgi?displaymode=n;start=2014-10-30%2007:30;end=now;target=Datacenters.RELENG-SCL3.fw1-releng-scl3~admin1b.private.releng.scl3.mozilla.com

That same graph shows another similar blip at 7:50am PT. I think it's safe to re-open the trees at this point, but I'll be watching the graphs at 11:50am PT to see whether we have a repeat performance.

Comment 2

[Chris Cooper [:coop] (he/him)]

Here's a better look at the ubiquity:

http://netops2.private.scl3.mozilla.com/smokeping/sm.cgi?target=Datacenters.RELENG-SCL3

I've pinged netops.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Things seem to be in better shape now. Trees reopened.

Comment 4

[:kanban-engops]

A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/81855346

Comment 5

[:kanban-engops]

A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/82064742