936248 - EC2 jobs failing due to lost network traffic

Status: RESOLVED DUPLICATE of bug 940356

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

See https://tbpl.mozilla.org/php/getParsedLog.php?id=30295185&tree=Fx-Team#error0 as an example

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Some of these were network timeouts, too.

This seems to be limited to use1, so I suspect this is the usual use1 connection problem.

Arzhel's seeing "%-RT_IPSEC_REPLAY: Replay packet detected on IPSec tunnel" suggesting that the Amazon end is re-transmitting packets and that both the original and re-transmitted packets are eventually arriving.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: nagios1-releng-use1~admin1b.private.releng.scl3.mozilla.com_last_10800.png

smokeping for the last 30h

Comment 3

[Chris Cooper [:coop] (he/him)]

Does this seem solvable? Do we still need to gather more data?

Should releng continue to setup in-house buildbot masters to minimize cross-colo traffic (bug 927129)?

Comment 4

[Armen [:armenzg]]

(In reply to Chris Cooper [:coop] from comment #3)
> Does this seem solvable? Do we still need to gather more data?
> 
> Should releng continue to setup in-house buildbot masters to minimize
> cross-colo traffic (bug 927129)?

I believe this was a different part of our system.

Ec2 host trying to read in-house host:
tst-linux32-ec2-134 -> http://pypi.pvt.build.mozilla.org

A better bug would be to setup a pypi host on EC2.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

That's true, but difficult since pypi is part of the releng web cluster and thus based on infra puppet and assuming the presence of a Zeus load balancer and a NetApp backend.  Following this line of thought, you'll need to move *everything* into AWS - clobberer, npm-mirror, ftp, hgmo, gitmo, ..

And maybe that is the right solution, but it's not a decision we should back into lightly or without some serious thought, as it will be a lot of work, and far *more* work if we don't approach it systematically.

Comment 6

[Chris Cooper [:coop] (he/him)]

Arzhel is debugging this in bug 940356.