1089747 - hitting timeouts while posting to graphserver and downloading from ftp

Status: RESOLVED WORKSFORME

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Jordan Lund (:jlund)]

e.g. https://treeherder.mozilla.org/ui/logviewer.html#?job_id=3325873&repo=mozilla-inbound

10:44:33 <jlund> http://netops2.private.scl3.mozilla.com/smokeping/sm.cgi?target=Datacenters.RELENG-SCL3.admin1-private-pek1 is not happy. not sure what that touches


10:48:43 <jlund> border1.phx1.mozilla.net @ 09:35 border1.sjc2.mozilla.net @ 08:40 core1.scl3.mozilla.net @ 10:10 nagios1.private.releng.usw1.mozilla.com @ 10:10 all had ~4% packet loss over last 3 hours. times are PDT

tracking in this bug in case there are further drops.

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I'm seeing several mochitest failures like https://treeherder.mozilla.org/ui/logviewer.html#?job_id=1008676&repo=fx-team
In the log, there's lines like 
13:44:13 INFO - [2233] WARNING: failed to bind socket: file /builds/slave/fx-team-lx-d-00000000000000000/build/netwerk/base/src/nsServerSocket.cpp, line 364
13:44:13 INFO - !!! could not start server on port 8888: [Exception... "Component returned failure code: 0x804b0036 (NS_ERROR_SOCKET_ADDRESS_IN_USE) [nsIServerSocket.init]" nsresult: "0x804b0036 (NS_ERROR_SOCKET_ADDRESS_IN_USE)" location: "JS frame :: /builds/slave/test/build/tests/bin/components/httpd.js :: nsHttpServer.prototype._start :: line 550" data: no]
13:44:13 INFO - JavaScript error: , line 0: uncaught exception: 2147746065


Would this be related to the timeout issues here?

Comment 2

[:kanban-engops]

A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/81854206

Comment 3

[:kanban-engops]

A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/82064752

Comment 4

[Pete Moore [:pmoore][:pete]]

Hi Jordan,

Have there been any more problems? Should we write this off as a one-off, or should we do more investigation?

Thanks,
Pete

Comment 5

[Jordan Lund (:jlund)]

(In reply to Pete Moore [:pete][:pmoore] from comment #4)
> Hi Jordan,
> 
> Have there been any more problems? Should we write this off as a one-off, or
> should we do more investigation?
> 
> Thanks,
> Pete

thanks for housecleaning. one off makes sense to me