979505 - make upload step failing on a few slaves

Status: RESOLVED WORKSFORME

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Jordan Lund (:jlund)]

around 11 PT today a few slaves failed the make upload step. Specifically around trying to ssh into stage.m.o:
Command ['ssh', '-o', 'IdentityFile=~/.ssh/ffxbld_dsa', '[e-mail]', 'mktemp -d'] returned non-zero exit code: 255

log example:
    - https://tbpl.mozilla.org/php/getParsedLog.php?id=35607636&tree=Mozilla-Inbound
    - https://tbpl.mozilla.org/php/getParsedLog.php?id=35607498&tree=Mozilla-Inbound

slaves involved:
    -  b2g_mozilla-inbound_macosx64_gecko build bld-linux64-spot-127
    -  b2g_mozilla-inbound_macosx64_gecko build bld-lion-r5-045

masters involved:
    - bm84
    - bm62

Seeing that these were close to each other and that there was some packet loss around that same time: in last 3 hours: ~1% avg ~28% peak:
    - http://netops2.private.scl3.mozilla.com/smokeping/sm.cgi?target=Datacenters.RELENG-SCL3.nagios1-releng-use1

chances are this is a network infra issue. In case this happens again, I'll keep this open for today.

Comment 1

[Jordan Lund (:jlund)]

bld-lion-r5-045 successfully ran 'make upload' just prior to the failure via a similar builder: b2g_mozilla-b2g28_v1_3_macosx64_gecko build

increases the chance this was a network blip

Comment 2

[John Hopkins (:jhopkins)]

RyanVM has not noticed any recent make upload failures so I'm going to close this bug.

Comment 3

[[e-mail] (:bhearsum)]

"ssh_exchange_identification: Connection closed by remote host" isn't usually indicative of a problem with the link AFAIK. Those usually result in timeouts. The logs you linked only have the upload step running for 30-40 seconds. It sounds like the server rejected them for whatever reason. Might be worth asking IT for server side logs next time this happens. Maybe we've hit maximum number of connections (per user?)?