783518 - Native Android Nightlies failing in uploadsymbols with ssh error: "Host key verification failed."

Status: VERIFIED FIXED

(Release Engineering :: General, defect, P1)

Description

[Ed Morley [:emorley]]

Bug 775620 fallout?

https://tbpl.mozilla.org/php/getParsedLog.php?id=14463602&tree=Firefox#error0
https://tbpl.mozilla.org/php/getParsedLog.php?id=14463897&tree=Firefox#error0

{
INFO: mock_mozilla.py version 1.0.3 starting...
State Changed: init plugins
INFO: selinux disabled
State Changed: start
State Changed: lock buildroot
State Changed: shell
/bin/sh /builds/slave/m-cen-andrd-ntly/build/toolkit/crashreporter/tools/upload_symbols.sh fennec-17.0a1-Android-20120817030555-arm-mozilla-central-symbols.txt "./dist/fennec-17.0a1.en-US.android-arm.crashreporter-symbols-full.zip"
Transferring symbols... ./dist/fennec-17.0a1.en-US.android-arm.crashreporter-symbols-full.zip
debug1: Reading configuration data /home/mock_mozilla/.ssh/config
debug1: Applying options for *mozilla.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to symbols1.dmz.phx1.mozilla.com [10.8.74.48] port 22.
debug1: Connection established.
debug1: identity file /home/mock_mozilla/.ssh/ffxbld_dsa type 2
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
Host key verification failed.
lost connection
make: *** [uploadsymbols] Error 1
State Changed: unlock buildroot
program finished with exit code 2
}

Comment 1

[Bill Gianopoulos [:WG9s]]

Seems to be mozilla-central only.  Android builds worked fine.

Comment 2

[Bill Gianopoulos [:WG9s]]

(In reply to Bill Gianopoulos [:WG9s] from comment #1)
> Seems to be mozilla-central only.  Android builds worked fine.
                                     ^^^^^^^
                                     Aurora

Comment 3

[Ed Morley [:emorley]]

Also fx-team:

https://tbpl.mozilla.org/?tree=Fx-Team&rev=78ad8bb8b030
https://tbpl.mozilla.org/php/getParsedLog.php?id=14465552&tree=Fx-Team
https://tbpl.mozilla.org/php/getParsedLog.php?id=14466018&tree=Fx-Team

Comment 4

[Chris AtLee [:catlee]]

Fallout from switching to new slaves. I've pushed out a puppet patch that will update known_hosts on the machines. Give it a few hours to make it onto all the machines, and then we can try again.

Comment 5

[Ed Morley [:emorley]]

Thank you :-)

Comment 6

[Phil Ringnalda (:philor)]

But nobody actually did try again.

https://tbpl.mozilla.org/php/getParsedLog.php?id=14493067&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=14492785&tree=Firefox

Warning: Identity file /home/cltbld/.ssh/auspush not accessible: No such file or directory.

Comment 7

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14506384&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=14506400&tree=Firefox

Comment 8

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14522839&tree=Firefox

Comment 9

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14523190&tree=Firefox

Comment 10

[Ed Morley [:emorley]]

This is still occurring - meaning we've had no Nightly coverage since 2012-08-16, CCing a few mobile people to make sure they are aware.

Comment 12

[Daniel Holbert [:dholbert]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14537838&tree=Mozilla-Inbound

Comment 13

[Daniel Holbert [:dholbert]]

sorry, Comment 12 was just for a normal tbpl per-push build, not for a nightly build. (only just noticed that this bug was nightly-specific)

Comment 14

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14556974&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=14556948&tree=Firefox

Comment 15

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14565410&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=14565475&tree=Firefox

Comment 16

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14569331&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=14568944&tree=Firefox

Comment 17

[Justin Wood (:Callek)]

:rail just deployed this key to aws slaves, I just deployed it to the hp slaves, I'm going to retrigger the Nightly build on [green] m-c right now to help verify.

Comment 18

[Rail Aliiev [:rail]]

Attachment: aus3-staging

Comment 19

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14574112&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=14573051&tree=Firefox

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

While we work through the issues with ssh, I just want to point out that the latest nightly builds are available via ftp.m.o and nightly.m.o. It's the update system that's not getting the new builds at the moment.

Comment 21

[Bill Gianopoulos [:WG9s]]

(In reply to Nick Thomas [:nthomas] from comment #20)
> While we work through the issues with ssh, I just want to point out that the
> latest nightly builds are available via ftp.m.o and nightly.m.o. It's the
> update system that's not getting the new builds at the moment.

I posted a message to that effect to the nightly builds thread to try to get more people testing the newer builds.

Comment 22

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14592917&tree=Firefox

Comment 23

[Ed Morley [:emorley]]

Retriggered the Nightly after bug 784521 was resolved, has succeeded:
https://tbpl.mozilla.org/php/getParsedLog.php?id=14595754&tree=Firefox

...and update is visible on phone.

Comment 24

[Rail Aliiev [:rail]]

Wooot!

Thanks everybody for helping debugging and fixing this!

Comment 25

[Kevin Brosnan [Ex-Mozilla]]

Updated Fennec Nightly from Aug-16 to today's nightly this morning using the update mechanism.