784177 - "Host key verification failed." for Android builds on multiple build slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Daniel Holbert [:dholbert]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14537838&full=1&branch=mozilla-inbound
Android Armv6 mozilla-inbound build on 2012-08-20 14:26:08 PDT for push 559c5895bad6
builder: mozilla-inbound-android-armv6
slave: bld-centos6-hp-013

{
Executing: ['make', 'upload']
make[1]: Entering directory `/builds/slave/m-in-andrd-armv6/build/obj-firefox/mobile/android/installer'
mkdir -p `dirname "../../../dist//fennec-17.0a1.en-US.android-arm-armv6.checksums"`
CHECKSUM FILE START
[...SNIP...]
CHECKSUM FILE END
/builds/slave/m-in-andrd-armv6/build/obj-firefox/_virtualenv/bin/python /builds/slave/m-in-andrd-armv6/build/build/upload.py --base-path ../../../dist \
		"../../../dist/fennec-17.0a1.en-US.android-arm-armv6.apk"     "../../../dist/fennec-17.0a1.en-US.android-arm-armv6.tests.zip" "../../../dist/fennec-17.0a1.en-US.android-arm-armv6.crashreporter-symbols.zip"  "../../../dist//fennec-17.0a1.en-US.android-arm-armv6.txt"    ../../../dist/gecko-unsigned-unaligned.apk  ../../../dist/robocop.apk  ../../../dist/fennec_ids.txt \
		"../../../dist//fennec-17.0a1.en-US.android-arm-armv6.checksums"
Host key verification failed.
Command ['ssh', '-o', 'IdentityFile=~/.ssh/ffxbld_dsa', '[e-mail]', 'mktemp -d'] returned non-zero exit code: 255
}

Comment 1

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14538510&tree=Try
bld-centos6-hp-033

https://tbpl.mozilla.org/php/getParsedLog.php?id=14538094&tree=Try
bld-centos6-hp-031

Comment 2

[Daniel Holbert [:dholbert]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14538410&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=14538503&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=14538462&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=14538776&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=14538975&tree=Mozilla-Inbound

This affects all non-XUL Android builds on the latest two mozilla-inbound pushes:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=559c5895bad6
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=70d0984bf533

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

This is fallout from bug 778303, which removed ~cltbld/.ssh/known_hosts and added /etc/ssh/ssh_known_hosts. The latter is not made available to the mock environment where make upload is called from.

Comment 4

[Phil Ringnalda (:philor)]

Closed m-c for this around 15:10, m-i was already closed but this overlaps on owning that closure too.

Comment 5

[Aki Sasaki (not active)]

Backed out bug 778303.
Puppet should be auto-updated shortly; slaves that reboot after that point will pick up the latest changes and should have the correct ssh settings.

Looking into ways to speed that process up.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Confirmed that fixes up bld-centos6-hp-013, provided you redo the copyin steps and chown steps in a clean mock environment.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

We've rebooted all the idle slaves, as of about 4:20 Pacific, so jobs starting after then are good. Very likely there will be continuing failures from jobs that started prior to that.

We have a green build for 'Android mozilla-inbound build' at 
https://tbpl.mozilla.org/php/getParsedLog.php?id=14540981&tree=Mozilla-Inbound

Comment 8

[Phil Ringnalda (:philor)]

Trees reopened at 16:53