1162148 - Warning: Identity file /builds/.ssh/{ffxbld_rsa,tbirdbld_dsa} not accessible: No such file or directory.

Status: RESOLVED DUPLICATE of bug 1173459

(Release Engineering :: Release Automation, defect)

Description

[Philipp Kewisch [:Fallen][📱📆]]

I found this during the Thunderbird l10n repacks.

latestmar=$(ssh -l tbirdbld -i ~/.ssh/tbirdbld_dsa stage.mozilla.org 'ls -1t  | grep linux-x86_64.complete.mar$ | head -n 1'); \
  if test -n "$latestmar"; then \
    wget -O /builds/slave/tb-rel-c-beta-l64_rpk_1-000000/comm-beta/obj-l10n/dist/update/linux-x86_64/ar//previous.mar http://stage.mozilla.org//$latestmar && \
    (cd previous; \
      MAR=/builds/slave/tb-rel-c-beta-l64_rpk_1-000000/comm-beta/obj-l10n/dist/host/bin/mar perl /builds/slave/tb-rel-c-beta-l64_rpk_1-000000/comm-beta/mozilla/tools/update-packaging/unwrap_full_update.pl '..//builds/slave/tb-rel-c-beta-l64_rpk_1-000000/comm-beta/obj-l10n/dist/update/linux-x86_64/ar//previous.mar') && \
    (cd current; \
      MAR=/builds/slave/tb-rel-c-beta-l64_rpk_1-000000/comm-beta/obj-l10n/dist/host/bin/mar perl /builds/slave/tb-rel-c-beta-l64_rpk_1-000000/comm-beta/mozilla/tools/update-packaging/unwrap_full_update.pl '..//builds/slave/tb-rel-c-beta-l64_rpk_1-000000/comm-beta/obj-l10n/dist/update/linux-x86_64/ar/thunderbird-38.0b4.complete.mar') && \
    find current -name \*.pgc -print -delete && \
    find previous -name \*.pgc -print -delete && \
    rm -f /builds/slave/tb-rel-c-beta-l64_rpk_1-000000/comm-beta/obj-l10n/dist/update/linux-x86_64/ar//*.partial.*.mar && \
    SRC_BUILD_ID=$(python /builds/slave/tb-rel-c-beta-l64_rpk_1-000000/comm-beta/mozilla/config/printconfigsetting.py $(find previous -maxdepth 4 -type f -name application.ini) App BuildID) \
    DST_BUILD_ID=$(cat ../../config/buildid) \
    SRC_BUILD=previous DST_BUILD=current \
      make partial-patch && \
    rm -f /builds/slave/tb-rel-c-beta-l64_rpk_1-000000/comm-beta/obj-l10n/dist/update/linux-x86_64/ar//previous.mar; \
  else \
    echo "No previous MAR found; not creating a partial MAR"; \
  fi
Warning: Identity file /builds/.ssh/tbirdbld_dsa not accessible: No such file or directory.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
No previous MAR found; not creating a partial MAR

http://ftp.mozilla.org/pub/mozilla.org/thunderbird/candidates/38.0b4-candidates/build5/logs/release-comm-beta-linux64_repack_1-bm71-build1-build5.txt.gz

Comment 1

[Philipp Kewisch [:Fallen][📱📆]]

rail, can you help out on this one? I'd like to figure out if this is an issue that needs to be fixed for Thunderbird 38 or not. If someone else is a better fit for needinfo please forward.

I see in the logs that there seem to be two paths referencing a .ssh directory, so I assume that either the home directory is set wrong for one of these scripts, or the key is not copied to both locations.

Comment 2

[Rail Aliiev [:rail]]

I'm not exactly sure why it happens and when started to fail. If the step runs in a mock environment, then we use /home/mock_mozilla/.ssh/... (see http://hg.mozilla.org/build/buildbot-configs/file/6b48f00ff479/mozilla/thunderbird_config.py#l274), otherwise it should be /home/cltbld/.ssh/. Looks like this step runs outside of mock.

I bet mgerva knows this code much better.

Comment 3

[Philipp Kewisch [:Fallen][📱📆]]

Looking at Firefox, it seems the same problem exists.

http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.0b4-candidates/build1/logs/release-mozilla-beta-linux64_repack_1-bm70-build1-build2.txt.gz

The variable it is taken from is UPLOAD_SSH_KEY. I haven't been able to figure out exactly where it is set to ~/.ssh unfortunately. mgerva, can you take a look?

Comment 4

[Massimo Gervasini [:massimo]]

We had a similar problem in l10n repacks on linux platforms (because of mock).

In l10n case, we had to set UPLOAD_SSH_KEY=/home/mock_mozilla/.ssh/<rsa_key>

I hope this helps!

Comment 5

[Philipp Kewisch [:Fallen][📱📆]]

It looks like you are still having this problem for Firefox, also happens for 38.0.5b1:

http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.0.5b1-candidates/build1/logs/release-mozilla-beta-linux64_repack_1-bm77-build1-build6.txt.gz

Are you suggesting to set this in mozilla/config.py in PLATFORM_VARS.*.env.UPLOAD_SSH_KEY ? If so, I am happy to spin up a patch that fixes both Firefox and Thunderbird.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

This doesn't actually cause any problems for beta or release builds, it's just the way we do updates for nightly/aurora leaking through into the release automation. Specifically, the code block is
  http://hg.mozilla.org/mozilla-central/file/66a95a483d2c/tools/update-packaging/Makefile.in#l81

So we could set 'MOZ_AUTOMATION_UPDATE_PACKAGING=0' in <app>/configs/mozconfigs/*/l10n-mozconfig, except that's used for the nightly/aurora case too. So we'd have to key that off some other variable or duplicate the mozconfigs.

Comment 7

[Philipp Kewisch [:Fallen][📱📆]]

Ok, as long as this doesn't cause any issues I'll just leave it open for now, I might get back to this when the our 38 release is out of the door. Thanks for the update!

Comment 8

[Rail Aliiev [:rail]]

Fallen, is there anything to be done here?

Comment 9

[Philipp Kewisch [:Fallen][📱📆]]

From what I've understood, the error message does not cause any issues. We could leave this open as an "enhancement" to take steps to remove the error message, either by setting MOZ_AUTOMATION_UPDATE_PACKAGING=0 in the l10n repacks or by extending the environment to set PLATFORM_VARS.*.env.UPLOAD_SSH_KEY. 

If this will go away in the next months due to moving to TC then it is probably not worth the effort.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

This goes away in 42 with http://hg.mozilla.org/releases/mozilla-aurora/rev/60302e9a0a6d.

Comment 11

[aleth [:aleth]]

As per comment 9, fixed for 42.