Closed Bug 1783883 Opened 3 years ago Closed 3 years ago

Fix ssh access for git operations on the firefox-android monorepo

Categories

(Release Engineering :: General, task)

Product:
Release Engineering
Component:
General
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jlorenzo, Assigned: gbrown)

References

Details

run-task is in charge of cloning repositories. In bug 1783690, we want to clone a (temporarily) private Github repo. For an unknown reason, the task doesn't get presented the allowlisted RSA key[1]. Instead, the task gets the ECDSA one[2]:

[vcs 2022-08-09T13:02:19.386Z] executing ['git', 'clone', 'git@github.com:$REPO.git', '/builds/worker/checkouts/vcs']
[vcs 2022-08-09T13:02:19.388Z] Cloning into '/builds/worker/checkouts/vcs'...
The authenticity of host 'github.com (192.30.255.113)' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
Are you sure you want to continue connecting (yes/no)? 
[taskcluster:error] Task timeout after 600 seconds. Force killing container.

On my machine, I even get the ed25519 one. RSA, ECDSA, and ed25519 are the only keys documented on Github's website[3]:

These are GitHub's public key fingerprints:

  • SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8 (RSA)
  • SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM (ECDSA)
  • SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU (Ed25519)

Thus, it's should a matter of updating the allowlisted key to support them all. I managed to get the public and the fingerprints this way:

ssh-keyscan -t rsa,ecdsa,ed25519 github.com > github_known_hosts
ssh-keygen -l -f github_known_hosts

They matched the documented ones.

[1] https://github.com/taskcluster/taskgraph/blob/ec733aa2c331390a0af079fa4a71e47c4622468f/src/taskgraph/run-task/run-task#L51-L58
[2] https://firefox-ci-tc.services.mozilla.com/tasks/ZfAS8kNzT_SSl6xfhKRObg/runs/1/logs/public/logs/live.log
[3] https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints

Blocks: 1782736
Assignee: nobody → gbrown

I tried updating the GITHUB_SSH_FINGERPRINT at [1], but that didn't have any effect: The hosts file was not created because I was not entering the block at [2] -- because the ssh-secret-name option was not set. The ssh-secret-name option derives from env variable <project>_SSH_SECRET_NAME [3].

I wasn't setting MOBILE_SSH_SECRET_NAME (name of the secret that provides the ssh private key) locally because it did not appear in the task definition [4]. But is that just the taskcluster UI suppressing secret references?? Now I see [5]

[1] https://github.com/taskcluster/taskgraph/blob/ec733aa2c331390a0af079fa4a71e47c4622468f/src/taskgraph/run-task/run-task#L51-L58
[2] https://github.com/taskcluster/taskgraph/blob/ec733aa2c331390a0af079fa4a71e47c4622468f/src/taskgraph/run-task/run-task#L876
[3] https://github.com/taskcluster/taskgraph/blob/ec733aa2c331390a0af079fa4a71e47c4622468f/src/taskgraph/run-task/run-task#L825
[4] https://firefox-ci-tc.services.mozilla.com/tasks/ZfAS8kNzT_SSl6xfhKRObg/definition
[5] https://github.com/mozilla-mobile/firefox-android/commit/2db89572554cd312a4b0299c53aafc85ae8fa039#diff-a728f7e52d751b98eafa856e45594533339b44f229d7b83f930df335391e7e15R212

How strongly do we feel about using ssh? It looks like many mozilla-releng and mozilla-mobile projects use https; in fact, I haven't found any other project using ssh. Do we really want firefox-android to be an outlier in this way?

(In reply to Geoff Brown [:gbrown] from comment #2)

How strongly do we feel about using ssh? It looks like many mozilla-releng and mozilla-mobile projects use https; in fact, I haven't found any other project using ssh. Do we really want firefox-android to be an outlier in this way?

Aiui https will not work if it's a private repo (unless we hit github's API rather than its repo url -- we don't have any support for this).

(In reply to Aki Sasaki [:aki] (he/him) (UTC-6) from comment #3)

I see. Thanks. I'll concentrate on getting the ssh case working then.

If I set MOBILE_SSH_SECRET_NAME and run run-task locally (with a small hack to get the secret data), I can successfully clone and fetch, even without changing GITHUB_SSH_FINGERPRINT.

setup 2022-08-10T21:59:51.419Z] run-task started in /home/gbrown/taskgraph/src/taskgraph/run-task
[vcs 2022-08-10T21:59:51.420Z] WARNING: vcs checkout path (/home/gbrown/taskgraph/src/taskgraph/run-task/checkouts) not in cache or volume; performance will likely suffer
[vcs 2022-08-10T21:59:51.420Z] fetching secret project/mobile/firefox-android/github-clone-ssh from https://firefox-ci-tc.services.mozilla.com/secrets/v1/secret/project/mobile/firefox-android/github-clone-ssh
[vcs 2022-08-10T21:59:56.741Z] executing ['git', 'clone', 'git@github.com:mozilla-mobile/firefox-android.git', '/home/gbrown/taskgraph/src/taskgraph/run-task/checkouts']
[vcs 2022-08-10T21:59:56.748Z] Cloning into '/home/gbrown/taskgraph/src/taskgraph/run-task/checkouts'...
[vcs 2022-08-10T21:59:57.076Z] Warning: Permanently added the RSA host key for IP address '140.82.114.3' to the list of known hosts.
[vcs 2022-08-10T22:00:58.554Z] executing ['git', 'fetch', '--tags', 'git@github.com:mozilla-mobile/firefox-android.git', 'refs/heads/bug-1783690']
[vcs 2022-08-10T22:00:58.892Z] Warning: Permanently added the RSA host key for IP address '140.82.112.4' to the list of known hosts.
[vcs 2022-08-10T22:00:59.551Z] From github.com:mozilla-mobile/firefox-android
[vcs 2022-08-10T22:00:59.551Z]  * branch                  bug-1783690 -> FETCH_HEAD
...

I note some differences compared to https://firefox-ci-tc.services.mozilla.com/tasks/ZfAS8kNzT_SSl6xfhKRObg/runs/1/logs/public/logs/live.log:

...
[setup 2022-08-09T13:02:19.384Z] run-task started in /builds/worker
[cache 2022-08-09T13:02:19.385Z] cache /builds/worker/checkouts is empty; writing requirements: gid=1000 uid=1000 version=1
[volume 2022-08-09T13:02:19.386Z] changing ownership of volume /builds/worker/.cache to 1000:1000
[volume 2022-08-09T13:02:19.386Z] volume /builds/worker/checkouts is a cache
[setup 2022-08-09T13:02:19.386Z] running as worker:worker
[vcs 2022-08-09T13:02:19.386Z] executing ['git', 'clone', 'git@github.com:mozilla-mobile/firefox-android.git', '/builds/worker/checkouts/vcs']
[vcs 2022-08-09T13:02:19.388Z] Cloning into '/builds/worker/checkouts/vcs'...
The authenticity of host 'github.com (192.30.255.113)' can't be established.
ECDSA key fingerprint is SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
Are you sure you want to continue connecting (yes/no)?
  • missing "fetching secret project/mobile/firefox-android/github-clone-ssh from ..."
  • missing "Warning: Permanently added the RSA host key ..."
    ...as though MOBILE_SSH_SECRET_NAME is not defined in the environment.

Johan thought the github branches might be out of sync; he force pushed, then commited https://github.com/mozilla-mobile/firefox-android/commit/d644793757500daf0364dd3fbdb7a22918b97119 -- and it worked!

https://firefox-ci-tc.services.mozilla.com/tasks/MAOVMUfaQgC6th48MrVLaA/definition does have MOBILE_SSH_SECRET_NAME defined, and https://firefox-ci-tc.services.mozilla.com/tasks/MAOVMUfaQgC6th48MrVLaA/runs/0/logs/public/logs/live.log shows the ssh secret being retrieved, the known_hosts being set up and the fetch succeeding:

[taskcluster 2022-08-11 16:17:55.349Z] === Task Starting ===
[setup 2022-08-11T16:17:55.746Z] run-task started in /
[vcs 2022-08-11T16:17:55.748Z] WARNING: vcs checkout path (/builds/worker/checkouts/vcs) not in cache or volume; performance will likely suffer
[setup 2022-08-11T16:17:55.748Z] running as worker:worker
[vcs 2022-08-11T16:17:55.749Z] fetching secret project/mobile/firefox-android/github-clone-ssh from http://taskcluster/secrets/v1/secret/project/mobile/firefox-android/github-clone-ssh
[vcs 2022-08-11T16:17:55.851Z] executing ['git', 'clone', 'git@github.com:mozilla-mobile/firefox-android.git', '/builds/worker/checkouts/vcs']
[vcs 2022-08-11T16:17:55.853Z] Cloning into '/builds/worker/checkouts/vcs'...
[vcs 2022-08-11T16:17:55.913Z] Warning: Permanently added the RSA host key for IP address '140.82.112.4' to the list of known hosts.
[vcs 2022-08-11T16:18:02.476Z] executing ['git', 'fetch', '--tags', 'git@github.com:mozilla-mobile/firefox-android.git', 'refs/heads/bug-1783690']
[vcs 2022-08-11T16:18:02.747Z] From github.com:mozilla-mobile/firefox-android
[vcs 2022-08-11T16:18:02.747Z]  * branch                  bug-1783690 -> FETCH_HEAD
[vcs 2022-08-11T16:18:02.796Z] executing ['git', 'checkout', '-f', 'd644793757500daf0364dd3fbdb7a22918b97119']
[vcs 2022-08-11T16:18:02.803Z] Note: checking out 'd644793757500daf0364dd3fbdb7a22918b97119'.
[vcs 2022-08-11T16:18:02.803Z] 
[vcs 2022-08-11T16:18:02.803Z] You are in 'detached HEAD' state. You can look around, make experimental
[vcs 2022-08-11T16:18:02.803Z] changes and commit them, and you can discard any commits you make in this
[vcs 2022-08-11T16:18:02.803Z] state without impacting any branches by performing another checkout.
[vcs 2022-08-11T16:18:02.804Z] 
[vcs 2022-08-11T16:18:02.804Z] If you want to create a new branch to retain commits you create, you may
[vcs 2022-08-11T16:18:02.804Z] do so (now or later) by using -b with the checkout command again. Example:
[vcs 2022-08-11T16:18:02.804Z] 
[vcs 2022-08-11T16:18:02.804Z]   git checkout -b <new-branch-name>
[vcs 2022-08-11T16:18:02.804Z] 
[vcs 2022-08-11T16:18:02.804Z] HEAD is now at d644793757 Bug 1783690 - part 6: Use firebase secret under the firefox-android namespace
[vcs 2022-08-11T16:18:02.804Z] executing ['git', 'clean', '-xdff']

I've been chasing ghosts.

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → WORKSFORME

Wait - we think the problem only manifest for non-decision tasks. .taskgraph.yml sets MOBILE_SSH_SECRET_NAME for the decision task only ... but we need to have that set for all tasks.

Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---

(In reply to Aki Sasaki [:aki] (he/him) (UTC-6) from comment #3)

Aiui https will not work if it's a private repo (unless we hit github's API rather than its repo url -- we don't have any support for this).

+1!

(In reply to Geoff Brown [:gbrown] from comment #8)

I noticed https://github.com/taskcluster/taskgraph/blob/main/src/taskgraph/transforms/job/common.py#L186 -- almost useful?

TIL! I think you found the right track! repo_configs[1] reads data from taskcluster/ci/config.yml[2]. So if I understand correctly, it should be a matter of changing this block[3] into:

taskgraph:
    repositories:
        mobile:
            name: firefox-android
            ssh-secret-name: project/mobile/firefox-android/github-clone-ssh

The good news is: we have an example on the VPN right there[4]

[1] https://github.com/taskcluster/taskgraph/blob/3a4b8cc79312a56dfb6291e21aaae7a1563c769d/src/taskgraph/transforms/base.py#L87
[2] https://github.com/taskcluster/taskgraph/blob/59917336bce0e77854f5e20b5d26365ac34a44b1/src/taskgraph/config.py#L128
[3] https://github.com/mozilla-mobile/firefox-android/blob/324d8a06323f995fa1fdc4a97c6ef33926e7f769/taskcluster/ci/config.yml#L149-L152
[4] https://github.com/mozilla-services/guardian-vpn/blob/8769a282864f6f8c23134430e3b772b6c8703f5d/taskcluster/ci/config.yml#L10

Component: Task Configuration → General
Product: Firefox Build System → Release Engineering
QA Contact: jlorenzo
Summary: taskgraph: Provide all known SSH public keys of github in run-task → Fix ssh access for git operations on the firefox-android monorepo

Perfect. Simply adding the ssh-secret-name to config.yml seems to work:

https://firefox-ci-tc.services.mozilla.com/tasks/OhzeG3WiRoaV2o07k_ZHww has MOBILE_SSH_SECRET_NAME defined and

https://firefox-ci-tc.services.mozilla.com/tasks/OhzeG3WiRoaV2o07k_ZHww/runs/0/logs/public/logs/live.log shows

[vcs 2022-08-12T17:22:27.776Z] fetching secret project/mobile/firefox-android/github-clone-ssh from http://taskcluster/secrets/v1/secret/project/mobile/firefox-android/github-clone-ssh
[vcs 2022-08-12T17:22:27.886Z] executing ['git', 'clone', 'git@github.com:mozilla-mobile/firefox-android.git', '/builds/worker/checkouts/vcs']
[vcs 2022-08-12T17:22:27.888Z] Cloning into '/builds/worker/checkouts/vcs'...
[vcs 2022-08-12T17:22:27.948Z] Warning: Permanently added the RSA host key for IP address '140.82.114.3' to the list of known hosts.
[vcs 2022-08-12T17:22:31.853Z] executing ['git', 'fetch', '--tags', 'git@github.com:mozilla-mobile/firefox-android.git', 'refs/heads/bug1783883']
[vcs 2022-08-12T17:22:32.098Z] From github.com:mozilla-mobile/firefox-android
[vcs 2022-08-12T17:22:32.098Z]  * branch                  bug1783883 -> FETCH_HEAD
[vcs 2022-08-12T17:22:32.145Z] executing ['git', 'checkout', '-f', '52c9a8705ddf302685b66f30fc848172e3e12722']
[vcs 2022-08-12T17:22:32.164Z] Note: checking out '52c9a8705ddf302685b66f30fc848172e3e12722'.
...

and the task succeeds!

Status: REOPENED → RESOLVED
Closed: 3 years ago → 3 years ago
Resolution: --- → FIXED
Blocks: 1819403
You need to log in before you can comment on or make changes to this bug.