1231750 - run reconfigs from cruncher

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

use ssh agent with windows to pass credentials for buildduty folks
verify they can tag
verify they can update wiki and bugzilla

Comment 1

[Vlad Ciobancai [:vladC] [:vciobancai]]

(In reply to Kim Moir [:kmoir] from comment #0)
> use ssh agent with windows to pass credentials for buildduty folks
> verify they can tag
> verify they can update wiki and bugzilla

So to understand, all the credentials to be sent via ssh to cruncher server and after that the script to be run with the credentials ?

Comment 2

[Kim Moir [:kmoir] ET]

a good test would be to setup ssh agent with forwarding and then try to checkout a hg repo on cruncher with your credentials via ssh

Comment 3

[Alin Selagea [:aselagea]]

Okay, so we managed to setup our Cygwin clients to automatically start ssh-agent when opening a terminal and then store the private key. Also, we enabled forwarding for cruncher. 
On cruncher, we created a config file on ~/.ssh/ where we added the user name (softvision e-mail address) in order to avoid using such a long name each time we try to authenticate.

As a result, we managed to clone a hg repository on cruncher, do some test changes and commit them afterwards without any issues.

Comment 4

[Alin Selagea [:aselagea]]

When running the end_to_end.sh script, we will most likely need to issue the required commands to connect to cruncher, go to the location of the tools repo, do a pull and update operation in order to ensure that the repo is up-to-date and only after that run the mentioned script.

:coop we see two solutions for the location of the tools repo that can be used for both running reconfigs and dealing with master restarts:

1. use the tools repo from /home/buildduty --> in that case, we will need some permission changes, as the buildduty user is in a different group than ours.
2. create a folder in a neutral location (for example /opt) and clone the tools repo there.
What do you think?

Comment 5

[Chris Cooper [:coop] (he/him)]

(In reply to Alin Selagea [:aselagea][:buildduty] from comment #4) 
> :coop we see two solutions for the location of the tools repo that can be
> used for both running reconfigs and dealing with master restarts:
> 
> 1. use the tools repo from /home/buildduty --> in that case, we will need
> some permission changes, as the buildduty user is in a different group than
> ours.
> 2. create a folder in a neutral location (for example /opt) and clone the
> tools repo there.
> What do you think?

In case it's unclear, this is the first step towards removing human interaction from this merge process entirely. Eventually we'll have this running automatically on some cadence, without human involvement, gated on travis results or whatever.

Given that, I think we should align on using the shared buildduty user to run this process. Both your keys (Alin and Vlad) are already added to the buildduty authorized_keys file on cruncher. I would advocate option #1, i.e. connecting as the buildduty user to get the venv, etc setup for a shared reconfig location. We should discuss how the merge will actually happen, since I don't think we have a shared user that is able to perform the hg merge yet.

Comment 6

[Vlad Ciobancai [:vladC] [:vciobancai]]

I have started working on a bash script but I found a problem, the ssh connection from the cruncher server to a buildbot is not working

[[e-mail]3 ~]$ telnet buildbot-master01.bb.releng.use1.mozilla.com 22
Trying 10.134.68.17...
telnet: connect to address 10.134.68.17: Connection timed out


(In reply to Chris Cooper [:coop] from comment #5)
> We should discuss how the merge will actually happen,since I don't think we have a shared 
> user that is able to perform the hg merge yet.
We can use ssh agent to forward the private key in order to make connection from cruncher to buildbot slaves

Comment 7

[Vlad Ciobancai [:vladC] [:vciobancai]]

At yesterday's meeting we discussed about the ssh connection from cruncher to buildbot masters and we decided to use dev-master2 in stead of cruncher server.

There's seems to be a problem with python : ERROR: Python version 2.7 not found - please make sure python 2.7 is in your PATH

I tried in the past to resolve this issue but I was not able, :coop can you please help us

Comment 8

[Justin Wood (:Callek)]

9:28 AM <rail> try to "export PATH=/tools/python27/bin:$PATH" and then run it?
9:30 AM <vladC|buildduty> rail: thank you worked

Comment 9

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: run_remotely_reconfig.sh

Attached you can find a first draft of the script

Comment 10

[Vlad Ciobancai [:vladC] [:vciobancai]]

Attachment: bug1231750_run_remotely_reconfig.output

Attached you can find the output of the script

Comment 11

[Vlad Ciobancai [:vladC] [:vciobancai]]

Comment on attachment 8703606 [details]
run_remotely_reconfig.sh

it will be great your feedback

Comment 12

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8703606 [details]
run_remotely_reconfig.sh

I'm unsure about the utility of this script. 

The script introduces a persistent ssh connection as a new point of failure. This isn't a big deal in the default case, i.e. when we merge the repos and let the reconfig happen on its own, because the process only takes a few minutes. When we need to run a full reconfig immediately, the process can take over an hour.

Ideally what we want is a single, shared environment that can run the reconfig process hands-off or with minimal interaction, e.g. when prompted by an IRC bot.

Does that make sense?

Comment 13

[Vlad Ciobancai [:vladC] [:vciobancai]]

(In reply to Chris Cooper [:coop] from comment #12)
> Ideally what we want is a single, shared environment that can run the
> reconfig process hands-off or with minimal interaction, e.g. when prompted
> by an IRC bot.
> 
> Does that make sense?

If we have a shared environment, how we manage the private credentials ?
Should we continue working on this bug only when the following bug 1235851 is resolved ?

Comment 14

[Chris Cooper [:coop] (he/him)]

(In reply to Vlad Ciobancai [:vladC] from comment #13) 
> If we have a shared environment, how we manage the private credentials ?

Sorry, I'm getting ahead of myself here. The end state I want is the the shared reconfig env that can run reconfigs triggered via IRC. I know you and Alin just need a consistent place to run reconfigs for now.

What I _do_ want to avoid is having a config file sitting around on disk with someone's personal LDAP/bugzilla.wiki credentials in it. Running the reconfig remotely from your laptop could solve this problem, provided your connection doesn't timeout, the VPN doesn't go down, etc. This probably isn't going to matter for the standard case of a simple merge, but for the much longer execution time of a forced reconfig, it could absolutely happen. 

I would advocate connecting to the remote machine and establishing a screen session to insulate against a disconnect.

> Should we continue working on this bug only when the following bug 1235851
> is resolved ?

Didn't we decide to use dev-master2 instead of cruncher since it already had the necessary network flows? Maybe we should update the bug summary if so.

Comment 15

[Vlad Ciobancai [:vladC] [:vciobancai]]

:coop yesterday I run the reconfig script without force option from my computer and worked without problems.

I know we discussed about this in the last weeks but I do not see any issue if we run the script from our computers. 
My suggestion is to start working on releng bot in order to add the reconfig future in the bot and the other futures.

Please let me know if it's ok if not I will attach the latest version of the script to be reviewed and to be pushed in repository.

Comment 16

[Chris Cooper [:coop] (he/him)]

If we're going to WONTFIX this bug, that's fine. I want to make sure we have a bug on file for for the relengbot reconfig work first though, please.

Comment 17

[Vlad Ciobancai [:vladC] [:vciobancai]]

I created the following bug 1239263 to keep track the new changes that we will make for releng bot