Closed Bug 1409008 Opened 7 years ago Closed 7 years ago

Slave loan request for wcosta

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: fubar, Assigned: wcosta)

Details

(Whiteboard: [buildduty][capacity][buildslaves][loaner]) 

TC would like four ubuntu 16.04 hosts to verify the work they did earlier this year to support the new OS. Please loan out the following hosts for :wcosta

t-linux64-xe-297.test.releng.mdc1.mozilla.com
t-linux64-xe-298.test.releng.mdc1.mozilla.com
t-linux64-xe-299.test.releng.mdc1.mozilla.com
t-linux64-xe-300.test.releng.mdc1.mozilla.com
These machines are not yet taking jobs,they are not enabled in production so until then Wander can take any machine mentioned above to test.
Wander since you already have rights to access these machines using 'root' and 'cltbld' users I will not reset the password for these machines.

When you are finished with the loan forever, please comment stating so here in the bug,and we will re-image the machines and mark the bug as RESOLVED.
Assignee: nobody → wcosta
(In reply to Andrei Obreja [:aobreja][:buildduty] from comment #1)
> These machines are not yet taking jobs,they are not enabled in production so
> until then Wander can take any machine mentioned above to test.
> Wander since you already have rights to access these machines using 'root'
> and 'cltbld' users I will not reset the password for these machines.
> 
> When you are finished with the loan forever, please comment stating so here
> in the bug,and we will re-image the machines and mark the bug as RESOLVED.

Thanks!
Status: NEW → ASSIGNED
I am getting timeouts when trying to connect to the machines:

wcosta@Wanders-MBP:work $ ssh cltbld@t-linux64-xe-297.test.releng.mdc1.mozilla.com
ssh: connect to host t-linux64-xe-297.test.releng.mdc1.mozilla.com port 22: Operation timed out
wcosta@Wanders-MBP:work $
Flags: needinfo?(aobreja)
It sounds like the firewall needs to be disabled.
Wander, these should work for you now.

I updated the loaner steps in the wiki to allow access from the corp. vpn and jumphost. I only had the CentOS-specific commands in there before:
https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#talos-linux32-ix.2C_talos-linux64-ix.2C_tst-linux32-ec2.2C_tst-linux64-ec2
```
iptables -I INPUT -s 10.22.240.0/20,10.22.72.158/32 -p tcp -m multiport --dports 22,5900 -m comment --comment "loaner vpn access" -j ACCEPT
iptables-save > /etc/iptables/rules.v4 || iptables-save > /etc/sysconfig/iptables
```

I applied this on 297-300 and confirmed that I can ssh into #300 from the corp vpn and jumphost:
```jumphost
[dhouse@ssh1.corpdmz.scl3 ~]$ ssh root@10.49.58.145
Last login: Mon Oct 16 15:54:24 2017 from 10.49.48.100
This host is set to follow security level "low"
Unauthorized access prohibited
[root@t-linux64-xe-300 ~]# exit
logout
Connection to 10.49.58.145 closed.
[dhouse@ssh1.corpdmz.scl3 ~]$ exit
logout
Shared connection to ssh.mozilla.com closed.
```

```vpn
$ ssh root@10.49.58.145
The authenticity of host '10.49.58.145 (10.49.58.145)' can't be established.
ECDSA key fingerprint is SHA256:ntaqdnzDOvFBidcsKlSeRwxledATvyuRWAlG/B1JpWA.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.49.58.145' (ECDSA) to the list of known hosts.
Last login: Mon Oct 16 15:56:08 2017 from 10.22.72.158
This host is set to follow security level "low"
Unauthorized access prohibited
[root@t-linux64-xe-300 ~]# exit
logout
Connection to 10.49.58.145 closed.
```
Andrei, I'm sorry I missed this firewall piece. Please feel free to move or add notes to the iptables commands I added to the wiki https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#talos-linux32-ix.2C_talos-linux64-ix.2C_tst-linux32-ec2.2C_tst-linux64-ec2

I've set the host firewall to allow corp vpn/jumphost ssh and vnc now. I'll work on it if Wander still has access issues.
Flags: needinfo?(aobreja)
Thank you Dave.I think everything is set fine now.
(In reply to Dave House [:dhouse] from comment #5)
> Wander, these should work for you now.
> 
> I updated the loaner steps in the wiki to allow access from the corp. vpn
> and jumphost. I only had the CentOS-specific commands in there before:
> https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#talos-
> linux32-ix.2C_talos-linux64-ix.2C_tst-linux32-ec2.2C_tst-linux64-ec2
> ```
> iptables -I INPUT -s 10.22.240.0/20,10.22.72.158/32 -p tcp -m multiport
> --dports 22,5900 -m comment --comment "loaner vpn access" -j ACCEPT
> iptables-save > /etc/iptables/rules.v4 || iptables-save >
> /etc/sysconfig/iptables
> ```
> 
> I applied this on 297-300 and confirmed that I can ssh into #300 from the
> corp vpn and jumphost:
> ```jumphost
> [dhouse@ssh1.corpdmz.scl3 ~]$ ssh root@10.49.58.145
> Last login: Mon Oct 16 15:54:24 2017 from 10.49.48.100
> This host is set to follow security level "low"
> Unauthorized access prohibited
> [root@t-linux64-xe-300 ~]# exit
> logout
> Connection to 10.49.58.145 closed.
> [dhouse@ssh1.corpdmz.scl3 ~]$ exit
> logout
> Shared connection to ssh.mozilla.com closed.
> ```
> 
> ```vpn
> $ ssh root@10.49.58.145
> The authenticity of host '10.49.58.145 (10.49.58.145)' can't be established.
> ECDSA key fingerprint is SHA256:ntaqdnzDOvFBidcsKlSeRwxledATvyuRWAlG/B1JpWA.
> Are you sure you want to continue connecting (yes/no)? yes
> Warning: Permanently added '10.49.58.145' (ECDSA) to the list of known hosts.
> Last login: Mon Oct 16 15:56:08 2017 from 10.22.72.158
> This host is set to follow security level "low"
> Unauthorized access prohibited
> [root@t-linux64-xe-300 ~]# exit
> logout
> Connection to 10.49.58.145 closed.
> ```

Thanks a lot, Dave, it works now
I am done with the loaners. Just landed the latest patch to make machines green from Taskcluster PoV. We made some improvements to taskcluster-worker for better status reporting and so but jonasfj hasn't released a new version yet. Nice if we upgrade it before going to production.
Flags: needinfo?(aobreja)
Good news indeed.
The machines are not in production so I will let Dave to decide what should we do with them,re-image is not a option since /usr/sbin/bless does not exist for these instances.
Flags: needinfo?(aobreja) → needinfo?(dhouse)
So I found that bless is a macos thing for choosing the boot disk. For the linux hosts, we're depending on using a kvm/ipmi interface to get a pxeboot and then choose the pxeboot kickstart imaging from there (like linked to in the reclaiming doc, https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#Reclaiming, https://mana.mozilla.org/wiki/display/DC/How+To+Reimage+Releng+iX+and+HP+Linux+Machines). but the docs do not have notes for imaging on the moonshots yet (similar but through the moonshot+xen interface). I think that ideally no-one will ever need to use the xencenter interface.

RelOps will need to perform imaging on the moonshots with the current configuration.
Flags: needinfo?(dhouse)
I re-imaged the vm's through the xencenter console to pxeboot+kickstart them. So they are no longer loaner.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Component: Loan Requests → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.