Slave loan request for wcosta

RESOLVED FIXED

Status

Release Engineering
Loan Requests
RESOLVED FIXED
a month ago
13 days ago

People

(Reporter: fubar, Assigned: wcosta)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [buildduty][capacity][buildslaves][loaner])

(Reporter)

Description

a month ago
TC would like four ubuntu 16.04 hosts to verify the work they did earlier this year to support the new OS. Please loan out the following hosts for :wcosta

t-linux64-xe-297.test.releng.mdc1.mozilla.com
t-linux64-xe-298.test.releng.mdc1.mozilla.com
t-linux64-xe-299.test.releng.mdc1.mozilla.com
t-linux64-xe-300.test.releng.mdc1.mozilla.com
These machines are not yet taking jobs,they are not enabled in production so until then Wander can take any machine mentioned above to test.
Wander since you already have rights to access these machines using 'root' and 'cltbld' users I will not reset the password for these machines.

When you are finished with the loan forever, please comment stating so here in the bug,and we will re-image the machines and mark the bug as RESOLVED.
Assignee: nobody → wcosta
(Assignee)

Comment 2

a month ago
(In reply to Andrei Obreja [:aobreja][:buildduty] from comment #1)
> These machines are not yet taking jobs,they are not enabled in production so
> until then Wander can take any machine mentioned above to test.
> Wander since you already have rights to access these machines using 'root'
> and 'cltbld' users I will not reset the password for these machines.
> 
> When you are finished with the loan forever, please comment stating so here
> in the bug,and we will re-image the machines and mark the bug as RESOLVED.

Thanks!
(Assignee)

Updated

a month ago
Status: NEW → ASSIGNED
(Assignee)

Comment 3

a month ago
I am getting timeouts when trying to connect to the machines:

wcosta@Wanders-MBP:work $ ssh cltbld@t-linux64-xe-297.test.releng.mdc1.mozilla.com
ssh: connect to host t-linux64-xe-297.test.releng.mdc1.mozilla.com port 22: Operation timed out
wcosta@Wanders-MBP:work $
Flags: needinfo?(aobreja)
(Reporter)

Comment 4

a month ago
It sounds like the firewall needs to be disabled.

Comment 5

a month ago
Wander, these should work for you now.

I updated the loaner steps in the wiki to allow access from the corp. vpn and jumphost. I only had the CentOS-specific commands in there before:
https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#talos-linux32-ix.2C_talos-linux64-ix.2C_tst-linux32-ec2.2C_tst-linux64-ec2
```
iptables -I INPUT -s 10.22.240.0/20,10.22.72.158/32 -p tcp -m multiport --dports 22,5900 -m comment --comment "loaner vpn access" -j ACCEPT
iptables-save > /etc/iptables/rules.v4 || iptables-save > /etc/sysconfig/iptables
```

I applied this on 297-300 and confirmed that I can ssh into #300 from the corp vpn and jumphost:
```jumphost
[dhouse@ssh1.corpdmz.scl3 ~]$ ssh root@10.49.58.145
Last login: Mon Oct 16 15:54:24 2017 from 10.49.48.100
This host is set to follow security level "low"
Unauthorized access prohibited
[root@t-linux64-xe-300 ~]# exit
logout
Connection to 10.49.58.145 closed.
[dhouse@ssh1.corpdmz.scl3 ~]$ exit
logout
Shared connection to ssh.mozilla.com closed.
```

```vpn
$ ssh root@10.49.58.145
The authenticity of host '10.49.58.145 (10.49.58.145)' can't be established.
ECDSA key fingerprint is SHA256:ntaqdnzDOvFBidcsKlSeRwxledATvyuRWAlG/B1JpWA.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.49.58.145' (ECDSA) to the list of known hosts.
Last login: Mon Oct 16 15:56:08 2017 from 10.22.72.158
This host is set to follow security level "low"
Unauthorized access prohibited
[root@t-linux64-xe-300 ~]# exit
logout
Connection to 10.49.58.145 closed.
```

Comment 6

a month ago
Andrei, I'm sorry I missed this firewall piece. Please feel free to move or add notes to the iptables commands I added to the wiki https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#talos-linux32-ix.2C_talos-linux64-ix.2C_tst-linux32-ec2.2C_tst-linux64-ec2

I've set the host firewall to allow corp vpn/jumphost ssh and vnc now. I'll work on it if Wander still has access issues.
Flags: needinfo?(aobreja)
Thank you Dave.I think everything is set fine now.
(Assignee)

Comment 8

a month ago
(In reply to Dave House [:dhouse] from comment #5)
> Wander, these should work for you now.
> 
> I updated the loaner steps in the wiki to allow access from the corp. vpn
> and jumphost. I only had the CentOS-specific commands in there before:
> https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#talos-
> linux32-ix.2C_talos-linux64-ix.2C_tst-linux32-ec2.2C_tst-linux64-ec2
> ```
> iptables -I INPUT -s 10.22.240.0/20,10.22.72.158/32 -p tcp -m multiport
> --dports 22,5900 -m comment --comment "loaner vpn access" -j ACCEPT
> iptables-save > /etc/iptables/rules.v4 || iptables-save >
> /etc/sysconfig/iptables
> ```
> 
> I applied this on 297-300 and confirmed that I can ssh into #300 from the
> corp vpn and jumphost:
> ```jumphost
> [dhouse@ssh1.corpdmz.scl3 ~]$ ssh root@10.49.58.145
> Last login: Mon Oct 16 15:54:24 2017 from 10.49.48.100
> This host is set to follow security level "low"
> Unauthorized access prohibited
> [root@t-linux64-xe-300 ~]# exit
> logout
> Connection to 10.49.58.145 closed.
> [dhouse@ssh1.corpdmz.scl3 ~]$ exit
> logout
> Shared connection to ssh.mozilla.com closed.
> ```
> 
> ```vpn
> $ ssh root@10.49.58.145
> The authenticity of host '10.49.58.145 (10.49.58.145)' can't be established.
> ECDSA key fingerprint is SHA256:ntaqdnzDOvFBidcsKlSeRwxledATvyuRWAlG/B1JpWA.
> Are you sure you want to continue connecting (yes/no)? yes
> Warning: Permanently added '10.49.58.145' (ECDSA) to the list of known hosts.
> Last login: Mon Oct 16 15:56:08 2017 from 10.22.72.158
> This host is set to follow security level "low"
> Unauthorized access prohibited
> [root@t-linux64-xe-300 ~]# exit
> logout
> Connection to 10.49.58.145 closed.
> ```

Thanks a lot, Dave, it works now

Updated

a month ago
Depends on: 1409474
(Assignee)

Comment 9

18 days ago
I am done with the loaners. Just landed the latest patch to make machines green from Taskcluster PoV. We made some improvements to taskcluster-worker for better status reporting and so but jonasfj hasn't released a new version yet. Nice if we upgrade it before going to production.
Flags: needinfo?(aobreja)
Good news indeed.
The machines are not in production so I will let Dave to decide what should we do with them,re-image is not a option since /usr/sbin/bless does not exist for these instances.
Flags: needinfo?(aobreja) → needinfo?(dhouse)
So I found that bless is a macos thing for choosing the boot disk. For the linux hosts, we're depending on using a kvm/ipmi interface to get a pxeboot and then choose the pxeboot kickstart imaging from there (like linked to in the reclaiming doc, https://wiki.mozilla.org/ReleaseEngineering/How_To/Loan_a_Slave#Reclaiming, https://mana.mozilla.org/wiki/display/DC/How+To+Reimage+Releng+iX+and+HP+Linux+Machines). but the docs do not have notes for imaging on the moonshots yet (similar but through the moonshot+xen interface). I think that ideally no-one will ever need to use the xencenter interface.

RelOps will need to perform imaging on the moonshots with the current configuration.
Flags: needinfo?(dhouse)
I re-imaged the vm's through the xencenter console to pxeboot+kickstart them. So they are no longer loaner.
Status: ASSIGNED → RESOLVED
Last Resolved: 13 days ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.