Closed Bug 1398224 Opened 8 years ago Closed 8 years ago

[MDC1] test.releng.mdc1 linux testing

Categories

(Infrastructure & Operations :: RelOps: General, task)

Product:
Infrastructure & Operations
Component:
RelOps: General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dhouse, Assigned: dhouse)

References

Details

Attachments

(2 files)

test.releng.mdc1_pxe-dhcp-fail_2017-09-13.png
407.40 KB, image/png
Details
test.releng.mdc1_kickstart-arp-timeout.png
596.73 KB, image/png
Details
Test releng kickstarts and connectivity from test.releng.mdc1.
I kickstarted a centos 6 base server and encountered no errors with puppet. I also kickstarted the vm on Ubuntu 16.04, to test it as a worker for taskcluster. No problems from puppet, but it is not taking jobs.
It fails to get host-secrets (kickstart-test1.test.releng.mdc1 -> releng-puppet2.srv.releng.mdc1 port 8020) port 8020, failing: [root@kickstart-test1 ~]# nc -rzw2 releng-puppet2.srv.releng.mdc1.mozilla.com 8020 &&echo pass||echo fail fail [root@kickstart-test1 ~]# nc -rzw2 releng-puppet2.srv.releng.mdc1.mozilla.com 80 &&echo pass||echo fail pass [root@kickstart-test1 ~]# nc -rzw2 releng-puppet2.srv.releng.scl3.mozilla.com 8020 &&echo pass||echo fail fail # host-secrets is moving to a separate host on port 80 and that works: [root@kickstart-test1 ~]# nc -rzw2 tc-host-secrets2.srv.releng.mdc1.mozilla.com 80 &&echo pass||echo fail pass [root@kickstart-test1 ~]# nc -rzw2 tc-host-secrets1.srv.releng.mdc1.mozilla.com 80 &&echo pass||echo fail pass
Actual service connection timeouts from the taskcluster worker log: Sep 8 09:36:15 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:36:15-07:00" level=info msg="Trying host-secrets server http://releng-puppet2.srv.releng.mdc1.mozilla.com:8020/v1/credentials" Sep 8 09:36:15 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T09:36:15-07:00" level=info msg="Trying host-secrets server http://releng-puppet2.srv.releng.mdc1.mozilla.com:8020/v1/credentials" Sep 8 09:39:18 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:39:18-07:00" level=error msg="error fetching secrets; continuing to next server" error="Get http://releng-puppet2.srv.releng.mdc1.mozilla.com:8020/v1/credentials: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" incidentId=b326fc96-5603-44e9-969a-104d224a4c19 Sep 8 09:39:18 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T09:39:18-07:00" level=error msg="error fetching secrets; continuing to next server" error="Get http://releng-puppet2.srv.releng.mdc1.mozilla.com:8020/v1/credentials: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" incidentId=b326fc96-5603-44e9-969a-104d224a4c19 Sep 8 09:39:18 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:39:18-07:00" level=info msg="Trying host-secrets server http://releng-puppet1.srv.releng.mdc1.mozilla.com:8020/v1/credentials" Sep 8 09:39:18 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T09:39:18-07:00" level=info msg="Trying host-secrets server http://releng-puppet1.srv.releng.mdc1.mozilla.com:8020/v1/credentials" Sep 8 09:42:21 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:42:21-07:00" level=error msg="error fetching secrets; continuing to next server" error="Get http://releng-puppet1.srv.releng.mdc1.mozilla.com:8020/v1/credentials: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" incidentId=3747ac22-ea64-4a0c-987b-72cbfbaee8a2 Sep 8 09:42:21 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T09:42:21-07:00" level=error msg="error fetching secrets; continuing to next server" error="Get http://releng-puppet1.srv.releng.mdc1.mozilla.com:8020/v1/credentials: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" incidentId=3747ac22-ea64-4a0c-987b-72cbfbaee8a2 Sep 8 09:42:21 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:42:21-07:00" level=info msg="Trying host-secrets server http://releng-puppet1.srv.releng.scl3.mozilla.com:8020/v1/credentials" Sep 8 09:42:21 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T09:42:21-07:00" level=info msg="Trying host-secrets server http://releng-puppet1.srv.releng.scl3.mozilla.com:8020/v1/credentials" Sep 8 09:45:24 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:45:24-07:00" level=error msg="error fetching secrets; continuing to next server" error="Get http://releng-puppet1.srv.releng.scl3.mozilla.com:8020/v1/credentials: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" incidentId=0e6a6c1d-8ef8-427e-a34d-2ad123c49d90 Sep 8 09:45:24 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T09:45:24-07:00" level=error msg="error fetching secrets; continuing to next server" error="Get http://releng-puppet1.srv.releng.scl3.mozilla.com:8020/v1/credentials: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" incidentId=0e6a6c1d-8ef8-427e-a34d-2ad123c49d90 Sep 8 09:45:24 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:45:24-07:00" level=info msg="Trying host-secrets server http://releng-puppet2.srv.releng.scl3.mozilla.com:8020/v1/credentials" Sep 8 09:45:24 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T09:45:24-07:00" level=info msg="Trying host-secrets server http://releng-puppet2.srv.releng.scl3.mozilla.com:8020/v1/credentials" Sep 8 09:48:27 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:48:27-07:00" level=error msg="error fetching secrets; continuing to next server" error="Get http://releng-puppet2.srv.releng.scl3.mozilla.com:8020/v1/credentials: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" incidentId=374711e2-b4b1-472e-a9d0-fa9a7dccebc9 Sep 8 09:48:27 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T09:48:27-07:00" level=error msg="error fetching secrets; continuing to next server" error="Get http://releng-puppet2.srv.releng.scl3.mozilla.com:8020/v1/credentials: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" incidentId=374711e2-b4b1-472e-a9d0-fa9a7dccebc9 Sep 8 09:48:27 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:48:27-07:00" level=info msg="Trying host-secrets server http://releng-puppet1.srv.releng.use1.mozilla.com:8020/v1/credentials" Sep 8 09:48:27 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T09:48:27-07:00" level=info msg="Trying host-secrets server http://releng-puppet1.srv.releng.use1.mozilla.com:8020/v1/credentials" Sep 8 09:51:30 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:51:30-07:00" level=error msg="error fetching secrets; continuing to next server" error="Get http://releng-puppet1.srv.releng.use1.mozilla.com:8020/v1/credentials: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" incidentId=9a6129f7-5bab-43ae-a34d-8fb217397aca Sep 8 09:51:30 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T09:51:30-07:00" level=error msg="error fetching secrets; continuing to next server" error="Get http://releng-puppet1.srv.releng.use1.mozilla.com:8020/v1/credentials: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" incidentId=9a6129f7-5bab-43ae-a34d-8fb217397aca Sep 8 09:51:30 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:51:30-07:00" level=info msg="Trying host-secrets server http://releng-puppet1.srv.releng.usw2.mozilla.com:8020/v1/credentials" Sep 8 09:51:30 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T09:51:30-07:00" level=info msg="Trying host-secrets server http://releng-puppet1.srv.releng.usw2.mozilla.com:8020/v1/credentials" Sep 8 09:54:33 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:54:33-07:00" level=error msg="error fetching secrets; continuing to next server" error="Get http://releng-puppet1.srv.releng.usw2.mozilla.com:8020/v1/credentials: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" incidentId=5d7f57af-7f1c-425f-aa65-654483e696f1 Sep 8 09:54:33 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T09:54:33-07:00" level=error msg="error fetching secrets; continuing to next server" error="Get http://releng-puppet1.srv.releng.usw2.mozilla.com:8020/v1/credentials: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" incidentId=5d7f57af-7f1c-425f-aa65-654483e696f1 Sep 8 09:54:33 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:54:33-07:00" level=info msg="list of servers exhausted; sleeping before starting again" Sep 8 09:54:33 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T09:54:33-07:00" level=info msg="list of servers exhausted; sleeping before starting again" Sep 8 09:55:33 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker[1674]: time="2017-09-08T09:55:33-07:00" level=info msg="Trying host-secrets server http://releng-puppet2.srv.releng.mdc1.mozilla.com:8020/v1/credentials"
Manually testing it against the tc-host-secrets1.srv.releng.mdc1:80 appears to show that the client is not set up for auth/secrets in tc-host-secrets: Sep 8 10:36:51 kickstart-test1.test.releng.mdc1.mozilla.com taskcluster-worker: time="2017-09-08T10:36:51-07:00" level=error msg="failed to ClaimWork" error="(Permanent) HTTP response code 401\nHTTP/1.1 401 Unauthorized\r\nContent-Length: 673\r\nAccess-Control-Allow-Headers: X-Requested-With,Content-Type,Authorization,Accept,Origin\r\nAccess-Control-Allow-Methods: OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT\r\nAccess-Control-Allow-Origin: *\r\nAccess-Control-Request-Method: *\r\nConnection: keep-alive\r\nContent-Type: application/json; charset=utf-8\r\nDate: Fri, 08 Sep 2017 17:36:51 GMT\r\nServer: Cowboy\r\nStrict-Transport-Security: max-age=7776000\r\nVia: 1.1 vegur\r\nWww-Authenticate: hawk\r\nX-Powered-By: Express\r\n\r\n{\n \"code\": \"AuthenticationFailed\",\n \"message\": \"ext.certificate issuer `project/releng/host-secrets/production` doesn't have `auth:create-client:project/releng/host-secrets/host/com.mozilla.mdc1.releng.test.kickstart-test1` for supplied clientId.\\n----\\nmethod: claimWork\\nerrorCode: AuthenticationFailed\\n
Dragos pointed out that he tested Ubuntu 16 taskcluster worker on moonshot-test3.test.releng.scl3. So I checked there and the logs show a non-timed-out connection for host-secrets to releng-puppet2.scl3 and I am able to netcat on port 8020 to there also (so, the same tests I did on the kickstart-test1 machine give the correct results showing it is working there but not on kickstart-test1). So it looks like I need to request port 8020 be opened from vlan256/test.releng.mdc1 to releng-puppet2.srv.releng.mdc1. That can give us the short-term(running on the puppet master) host secrets. long-term, port 80 to the new tc-host-secrets servers is already open by bug 1389533
I created bug https://bugzilla.mozilla.org/show_bug.cgi?id=1398265 to get port 8020/tcp open from test.releng.mdc1 to the mdc1 releng puppet masters for taskcluster host-secrets. When that is done, I can test the worker again.
mdc1 move is in progress. the VM's are up and running so I tested kickstarts in srv.releng.mdc1 and test.releng.mdc1 The kickstart+puppetagain in srv.releng.mdc1 had no problems. The kickstart fails to get a response for dhcp in test.releng.mdc1. I'll wait and test again in a few days. Maybe something is not yet in place. Testing isn't planned until Friday+
PXE-E11: ARP timeout PXE-E38: TFTP cannot open connection The test kickstart in test.releng.mdc1 gets an IP from dhcp, but fails on tftp.
Tested this morning with ip 10.48.75.120 between 5:34 and 5:39 PDT
We may need firewall changes like done in bug 1394529. The pxe boot does not get past the initial tftp, and so I don't know if port 80 is open to the admin host (like requested in bug 1394529).
So, we can tftp to the admin servers from other VLANs, (private, srv.releng, etc) but not from the test.releng The network firewalls appear to have the appropriately permissive rules. (anything in 10.49.0.0/16 can tftp from the admin server) and the DHCP server is setup the same between working vlans and test.releng. So I'm not sure where things are broken.
We need to get some of these up and running to verify talos testing; can we get netops involved and sort out where the issue is soonest?
Flags: needinfo?(dhouse)
netops is aware and it's on their plate, right after they figure out some dhcp related issues with ldap vlan in MDC1. bug 1401327 for that. cc;ing :rmfd for his edification
(In reply to Dave House [:dhouse] from comment #8) > Created attachment 8911127 [details] > test.releng.mdc1_kickstart-arp-timeout.png > > PXE-E11: ARP timeout > PXE-E38: TFTP cannot open connection > > The test kickstart in test.releng.mdc1 gets an IP from dhcp, but fails on > tftp. Just to confirm state: I tested this again today and get the same arp timeout error.
Flags: needinfo?(dhouse)
Was asked by :rmfd to test again - testing, same timeout occurs.
This has been resolved. The following line was missing from access6.mdc1: vxlan vlan 256 flood vtep 10.48.250.1 10.48.250.3 10.48.250.4 10.48.250.5 10.48.250.6 10.48.250.7 10.48.250.8 10.48.250.9 10.48.250.10 10.48.250.11 10.48.250.12 This has been added and confirmed working with Chris Knowles.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: