1175150 - new PTR records not resolving for releng loaner AWS instances

Status: RESOLVED FIXED

(Infrastructure & Operations :: DNS and Domain Registration, task)

Description

[Kim Moir [:kmoir] ET]

I recently added some entries for a new AWS instance I'm about to create. I can see the records in inventory

https://inventory.mozilla.org/en-US/mozdns/record/update/A/32668/

However, the PTR record does not resolve.  I created the record over and hour ago

(aws_manager)[[e-mail] aws_manager]$ nslookup  tst-linux64-ec2-fkiefer.build.mozilla.org
Server:		10.26.75.40
Address:	10.26.75.40#53

tst-linux64-ec2-fkiefer.build.mozilla.org	canonical name = tst-linux64-ec2-fkiefer.test.releng.use1.mozilla.com.
Name:	tst-linux64-ec2-fkiefer.test.releng.use1.mozilla.com
Address: 10.134.42.62

(aws_manager)[[e-mail] aws_manager]$ nslookup 10.134.42.62
Server:		10.26.75.40
Address:	10.26.75.40#53

** server can't find 62.42.134.10.in-addr.arpa.: NXDOMAIN

(aws_manager)[[e-mail] aws_manager]$

Comment 1

[Kim Moir [:kmoir] ET]

I can't loan EC2 slaves until this is fixed

Comment 2

[:Atoll]

Out of curiosity, what's this host's netmask? There's no 10.134.42 network in Inventory:

https://inventory.mozilla.org/en-US/core/search/#q=10.134

(This may or may not be related.)

Comment 3

[Amy Rich [:arr] [:arich]]

:kmoir is this a new network that got added to AWS recently? 

There wasn't a corresponding network for it in inventory, and there doesn't seem to be a global zone for zone 134.10.in-addr.arpa listed in the config like there is for 26.10.in-addr.arpa as a catch all.

I created:

https://inventory.mozilla.org/en-US/core/network/630/

I'm not sure how to make the correspond entry to the zone config file, though, and I don't want to mess that up. I think a better option is adding 134.10.in-addr.arpa, 132.10.in-addr.arpa, and 130.10.in-addr.arpa zones that can catch all of the addresses outside the defined networks we have listed.

Comment 4

[Kim Moir [:kmoir] ET]

I don't know if it was added recently.  Looking at the AWS console, addresses in this range are associated with subnet subnet-f42fb5ad and VPC vpc-b42100df.  This was just the address that was assigned when I ran the cloud tools to generate an ip for a loaner instance type. I don't know the netmask.

Comment 5

[Ryan Watson [:w0ts0n]]

I'm moving this down to "Normal" priority. It is paging the IT on call and it appears as though Arr is working on it. 

Arr is you are actively working on this bug, can you pick it up from Eugene please?

Comment 6

[Amy Rich [:arr] [:arich]]

No, I'm waiting for IT to work on it and offering advice as to how I think it should be fixed. I don't know how to correctly implement the fix. And this is still blocking releng, so the priority shouldn't be bumped down.

Comment 7

[Brian Hourigan [:digi]]

Good morning

I agree it would be best to create 134.10.in-addr.arpa and this would eliminate the need to create new zone and domain scopes for each /24 under this network. However, 28 zones for /24s in this network already exist. We would need to flatten all of these records prior to adding 134.10.in-addr.arpa

Adding a network to inventory does not create the zones and SOAs to create a new DNS scope. It's a little ambiguous but my understanding is that networks are used for automatic IP assignment when adding A records and also perhaps for DHCP management. Not DNS.

I went ahead and added 42.134.10.in-addr.arpa with a SOA so the records in this particular /24 can be added and work. We can set aside some time later to flatten everything into 134.10.in-addr.arpa but it will likely require a ~30-60 minute maintenance window where DNS for 134.10.in-addr.arpa may not work. 

https://inventory.mozilla.org/en-US/core/search/#q=zone=:42.134.10.in-addr.arpa

Test:

bhourigan@Brians-MacBook ~/svn/dnsconfig/invzones » nslookup 10.134.42.62
Server:		10.22.72.136
Address:	10.22.72.136#53

62.42.134.10.in-addr.arpa	name = tst-linux64-ec2-fkiefer.test.releng.use1.mozilla.com.

Comment 8

[Brian Hourigan [:digi]]

> We would need to flatten all of these records prior to adding
> 134.10.in-addr.arpa

After speaking with :arr on IRC it looks like 26.10.in-addr.arpa has overlapping reverse scopes, and this seems to work. We could add 134.10.in-addr.arpa as a new zone and domain and it should provide a scope for any future subnets in this network.

Comment 9

[Amy Rich [:arr] [:arich]]

Thanks, digi! Kim, the address should be working now.

Once we verify the long term fix from comment 8 works, we should make sure the reverse scopes exist for all three releng AWS networks:

134.10.in-addr.arpa, 132.10.in-addr.arpa, and 130.10.in-addr.arpa

That covers all our bases in all three regions we're in.

Comment 10

[Kim Moir [:kmoir] ET]

Thanks so much!  I can create instances now:-)

Comment 11

[Kim Moir [:kmoir] ET]

Actually I still have a problem with this record for bug  1172749

https://inventory.mozilla.org/en-US/mozdns/record/update/PTR/33261/

Comment 12

[Amy Rich [:arr] [:arich]]

That's in a different subnet. The longer term fix that digi's going to implement tonight should fix them all. Can this wait till then?

Comment 13

[Kim Moir [:kmoir] ET]

Sure, it can wait.

Comment 14

[Brian Hourigan [:digi]]

(In reply to Kim Moir [:kmoir] from comment #11)
> Actually I still have a problem with this record for bug  1172749
> 
> https://inventory.mozilla.org/en-US/mozdns/record/update/PTR/33261/

Hi Kim

Thanks for your patience today - I went ahead and added the zones as :arr mentioned and your entry is now resolving for me.

bhourigan@Brians-MacBook ~ » nslookup 10.134.40.86
Server:		10.22.72.136
Address:	10.22.72.136#53

86.40.134.10.in-addr.arpa	name = tst-linux64-ec2-gbrown.test.releng.use1.mozilla.com.

Moving forward you can add reverse entries for all of 10.134, 10.132, and 10.130. Please reopen the bug if you need anything else.