735238 - khan move to scl3, lingering issues

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Corey Shields [:cshields]]

Please add lingering issues to this bug or file a new dep bug.

Comment 1

[Corey Shields [:cshields]]

we will still need to work out mounts.  Please help us out
and review the list of network mounts at the etherpad below, bold any
that are still necessary.  We want to cut out anything that is no longer
needed.  In fact, if you know it is no longer needed strike it out.

https://etherpad.mozilla.org/khan-mounts

Comment 2

[Chris More [:cmore]]

New issue with khan: I am unable to curl to any website outside of the network. Could be a network routing issue.

Comment 3

[casey ransom [:casey]]

[root@khan ~]# fping yahoo.com
yahoo.com is alive

and as I saw the fping return, I felt a very heavy jolt of irony, but, anyway, internet access is up now.

Comment 4

[Corey Shields [:cshields]]

:D  thanks Casey! 

I filed a bug for this but left it unlinked, going to assign and resolve that one for you, and keep this one open for other lingering issues.

Comment 5

[Chris More [:cmore]]

Confirmed! Thanks

Comment 6

[:glob ✱]

http://amckay-arecibo.khan.mozilla.org/list/ is throwing "Unknown MySQL server host 'cm-webdev01-master01'".

Comment 7

[Wil Clouser [:clouserw]]

(In reply to Byron Jones ‹:glob› from comment #6)
> http://amckay-arecibo.khan.mozilla.org/list/ is throwing "Unknown MySQL
> server host 'cm-webdev01-master01'".

Yep, I can't resolve that hostname from the box either

Comment 8

[Rick Bryce [:rbryce]]

For tracking purposes--

khan's monitoring checks have been commented out in Nagios

Comment 9

[Chris More [:cmore]]

Another issue: Khan is unable to resolve certain domains thus it must be using name servers that are not updated.

Example: curl mozilla.lt

Expected: website contents

Actual: curl: (6) Couldn't resolve host 'mozilla.lt'

When I an scanning websites, I use the following name servers to verify if they are our websites:

ns1.mozilla.org
ns2.mozilla.org
ns3.mozilla.org
ns0.mozilla.or.jp
ns1.mvs.ne.jp
ns1.mozilla.net
ns2.mozilla.net
ns3.mozilla.net
ns1-glb.mozilla.net
ns2-glb.mozilla.net
ns3-glb.mozilla.net
ns4-glb.mozilla.net

It looks like khan is using these name servers:

nameserver 10.22.75.40
nameserver 10.22.75.41

It appears that 10.22.75.40|41 have old cache.

Comment 10

[Brian Hourigan [:digi]]

The issue with mozilla.lt is because we're slaving this zone data from a third party server:

        zone "mozilla.lt" IN { /* zone contact: Rimas Kudelis <[e-mail]> */
                type slave;
        masters { 193.219.42.13; }; /* ideja.akl.lt */
                file "slaves/mozilla.lt";
        };

The new IPs of ns1/ns2 in SCL3 are not on this hosts access list. I've emailed Rimas with the new addresses requesting that authorize our new DNS servers to perform zone transfers.

Comment 11

[Brian Hourigan [:digi]]

You should now be able to resolve mozilla.lt from ns1/ns2 in scl3:

[[e-mail]3 named]# dig +short mozilla.lt
193.219.42.40

Comment 12

[Sheeri Cabral [:sheeri]]

Since rbryce mentioned it above for tracking purposes, I thought I'd mention it here:

I will note that khan's monitoring checks are in scl3's nagios and look like they're all green.

As for general status, looks like http://amckay-arecibo.khan.mozilla.org/list/ is still throwing "Unknown MySQL server host 'cm-webdev01-master01'". 

It needs an fqdn - cm-webdev01-master01.mozilla.org

Comment 13

[:glob ✱]

the file which needs updating to fix arecibo is:
  /home/amckay/arecibo/listener/normal/local_settings.py

Comment 14

[Andy McKay]

Updated arecibo.

Comment 15

[Sheeri Cabral [:sheeri]]

Is there anything more on this bug?

Comment 16

[Sheeri Cabral [:sheeri]]

cshields says this is closeable.