Closed
Bug 509591
Opened 15 years ago
Closed 15 years ago
SiteSpect Implementation [ ref:00D7JfQw.50079No58:ref ]
Categories
(Infrastructure & Operations Graveyard :: WebOps: Other, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: bcutler, Assigned: oremj)
Details
Attachments
(1 file)
248.48 KB,
image/png
|
Details |
Tal Cohen, from SiteSpect, is helping us implement their service. He currently needs: "Your NTP server IP address. Smartrelay host name (if needed for outbound emails). In addition to this, our systems now come with a Dell Remote Administration Card (DRAC) that allows out of band management of the server. Think KVM over internet. I need 1 additional IP/Netmask/Gateway configuration for the DRAC card." Instead of sending email back and forth, Matthew suggested we communicate over bugzilla.
Reporter | ||
Comment 1•15 years ago
|
||
um, please ignore the typo in the title.
Comment 2•15 years ago
|
||
I've created a Bugzilla account for SiteSpect per Matthew's suggestion.
Reporter | ||
Comment 3•15 years ago
|
||
Any progress on getting the information above? We would like to have SiteSpect up and running my the end of August at the latest.
Comment 4•15 years ago
|
||
I think I'm supposed to answer this... If you guys can do DHCP all of thise should come from DHCP. You should be able to relay email through smtp.mozilla.org.
Comment 5•15 years ago
|
||
Our system requires static IP addresses. DHCP will not work. Sorry.
Comment 6•15 years ago
|
||
Also, can you update the title/subject of this case to include the following text: [ ref:00D7JfQw.50079No58:ref ] Otherwise our tracking system assigns a new case each time someone updates this bug. Tal
Updated•15 years ago
|
Summary: SiteSpect Implementaiton → SiteSpect Implementation [ ref:00D7JfQw.50079No58:ref ]
Comment 7•15 years ago
|
||
We talked about security issues and how/where this would be best deployed. Passing to Derek for network info.
Assignee: server-ops → dmoore
Comment 8•15 years ago
|
||
I've spoken with Blake, and we should have the necessary information over to him around mid-day tomorrow.
Comment 9•15 years ago
|
||
I still have not received the information. Can you please update this ticket with it?
Comment 10•15 years ago
|
||
NTP server: 63.245.208.5 DNS server: 63.245.208.5 SMTP relay: mail.mozilla.org IP addresses: 63.245.208.80, 63.245.208.81 Netmask: 255.255.255.128 Gateway: 63.245.208.1
Comment 11•15 years ago
|
||
OK, the system is shipping today. The FedEx tracking number is 797866973876. You should expect the system to arrive on Tuesday. I will send an email out with usernames and passwords directly to Blake and Matt. We should schedule some configuration and validation time for the week of 8/31. let me know when is convenient for you. Tal
Comment 12•15 years ago
|
||
Is this shipping to Mountain View or to the San Jose data center?
Comment 13•15 years ago
|
||
It is shipping to: Matthew Zeier Mozilla Corporation 55 South Market Street San Jose, CA 95113
Assignee | ||
Updated•15 years ago
|
Assignee: dmoore → jeremy.orem+bugs
Assignee | ||
Comment 14•15 years ago
|
||
Derek, did you power on these servers? How do I access them?
Comment 15•15 years ago
|
||
Server is now online and accessible: sitespect.mozilla.com (10.2.80.230) OOB management (DRAC): sitespect-drac.mozilla.com (10.2.80.233)
Assignee | ||
Comment 16•15 years ago
|
||
Should be ready for Blake to start configuring.
Comment 17•15 years ago
|
||
Blake, I just got back from vacation and want to continue your deployment of SiteSpect. Where you able to log into the control panel? When you have a few minutes, give me a call to review your current status. I can be reached at 617-859-1900 x 705 - Tal
Reporter | ||
Comment 18•15 years ago
|
||
I won't have access to the control panel until tomorrow. Are you free for a call after 12pm tomorrow?
Comment 19•15 years ago
|
||
(In reply to comment #18) > I won't have access to the control panel until tomorrow. Are you free for a > call after 12pm tomorrow? 12pm EST or 12pm PDT?
Reporter | ||
Comment 20•15 years ago
|
||
PDT
Comment 21•15 years ago
|
||
(In reply to comment #20) > PDT 12 pm PDT / 3 pm EST works for me. Let me know if you want me to setup a phone bridge. Otherwise, give me a call at 617-859-1900 x 705.
Comment 22•15 years ago
|
||
VPN was restarted and I tested connectivity.
Reporter | ||
Comment 23•15 years ago
|
||
I'm still unable to access the control panel. Jeremy, are you available to troubleshoot with Tal Cohen tomorrow? He's available by phone from 9am-4pm.
Assignee | ||
Comment 24•15 years ago
|
||
Yeah, I'll be around.
Reporter | ||
Comment 25•15 years ago
|
||
Does a 10:30 call work for you, Jeremy? Tal?
Assignee | ||
Comment 26•15 years ago
|
||
Yeah, works for me. Will you send out a meeting request for whatever time you decide on.
Comment 27•15 years ago
|
||
Jeremy, when ever you are ready, just call me at 617-859-1900 x705
Comment 28•15 years ago
|
||
Blake, you can now log into SiteSpect at https://sitespect.mozilla.com:9443. You may need the following local host entry: 62.27.48.226 sitespect.mozilla.com Jeremy and I were able to work through most of the open issues. The only things that are left are Failover and Load Balancing. The issue with Failover is that when SiteSpect fails heartbeat checks, the load balancer is not bypassing SiteSpect. It just stops traffic. Jeremy is looking into this. I will need to test out your load balancing rules prior to your going live. In the meantime you can access the system and start configuring test objects and test campaigns.
Assignee | ||
Comment 29•15 years ago
|
||
I think he meant this entry in /etc/hosts: 63.245.209.10 sitespect.mozilla.com
Reporter | ||
Comment 30•15 years ago
|
||
it works! thanks for looking into this Jeremy. when do you expect to resolve the Failover and Load Balancing issues?
Assignee | ||
Comment 31•15 years ago
|
||
Failover is pretty easy, but I'm kind of confused on the load balancing part. We don't want to point all the traffic at the sitespect box? How much traffic should we send there?
Comment 32•15 years ago
|
||
Attached is the SiteSpect Load Balancer Guide. It provides guidance and suggestions on how to configure your load balancer to function with SiteSpect.
Comment 33•15 years ago
|
||
I've attached a load balancer guide. The actual percentage of traffic to send through SiteSpect is not discussed, but I'd suggest determining a desired based on a combination of business needs, Net-Opps requirements, and system capacity. As a general rule, you want to start with a low percentage and ramp up over a period of time until the desired percentage has been reached (or until SiteSpect has reached 75% capacity).
Comment 34•15 years ago
|
||
Re /etc/hosts: Jeremy is correct Please use: 63.245.209.10 sitespect.mozilla.com
Comment 35•15 years ago
|
||
oremj - we did percentage based on weighted round robin last time. The config is still there in the netscaler for this - look at cs-mozcom IIRC, you'll need to bind mozcom-sitespect to cs-mozcom. cs-mozcom would basically have two bound services - one is just sitespect and the other is the static cluster. add cs vserver "cs-mozcom" HTTP 63.245.209.10 80 -cltTimeout 180 bind cs vserver "cs-mozcom" "mozcom-inside-vs" add lb vserver "mozcom-inside-vs" HTTP 0.0.0.0 0 -persistenceType NONE -lbMethod ROUNDROBIN -cltTimeout 180 add lb vserver "mozcom-sitespect" HTTP 0.0.0.0 0 -persistenceType NONE -lbMethod LEASTRESPONSETIME -cltTimeout 180 bind lb vserver "mozcom-inside-vs" "svc-group-static" bind lb vserver "mozcom-sitespect" "svc-sitespect-80-web"
Comment 36•15 years ago
|
||
Orem / Matthew, Any idea when you plan to deploy the load balancer rules? Tal
Comment 37•15 years ago
|
||
They are still in place from the last time we did this - I guess I was lazy and never took them out.
Comment 38•15 years ago
|
||
So when will you start sending live traffic through SiteSpect?
Reporter | ||
Comment 39•15 years ago
|
||
We would like to begin sending traffic through SiteSpect early next week.
Comment 40•15 years ago
|
||
I would like to run a validation test of the load balancer configuration, including the splitting of traffic, affinity to SiteSpect, affinity away from SiteSpect, heart beating and failover. Is there a publicly accessible VIP on the load balancer that I can currently hit to test? If not I can run these tests once you turn on live traffic.
Reporter | ||
Comment 41•15 years ago
|
||
We would like to run tests on two additional Mozilla domains: addons.mozilla.org and support.mozilla.com. What steps are required to make this happen?
Comment 42•15 years ago
|
||
The answer is more involved than what I can properly document in this system, and really justifies a phone call. Can we setup a 15 minute call with Blake and Orem tomorrow in the morning?
Comment 43•15 years ago
|
||
Er, yeah, what Tal says. AMO's on a different load balancer, would have different engineering around it.
Reporter | ||
Comment 44•15 years ago
|
||
I'm free for a call anytime tomorrow. How does a 2pm PDT call work?
Comment 45•15 years ago
|
||
2PM PDT works for me. Here is the phone bridge dial in number: Dial 213-286-1202 Access Code: 526-783-859 Also, if needed, I do have a GotoMeeting session scheduled with this call. If we need to share desktops. The link is https://www2.gotomeeting.com/join/526783859 Tal
Comment 46•15 years ago
|
||
We have detected a number of attempts to break into the SiteSpect system. All of the attempts are comming from 10.2.80.4. If I'm not mistaken, that IP is one of your internal gateway devices (when I log in my IP address also appears to be 10.2.80.4). The system has not been compromised, but I'd like you to restrict access on 63.245.209.10 port 9022 to only allow connections from corp.sitespect.com, frmtca1-sm02.sitespect.com, bostma1-sm01.sitespect.com, onduk1-sm01.sitespect.com, and ogilvy.sitespect.com. Thanks, Tal
Comment 47•15 years ago
|
||
Define "break into" - 10.2.80.4 is the load balancer which is running health checks.
Comment 48•15 years ago
|
||
but by all means add these acls. honestly quite surprised they aren't there by default...
Updated•15 years ago
|
Group: infra
Comment 49•15 years ago
|
||
I figured that 10.2.80.4 is the load balancer. We were seeing entries in /var/log/auth: Sep 14 14:05:03 sitespect sshd[15897]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 14 14:05:06 sitespect sshd[15897]: Accepted password for ss_tcohen from 10.2.80.4 port 22720 ssh2 Sep 14 14:05:08 sitespect su[15927]: Successful su for root by root Sep 14 14:05:08 sitespect su[15927]: + pts/1 root:root Sep 15 00:44:13 sitespect sshd[21507]: Did not receive identification string from 10.2.80.4 Sep 15 00:46:23 sitespect sshd[21540]: Did not receive identification string from 10.2.80.4 Sep 15 00:46:23 sitespect sshd[21547]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 00:46:23 sitespect sshd[21547]: Invalid user zZyYxXwW from 10.2.80.4 Sep 15 00:46:23 sitespect sshd[21547]: Failed none for invalid user zZyYxXwW from 10.2.80.4 port 9469 ssh2 Sep 15 00:46:41 sitespect sshd[21551]: Bad protocol version identification 'id' from 10.2.80.4 Sep 15 11:00:02 sitespect sshd[27085]: Did not receive identification string from 10.2.80.4 Sep 15 11:03:09 sitespect sshd[27104]: Bad protocol version identification 'GET / HTTP/1.0' from 10.2.80.4 Sep 15 11:03:58 sitespect sshd[27107]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:04:04 sitespect sshd[27118]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:04:04 sitespect sshd[27118]: Invalid user operator from 10.2.80.4 Sep 15 11:04:04 sitespect sshd[27118]: Failed none for invalid user operator from 10.2.80.4 port 41078 ssh2 Sep 15 11:04:04 sitespect sshd[27118]: error: Could not get shadow information for NOUSER Sep 15 11:04:04 sitespect sshd[27118]: Failed password for invalid user operator from 10.2.80.4 port 41078 ssh2 Sep 15 11:04:05 sitespect sshd[27122]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:04:05 sitespect sshd[27122]: Failed password for root from 10.2.80.4 port 37616 ssh2 Sep 15 11:04:05 sitespect sshd[27128]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:04:05 sitespect sshd[27128]: Invalid user r00t from 10.2.80.4 Sep 15 11:04:05 sitespect sshd[27128]: Failed none for invalid user r00t from 10.2.80.4 port 42204 ssh2 Sep 15 11:04:05 sitespect sshd[27132]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:04:05 sitespect sshd[27132]: Invalid user glftpd from 10.2.80.4 Sep 15 11:04:05 sitespect sshd[27132]: Failed none for invalid user glftpd from 10.2.80.4 port 36461 ssh2 Sep 15 11:04:05 sitespect sshd[27132]: error: Could not get shadow information for NOUSER Sep 15 11:04:05 sitespect sshd[27132]: Failed password for invalid user glftpd from 10.2.80.4 port 36461 ssh2 Sep 15 11:04:36 sitespect sshd[27140]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:04:36 sitespect sshd[27140]: Failed password for root from 10.2.80.4 port 64060 ssh2 Sep 15 11:04:51 sitespect sshd[27147]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:04:51 sitespect sshd[27147]: Invalid user trans from 10.2.80.4 Sep 15 11:04:51 sitespect sshd[27147]: Failed none for invalid user trans from 10.2.80.4 port 9636 ssh2 Sep 15 11:04:51 sitespect sshd[27147]: error: Could not get shadow information for NOUSER Sep 15 11:04:51 sitespect sshd[27147]: Failed password for invalid user trans from 10.2.80.4 port 9636 ssh2 Sep 15 11:05:07 sitespect sshd[27165]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:05:07 sitespect sshd[27165]: Invalid user db2inst1 from 10.2.80.4 Sep 15 11:05:07 sitespect sshd[27165]: Failed none for invalid user db2inst1 from 10.2.80.4 port 43320 ssh2 Sep 15 11:05:07 sitespect sshd[27165]: error: Could not get shadow information for NOUSER Sep 15 11:05:07 sitespect sshd[27165]: Failed password for invalid user db2inst1 from 10.2.80.4 port 43320 ssh2 Sep 15 11:06:22 sitespect sshd[27180]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:06:22 sitespect sshd[27180]: Invalid user mpi from 10.2.80.4 Sep 15 11:06:22 sitespect sshd[27180]: Failed none for invalid user mpi from 10.2.80.4 port 35380 ssh2 Sep 15 11:06:44 sitespect sshd[27185]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:06:44 sitespect sshd[27185]: Invalid user debug from 10.2.80.4 Sep 15 11:06:44 sitespect sshd[27185]: Failed none for invalid user debug from 10.2.80.4 port 30919 ssh2 Sep 15 11:06:44 sitespect sshd[27185]: error: Could not get shadow information for NOUSER Sep 15 11:06:44 sitespect sshd[27185]: Failed password for invalid user debug from 10.2.80.4 port 30919 ssh2 Sep 15 11:07:38 sitespect sshd[27201]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:07:38 sitespect sshd[27201]: Invalid user oracle from 10.2.80.4 Sep 15 11:07:38 sitespect sshd[27201]: Failed none for invalid user oracle from 10.2.80.4 port 17898 ssh2 Sep 15 11:07:38 sitespect sshd[27201]: error: Could not get shadow information for NOUSER Sep 15 11:07:38 sitespect sshd[27201]: Failed password for invalid user oracle from 10.2.80.4 port 17898 ssh2 Sep 15 11:07:45 sitespect sshd[27207]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:07:45 sitespect sshd[27207]: Invalid user bash from 10.2.80.4 Sep 15 11:07:45 sitespect sshd[27207]: Failed none for invalid user bash from 10.2.80.4 port 56623 ssh2 Sep 15 11:07:48 sitespect sshd[27211]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:07:48 sitespect sshd[27211]: Invalid user friday from 10.2.80.4 Sep 15 11:07:48 sitespect sshd[27211]: Failed none for invalid user friday from 10.2.80.4 port 12306 ssh2 Sep 15 11:07:51 sitespect sshd[27215]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:07:51 sitespect sshd[27215]: Invalid user backdoor from 10.2.80.4 Sep 15 11:07:51 sitespect sshd[27215]: Failed none for invalid user backdoor from 10.2.80.4 port 52005 ssh2 Sep 15 11:08:06 sitespect sshd[27228]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:08:06 sitespect sshd[27228]: Invalid user tutor from 10.2.80.4 Sep 15 11:08:06 sitespect sshd[27228]: Failed none for invalid user tutor from 10.2.80.4 port 53532 ssh2 Sep 15 11:08:15 sitespect sshd[27232]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:08:15 sitespect sshd[27232]: Invalid user gamez from 10.2.80.4 Sep 15 11:08:15 sitespect sshd[27232]: Failed none for invalid user gamez from 10.2.80.4 port 53055 ssh2 Sep 15 11:08:15 sitespect sshd[27232]: error: Could not get shadow information for NOUSER Sep 15 11:08:15 sitespect sshd[27232]: Failed password for invalid user gamez from 10.2.80.4 port 53055 ssh2 Sep 15 11:08:23 sitespect sshd[27238]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:08:23 sitespect sshd[27238]: Invalid user toor from 10.2.80.4 Sep 15 11:08:23 sitespect sshd[27238]: Failed none for invalid user toor from 10.2.80.4 port 11276 ssh2 Sep 15 11:08:39 sitespect sshd[27242]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:08:39 sitespect sshd[27242]: Invalid user bank from 10.2.80.4 Sep 15 11:08:39 sitespect sshd[27242]: Failed none for invalid user bank from 10.2.80.4 port 3940 ssh2 Sep 15 11:08:39 sitespect sshd[27242]: error: Could not get shadow information for NOUSER Sep 15 11:08:39 sitespect sshd[27242]: Failed password for invalid user bank from 10.2.80.4 port 3940 ssh2 Sep 15 11:08:40 sitespect sshd[27248]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:08:40 sitespect sshd[27248]: Invalid user db2fenc1 from 10.2.80.4 Sep 15 11:08:40 sitespect sshd[27248]: Failed none for invalid user db2fenc1 from 10.2.80.4 port 43240 ssh2 Sep 15 11:08:40 sitespect sshd[27248]: error: Could not get shadow information for NOUSER Sep 15 11:08:40 sitespect sshd[27248]: Failed password for invalid user db2fenc1 from 10.2.80.4 port 43240 ssh2 Sep 15 11:09:01 sitespect sshd[27254]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:09:01 sitespect sshd[27254]: Failed password for root from 10.2.80.4 port 1621 ssh2 Sep 15 11:09:08 sitespect sshd[27265]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:09:08 sitespect sshd[27265]: Invalid user jill from 10.2.80.4 Sep 15 11:09:08 sitespect sshd[27265]: Failed none for invalid user jill from 10.2.80.4 port 58271 ssh2 Sep 15 11:09:09 sitespect sshd[27269]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:09:09 sitespect sshd[27269]: Invalid user guest from 10.2.80.4 Sep 15 11:09:09 sitespect sshd[27269]: Failed none for invalid user guest from 10.2.80.4 port 22728 ssh2 Sep 15 11:09:24 sitespect sshd[27273]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:09:24 sitespect sshd[27273]: Failed password for root from 10.2.80.4 port 53356 ssh2 Sep 15 11:09:27 sitespect sshd[27279]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:09:27 sitespect sshd[27279]: Invalid user **** from 10.2.80.4 Sep 15 11:09:27 sitespect sshd[27279]: Failed none for invalid user **** from 10.2.80.4 port 51755 ssh2 Sep 15 11:09:27 sitespect sshd[27279]: error: Could not get shadow information for NOUSER Sep 15 11:09:27 sitespect sshd[27279]: Failed password for invalid user **** from 10.2.80.4 port 51755 ssh2 Sep 15 11:09:46 sitespect sshd[27286]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:09:46 sitespect sshd[27286]: Invalid user db2as from 10.2.80.4 Sep 15 11:09:46 sitespect sshd[27286]: Failed none for invalid user db2as from 10.2.80.4 port 15096 ssh2 Sep 15 11:09:46 sitespect sshd[27286]: error: Could not get shadow information for NOUSER Sep 15 11:09:46 sitespect sshd[27286]: Failed password for invalid user db2as from 10.2.80.4 port 15096 ssh2 Sep 15 11:10:03 sitespect sshd[27317]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:10:03 sitespect sshd[27317]: Failed password for root from 10.2.80.4 port 6844 ssh2 Sep 15 11:10:03 sitespect sshd[27321]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:10:04 sitespect sshd[27321]: Failed password for root from 10.2.80.4 port 29346 ssh2 Sep 15 11:10:10 sitespect sshd[27329]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:10:10 sitespect sshd[27329]: Invalid user jack from 10.2.80.4 Sep 15 11:10:10 sitespect sshd[27329]: Failed none for invalid user jack from 10.2.80.4 port 47108 ssh2 Sep 15 11:10:17 sitespect sshd[27333]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:10:17 sitespect sshd[27333]: Invalid user sync from 10.2.80.4 Sep 15 11:10:17 sitespect sshd[27333]: Failed none for invalid user sync from 10.2.80.4 port 33023 ssh2 Sep 15 11:10:23 sitespect sshd[27337]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:10:23 sitespect sshd[27337]: Invalid user db2inst1 from 10.2.80.4 Sep 15 11:10:23 sitespect sshd[27337]: Failed none for invalid user db2inst1 from 10.2.80.4 port 7178 ssh2 Sep 15 11:10:23 sitespect sshd[27337]: error: Could not get shadow information for NOUSER Sep 15 11:10:23 sitespect sshd[27337]: Failed password for invalid user db2inst1 from 10.2.80.4 port 7178 ssh2 Sep 15 11:10:32 sitespect sshd[27344]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:10:32 sitespect sshd[27344]: Invalid user super from 10.2.80.4 Sep 15 11:10:32 sitespect sshd[27344]: Failed none for invalid user super from 10.2.80.4 port 55188 ssh2 Sep 15 11:10:32 sitespect sshd[27344]: error: Could not get shadow information for NOUSER Sep 15 11:10:32 sitespect sshd[27344]: Failed password for invalid user super from 10.2.80.4 port 55188 ssh2 Sep 15 11:10:35 sitespect sshd[27350]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:10:35 sitespect sshd[27350]: Invalid user demos from 10.2.80.4 Sep 15 11:10:35 sitespect sshd[27350]: Failed none for invalid user demos from 10.2.80.4 port 18766 ssh2 Sep 15 11:10:35 sitespect sshd[27354]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:10:35 sitespect sshd[27354]: Invalid user user from 10.2.80.4 Sep 15 11:10:35 sitespect sshd[27354]: Failed none for invalid user user from 10.2.80.4 port 31441 ssh2 Sep 15 11:10:35 sitespect sshd[27354]: error: Could not get shadow information for NOUSER Sep 15 11:10:35 sitespect sshd[27354]: Failed password for invalid user user from 10.2.80.4 port 31441 ssh2 Sep 15 11:10:37 sitespect sshd[27361]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:10:37 sitespect sshd[27361]: Invalid user db2fenc1 from 10.2.80.4 Sep 15 11:10:37 sitespect sshd[27361]: Failed none for invalid user db2fenc1 from 10.2.80.4 port 7097 ssh2 Sep 15 11:10:37 sitespect sshd[27361]: error: Could not get shadow information for NOUSER Sep 15 11:10:37 sitespect sshd[27361]: Failed password for invalid user db2fenc1 from 10.2.80.4 port 7097 ssh2 Sep 15 11:10:51 sitespect sshd[27367]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:10:51 sitespect sshd[27367]: Invalid user OutOfBox from 10.2.80.4 Sep 15 11:10:51 sitespect sshd[27367]: Failed none for invalid user OutOfBox from 10.2.80.4 port 6628 ssh2 Sep 15 11:10:55 sitespect sshd[27372]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:10:55 sitespect sshd[27372]: Invalid user 4Dgifts from 10.2.80.4 Sep 15 11:10:55 sitespect sshd[27372]: Failed none for invalid user 4Dgifts from 10.2.80.4 port 63830 ssh2 Sep 15 11:11:12 sitespect sshd[27379]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:11:12 sitespect sshd[27379]: Invalid user system from 10.2.80.4 Sep 15 11:11:12 sitespect sshd[27379]: Failed none for invalid user system from 10.2.80.4 port 36212 ssh2 Sep 15 11:11:12 sitespect sshd[27379]: error: Could not get shadow information for NOUSER Sep 15 11:11:12 sitespect sshd[27379]: Failed password for invalid user system from 10.2.80.4 port 36212 ssh2 Sep 15 11:11:13 sitespect sshd[27385]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:11:13 sitespect sshd[27385]: Invalid user admin from 10.2.80.4 Sep 15 11:11:13 sitespect sshd[27385]: Failed none for invalid user admin from 10.2.80.4 port 51149 ssh2 Sep 15 11:11:13 sitespect sshd[27385]: error: Could not get shadow information for NOUSER Sep 15 11:11:13 sitespect sshd[27385]: Failed password for invalid user admin from 10.2.80.4 port 51149 ssh2 Sep 15 11:11:13 sitespect sshd[27389]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:11:13 sitespect sshd[27389]: Invalid user rewt from 10.2.80.4 Sep 15 11:11:13 sitespect sshd[27389]: Failed none for invalid user rewt from 10.2.80.4 port 38616 ssh2 Sep 15 11:11:13 sitespect sshd[27389]: error: Could not get shadow information for NOUSER Sep 15 11:11:13 sitespect sshd[27389]: Failed password for invalid user rewt from 10.2.80.4 port 38616 ssh2 Sep 15 11:11:27 sitespect sshd[27397]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:11:27 sitespect sshd[27397]: Invalid user guest from 10.2.80.4 Sep 15 11:11:27 sitespect sshd[27397]: Failed none for invalid user guest from 10.2.80.4 port 63153 ssh2 Sep 15 11:11:27 sitespect sshd[27397]: error: Could not get shadow information for NOUSER Sep 15 11:11:27 sitespect sshd[27397]: Failed password for invalid user guest from 10.2.80.4 port 63153 ssh2 Sep 15 11:11:55 sitespect sshd[27403]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:11:55 sitespect sshd[27403]: Invalid user help from 10.2.80.4 Sep 15 11:11:55 sitespect sshd[27403]: Failed none for invalid user help from 10.2.80.4 port 55074 ssh2 Sep 15 11:12:04 sitespect sshd[27414]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:12:04 sitespect sshd[27414]: Invalid user admin from 10.2.80.4 Sep 15 11:12:04 sitespect sshd[27414]: Failed none for invalid user admin from 10.2.80.4 port 42262 ssh2 Sep 15 11:12:04 sitespect sshd[27414]: error: Could not get shadow information for NOUSER Sep 15 11:12:04 sitespect sshd[27414]: Failed password for invalid user admin from 10.2.80.4 port 42262 ssh2 Sep 15 11:12:12 sitespect sshd[27420]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:12:12 sitespect sshd[27420]: Failed password for root from 10.2.80.4 port 50906 ssh2 Sep 15 11:12:35 sitespect sshd[27426]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:12:35 sitespect sshd[27426]: Invalid user StoogR from 10.2.80.4 Sep 15 11:12:35 sitespect sshd[27426]: Failed none for invalid user StoogR from 10.2.80.4 port 55590 ssh2 Sep 15 11:12:35 sitespect sshd[27430]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:12:35 sitespect sshd[27430]: Failed password for root from 10.2.80.4 port 23186 ssh2 Sep 15 11:13:32 sitespect sshd[27439]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:13:32 sitespect sshd[27439]: Invalid user EZsetup from 10.2.80.4 Sep 15 11:13:32 sitespect sshd[27439]: Failed none for invalid user EZsetup from 10.2.80.4 port 44977 ssh2 Sep 15 11:13:32 sitespect sshd[27443]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:13:32 sitespect sshd[27443]: Invalid user lp from 10.2.80.4 Sep 15 11:13:32 sitespect sshd[27443]: Failed none for invalid user lp from 10.2.80.4 port 47464 ssh2 Sep 15 11:13:33 sitespect sshd[27447]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:13:33 sitespect sshd[27447]: Invalid user hax0r from 10.2.80.4 Sep 15 11:13:33 sitespect sshd[27447]: Failed none for invalid user hax0r from 10.2.80.4 port 16313 ssh2 Sep 15 11:13:35 sitespect sshd[27451]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:13:35 sitespect sshd[27451]: Invalid user user from 10.2.80.4 Sep 15 11:13:35 sitespect sshd[27451]: Failed none for invalid user user from 10.2.80.4 port 42617 ssh2 Sep 15 11:13:35 sitespect sshd[27451]: error: Could not get shadow information for NOUSER Sep 15 11:13:35 sitespect sshd[27451]: Failed password for invalid user user from 10.2.80.4 port 42617 ssh2 Sep 15 11:13:39 sitespect sshd[27460]: Protocol major versions differ for 10.2.80.4: SSH-2.0-OpenSSH_5.2 vs. SSH-9.9-NessusSSH_1.0 Sep 15 11:13:39 sitespect sshd[27463]: Protocol major versions differ for 10.2.80.4: SSH-2.0-OpenSSH_5.2 vs. SSH-1.33-NessusSSH_1.0 Sep 15 11:13:39 sitespect sshd[27466]: Protocol major versions differ for 10.2.80.4: SSH-2.0-OpenSSH_5.2 vs. SSH-1.5-NessusSSH_1.0 Sep 15 11:13:39 sitespect sshd[27477]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:13:39 sitespect sshd[27477]: Invalid user from 10.2.80.4 Sep 15 11:13:39 sitespect sshd[27477]: Failed none for invalid user from 10.2.80.4 port 55143 ssh2 Sep 15 11:13:39 sitespect sshd[27457]: Did not receive identification string from 10.2.80.4 Sep 15 11:14:41 sitespect sshd[27488]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:14:41 sitespect sshd[27488]: Invalid user informix from 10.2.80.4 Sep 15 11:14:41 sitespect sshd[27488]: Failed none for invalid user informix from 10.2.80.4 port 56713 ssh2 Sep 15 11:14:41 sitespect sshd[27488]: error: Could not get shadow information for NOUSER Sep 15 11:14:41 sitespect sshd[27488]: Failed password for invalid user informix from 10.2.80.4 port 56713 ssh2 Sep 15 11:14:42 sitespect sshd[27494]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:14:42 sitespect sshd[27494]: Failed password for root from 10.2.80.4 port 55527 ssh2 Sep 15 11:15:05 sitespect sshd[27515]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:15:05 sitespect sshd[27515]: Invalid user date from 10.2.80.4 Sep 15 11:15:05 sitespect sshd[27515]: Failed none for invalid user date from 10.2.80.4 port 11687 ssh2 Sep 15 11:15:19 sitespect sshd[27519]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:15:19 sitespect sshd[27519]: Failed password for root from 10.2.80.4 port 39043 ssh2 Sep 15 11:15:29 sitespect sshd[27525]: Protocol major versions differ for 10.2.80.4: SSH-2.0-OpenSSH_5.2 vs. SSH-9.9-NessusSSH_1.0 Sep 15 11:15:29 sitespect sshd[27528]: Protocol major versions differ for 10.2.80.4: SSH-2.0-OpenSSH_5.2 vs. SSH-1.33-NessusSSH_1.0 Sep 15 11:15:29 sitespect sshd[27531]: Protocol major versions differ for 10.2.80.4: SSH-2.0-OpenSSH_5.2 vs. SSH-1.5-NessusSSH_1.0 Sep 15 11:15:34 sitespect sshd[27534]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:15:34 sitespect sshd[27534]: Invalid user super from 10.2.80.4 Sep 15 11:15:34 sitespect sshd[27534]: Failed none for invalid user super from 10.2.80.4 port 39055 ssh2 Sep 15 11:15:34 sitespect sshd[27534]: error: Could not get shadow information for NOUSER Sep 15 11:15:34 sitespect sshd[27534]: Failed password for invalid user super from 10.2.80.4 port 39055 ssh2 Sep 15 11:15:35 sitespect sshd[27540]: Did not receive identification string from 10.2.80.4 Sep 15 11:15:47 sitespect sshd[27545]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:15:48 sitespect sshd[27549]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:15:48 sitespect sshd[27549]: Invalid user pam_ssh_user_enumeration.nasl from 10.2.80.4 Sep 15 11:16:12 sitespect sshd[27560]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:16:12 sitespect sshd[27560]: Invalid user swift from 10.2.80.4 Sep 15 11:16:12 sitespect sshd[27560]: Failed none for invalid user swift from 10.2.80.4 port 9052 ssh2 Sep 15 11:16:12 sitespect sshd[27560]: error: Could not get shadow information for NOUSER Sep 15 11:16:12 sitespect sshd[27560]: Failed password for invalid user swift from 10.2.80.4 port 9052 ssh2 Sep 15 11:16:15 sitespect sshd[27566]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:16:15 sitespect sshd[27566]: Invalid user db2as from 10.2.80.4 Sep 15 11:16:15 sitespect sshd[27566]: Failed none for invalid user db2as from 10.2.80.4 port 25858 ssh2 Sep 15 11:16:15 sitespect sshd[27566]: error: Could not get shadow information for NOUSER Sep 15 11:16:15 sitespect sshd[27566]: Failed password for invalid user db2as from 10.2.80.4 port 25858 ssh2 Sep 15 11:16:17 sitespect sshd[27572]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:16:17 sitespect sshd[27572]: Failed password for root from 10.2.80.4 port 26945 ssh2 Sep 15 11:16:25 sitespect sshd[27578]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:16:32 sitespect sshd[27582]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 11:16:32 sitespect sshd[27582]: Invalid user public from 10.2.80.4 Sep 15 11:16:32 sitespect sshd[27582]: Failed none for invalid user public from 10.2.80.4 port 28077 ssh2 Sep 15 11:16:32 sitespect sshd[27582]: error: Could not get shadow information for NOUSER Sep 15 11:16:32 sitespect sshd[27582]: Failed password for invalid user public from 10.2.80.4 port 28077 ssh2 Sep 15 11:16:36 sitespect sshd[27589]: Did not receive identification string from 10.2.80.4 Sep 15 11:19:26 sitespect sshd[27623]: Bad protocol version identification '`' from 10.2.80.4 Sep 15 11:19:26 sitespect sshd[27626]: Bad protocol version identification '`/bin/id` #' from 10.2.80.4 Sep 15 11:19:26 sitespect sshd[27629]: Bad protocol version identification '`/usr/bin/id` #' from 10.2.80.4 Sep 15 11:19:28 sitespect sshd[27633]: Did not receive identification string from 10.2.80.4 Sep 15 12:33:51 sitespect sshd[28280]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 12:33:54 sitespect sshd[28280]: Accepted password for ss_tcohen from 10.2.80.4 port 31888 ssh2 Sep 15 12:33:56 sitespect su[28298]: Successful su for root by root Sep 15 12:33:56 sitespect su[28298]: + pts/1 root:root Sep 15 12:40:13 sitespect sshd[28380]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 12:40:15 sitespect sshd[28380]: Accepted password for ss_tcohen from 10.2.80.4 port 57042 ssh2 Sep 15 12:40:17 sitespect su[28398]: Successful su for root by root Sep 15 12:40:17 sitespect su[28398]: + pts/1 root:root Sep 15 14:41:46 sitespect sshd[29454]: Bad protocol version identification 'GET / HTTP/1.1' from 10.2.80.4 Sep 15 14:41:49 sitespect sshd[29457]: Bad protocol version identification 'GET / HTTP/1.1' from 10.2.80.4 Sep 15 14:41:55 sitespect sshd[29460]: Bad protocol version identification '\026\003\001' from 10.2.80.4 Sep 15 14:41:55 sitespect sshd[29463]: Bad protocol version identification '\200R\001\003' from 10.2.80.4 Sep 15 14:41:56 sitespect sshd[29466]: Bad protocol version identification '\026\003\001' from 10.2.80.4 Sep 15 14:41:56 sitespect sshd[29469]: Bad protocol version identification '\200R\001\003' from 10.2.80.4 Sep 15 14:41:56 sitespect sshd[29472]: Bad protocol version identification '\026\003\001' from 10.2.80.4 Sep 15 14:41:56 sitespect sshd[29475]: Bad protocol version identification '\200R\001\003' from 10.2.80.4 Sep 15 14:41:56 sitespect sshd[29478]: Bad protocol version identification '\026\003\001' from 10.2.80.4 Sep 15 14:41:59 sitespect sshd[29481]: Bad protocol version identification '\200R\001\003' from 10.2.80.4 Sep 15 14:42:00 sitespect sshd[29484]: Bad protocol version identification '\026\003\001' from 10.2.80.4 Sep 15 14:42:00 sitespect sshd[29487]: Bad protocol version identification '\200R\001\003' from 10.2.80.4 Sep 15 14:42:19 sitespect sshd[29497]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 14:42:19 sitespect sshd[29497]: Invalid user reed from 10.2.80.4 Sep 15 14:42:19 sitespect sshd[29497]: Failed none for invalid user reed from 10.2.80.4 port 57711 ssh2 Sep 15 16:24:27 sitespect sshd[30392]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT! Sep 15 16:24:29 sitespect sshd[30392]: Accepted password for ss_tcohen from 10.2.80.4 port 10608 ssh2 Sep 15 16:24:31 sitespect su[30410]: Successful su for root by root Sep 15 16:24:31 sitespect su[30410]: + pts/1 root:root
Assignee | ||
Comment 50•15 years ago
|
||
Note that the ssh port is exposed through the netscaler, so this makes sense.
Comment 51•15 years ago
|
||
and that's definitely not the health checks. Are acl's in place yet? let's get this done asap.
Assignee | ||
Comment 52•15 years ago
|
||
add ns acl "sitespect-corp" ALLOW -srcIP = 72.85.233.34 -destIP = 63.245.209.10 -destPort = 9022 -protocol TCP -priority 1010 -kernelstate SFAPPLIED61 add ns acl "sitespect-frmtca1" ALLOW -srcIP = 64.71.151.28 -destIP = 63.245.209.10 -destPort = 9022 -protocol TCP -priority 1011 -kernelstate SFAPPLIED61 add ns acl "sitespect-bostma1" ALLOW -srcIP = 70.42.51.67 -destIP = 63.245.209.10 -destPort = 9022 -protocol TCP -priority 1012 -kernelstate SFAPPLIED61 add ns acl "sitespect-ogilvy" ALLOW -srcIP = 66.30.63.104 -destIP = 63.245.209.10 -destPort = 9022 -protocol TCP -priority 1013 -kernelstate SFAPPLIED61 add ns acl "sitespect-ssh-deny-all" DENY -destIP = 63.245.209.10 -destPort = 9022 -protocol TCP -priority 1014 -kernelstate SFAPPLIED61
Comment 53•15 years ago
|
||
Looks good. I'll keep an eye on the system and let you know if I continue to see anything suspicious.
Comment 54•15 years ago
|
||
Going back to the deployment questions: When do you plan to bring SiteSpect live, and when can I validate the rule sets? Tal
Reporter | ||
Comment 55•15 years ago
|
||
We would like to bring SiteSpect live next week.
Updated•15 years ago
|
Severity: minor → blocker
Assignee | ||
Comment 56•15 years ago
|
||
I think I have this set up correctly for mozilla.com on 63.245.209.112. Right now I only have 1 webhead + the sitespect box in rotation (mozcom-inside-vs-test) with RR load balancing. add cs policy "sitespect-cookieA" -rule "REQ.HTTP.HEADER Cookie CONTAINS SSLB=A" add cs vserver "cs-mozcom-test" HTTP 63.245.209.112 80 -cltTimeout 180 bind cs vserver "cs-mozcom-test" "mozcom-sitespect" -policyName "sitespect-cookieA" bind cs vserver "cs-mozcom-test" "mozcom-inside-vs-test"
Reporter | ||
Comment 57•15 years ago
|
||
Will it be possible to begin sending live traffic through SiteSpect tomorrow?
Comment 58•15 years ago
|
||
I'm working on validating the load balancer configuration and am finding some issues. 1. Please update the load balancer rule so that when SSLB=1 it sends the request to SiteSpect (instead of SSLB=A). 2. Add a rule so that if SSLB exists with any other value, the request does not route through SiteSpect. 3. When splitting the traffic to the non-SiteSpect route, the load balancer should set SSLB=0 as a session only cookie if no SSLB cookie exists (the path should be / and the domain should be .mozilla.com). Please let me know when these ruleset are in place and I'll re-test. Tal
Assignee | ||
Comment 59•15 years ago
|
||
I'm not sure that the load balancer can easily set a session cookie. Is this a big deal? Here are the new rules: add cs vserver "cs-mozcom-test" HTTP 63.245.209.112 80 -cltTimeout 180 bind cs vserver "cs-mozcom-test" "mozcom-sitespect" -policyName "sitespect-cookieA" bind cs vserver "cs-mozcom-test" "mozcom-inside-vs-test" bind cs vserver "cs-mozcom-test" "mozcom-inside-vs" -policyName "sitespect-cookie-nosend" add cs policy "sitespect-cookieA" -rule "REQ.HTTP.HEADER Cookie CONTAINS SSLB=1" add cs policy "sitespect-cookie-nosend" -rule "REQ.HTTP.HEADER Cookie CONTAINS SSLB && REQ.HTTP.HEADER Cookie NOTCONTAINS SSLB=1"
Comment 60•15 years ago
|
||
I'm not sure what's going on, but when I test it seems that at first things appear to work - I am able to see the split and also able to maintain affinity to or away from SiteSpect based on cookies. Then I go to retest and I can not direct any traffic to SiteSpect at all. No split, no cookie affinity. Can you take a look? Tal
Comment 61•15 years ago
|
||
oremj, engage edgecast?
Comment 62•15 years ago
|
||
Jeremy, mrz, and Tal -- can we resolved this by tomorrow (Tuesday) morning? We'd really like to start running our first test tomorrow afternoon. If it would be helpful to schedule a phone call, just say the word.
Assignee | ||
Comment 63•15 years ago
|
||
(In reply to comment #61) > oremj, engage edgecast? He should be using the testing IP which doesn't include edgecast. (In reply to comment #60) > I'm not sure what's going on, but when I test it seems that at first things > appear to work - I am able to see the split and also able to maintain affinity > to or away from SiteSpect based on cookies. Then I go to retest and I can not > direct any traffic to SiteSpect at all. No split, no cookie affinity. > When was the first test and when did you retest? If you retest now do you see sitespect cookie?
Comment 64•15 years ago
|
||
I just ran into this again - it is the weirdest thing! I sent 10 requests to http://63.245.209.112/en-US/?taltest, passing www. mozilla.com for the host header and User-Agent: M ozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729) for the user agent string. The load balancer appropriatly split the traffic 50% to SiteSpect. Then I sent 6 request, but this time included the SSLB=1 cookie. The load balancer correctly maintained affinity to SiteSpect. Then I sent 12 requests with the SSLB cookie = 0. Again, the load balancer correctly maintained affinity away from SiteSpect. Then I retried both the split (no cookies) and affinity to SiteSpect (SSLB=1) both failed - I never got to SiteSpect. Is it possible that there is some kind of Layer4 affinity going on?
Assignee | ||
Comment 65•15 years ago
|
||
Matthew, do you remember how we did this before? It appears the IC is hit before Content switches or Virtual Servers. Only thing I can think of is turning off caching completely for all users without a SSLB cookie.
Comment 66•15 years ago
|
||
Certainly looks that way: root@nslb01# nscachemgr -a | grep talt 0x00000003b9393672495b MOZ GET //www.mozilla.com:80/en-US/?taltest I found this in an email to Tal back on Sept 20 2007: *** I added a specific rule to exclude anything matching a host header that containts "www.mozilla.com" and a cookie that matches "SSLB=A" from caching. Tal Cohen wrote: Matthew, It looks like when users are going to www.mozilla.com <http://www.mozilla.com/>, even if their SSLB cookie = A, they are hitting the cache rather than SiteSpect. Please call me to discuss. *** That rule's still in place along with the =1 one: nslb01> sh run | grep SSLB | grep cache add cache policy "mozcom-sitespect-no" -rule "REQ.HTTP.HEADER Host CONTAINS www.mozilla.com && REQ.HTTP.HEADER Cookie CONTAINS SSLB=A" -action NOCACHE add cache policy "mozcom-sitespect-no2" -rule "REQ.HTTP.HEADER Host CONTAINS www.mozilla.com && REQ.HTTP.HEADER Cookie CONTAINS SSLB=1" -action NOCACHE Hitting: GET -dsSe http://63.245.209.112/en-US/\?taltest -H'Host: www.mozilla.com' -H 'Cookie: SSLB=1' -H 'User-Agent: Mozilla' Consistently hits the origin server.
Comment 67•15 years ago
|
||
Not getting any Set-Cookie headers back from your box - should I be? mrz@boris [~/] 52> curl -o /dev/null -s --dump-header - -H'Cookie: SSLB=A' -H'User-Agent: Mozilla' -H'Host: www.mozilla.com' http://10.2.80.232/en-US/about/legal.html HTTP/1.1 200 OK Date: Tue, 22 Sep 2009 03:14:18 GMT Server: Apache/2.2.3 (Red Hat) X-Powered-By: PHP/5.1.6 Cache-Control: private, no-store, no-cache, max-age=0, must-revalidate, proxy-revalidate Expires: Wed, 25 Mar 2009 17:54:01 GMT Content-Type: text/html; charset=UTF-8 Pragma: no-cache Vary: Accept-Encoding Transfer-Encoding: chunked
Comment 68•15 years ago
|
||
OK, so as long as there is no caching in front of SiteSpect when SSLB=1 that should work. Is it possible for your application/web server to set SSLB=0 as a session only cookie (path=/ domain=.mozilla.com)? SiteSpect will over write the cookie, so always setting it will prevent users from drifting into a test. Alternately, is there currently any session based affinity being done in the load balancer that would keep a user away from SiteSpect if they did not start their session with SiteSpect?
Comment 69•15 years ago
|
||
Matt - When testing, use SSLB=1 cookie and make sure to use a fully qualified user agent string. SiteSpect will treat partial user agent strings (like just "Mozilla") as bots and just pass them through. Here is the user agent string that I use for testing: "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)"
Comment 70•15 years ago
|
||
Heads up - I'm going to test failover. I will be bringing the system into and out of failover mode for the next 30 minutes or so. I'll let you know once I'm done.
Comment 71•15 years ago
|
||
I've completed my failover testing. The system passed with flying colors. At this time I've removed the failover condition.
Comment 72•15 years ago
|
||
So, from what I can see everyting looks good. The only open issue is to add a session cookie as I described in comment #68. This cookie is important to prevent users from drifting into a test mid-session.
Assignee | ||
Comment 73•15 years ago
|
||
Changed vserver to: add lb vserver "mozcom-inside-vs-test" HTTP 0.0.0.0 0 -persistenceType COOKIEINSERT -timeout 15 -lbMethod ROUNDROBIN -cltTimeout 180 Note "-persistenceType COOKIEINSERT -timeout 15" which the docs say will lock a user down to a service for 15 minutes in this case.
Comment 74•15 years ago
|
||
Can that cookie be set as session only?
Comment 75•15 years ago
|
||
If session only is possible that is recommended. However, based on my testing you should be able to launch with the current config (after adding additional web servers of course).
Assignee | ||
Comment 76•15 years ago
|
||
Unfortunately, session cookies aren't an option.
Comment 77•15 years ago
|
||
We didn't do session cookies last time - just different implementation now?
Comment 78•15 years ago
|
||
OK, keeping them at 15 minutes should be OK. Not so much a different implementation...a more robust implementation. Anyway, I'm OK with the current config. Let me know when you plan to send live traffic. Tal
Assignee | ||
Comment 79•15 years ago
|
||
We plan on going live tonight @ 6pm.
Assignee | ||
Comment 80•15 years ago
|
||
Production .com now has sitespect in the mix.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Comment 81•15 years ago
|
||
It looks like some of the caching rules did not make it into production. Some requests with the SSLB=1 cookie are not making it into SiteSpect. Can you verify that the SSLB=1 nocache rules are in place? Also, what is the percentage of traffic that will go to SiteSpect?
Comment 82•15 years ago
|
||
(In reply to comment #81) > It looks like some of the caching rules did not make it into production. > Some requests with the SSLB=1 cookie are not making it into SiteSpect. > Can you verify that the SSLB=1 nocache rules are in place? Do you have a test GET I can use to duplicate? Rules all look the same. > Also, what is the percentage of traffic that will go to SiteSpect? Tough to say. I'd say 1/5 but it's fronted by EdgeCast so you'll only see 1/5 of the cache misses. If you're not getting enough traffic we'll tweak EdgeCast. That reminds me though - the cookie changed in the middle of implementation and EdgeCast had configured their end to match on SSLB=A or SSLB=B. That changed to =1 & =0. I've asked EdgeCast to update their end.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment 83•15 years ago
|
||
Here is the GET that I'm doing: export COUNTER=1;while [ $COUNTER -le 100 ]; do echo && echo -n "$COUNTER ";GET -Ssed -H 'User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)' -H 'Host: www.mozilla.com' -H 'Cookie: SSLB=1' http://www.mozilla.com/en-US/?taltest=$COUNTER | grep -i ssid ; COUNTER=`expr $COUNTER + 1`; done Set SSLB=1 for SiteSpect, SSLB=0 for no SiteSpect, or remove the SSLB cookie to simulate a new request (traffic should get slpit). Due to the nature of the caching, you may need to run a few times before you see the failure. Let me know once EdgeCast has been updated with the SSLB=1/0 rules.
Comment 85•15 years ago
|
||
The changes that EdgeCast deployed look good. Here is what I'm seeing: 1. Requests with SSLB=1 always go to SiteSpect and bypass cache 2. Requests with SSLB=0 never go through SiteSpect 3. Requests with no SSLB cookie either hit the cache, or get split between SiteSpect and non-SiteSpect routes. The fact that they hit the cache instead of being split should be OK given your traffic levels. At this point I'm OK signing off on your deployment. You should feel free to start testing at any time.
Comment 86•15 years ago
|
||
Tal, Jeremy, and mrz -- thanks for all your work on this! we're excited to start testing!
Reporter | ||
Comment 87•15 years ago
|
||
A big second on the thanks! I have one last issue; I am no longer able to preview my experiment variations. After I click on the preview button (http://www.mozilla.com:9081/en-US/products/download.html?product=firefox-3.5.2&os=win&lang=en-US), I'm directed to https://sitespect.mozilla.com:9443/Variation_List which times out. Tal, any idea what's going on here? I assume it's user error.
Comment 88•15 years ago
|
||
You could be hitting the EdgeCast cache. Try setting a cookie, SSLB=1, and retry.
Reporter | ||
Comment 89•15 years ago
|
||
Doesn't seem to work. Here's the info for the cookie I set: Name: SSLB Content: 1 Domain: .mozilla.com Path: / Send For: Any type of connection Any other ideas?
Comment 90•15 years ago
|
||
Previews need to run through the front end IP on SiteSpect (10.2.80.232:80). Prior to going live there was a rule that directed requests to www.mozilla.com:9081 to SiteSpect. Is that rule still in place?
Comment 91•15 years ago
|
||
To clarify - the rule was on your NetScalar.
Assignee | ||
Comment 92•15 years ago
|
||
You aren't able to access the preview, because DNS for www.mozilla.com is pointing to edgecast. Adding the following to /etc/hosts should fix the problem: 63.245.209.10 www.mozilla.com
Status: REOPENED → RESOLVED
Closed: 15 years ago → 15 years ago
Resolution: --- → FIXED
Comment 93•15 years ago
|
||
I agree, that resolves the issue for me.
Reporter | ||
Comment 94•15 years ago
|
||
success! thanks Jeremy.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Reporter | ||
Updated•15 years ago
|
Status: REOPENED → RESOLVED
Closed: 15 years ago → 15 years ago
Resolution: --- → FIXED
Updated•11 years ago
|
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Updated•5 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•