Closed Bug 509591 Opened 15 years ago Closed 15 years ago

SiteSpect Implementation [ ref:00D7JfQw.50079No58:ref ]

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

All
Other
task
Not set
blocker

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bcutler, Assigned: oremj)

Details

Attachments

(1 file)

Tal Cohen, from SiteSpect, is helping us implement their service.  He currently needs:

"Your NTP server IP address. 
Smartrelay host name (if needed for outbound emails). 

In addition to this, our systems now come with a Dell Remote Administration Card (DRAC) that allows out of band management of the server. Think KVM over internet. I need 1 additional IP/Netmask/Gateway configuration for the DRAC card."

Instead of sending email back and forth, Matthew suggested we communicate over bugzilla.
um, please ignore the typo in the title.
I've created a Bugzilla account for SiteSpect per Matthew's suggestion.
Any progress on getting the information above?  We would like to have SiteSpect up and running my the end of August at the latest.
I think I'm supposed to answer this...

If you guys can do DHCP all of thise should come from DHCP.  You should be able to relay email through smtp.mozilla.org.
Our system requires static IP addresses. DHCP will  not work. Sorry.
Also, can you update the title/subject of this case to include the following text:

 [ ref:00D7JfQw.50079No58:ref ]

Otherwise our tracking system assigns a new case each time someone updates this bug.

Tal
Summary: SiteSpect Implementaiton → SiteSpect Implementation [ ref:00D7JfQw.50079No58:ref ]
We talked about security issues and how/where this would be best deployed.  Passing to Derek for network info.
Assignee: server-ops → dmoore
I've spoken with Blake, and we should have the necessary information over to him around mid-day tomorrow.
I still have not received the information. Can you please update this ticket with it?
NTP server: 63.245.208.5
DNS server: 63.245.208.5
SMTP relay: mail.mozilla.org

IP addresses: 63.245.208.80, 63.245.208.81
Netmask: 255.255.255.128
Gateway: 63.245.208.1
OK, the system is shipping today.

The FedEx tracking number is 797866973876.

You should expect the system to arrive on Tuesday.

I will send an email out with usernames and passwords directly to Blake and Matt.

We should schedule some configuration and validation time for the week of 8/31. let me know when is convenient for you.

Tal
Is this shipping to Mountain View or to the San Jose data center?
It is shipping to:

Matthew Zeier
Mozilla Corporation
55 South Market Street
San Jose, CA 95113
Assignee: dmoore → jeremy.orem+bugs
Derek, did you power on these servers? How do I access them?
Server is now online and accessible: sitespect.mozilla.com (10.2.80.230)

OOB management (DRAC): sitespect-drac.mozilla.com (10.2.80.233)
Should be ready for Blake to start configuring.
Blake, I just got back from vacation and want to continue your deployment of SiteSpect. Where you able to log into the control panel? When you have a few minutes, give me a call to review your current status. I can be reached at 617-859-1900 x 705 - Tal
I won't have access to the control panel until tomorrow.  Are you free for a call after 12pm tomorrow?
(In reply to comment #18)
> I won't have access to the control panel until tomorrow.  Are you free for a
> call after 12pm tomorrow?

12pm EST or 12pm PDT?
PDT
(In reply to comment #20)
> PDT

12 pm PDT / 3 pm EST works for me. Let me know if you want me to setup a phone bridge. Otherwise, give me a call at 617-859-1900 x 705.
VPN was restarted and I tested connectivity.
I'm still unable to access the control panel.  Jeremy, are you available to troubleshoot with Tal Cohen tomorrow?  He's available by phone from 9am-4pm.
Yeah, I'll be around.
Does a 10:30 call work for you, Jeremy?  Tal?
Yeah, works for me. Will you send out a meeting request for whatever time you decide on.
Jeremy, when ever you are ready, just call me at 617-859-1900 x705
Blake, you can now log into SiteSpect at https://sitespect.mozilla.com:9443. You may need the following local host entry:

62.27.48.226 sitespect.mozilla.com

Jeremy and I were able to work through most of the open issues. The only things that are left are Failover and Load Balancing. 

The issue with Failover is that when SiteSpect fails heartbeat checks, the load balancer is not bypassing SiteSpect. It just stops traffic. Jeremy is looking into this.

I will need to test out your load balancing rules prior to your going live. 

In the meantime you can access the system and start configuring test objects and test campaigns.
I think he meant this entry in /etc/hosts:

63.245.209.10 sitespect.mozilla.com
it works! thanks for looking into this Jeremy.  when do you expect to resolve the Failover and Load Balancing issues?
Failover is pretty easy, but I'm kind of confused on the load balancing part. We don't want to point all the traffic at the sitespect box? How much traffic should we send there?
Attached image SiteSpect Load Balancer Guide —
Attached is the SiteSpect Load Balancer Guide. It provides guidance and suggestions on how to configure your load balancer to function with SiteSpect.
I've attached a load balancer guide. The actual percentage of traffic to send through SiteSpect is not discussed, but I'd suggest determining a desired based on a combination of business needs, Net-Opps requirements, and system capacity. As a general rule, you want to start with a low percentage and ramp up over a period of time until the desired percentage has been reached (or until SiteSpect has reached 75% capacity).
Re /etc/hosts: Jeremy is correct Please use:

63.245.209.10		sitespect.mozilla.com
oremj - we did percentage based on weighted round robin last time.  The config is still there in the netscaler for this - look at cs-mozcom

IIRC, you'll need to bind mozcom-sitespect to cs-mozcom.  cs-mozcom would basically have two bound services - one is just sitespect and the other is the static cluster.

add cs vserver "cs-mozcom" HTTP 63.245.209.10 80 -cltTimeout 180
bind cs vserver "cs-mozcom" "mozcom-inside-vs"

add lb vserver "mozcom-inside-vs" HTTP 0.0.0.0 0 -persistenceType NONE -lbMethod ROUNDROBIN -cltTimeout 180
add lb vserver "mozcom-sitespect" HTTP 0.0.0.0 0 -persistenceType NONE -lbMethod LEASTRESPONSETIME -cltTimeout 180

bind lb vserver "mozcom-inside-vs" "svc-group-static"
bind lb vserver "mozcom-sitespect" "svc-sitespect-80-web"
Orem / Matthew,
   Any idea when you plan to deploy the load balancer rules?
Tal
They are still in place from the last time we did this - I guess I was lazy and never took them out.
So when will you start sending live traffic through SiteSpect?
We would like to begin sending traffic through SiteSpect early next week.
I would like to run a validation test of the load balancer configuration, including the splitting of traffic, affinity to SiteSpect, affinity away from SiteSpect, heart beating and failover. Is there a publicly accessible VIP on the load balancer that I can currently hit to test? If not I can run these tests once you turn on live traffic.
We would like to run tests on two additional Mozilla domains: addons.mozilla.org and support.mozilla.com.

What steps are required to make this happen?
The answer is more involved than what I can properly document in this system, and really justifies a phone call. Can we setup a 15 minute call with Blake and Orem tomorrow in the morning?
Er, yeah, what Tal says.  AMO's on a different load balancer, would have different engineering around it.
I'm free for a call anytime tomorrow.  How does a 2pm PDT call work?
2PM PDT works for me.
Here is the phone bridge dial in number: 

Dial 213-286-1202
Access Code: 526-783-859

Also, if needed, I do have a GotoMeeting session scheduled with this call. If we need to share desktops. The link is https://www2.gotomeeting.com/join/526783859

Tal
We have detected a number of attempts to break into the SiteSpect system. All of the attempts are comming from 10.2.80.4. If I'm not mistaken, that IP is one of your internal gateway devices (when I log in my IP address also appears to be 10.2.80.4). 

The system has not been compromised, but I'd like you to restrict access on 63.245.209.10 port 9022 to only allow connections from corp.sitespect.com, frmtca1-sm02.sitespect.com, bostma1-sm01.sitespect.com, onduk1-sm01.sitespect.com, and ogilvy.sitespect.com.

Thanks,

Tal
Define "break into" - 10.2.80.4 is the load balancer which is running health checks.
but by all means add these acls.  honestly quite surprised they aren't there by default...
Group: infra
I figured that 10.2.80.4 is the load balancer.

We were seeing entries in /var/log/auth:

Sep 14 14:05:03 sitespect sshd[15897]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 14 14:05:06 sitespect sshd[15897]: Accepted password for ss_tcohen from 10.2.80.4 port 22720 ssh2
Sep 14 14:05:08 sitespect su[15927]: Successful su for root by root
Sep 14 14:05:08 sitespect su[15927]: + pts/1 root:root
Sep 15 00:44:13 sitespect sshd[21507]: Did not receive identification string from 10.2.80.4
Sep 15 00:46:23 sitespect sshd[21540]: Did not receive identification string from 10.2.80.4
Sep 15 00:46:23 sitespect sshd[21547]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 00:46:23 sitespect sshd[21547]: Invalid user zZyYxXwW from 10.2.80.4
Sep 15 00:46:23 sitespect sshd[21547]: Failed none for invalid user zZyYxXwW from 10.2.80.4 port 9469 ssh2
Sep 15 00:46:41 sitespect sshd[21551]: Bad protocol version identification 'id' from 10.2.80.4
Sep 15 11:00:02 sitespect sshd[27085]: Did not receive identification string from 10.2.80.4
Sep 15 11:03:09 sitespect sshd[27104]: Bad protocol version identification 'GET / HTTP/1.0' from 10.2.80.4
Sep 15 11:03:58 sitespect sshd[27107]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:04:04 sitespect sshd[27118]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:04:04 sitespect sshd[27118]: Invalid user operator from 10.2.80.4
Sep 15 11:04:04 sitespect sshd[27118]: Failed none for invalid user operator from 10.2.80.4 port 41078 ssh2
Sep 15 11:04:04 sitespect sshd[27118]: error: Could not get shadow information for NOUSER
Sep 15 11:04:04 sitespect sshd[27118]: Failed password for invalid user operator from 10.2.80.4 port 41078 ssh2
Sep 15 11:04:05 sitespect sshd[27122]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:04:05 sitespect sshd[27122]: Failed password for root from 10.2.80.4 port 37616 ssh2
Sep 15 11:04:05 sitespect sshd[27128]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:04:05 sitespect sshd[27128]: Invalid user r00t from 10.2.80.4
Sep 15 11:04:05 sitespect sshd[27128]: Failed none for invalid user r00t from 10.2.80.4 port 42204 ssh2
Sep 15 11:04:05 sitespect sshd[27132]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:04:05 sitespect sshd[27132]: Invalid user glftpd from 10.2.80.4
Sep 15 11:04:05 sitespect sshd[27132]: Failed none for invalid user glftpd from 10.2.80.4 port 36461 ssh2
Sep 15 11:04:05 sitespect sshd[27132]: error: Could not get shadow information for NOUSER
Sep 15 11:04:05 sitespect sshd[27132]: Failed password for invalid user glftpd from 10.2.80.4 port 36461 ssh2
Sep 15 11:04:36 sitespect sshd[27140]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:04:36 sitespect sshd[27140]: Failed password for root from 10.2.80.4 port 64060 ssh2
Sep 15 11:04:51 sitespect sshd[27147]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:04:51 sitespect sshd[27147]: Invalid user trans from 10.2.80.4
Sep 15 11:04:51 sitespect sshd[27147]: Failed none for invalid user trans from 10.2.80.4 port 9636 ssh2
Sep 15 11:04:51 sitespect sshd[27147]: error: Could not get shadow information for NOUSER
Sep 15 11:04:51 sitespect sshd[27147]: Failed password for invalid user trans from 10.2.80.4 port 9636 ssh2
Sep 15 11:05:07 sitespect sshd[27165]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:05:07 sitespect sshd[27165]: Invalid user db2inst1 from 10.2.80.4
Sep 15 11:05:07 sitespect sshd[27165]: Failed none for invalid user db2inst1 from 10.2.80.4 port 43320 ssh2
Sep 15 11:05:07 sitespect sshd[27165]: error: Could not get shadow information for NOUSER
Sep 15 11:05:07 sitespect sshd[27165]: Failed password for invalid user db2inst1 from 10.2.80.4 port 43320 ssh2
Sep 15 11:06:22 sitespect sshd[27180]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:06:22 sitespect sshd[27180]: Invalid user mpi from 10.2.80.4
Sep 15 11:06:22 sitespect sshd[27180]: Failed none for invalid user mpi from 10.2.80.4 port 35380 ssh2
Sep 15 11:06:44 sitespect sshd[27185]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:06:44 sitespect sshd[27185]: Invalid user debug from 10.2.80.4
Sep 15 11:06:44 sitespect sshd[27185]: Failed none for invalid user debug from 10.2.80.4 port 30919 ssh2
Sep 15 11:06:44 sitespect sshd[27185]: error: Could not get shadow information for NOUSER
Sep 15 11:06:44 sitespect sshd[27185]: Failed password for invalid user debug from 10.2.80.4 port 30919 ssh2
Sep 15 11:07:38 sitespect sshd[27201]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:07:38 sitespect sshd[27201]: Invalid user oracle from 10.2.80.4
Sep 15 11:07:38 sitespect sshd[27201]: Failed none for invalid user oracle from 10.2.80.4 port 17898 ssh2
Sep 15 11:07:38 sitespect sshd[27201]: error: Could not get shadow information for NOUSER
Sep 15 11:07:38 sitespect sshd[27201]: Failed password for invalid user oracle from 10.2.80.4 port 17898 ssh2
Sep 15 11:07:45 sitespect sshd[27207]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:07:45 sitespect sshd[27207]: Invalid user bash from 10.2.80.4
Sep 15 11:07:45 sitespect sshd[27207]: Failed none for invalid user bash from 10.2.80.4 port 56623 ssh2
Sep 15 11:07:48 sitespect sshd[27211]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:07:48 sitespect sshd[27211]: Invalid user friday from 10.2.80.4
Sep 15 11:07:48 sitespect sshd[27211]: Failed none for invalid user friday from 10.2.80.4 port 12306 ssh2
Sep 15 11:07:51 sitespect sshd[27215]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:07:51 sitespect sshd[27215]: Invalid user backdoor from 10.2.80.4
Sep 15 11:07:51 sitespect sshd[27215]: Failed none for invalid user backdoor from 10.2.80.4 port 52005 ssh2
Sep 15 11:08:06 sitespect sshd[27228]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:08:06 sitespect sshd[27228]: Invalid user tutor from 10.2.80.4
Sep 15 11:08:06 sitespect sshd[27228]: Failed none for invalid user tutor from 10.2.80.4 port 53532 ssh2
Sep 15 11:08:15 sitespect sshd[27232]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:08:15 sitespect sshd[27232]: Invalid user gamez from 10.2.80.4
Sep 15 11:08:15 sitespect sshd[27232]: Failed none for invalid user gamez from 10.2.80.4 port 53055 ssh2
Sep 15 11:08:15 sitespect sshd[27232]: error: Could not get shadow information for NOUSER
Sep 15 11:08:15 sitespect sshd[27232]: Failed password for invalid user gamez from 10.2.80.4 port 53055 ssh2
Sep 15 11:08:23 sitespect sshd[27238]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:08:23 sitespect sshd[27238]: Invalid user toor from 10.2.80.4
Sep 15 11:08:23 sitespect sshd[27238]: Failed none for invalid user toor from 10.2.80.4 port 11276 ssh2
Sep 15 11:08:39 sitespect sshd[27242]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:08:39 sitespect sshd[27242]: Invalid user bank from 10.2.80.4
Sep 15 11:08:39 sitespect sshd[27242]: Failed none for invalid user bank from 10.2.80.4 port 3940 ssh2
Sep 15 11:08:39 sitespect sshd[27242]: error: Could not get shadow information for NOUSER
Sep 15 11:08:39 sitespect sshd[27242]: Failed password for invalid user bank from 10.2.80.4 port 3940 ssh2
Sep 15 11:08:40 sitespect sshd[27248]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:08:40 sitespect sshd[27248]: Invalid user db2fenc1 from 10.2.80.4
Sep 15 11:08:40 sitespect sshd[27248]: Failed none for invalid user db2fenc1 from 10.2.80.4 port 43240 ssh2
Sep 15 11:08:40 sitespect sshd[27248]: error: Could not get shadow information for NOUSER
Sep 15 11:08:40 sitespect sshd[27248]: Failed password for invalid user db2fenc1 from 10.2.80.4 port 43240 ssh2
Sep 15 11:09:01 sitespect sshd[27254]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:09:01 sitespect sshd[27254]: Failed password for root from 10.2.80.4 port 1621 ssh2
Sep 15 11:09:08 sitespect sshd[27265]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:09:08 sitespect sshd[27265]: Invalid user jill from 10.2.80.4
Sep 15 11:09:08 sitespect sshd[27265]: Failed none for invalid user jill from 10.2.80.4 port 58271 ssh2
Sep 15 11:09:09 sitespect sshd[27269]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:09:09 sitespect sshd[27269]: Invalid user guest from 10.2.80.4
Sep 15 11:09:09 sitespect sshd[27269]: Failed none for invalid user guest from 10.2.80.4 port 22728 ssh2
Sep 15 11:09:24 sitespect sshd[27273]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:09:24 sitespect sshd[27273]: Failed password for root from 10.2.80.4 port 53356 ssh2
Sep 15 11:09:27 sitespect sshd[27279]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:09:27 sitespect sshd[27279]: Invalid user **** from 10.2.80.4
Sep 15 11:09:27 sitespect sshd[27279]: Failed none for invalid user **** from 10.2.80.4 port 51755 ssh2
Sep 15 11:09:27 sitespect sshd[27279]: error: Could not get shadow information for NOUSER
Sep 15 11:09:27 sitespect sshd[27279]: Failed password for invalid user **** from 10.2.80.4 port 51755 ssh2
Sep 15 11:09:46 sitespect sshd[27286]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:09:46 sitespect sshd[27286]: Invalid user db2as from 10.2.80.4
Sep 15 11:09:46 sitespect sshd[27286]: Failed none for invalid user db2as from 10.2.80.4 port 15096 ssh2
Sep 15 11:09:46 sitespect sshd[27286]: error: Could not get shadow information for NOUSER
Sep 15 11:09:46 sitespect sshd[27286]: Failed password for invalid user db2as from 10.2.80.4 port 15096 ssh2
Sep 15 11:10:03 sitespect sshd[27317]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:10:03 sitespect sshd[27317]: Failed password for root from 10.2.80.4 port 6844 ssh2
Sep 15 11:10:03 sitespect sshd[27321]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:10:04 sitespect sshd[27321]: Failed password for root from 10.2.80.4 port 29346 ssh2
Sep 15 11:10:10 sitespect sshd[27329]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:10:10 sitespect sshd[27329]: Invalid user jack from 10.2.80.4
Sep 15 11:10:10 sitespect sshd[27329]: Failed none for invalid user jack from 10.2.80.4 port 47108 ssh2
Sep 15 11:10:17 sitespect sshd[27333]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:10:17 sitespect sshd[27333]: Invalid user sync from 10.2.80.4
Sep 15 11:10:17 sitespect sshd[27333]: Failed none for invalid user sync from 10.2.80.4 port 33023 ssh2
Sep 15 11:10:23 sitespect sshd[27337]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:10:23 sitespect sshd[27337]: Invalid user db2inst1 from 10.2.80.4
Sep 15 11:10:23 sitespect sshd[27337]: Failed none for invalid user db2inst1 from 10.2.80.4 port 7178 ssh2
Sep 15 11:10:23 sitespect sshd[27337]: error: Could not get shadow information for NOUSER
Sep 15 11:10:23 sitespect sshd[27337]: Failed password for invalid user db2inst1 from 10.2.80.4 port 7178 ssh2
Sep 15 11:10:32 sitespect sshd[27344]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:10:32 sitespect sshd[27344]: Invalid user super from 10.2.80.4
Sep 15 11:10:32 sitespect sshd[27344]: Failed none for invalid user super from 10.2.80.4 port 55188 ssh2
Sep 15 11:10:32 sitespect sshd[27344]: error: Could not get shadow information for NOUSER
Sep 15 11:10:32 sitespect sshd[27344]: Failed password for invalid user super from 10.2.80.4 port 55188 ssh2
Sep 15 11:10:35 sitespect sshd[27350]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:10:35 sitespect sshd[27350]: Invalid user demos from 10.2.80.4
Sep 15 11:10:35 sitespect sshd[27350]: Failed none for invalid user demos from 10.2.80.4 port 18766 ssh2
Sep 15 11:10:35 sitespect sshd[27354]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:10:35 sitespect sshd[27354]: Invalid user user from 10.2.80.4
Sep 15 11:10:35 sitespect sshd[27354]: Failed none for invalid user user from 10.2.80.4 port 31441 ssh2
Sep 15 11:10:35 sitespect sshd[27354]: error: Could not get shadow information for NOUSER
Sep 15 11:10:35 sitespect sshd[27354]: Failed password for invalid user user from 10.2.80.4 port 31441 ssh2
Sep 15 11:10:37 sitespect sshd[27361]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:10:37 sitespect sshd[27361]: Invalid user db2fenc1 from 10.2.80.4
Sep 15 11:10:37 sitespect sshd[27361]: Failed none for invalid user db2fenc1 from 10.2.80.4 port 7097 ssh2
Sep 15 11:10:37 sitespect sshd[27361]: error: Could not get shadow information for NOUSER
Sep 15 11:10:37 sitespect sshd[27361]: Failed password for invalid user db2fenc1 from 10.2.80.4 port 7097 ssh2
Sep 15 11:10:51 sitespect sshd[27367]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:10:51 sitespect sshd[27367]: Invalid user OutOfBox from 10.2.80.4
Sep 15 11:10:51 sitespect sshd[27367]: Failed none for invalid user OutOfBox from 10.2.80.4 port 6628 ssh2
Sep 15 11:10:55 sitespect sshd[27372]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:10:55 sitespect sshd[27372]: Invalid user 4Dgifts from 10.2.80.4
Sep 15 11:10:55 sitespect sshd[27372]: Failed none for invalid user 4Dgifts from 10.2.80.4 port 63830 ssh2
Sep 15 11:11:12 sitespect sshd[27379]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:11:12 sitespect sshd[27379]: Invalid user system from 10.2.80.4
Sep 15 11:11:12 sitespect sshd[27379]: Failed none for invalid user system from 10.2.80.4 port 36212 ssh2
Sep 15 11:11:12 sitespect sshd[27379]: error: Could not get shadow information for NOUSER
Sep 15 11:11:12 sitespect sshd[27379]: Failed password for invalid user system from 10.2.80.4 port 36212 ssh2
Sep 15 11:11:13 sitespect sshd[27385]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:11:13 sitespect sshd[27385]: Invalid user admin from 10.2.80.4
Sep 15 11:11:13 sitespect sshd[27385]: Failed none for invalid user admin from 10.2.80.4 port 51149 ssh2
Sep 15 11:11:13 sitespect sshd[27385]: error: Could not get shadow information for NOUSER
Sep 15 11:11:13 sitespect sshd[27385]: Failed password for invalid user admin from 10.2.80.4 port 51149 ssh2
Sep 15 11:11:13 sitespect sshd[27389]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:11:13 sitespect sshd[27389]: Invalid user rewt from 10.2.80.4
Sep 15 11:11:13 sitespect sshd[27389]: Failed none for invalid user rewt from 10.2.80.4 port 38616 ssh2
Sep 15 11:11:13 sitespect sshd[27389]: error: Could not get shadow information for NOUSER
Sep 15 11:11:13 sitespect sshd[27389]: Failed password for invalid user rewt from 10.2.80.4 port 38616 ssh2
Sep 15 11:11:27 sitespect sshd[27397]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:11:27 sitespect sshd[27397]: Invalid user guest from 10.2.80.4
Sep 15 11:11:27 sitespect sshd[27397]: Failed none for invalid user guest from 10.2.80.4 port 63153 ssh2
Sep 15 11:11:27 sitespect sshd[27397]: error: Could not get shadow information for NOUSER
Sep 15 11:11:27 sitespect sshd[27397]: Failed password for invalid user guest from 10.2.80.4 port 63153 ssh2
Sep 15 11:11:55 sitespect sshd[27403]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:11:55 sitespect sshd[27403]: Invalid user help from 10.2.80.4
Sep 15 11:11:55 sitespect sshd[27403]: Failed none for invalid user help from 10.2.80.4 port 55074 ssh2
Sep 15 11:12:04 sitespect sshd[27414]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:12:04 sitespect sshd[27414]: Invalid user admin from 10.2.80.4
Sep 15 11:12:04 sitespect sshd[27414]: Failed none for invalid user admin from 10.2.80.4 port 42262 ssh2
Sep 15 11:12:04 sitespect sshd[27414]: error: Could not get shadow information for NOUSER
Sep 15 11:12:04 sitespect sshd[27414]: Failed password for invalid user admin from 10.2.80.4 port 42262 ssh2
Sep 15 11:12:12 sitespect sshd[27420]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:12:12 sitespect sshd[27420]: Failed password for root from 10.2.80.4 port 50906 ssh2
Sep 15 11:12:35 sitespect sshd[27426]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:12:35 sitespect sshd[27426]: Invalid user StoogR from 10.2.80.4
Sep 15 11:12:35 sitespect sshd[27426]: Failed none for invalid user StoogR from 10.2.80.4 port 55590 ssh2
Sep 15 11:12:35 sitespect sshd[27430]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:12:35 sitespect sshd[27430]: Failed password for root from 10.2.80.4 port 23186 ssh2
Sep 15 11:13:32 sitespect sshd[27439]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:13:32 sitespect sshd[27439]: Invalid user EZsetup from 10.2.80.4
Sep 15 11:13:32 sitespect sshd[27439]: Failed none for invalid user EZsetup from 10.2.80.4 port 44977 ssh2
Sep 15 11:13:32 sitespect sshd[27443]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:13:32 sitespect sshd[27443]: Invalid user lp from 10.2.80.4
Sep 15 11:13:32 sitespect sshd[27443]: Failed none for invalid user lp from 10.2.80.4 port 47464 ssh2
Sep 15 11:13:33 sitespect sshd[27447]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:13:33 sitespect sshd[27447]: Invalid user hax0r from 10.2.80.4
Sep 15 11:13:33 sitespect sshd[27447]: Failed none for invalid user hax0r from 10.2.80.4 port 16313 ssh2
Sep 15 11:13:35 sitespect sshd[27451]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:13:35 sitespect sshd[27451]: Invalid user user from 10.2.80.4
Sep 15 11:13:35 sitespect sshd[27451]: Failed none for invalid user user from 10.2.80.4 port 42617 ssh2
Sep 15 11:13:35 sitespect sshd[27451]: error: Could not get shadow information for NOUSER
Sep 15 11:13:35 sitespect sshd[27451]: Failed password for invalid user user from 10.2.80.4 port 42617 ssh2
Sep 15 11:13:39 sitespect sshd[27460]: Protocol major versions differ for 10.2.80.4: SSH-2.0-OpenSSH_5.2 vs. SSH-9.9-NessusSSH_1.0
Sep 15 11:13:39 sitespect sshd[27463]: Protocol major versions differ for 10.2.80.4: SSH-2.0-OpenSSH_5.2 vs. SSH-1.33-NessusSSH_1.0
Sep 15 11:13:39 sitespect sshd[27466]: Protocol major versions differ for 10.2.80.4: SSH-2.0-OpenSSH_5.2 vs. SSH-1.5-NessusSSH_1.0
Sep 15 11:13:39 sitespect sshd[27477]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:13:39 sitespect sshd[27477]: Invalid user  from 10.2.80.4
Sep 15 11:13:39 sitespect sshd[27477]: Failed none for invalid user  from 10.2.80.4 port 55143 ssh2
Sep 15 11:13:39 sitespect sshd[27457]: Did not receive identification string from 10.2.80.4
Sep 15 11:14:41 sitespect sshd[27488]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:14:41 sitespect sshd[27488]: Invalid user informix from 10.2.80.4
Sep 15 11:14:41 sitespect sshd[27488]: Failed none for invalid user informix from 10.2.80.4 port 56713 ssh2
Sep 15 11:14:41 sitespect sshd[27488]: error: Could not get shadow information for NOUSER
Sep 15 11:14:41 sitespect sshd[27488]: Failed password for invalid user informix from 10.2.80.4 port 56713 ssh2
Sep 15 11:14:42 sitespect sshd[27494]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:14:42 sitespect sshd[27494]: Failed password for root from 10.2.80.4 port 55527 ssh2
Sep 15 11:15:05 sitespect sshd[27515]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:15:05 sitespect sshd[27515]: Invalid user date from 10.2.80.4
Sep 15 11:15:05 sitespect sshd[27515]: Failed none for invalid user date from 10.2.80.4 port 11687 ssh2
Sep 15 11:15:19 sitespect sshd[27519]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:15:19 sitespect sshd[27519]: Failed password for root from 10.2.80.4 port 39043 ssh2
Sep 15 11:15:29 sitespect sshd[27525]: Protocol major versions differ for 10.2.80.4: SSH-2.0-OpenSSH_5.2 vs. SSH-9.9-NessusSSH_1.0
Sep 15 11:15:29 sitespect sshd[27528]: Protocol major versions differ for 10.2.80.4: SSH-2.0-OpenSSH_5.2 vs. SSH-1.33-NessusSSH_1.0
Sep 15 11:15:29 sitespect sshd[27531]: Protocol major versions differ for 10.2.80.4: SSH-2.0-OpenSSH_5.2 vs. SSH-1.5-NessusSSH_1.0
Sep 15 11:15:34 sitespect sshd[27534]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:15:34 sitespect sshd[27534]: Invalid user super from 10.2.80.4
Sep 15 11:15:34 sitespect sshd[27534]: Failed none for invalid user super from 10.2.80.4 port 39055 ssh2
Sep 15 11:15:34 sitespect sshd[27534]: error: Could not get shadow information for NOUSER
Sep 15 11:15:34 sitespect sshd[27534]: Failed password for invalid user super from 10.2.80.4 port 39055 ssh2
Sep 15 11:15:35 sitespect sshd[27540]: Did not receive identification string from 10.2.80.4
Sep 15 11:15:47 sitespect sshd[27545]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:15:48 sitespect sshd[27549]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:15:48 sitespect sshd[27549]: Invalid user pam_ssh_user_enumeration.nasl from 10.2.80.4
Sep 15 11:16:12 sitespect sshd[27560]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:16:12 sitespect sshd[27560]: Invalid user swift from 10.2.80.4
Sep 15 11:16:12 sitespect sshd[27560]: Failed none for invalid user swift from 10.2.80.4 port 9052 ssh2
Sep 15 11:16:12 sitespect sshd[27560]: error: Could not get shadow information for NOUSER
Sep 15 11:16:12 sitespect sshd[27560]: Failed password for invalid user swift from 10.2.80.4 port 9052 ssh2
Sep 15 11:16:15 sitespect sshd[27566]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:16:15 sitespect sshd[27566]: Invalid user db2as from 10.2.80.4
Sep 15 11:16:15 sitespect sshd[27566]: Failed none for invalid user db2as from 10.2.80.4 port 25858 ssh2
Sep 15 11:16:15 sitespect sshd[27566]: error: Could not get shadow information for NOUSER
Sep 15 11:16:15 sitespect sshd[27566]: Failed password for invalid user db2as from 10.2.80.4 port 25858 ssh2
Sep 15 11:16:17 sitespect sshd[27572]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:16:17 sitespect sshd[27572]: Failed password for root from 10.2.80.4 port 26945 ssh2
Sep 15 11:16:25 sitespect sshd[27578]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:16:32 sitespect sshd[27582]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 11:16:32 sitespect sshd[27582]: Invalid user public from 10.2.80.4
Sep 15 11:16:32 sitespect sshd[27582]: Failed none for invalid user public from 10.2.80.4 port 28077 ssh2
Sep 15 11:16:32 sitespect sshd[27582]: error: Could not get shadow information for NOUSER
Sep 15 11:16:32 sitespect sshd[27582]: Failed password for invalid user public from 10.2.80.4 port 28077 ssh2
Sep 15 11:16:36 sitespect sshd[27589]: Did not receive identification string from 10.2.80.4
Sep 15 11:19:26 sitespect sshd[27623]: Bad protocol version identification '`' from 10.2.80.4
Sep 15 11:19:26 sitespect sshd[27626]: Bad protocol version identification '`/bin/id` #' from 10.2.80.4
Sep 15 11:19:26 sitespect sshd[27629]: Bad protocol version identification '`/usr/bin/id` #' from 10.2.80.4
Sep 15 11:19:28 sitespect sshd[27633]: Did not receive identification string from 10.2.80.4
Sep 15 12:33:51 sitespect sshd[28280]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 12:33:54 sitespect sshd[28280]: Accepted password for ss_tcohen from 10.2.80.4 port 31888 ssh2
Sep 15 12:33:56 sitespect su[28298]: Successful su for root by root
Sep 15 12:33:56 sitespect su[28298]: + pts/1 root:root
Sep 15 12:40:13 sitespect sshd[28380]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 12:40:15 sitespect sshd[28380]: Accepted password for ss_tcohen from 10.2.80.4 port 57042 ssh2
Sep 15 12:40:17 sitespect su[28398]: Successful su for root by root
Sep 15 12:40:17 sitespect su[28398]: + pts/1 root:root
Sep 15 14:41:46 sitespect sshd[29454]: Bad protocol version identification 'GET / HTTP/1.1' from 10.2.80.4
Sep 15 14:41:49 sitespect sshd[29457]: Bad protocol version identification 'GET / HTTP/1.1' from 10.2.80.4
Sep 15 14:41:55 sitespect sshd[29460]: Bad protocol version identification '\026\003\001' from 10.2.80.4
Sep 15 14:41:55 sitespect sshd[29463]: Bad protocol version identification '\200R\001\003' from 10.2.80.4
Sep 15 14:41:56 sitespect sshd[29466]: Bad protocol version identification '\026\003\001' from 10.2.80.4
Sep 15 14:41:56 sitespect sshd[29469]: Bad protocol version identification '\200R\001\003' from 10.2.80.4
Sep 15 14:41:56 sitespect sshd[29472]: Bad protocol version identification '\026\003\001' from 10.2.80.4
Sep 15 14:41:56 sitespect sshd[29475]: Bad protocol version identification '\200R\001\003' from 10.2.80.4
Sep 15 14:41:56 sitespect sshd[29478]: Bad protocol version identification '\026\003\001' from 10.2.80.4
Sep 15 14:41:59 sitespect sshd[29481]: Bad protocol version identification '\200R\001\003' from 10.2.80.4
Sep 15 14:42:00 sitespect sshd[29484]: Bad protocol version identification '\026\003\001' from 10.2.80.4
Sep 15 14:42:00 sitespect sshd[29487]: Bad protocol version identification '\200R\001\003' from 10.2.80.4
Sep 15 14:42:19 sitespect sshd[29497]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 14:42:19 sitespect sshd[29497]: Invalid user reed from 10.2.80.4
Sep 15 14:42:19 sitespect sshd[29497]: Failed none for invalid user reed from 10.2.80.4 port 57711 ssh2
Sep 15 16:24:27 sitespect sshd[30392]: reverse mapping checking getaddrinfo for v80-nslb.mozilla.com [10.2.80.4] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 15 16:24:29 sitespect sshd[30392]: Accepted password for ss_tcohen from 10.2.80.4 port 10608 ssh2
Sep 15 16:24:31 sitespect su[30410]: Successful su for root by root
Sep 15 16:24:31 sitespect su[30410]: + pts/1 root:root
Note that the ssh port is exposed through the netscaler, so this makes sense.
and that's definitely not the health checks.  Are acl's in place yet?  let's get this done asap.
add ns acl "sitespect-corp" ALLOW -srcIP = 72.85.233.34 -destIP = 63.245.209.10 -destPort = 9022 -protocol TCP -priority 1010 -kernelstate SFAPPLIED61
add ns acl "sitespect-frmtca1" ALLOW -srcIP = 64.71.151.28 -destIP = 63.245.209.10 -destPort = 9022 -protocol TCP -priority 1011 -kernelstate SFAPPLIED61
add ns acl "sitespect-bostma1" ALLOW -srcIP = 70.42.51.67 -destIP = 63.245.209.10 -destPort = 9022 -protocol TCP -priority 1012 -kernelstate SFAPPLIED61
add ns acl "sitespect-ogilvy" ALLOW -srcIP = 66.30.63.104 -destIP = 63.245.209.10 -destPort = 9022 -protocol TCP -priority 1013 -kernelstate SFAPPLIED61
add ns acl "sitespect-ssh-deny-all" DENY -destIP = 63.245.209.10 -destPort = 9022 -protocol TCP -priority 1014 -kernelstate SFAPPLIED61
Looks good. I'll keep an eye on the system and let you know if I continue to see anything suspicious.
Going back to the deployment questions: When do you plan to bring SiteSpect live, and when can I validate the rule sets?

Tal
We would like to bring SiteSpect live next week.
Severity: minor → blocker
I think I have this set up correctly for mozilla.com on 63.245.209.112. Right now I only have 1 webhead + the sitespect box in rotation (mozcom-inside-vs-test) with RR load balancing.

add cs policy "sitespect-cookieA" -rule "REQ.HTTP.HEADER Cookie CONTAINS SSLB=A"
add cs vserver "cs-mozcom-test" HTTP 63.245.209.112 80 -cltTimeout 180
bind cs vserver "cs-mozcom-test" "mozcom-sitespect" -policyName "sitespect-cookieA"
bind cs vserver "cs-mozcom-test" "mozcom-inside-vs-test"
Will it be possible to begin sending live traffic through SiteSpect tomorrow?
I'm working on validating the load balancer configuration and am finding some issues.

1. Please update the load balancer rule so that when SSLB=1 it sends the request to SiteSpect (instead of SSLB=A).

2. Add a rule so that if SSLB exists with any other value, the request does not route through SiteSpect.

3. When splitting the traffic to the non-SiteSpect route, the load balancer should set SSLB=0 as a session only cookie if no SSLB cookie exists (the path should be / and the domain should be .mozilla.com).

Please let me know when these ruleset are in place and I'll re-test.

Tal
I'm not sure that the load balancer can easily set a session cookie. Is this a big deal? Here are the new rules:

add cs vserver "cs-mozcom-test" HTTP 63.245.209.112 80 -cltTimeout 180
bind cs vserver "cs-mozcom-test" "mozcom-sitespect" -policyName "sitespect-cookieA"
bind cs vserver "cs-mozcom-test" "mozcom-inside-vs-test"
bind cs vserver "cs-mozcom-test" "mozcom-inside-vs" -policyName "sitespect-cookie-nosend"
add cs policy "sitespect-cookieA" -rule "REQ.HTTP.HEADER Cookie CONTAINS SSLB=1"
add cs policy "sitespect-cookie-nosend" -rule "REQ.HTTP.HEADER Cookie CONTAINS SSLB && REQ.HTTP.HEADER Cookie NOTCONTAINS SSLB=1"
I'm not sure what's going on, but when I test it seems that at first things appear to work - I am able to see the split and also able to maintain affinity to or away from SiteSpect based on cookies. Then I go to retest and I can not direct any traffic to SiteSpect at all. No split, no cookie affinity. 

Can you take a look?

Tal
oremj, engage edgecast?
Jeremy, mrz, and Tal -- can we resolved this by tomorrow (Tuesday) morning?  We'd really like to start running our first test tomorrow afternoon.

If it would be helpful to schedule a phone call, just say the word.
(In reply to comment #61)
> oremj, engage edgecast?

He should be using the testing IP which doesn't include edgecast.

(In reply to comment #60)
> I'm not sure what's going on, but when I test it seems that at first things
> appear to work - I am able to see the split and also able to maintain affinity
> to or away from SiteSpect based on cookies. Then I go to retest and I can not
> direct any traffic to SiteSpect at all. No split, no cookie affinity. 
> 

When was the first test and when did you retest? If you retest now do you see sitespect cookie?
I just ran into this again - it is the weirdest thing!

I sent 10 requests to http://63.245.209.112/en-US/?taltest, passing www.  mozilla.com for the host header and User-Agent: M  ozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729) for the user agent string. The load balancer appropriatly split the traffic 50% to SiteSpect.

Then I sent 6 request, but this time included the SSLB=1 cookie. The load balancer correctly maintained affinity to SiteSpect.

Then I sent 12 requests with the SSLB cookie = 0. Again, the load balancer correctly maintained affinity away from SiteSpect.

Then I retried both the split (no cookies) and affinity to SiteSpect (SSLB=1) both failed - I never got to SiteSpect.

Is it possible that there is some kind of Layer4 affinity going on?
Matthew, do you remember how we did this before? It appears the IC is hit before Content switches or Virtual Servers. Only thing I can think of is turning off caching completely for all users without a SSLB cookie.
Certainly looks that way:

root@nslb01# nscachemgr  -a | grep talt
0x00000003b9393672495b  MOZ     GET     //www.mozilla.com:80/en-US/?taltest

I found this in an email to Tal back on Sept 20 2007:

*** 
I added a specific rule to exclude anything matching a host header that containts "www.mozilla.com" and a cookie that matches "SSLB=A" from caching.

Tal Cohen wrote:
Matthew,
  It looks like when users are going to www.mozilla.com <http://www.mozilla.com/>, even if their SSLB cookie = A, they are hitting the cache rather than SiteSpect. Please call me to discuss.
***

That rule's still in place along with the =1 one:

nslb01> sh run | grep SSLB | grep cache
add cache policy "mozcom-sitespect-no" -rule "REQ.HTTP.HEADER Host CONTAINS www.mozilla.com && REQ.HTTP.HEADER Cookie CONTAINS SSLB=A" -action NOCACHE
add cache policy "mozcom-sitespect-no2" -rule "REQ.HTTP.HEADER Host CONTAINS www.mozilla.com && REQ.HTTP.HEADER Cookie CONTAINS SSLB=1" -action NOCACHE

Hitting:

GET -dsSe http://63.245.209.112/en-US/\?taltest -H'Host: www.mozilla.com' -H 'Cookie: SSLB=1' -H 'User-Agent: Mozilla'

Consistently hits the origin server.
Not getting any Set-Cookie headers back from your box - should I be?

mrz@boris [~/] 52> curl -o /dev/null -s --dump-header - -H'Cookie: SSLB=A' -H'User-Agent: Mozilla' -H'Host: www.mozilla.com' http://10.2.80.232/en-US/about/legal.html
HTTP/1.1 200 OK
Date: Tue, 22 Sep 2009 03:14:18 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.1.6
Cache-Control: private, no-store, no-cache, max-age=0, must-revalidate, proxy-revalidate
Expires: Wed, 25 Mar 2009 17:54:01 GMT
Content-Type: text/html; charset=UTF-8
Pragma: no-cache
Vary: Accept-Encoding
Transfer-Encoding: chunked
OK, so as long as there is no caching in front of SiteSpect when SSLB=1 that should work.

Is it possible for your application/web server to set SSLB=0 as a session only cookie (path=/ domain=.mozilla.com)? SiteSpect will over write the cookie, so always setting it will prevent users from drifting into a test.

Alternately, is there currently any session based affinity being done in the load balancer that would keep a user away from SiteSpect if they did not start their session with SiteSpect?
Matt - When testing, use SSLB=1 cookie and make sure to use a fully qualified user agent string. SiteSpect will treat partial user agent strings (like just "Mozilla") as bots and just pass them through. Here is the user agent string that I use for testing: "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)"
Heads up - I'm going to test failover. I will be bringing the system into and out of failover mode for the next 30 minutes or so. I'll let you know once I'm done.
I've completed my failover testing. The system passed with flying colors.

At this time I've removed the failover condition.
So, from what I can see everyting looks good. 
The only open issue is to add a session cookie as I described in comment #68.
This cookie is important to prevent users from drifting into a test mid-session.
Changed vserver to:
add lb vserver "mozcom-inside-vs-test" HTTP 0.0.0.0 0 -persistenceType COOKIEINSERT -timeout 15 -lbMethod ROUNDROBIN -cltTimeout 180

Note "-persistenceType COOKIEINSERT -timeout 15" which the docs say will lock a user down to a service for 15 minutes in this case.
Can that cookie be set as session only?
If session only is possible that is recommended. However, based on my testing you should be able to launch with the current config (after adding additional web servers of course).
Unfortunately, session cookies aren't an option.
We didn't do session cookies last time - just different implementation now?
OK, keeping them at 15 minutes should be OK.

Not so much a different implementation...a more robust implementation.

Anyway, I'm OK with the current config. Let me know when you plan to send live traffic.

Tal
We plan on going live tonight @ 6pm.
Production .com now has sitespect in the mix.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
It looks like some of the caching rules did not make it into production.
Some requests with the SSLB=1 cookie are not making it into SiteSpect.
Can you verify that the SSLB=1 nocache rules are in place?

Also, what is the percentage of traffic that will go to SiteSpect?
(In reply to comment #81)
> It looks like some of the caching rules did not make it into production.
> Some requests with the SSLB=1 cookie are not making it into SiteSpect.
> Can you verify that the SSLB=1 nocache rules are in place?

Do you have a test GET I can use to duplicate?  Rules all look the same.

> Also, what is the percentage of traffic that will go to SiteSpect?

Tough to say.  I'd say 1/5 but it's fronted by EdgeCast so you'll only see 1/5 of the cache misses.  If you're not getting enough traffic we'll tweak EdgeCast.

That reminds me though - the cookie changed in the middle of implementation and EdgeCast had configured their end to match on SSLB=A or SSLB=B.  That changed to =1 & =0.  I've asked EdgeCast to update their end.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Here is the GET that I'm doing:

export COUNTER=1;while [ $COUNTER -le 100  ]; do echo && echo -n "$COUNTER ";GET -Ssed -H 'User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)' -H 'Host: www.mozilla.com' -H 'Cookie: SSLB=1' http://www.mozilla.com/en-US/?taltest=$COUNTER | grep -i ssid ; COUNTER=`expr $COUNTER + 1`; done

Set SSLB=1 for SiteSpect, SSLB=0 for no SiteSpect, or remove the SSLB cookie to simulate a new request (traffic should get slpit).

Due to the nature of the caching, you may need to run a few times before you see the failure.

Let me know once EdgeCast has been updated with the SSLB=1/0 rules.
Tal - I copied helpdesk@ on the email to EdgeCast.
Group: infra
The changes that EdgeCast deployed look good. Here is what I'm seeing:

1. Requests with SSLB=1 always go to SiteSpect and bypass cache
2. Requests with SSLB=0 never go through SiteSpect
3. Requests with no SSLB cookie either hit the cache, or get split between SiteSpect and non-SiteSpect routes. The fact that they hit the cache instead of being split should be OK given your traffic levels.

At this point I'm OK signing off on your deployment. You should feel free to start testing at any time.
Tal, Jeremy, and mrz -- thanks for all your work on this!  we're excited to start testing!
A big second on the thanks!  I have one last issue; I am no longer able to preview my experiment variations.

After I click on the preview button (http://www.mozilla.com:9081/en-US/products/download.html?product=firefox-3.5.2&os=win&lang=en-US), I'm directed to https://sitespect.mozilla.com:9443/Variation_List  which times out.  Tal, any idea what's going on here? I assume it's user error.
You could be hitting the EdgeCast cache. Try setting a cookie, SSLB=1, and retry.
Doesn't seem to work. Here's the info for the cookie I set:

Name: SSLB
Content: 1
Domain: .mozilla.com
Path: / 
Send For: Any type of connection

Any other ideas?
Previews need to run through the front end IP on SiteSpect (10.2.80.232:80). 

Prior to going live there was a rule that directed requests to www.mozilla.com:9081 to SiteSpect. Is that rule still in place?
To clarify - the rule was on your NetScalar.
You aren't able to access the preview, because DNS for www.mozilla.com is pointing to edgecast. Adding the following to /etc/hosts should fix the problem:

63.245.209.10 www.mozilla.com
Status: REOPENED → RESOLVED
Closed: 15 years ago15 years ago
Resolution: --- → FIXED
I agree, that resolves the issue for me.
success!  thanks Jeremy.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → RESOLVED
Closed: 15 years ago15 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: