984600 - [SUMO] Go full HTTPS.

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task)

Description

[Ricky Rosario [:rrosario, :r1cky]]

Right now we have separate processes for SSL and non-SSL. It appears that about 2/3rds or more of our traffic is on SSL already. This makes sense because all authenticated visitors will be on HTTPS, all users that come from within our products will be on HTTPS, and all visitors that ever see an HTTPS page will not be able to go back to HTTP because of the HTTP Strict Transport Security headers we use.

This bug is to:
* Redirect all http traffic to https. 
* Shut off the non-SSL apache/wsgi processes. This will give us more room to run more SSL processes.

This should be done on -dev, -stage and prod.

Comment 1

[Jake Maul [:jakem]]

Redirect in place for support-dev.allizom.org. Haven't changed server-side configs yet.

Comment 2

[Jake Maul [:jakem]]

stage and prod are flipped over now too (coordinated in IRC).

Comment 3

[Jake Maul [:jakem]]

Disabled the relevant apache configs for dev and stage in puppet, will propagate out within an hour.

This will trigger an apache restart, so I'm holding off on prod for a bit longer so traffic can drop off some more.

Comment 4

[Jake Maul [:jakem]]

TODO:

remove config on prod (soon... just waiting for traffic to fall off a little more for a smaller impact)

test switching to the Worker MPM to save memory - should reduce Apache's usage, and shouldn't cause problems because there is no PHP/scripting, except Python/WSGI, which is handled in separate processes

test the effects of the "stack-size" parameter to WSGIDaemonProcess... might save more memory

test the effects of the ThreadStackSize setting in Apache Worker MPM... should save memory

try to tune dev/stage/prod WSGIDaemonProcess settings to eliminate threading.. should have lots more RAM by this point. Remember in bug 968185 we're adding a support6 node, so requests per node is also dropping.

remove now-unnecessary Zeus pools - already disabled.

Comment 5

[Jake Maul [:jakem]]

I have removed the config on prod.

I have somewhat tuned the threading on prod... from 4 procs / 10 threads each to 8 procs / 5 threads each. This won't cure the threading problems, but it should ease the pain somewhat- work spread out over more processes should equate to less contention per-process. This is because with the elimination of the HTTP workers, we can easily afford 4 more procs without any concern at all. I didn't increase the total number of workers because in looking at the system load, I don't think we're running into trouble there (though I could be wrong, due to ES, MySQL, memcache, etc).

I have started tuning stack-size on support-dev. Initial results show no particular gain in "Resident Set Size" (about 130MB either way, right after a restart and visiting the home page), but a reasonable gain in overall process virtual memory size (1110MB -> 900MB). In practice, this probably won't help much. Still, I'll tune it on stage and prod anyway.

Comment 6

[Ricky Rosario [:rrosario, :r1cky]]

I think we can make some changes to the prime_app step of our deploy script?

The source is here: https://github.com/mozilla/kitsune/blob/master/scripts/update/deploy.py#L85

It is curl'ing both port 80 and 81 of all the webheads. Do we change this to do only port 81?

This is the output right now:

[2014-03-17 18:08:16] Running prime_app
[2014-03-17 18:08:16] [support1.webapp.phx1.mozilla.com] running: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:80/ & sleep 1; done
[2014-03-17 18:08:16] [support2.webapp.phx1.mozilla.com] running: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:80/ & sleep 1; done
[2014-03-17 18:08:16] [support3.webapp.phx1.mozilla.com] running: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:80/ & sleep 1; done
[2014-03-17 18:08:16] [support4.webapp.phx1.mozilla.com] running: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:80/ & sleep 1; done
[2014-03-17 18:08:16] [support5.webapp.phx1.mozilla.com] running: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:80/ & sleep 1; done
[2014-03-17 18:08:35] [support3.webapp.phx1.mozilla.com] finished: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:80/ & sleep 1; done (19.004s)
[2014-03-17 18:08:35] [support3.webapp.phx1.mozilla.com] running: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:81/ & sleep 1; done
[2014-03-17 18:08:37] [support2.webapp.phx1.mozilla.com] finished: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:80/ & sleep 1; done (21.611s)
[2014-03-17 18:08:37] [support2.webapp.phx1.mozilla.com] running: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:81/ & sleep 1; done
[2014-03-17 18:08:46] [support3.webapp.phx1.mozilla.com] finished: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:81/ & sleep 1; done (11.364s)
[2014-03-17 18:08:48] [support2.webapp.phx1.mozilla.com] finished: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:81/ & sleep 1; done (10.437s)
[2014-03-17 18:09:33] [support1.webapp.phx1.mozilla.com] finished: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:80/ & sleep 1; done (76.732s)
[2014-03-17 18:09:33] [support1.webapp.phx1.mozilla.com] running: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:81/ & sleep 1; done
[2014-03-17 18:09:43] [support1.webapp.phx1.mozilla.com] finished: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:81/ & sleep 1; done (10.399s)
[2014-03-17 18:10:32] [support5.webapp.phx1.mozilla.com] finished: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:80/ & sleep 1; done (136.042s)
[2014-03-17 18:10:32] [support5.webapp.phx1.mozilla.com] running: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:81/ & sleep 1; done
[2014-03-17 18:10:42] [support5.webapp.phx1.mozilla.com] finished: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:81/ & sleep 1; done (10.133s)
[2014-03-17 18:10:42] [support4.webapp.phx1.mozilla.com] finished: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:80/ & sleep 1; done (146.632s)
[2014-03-17 18:10:42] [support4.webapp.phx1.mozilla.com] running: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:81/ & sleep 1; done
[2014-03-17 18:10:53] [support4.webapp.phx1.mozilla.com] finished: for i in {1..10}; do curl -so /dev/null -H 'Host: support.mozilla.org' -I http://localhost:81/ & sleep 1; done (10.194s)
[2014-03-17 18:10:53] Finished prime_app (156.834s)

Comment 7

[Jake Maul [:jakem]]

I'm going to mark this as resolved. The stuff in comment 5 is really not directly related to HTTPS, so we can take that sort of discussion back to the main SUMO performance bug (934601).