Closed Bug 750798 Opened 13 years ago Closed 13 years ago

[download.allizom.org] stage environment for bouncer setup

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P2)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: laura, Assigned: cturra)

References

Details

(Whiteboard: [triaged 20120831])

bounceradmin.mozilla.com has the staging environment tuxedo.stage.mozilla.com. download.mozilla.org doesn't have a staging environment. The first two run the admin side (tuxedo, the django app), but download is the meat of the business. It needs a stage env so we can test out changes on the PHP side, for example: https://bugzilla.mozilla.org/show_bug.cgi?id=731299 and https://bugzilla.mozilla.org/show_bug.cgi?id=667152 Please set up a download.allizom.org as a staging site, using the same database as tuxedo.stage.mozilla.com.
Blocks: 749207
This won't happen for at least a week, as we're in the process of confirming all of bouncer is out of SJC and getting bounceradmin moved to SJC
Status: NEW → ASSIGNED
Component: Server Operations → Server Operations: Web Operations
QA Contact: phong → cshields
Assignee: server-ops → bburton
Any update on this? This blocks a few other bugs.
Corey: Any update on this?
I should be able to have this ready check out by the end of the week
Initial setup is done, tested with curl, once the flows for MySQL are in place, we should be able to test a URL that will "bounce" bburton@andesite ~$ curl -v -H "Host: download.allizom.org" https://download.allizom.org/ ‹1.9.2-p290› * About to connect() to download.allizom.org port 443 (#0) * Trying 63.245.217.83... connected * Connected to download.allizom.org (63.245.217.83) port 443 (#0) * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using RC4-SHA * Server certificate: * subject: serialNumber=QFblspylXort2BviK0LdJuDx7haU0SBy; C=US; ST=California; L=Mountain View; O=Mozilla Corporation; CN=*.allizom.org * start date: 2011-10-10 22:11:59 GMT * expire date: 2013-12-11 16:30:26 GMT * subjectAltName: download.allizom.org matched * issuer: C=US; O=GeoTrust, Inc.; CN=GeoTrust SSL CA * SSL certificate verify ok. > GET / HTTP/1.1 > User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5 > Accept: */* > Host: download.allizom.org > < HTTP/1.1 302 Found < Server: Apache < X-Backend-Server: node206.seamicro.phx1.mozilla.com < Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private < Content-Type: text/html; charset=UTF-8 < Date: Thu, 31 May 2012 06:31:45 GMT < Location: http://www.mozilla.com/ < Pragma: no-cache < X-Powered-By: PHP/5.3.3 < X-Cache-Info: not cacheable; response specified "Cache-Control: no-store" < Content-Length: 0 < * Connection #0 to host download.allizom.org left intact * Closing connection #0 * SSLv3, TLS alert, Client hello (1):
Network flows are in place. What's a good curl to test that requests are "bounced" correctly?
http://download.allizom.org/?product=firefox-12.0&os=osx&lang=en-US should be a good test. Although it returns an ISE 500: ➜ patcher-configs curl -I "https://download.allizom.org/?product=firefox-12.0&os=osx&lang=en-US" zsh: correct 'curl' to 'ul' [nyae]? n HTTP/1.0 500 Internal Server Error Server: Apache X-Backend-Server: generic2.stage.webapp.phx1.mozilla.com Vary: Accept-Encoding Content-Type: text/html; charset=UTF-8 Date: Fri, 01 Jun 2012 12:28:38 GMT Connection: close X-Powered-By: PHP/5.3.3 X-Cache-Info: not cacheable; response code not cacheable Whereas production looks ok: ➜ patcher-configs curl -I "http://download.mozilla.org/?product=firefox-12.0&os=osx&lang=en-US" HTTP/1.1 302 Found Server: Apache X-Backend-Server: pp-app-dist07 Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private Content-Type: text/html; charset=UTF-8 Date: Fri, 01 Jun 2012 12:32:45 GMT Location: http://mozilla.cdn.leaseweb.com/firefox/releases/12.0/mac/en-US/Firefox%2012.0.dmg Pragma: no-cache Transfer-Encoding: chunked Connection: Keep-Alive Set-Cookie: dmo=10.8.84.214.1338553965565605; path=/; expires=Sat, 01-Jun-13 12:32:45 GMT X-Powered-By: PHP/5.1.6
The stage database may not be up to date with the latest. You might inspect the database directly to see what versions are there, or try a few older versions. Alternatively, we can possibly add a version to test against.
I just added that product/OS and I'm still getting the same results. Bouncer returns a 404 for missing things when it's working, eg: ➜ patcher-configs curl -I "http://download.mozilla.org/?product=firefox-12.0abuteo&os=osx&lang=en-US" HTTP/1.1 404 Not Found Server: Apache X-Backend-Server: pp-app-dist01 Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private Content-Type: text/html; charset=UTF-8 Date: Fri, 01 Jun 2012 12:40:18 GMT Pragma: no-cache Transfer-Encoding: chunked Connection: Keep-Alive Set-Cookie: dmo=10.8.84.214.1338554418151853; path=/; expires=Sat, 01-Jun-13 12:40:18 GMT X-Powered-By: PHP/5.1.6
Had some bad info in the db u/p. Updated, now returning a 404, which I believe should be correct. bburton@andesite ~$ curl -v -H "Host: download.allizom.org" "http://generic1.stage.webapp.phx1.mozilla.com:81/?product=firefox-12.0&os=osx&lang=en-US" -o /dev/null * About to connect() to generic1.stage.webapp.phx1.mozilla.com port 81 (#0) * Trying 10.8.81.120... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0connected * Connected to generic1.stage.webapp.phx1.mozilla.com (10.8.81.120) port 81 (#0) > GET /?product=firefox-12.0&os=osx&lang=en-US HTTP/1.1 > User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5 > Accept: */* > Host: download.allizom.org > * HTTP 1.0, assume close after body < HTTP/1.0 404 Not Found < Date: Mon, 04 Jun 2012 17:23:55 GMT < Server: Apache < X-Backend-Server: generic1.stage.webapp.phx1.mozilla.com < X-Powered-By: PHP/5.3.3 < Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private < Pragma: no-cache < Content-Length: 3664 < Connection: close < Content-Type: text/html; charset=UTF-8 < { [data not shown] 100 3664 100 3664 0 0 9217 0 --:--:-- --:--:-- --:--:-- 9770* Closing connection #0 Do you want me to update the DB from prod?
I think we want to avoid updating the DB from prod because there are DB changes in this version. Can you create me a user account? The app is a django app so user accounts can be created through the django admin CLI tool.
Code and submodules seem up to date, but manage.py can't find Django, am I missing something? [root@oldstage2 tuxedo]# git pull Already up-to-date. [root@oldstage2 tuxedo]# git submodule init [root@oldstage2 tuxedo]# git submodule update --init --recursive [root@oldstage2 tuxedo]# ls apps __init__.py local_settings.pyc pip-log.txt sentry sql urls.pyc bouncer __init__.pyc manage.py README.md settings.py templates wsgi inc local_settings.py media requirements.txt settings.pyc urls.py [root@oldstage2 tuxedo]# ls -alh total 620K drwxr-xr-x 11 root root 4.0K May 15 11:34 . drwxr-xr-x 5 root root 4.0K Jun 4 14:13 .. drwxr-xr-x 8 root root 4.0K Mar 2 2010 apps drwxr-xr-x 4 root root 4.0K Feb 10 2010 bouncer drwxr-xr-x 8 root root 4.0K Jun 4 14:29 .git -rw-r--r-- 1 root root 69 Feb 10 2010 .gitignore drwxr-xr-x 3 root root 4.0K Mar 2 2010 inc -rw-r--r-- 1 root root 0 Feb 10 2010 __init__.py -rw-r--r-- 1 root root 150 Feb 10 2010 __init__.pyc -rw-r--r-- 1 root root 767 May 15 11:34 local_settings.py -rw-r--r-- 1 root root 660 Mar 30 10:31 local_settings.pyc -rwxr-xr-x 1 root root 656 Feb 10 2010 manage.py drwxr-xr-x 5 root root 4.0K Feb 10 2010 media -rw-r--r-- 1 root root 519K Feb 10 2010 pip-log.txt -rw-r--r-- 1 root root 4.3K Mar 2 2010 README.md -rw-r--r-- 1 root root 195 Mar 30 10:22 requirements.txt drwxr-xr-x 3 root root 4.0K Nov 9 2010 sentry -rw-r--r-- 1 root root 4.5K Jun 14 2011 settings.py -rw-r--r-- 1 root root 3.2K Mar 30 10:31 settings.pyc drwxr-xr-x 2 root root 4.0K Mar 30 10:22 sql drwxr-xr-x 3 root root 4.0K Mar 30 10:22 templates -rw-r--r-- 1 root root 829 Jul 28 2010 urls.py -rw-r--r-- 1 root root 920 Feb 10 2010 urls.pyc drwxr-xr-x 2 root root 4.0K Feb 10 2010 wsgi [root@oldstage2 tuxedo]# python26 manage.py Traceback (most recent call last): File "manage.py", line 4, in <module> from django.core.management import execute_manager ImportError: No module named django.core.management
:brandon, given c12, how should I proceed?
Sorry, I had come up with some ideas in my head but apparently forgot to record them. It's my guess that the app is running inside a virtualenv, while you are executing against the system-wide python. It wouldn't be a good idea to have django installed system-wide. The key is to find the virtualenv that is running this app, and use it to run the component.
I see, yeah, it's using that in production too, we don't use virtualenv anywhere, so I forgot about it I'll poke at it later this morning, thanks!
Any update here? curl -I "https://download.allizom.org/?product=firefox-12.0&os=osx&lang=en-US" now gives me a 404 instead of a 500 error. Do we need to get Sentry updating availability, too?
Sorry for the delay, I've gotten caught up in a number of equally important bugs in the last couple of weeks and pulled into dealing with some tasks that weren't part of bugs, but post SJC1 cleanup/improvements that suddenly were needed more urgently. Unfortunately this has resulted in my losing state on this bug. I'm triaging this and other bugs today, as I'll be on PTO getting LASIK tomorrow, Thursday, and potentially Friday. (In reply to Ben Hearsum [:bhearsum] from comment #16) > Any update here? curl -I > "https://download.allizom.org/?product=firefox-12.0&os=osx&lang=en-US" now > gives me a 404 instead of a 500 error. Do we need to get Sentry updating > availability, too? As in, the cron jobs running in prod? I plan to review and work on this on 6/18 and ideally wrap up by 6/22, or hand off to another webops team member to wrap up by then If this is needed more urgently, let me know and I'll try to hand it off today
Whiteboard: [review and work on 6/18]
> (In reply to Ben Hearsum [:bhearsum] from comment #16) > > Any update here? curl -I > > "https://download.allizom.org/?product=firefox-12.0&os=osx&lang=en-US" now > > gives me a 404 instead of a 500 error. Do we need to get Sentry updating > > availability, too? > > As in, the cron jobs running in prod? I guess so. I'm not really plugged in to how Sentry works -- I just know it runs periodically and I _think_ updates the database with availability information. Justdave probably knows more. > I plan to review and work on this on 6/18 and ideally wrap up by 6/22, or > hand off to another webops team member to wrap up by then > > If this is needed more urgently, let me know and I'll try to hand it off > today By the 22nd should be fine, thanks for the heads up.
(In reply to Ben Hearsum [:bhearsum] from comment #18) > By the 22nd should be fine, thanks for the heads up. solarce: are we still on track for today?
I thought I was, but as I dug into the cron jobs, I realized I need to run a copy of Sentry on the stage environment's admin node, so all the sentry cron jobs can run, which led to a pile of Perl module dependencies But I'm close to getting the dependencies Puppetized and expect to be able to manually run the scripts tonight and get cron jobs in place on Monday to keep stage updated
Cron job script is running in screen! will check progress in the morning
Unfortunately, the one kickoff I did is still running. this is due to the admin node on the cluster that download.allizom.org is one is overloaded, which we're addressing by getting new hardware. In the mean time, per irc discussion with :bhearsum, I will do the following 1. Refresh the stage DB from prod, will be done shortly 2. Request a VM in SCL3 to be dedicated to running Sentry crons for staging 3. Get Sentry, deps, crons, etc setup in the VM
Import is running, I didn't realize prod was a couple gigabytes, it's gonna be a while, but is running in screen [root@stage1 bburton]# mysql tuxedo < bouncer_2012-06-25.sql
Depends on: 768100
DB import finally finished. Curl is looking good for me ~ ❯ curl -v https://download.allizom.org/\?product\=firefox-12.0\&os\=osx\&lang\=en-US * About to connect() to download.allizom.org port 443 (#0) * Trying 63.245.217.83... connected * Connected to download.allizom.org (63.245.217.83) port 443 (#0) * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using RC4-SHA * Server certificate: * subject: serialNumber=QFblspylXort2BviK0LdJuDx7haU0SBy; C=US; ST=California; L=Mountain View; O=Mozilla Corporation; CN=*.allizom.org * start date: 2011-10-10 22:11:59 GMT * expire date: 2013-12-11 16:30:26 GMT * subjectAltName: download.allizom.org matched * issuer: C=US; O=GeoTrust, Inc.; CN=GeoTrust SSL CA * SSL certificate verify ok. > GET /?product=firefox-12.0&os=osx&lang=en-US HTTP/1.1 > User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5 > Host: download.allizom.org > Accept: */* > < HTTP/1.1 302 Found < Server: Apache < X-Backend-Server: generic2.stage.webapp.phx1.mozilla.com < Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private < Content-Type: text/html; charset=UTF-8 < Date: Mon, 25 Jun 2012 22:58:40 GMT < Location: http://ftp-zlb.vips.scl3.mozilla.com/pub/mozilla.org//firefox/releases/12.0/mac/en-US/Firefox%2012.0.dmg < Pragma: no-cache < X-Powered-By: PHP/5.3.3 < X-Cache-Info: not cacheable; response specified "Cache-Control: no-store" < Content-Length: 0 < * Connection #0 to host download.allizom.org left intact * Closing connection #0 * SSLv3, TLS alert, Client hello (1): And Firefox downloaded a 12.0 DMG
The curl output looks good to me. I need to do larger scale tests to fully verify these changes, I'll try to do that today or tomorrow. Thank you for your work here!
I just tried to configure tuxedo stage for bug 646076 again and I got an ISE 500. The stage code expects a different schema than production, did that get clobbered by the db import?
That would be it, I did not realize that As I took a dump of the previous stage DB, I can load it and only import specific tables. Do you know which ones should be loaded from the stage schema and which I could import from the prod dump? Thanks!
(In reply to Brandon Burton [:solarce] from comment #27) > That would be it, I did not realize that > > As I took a dump of the previous stage DB, I can load it and only import > specific tables. > > Do you know which ones should be loaded from the stage schema and which I > could import from the prod dump? > > Thanks! I don't know...you'd have to ask :wenzel or maybe :brandon. It would be better if we could upgrade the imported production DB rather than overwrite data though -- the uptake information is time-sensitive.
There's an upgrade script in the migrations directory. It should have the correct code to execute to upgrade the DB to the staged version.
(In reply to Brandon Savage [:brandon] from comment #29) > There's an upgrade script in the migrations directory. It should have the > correct code to execute to upgrade the DB to the staged version. Where in https://github.com/fwenzel/tuxedo should I find the migrations directory?
https://github.com/fwenzel/tuxedo/blob/master/sql/incremental.sql#L113 Here to the end of the file. Sorry, I forgot this is NOT a playdoh project.
(In reply to Brandon Savage [:brandon] from comment #31) > https://github.com/fwenzel/tuxedo/blob/master/sql/incremental.sql#L113 Here > to the end of the file. Sorry, I forgot this is NOT a playdoh project. Thank you! That's been tripping me up as well :) :bhearsum, I want the queries, all ran successfully mysql> use tuxedo; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> ALTER TABLE `geoip_regions` ADD COLUMN `fallback_id` integer; Query OK, 21 rows affected (0.08 sec) Records: 21 Duplicates: 0 Warnings: 0 mysql> ALTER TABLE `geoip_regions` ADD CONSTRAINT `fallback_id_refs_id_e6bfe66d` FOREIGN KEY (`fallback_id`) REFERENCES `geoip_regions` (`id`); Query OK, 21 rows affected (0.01 sec) Records: 21 Duplicates: 0 Warnings: 0 mysql> CREATE INDEX `geoip_regions_e28329c2` ON `geoip_regions` (`fallback_id`); Query OK, 21 rows affected (0.01 sec) Records: 21 Duplicates: 0 Warnings: 0 :bhearsum, can you also make :brandon an admin account, per https://bugzilla.mozilla.org/show_bug.cgi?id=750798#c11 ?
(In reply to Brandon Savage [:brandon] from comment #31) > https://github.com/fwenzel/tuxedo/blob/master/sql/incremental.sql#L113 Here > to the end of the file. Sorry, I forgot this is NOT a playdoh project. Medium-term, you might want to turn it *into* a playdoh project.
(In reply to Brandon Burton [:solarce] from comment #32) > :bhearsum, can you also make :brandon an admin account, per > https://bugzilla.mozilla.org/show_bug.cgi?id=750798#c11 ? There's already a bburton account, actually. I assume it got imported from production. Unfortunately, I'm still getting an ISE 500 from https://tuxedo.stage.mozilla.com/admin/mirror/mirror/418/, so I can't do any further testing as of yet.
(In reply to Ben Hearsum [:bhearsum] from comment #34) > (In reply to Brandon Burton [:solarce] from comment #32) > > :bhearsum, can you also make :brandon an admin account, per > > https://bugzilla.mozilla.org/show_bug.cgi?id=750798#c11 ? > > There's already a bburton account, actually. I assume it got imported from > production. > Thanks for the password reset > Unfortunately, I'm still getting an ISE 500 from > https://tuxedo.stage.mozilla.com/admin/mirror/mirror/418/, so I can't do any > further testing as of yet. I am digging into what's up with this
Whiteboard: [review and work on 6/18]
1. I made :brandonsavage an admin account 2. I enabled DEBUG=True as the 500 wasn't resulting in an error log entry Any mirror details url results in an error, e.g. https://tuxedo.stage.mozilla.com/admin/mirror/mirror/119/ (1054, "Unknown column 'geoip_regions.prevent_global_fallback' in 'field list'") However the code is all up to date [root@oldstage2 tuxedo]# git pull Already up-to-date. [root@oldstage2 tuxedo]# git status # On branch master # Changes not staged for commit: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: wsgi/tuxedo.wsgi # # Untracked files: # (use "git add <file>..." to include in what will be committed) # # pip-log.txt no changes added to commit (use "git add" and/or "git commit -a") [root@oldstage2 tuxedo]# For verboseness, here is the .git/config [root@oldstage2 tuxedo]# cat .git/config [core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true [remote "origin"] fetch = +refs/heads/*:refs/remotes/origin/* url = https://github.com/fwenzel/tuxedo.git [branch "master"] remote = origin merge = refs/heads/master [submodule "inc/product-details/json"] url = https://github.com/jbalogh/mozilla-product-details.git [root@oldstage2 tuxedo]# Is this a code thing? Or Did I miss a SQL query I should have run?
Brandon, Who are you waiting to hear back from on your questions in the ticket?
:fwenzel / :brandonsavage , any insight into the error we're seeing from Django? OperationalError at /admin/mirror/mirror/119/ (1054, "Unknown column 'geoip_regions.prevent_global_fallback' in 'field list'") Request Method: GET Request URL: https://tuxedo.stage.mozilla.com/admin/mirror/mirror/119/ Django Version: 1.2 Exception Type: OperationalError Exception Value: (1054, "Unknown column 'geoip_regions.prevent_global_fallback' in 'field list'") Exception Location: /usr/lib/python2.6/site-packages/MySQLdb/connections.py in defaulterrorhandler, line 36 Python Executable: /usr/bin/python Python Version: 2.6.8 Python Path: ['/usr/lib/python2.6/site-packages/virtualenv-1.4.5-py2.6.egg', '/usr/lib/python2.6/site-packages/pip-0.8-py2.6.egg', '/data/virtualenvs/tuxedo/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg', '/data/virtualenvs/tuxedo/lib/python2.6/site-packages/pip-0.6.3-py2.6.egg', '/usr/lib/python26.zip', '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', '/usr/lib/python2.6/lib-dynload', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/PIL', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info', '/data/virtualenvs/tuxedo/lib/python2.6', '/data/virtualenvs/tuxedo/lib/python2.6/site-packages', '/data/www/tuxedo.stage.mozilla.com', '/data/www/tuxedo.stage.mozilla.com/tuxedo/apps'] Server time: Mon, 16 Jul 2012 10:28:57 -0700 Thanks
Whiteboard: [pending dev feedback]
No feedback received yet from :fwenzel / :brandonsavage, I've emailed them as well to ping for feedback.
:Rik, can you take a look? Thanks!
Solarce: Looks like this needs to run the DB migrations. Specifically, those in apps/geoip/migrations/. ./manage.py migrate geoip should do the trick.
Summary: Need a stage environment for bouncer → [download.allizom.org] stage environment for bouncer setup
(In reply to Anthony Ricaud (:rik) from comment #41) > Solarce: Looks like this needs to run the DB migrations. Specifically, those > in apps/geoip/migrations/. > > ./manage.py migrate geoip should do the trick. Thanks Rik, I did get an error the first time and had to drop three tables mysql> DROP TABLE `geoip_country_to_region` CASCADE; Query OK, 0 rows affected (0.02 sec) mysql> DROP TABLE `geoip_ip_to_country` CASCADE; Query OK, 0 rows affected (0.03 sec) mysql> DROP TABLE `geoip_regions` CASCADE; Query OK, 0 rows affected (0.01 sec) mysql> But after that is was successful (tuxedo)[root@oldstage2 tuxedo]# python26 manage.py migrate geoip Running migrations for geoip: - Migrating forwards to 0003_auto__add_field_region_prevent_global_fallback. > geoip:0001_initial DEBUG:south:south execute "CREATE TABLE `geoip_country_to_region` (`country_code` varchar(2) NOT NULL PRIMARY KEY, `region_id` integer NULL, `country_name` varchar(255) NOT NULL, `continent` varchar(2) NOT NULL);" with params "[]" DEBUG:south:south execute "CREATE TABLE `geoip_ip_to_country` (`id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY, `ip_start` numeric(12, 0) NOT NULL, `ip_end` numeric(12, 0) NOT NULL, `country_code` varchar(2) NOT NULL);" with params "[]" DEBUG:south:south execute "CREATE TABLE `geoip_regions` (`id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY, `name` varchar(255) NOT NULL, `priority` integer NOT NULL, `throttle` integer NOT NULL);" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_country_to_region` ADD CONSTRAINT `region_id_refs_id_1969eae5` FOREIGN KEY (`region_id`) REFERENCES `geoip_regions` (`id`);" with params "[]" DEBUG:south:south execute "CREATE INDEX `geoip_country_to_region_9574fce` ON `geoip_country_to_region` (`region_id`);" with params "[]" DEBUG:south:south execute "SET FOREIGN_KEY_CHECKS=1;" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_ip_to_country` ADD CONSTRAINT `country_code_refs_country_code_139fcdf6` FOREIGN KEY (`country_code`) REFERENCES `geoip_country_to_region` (`country_code`);" with params "[]" DEBUG:south:south execute "CREATE INDEX `geoip_ip_to_country_2d3827c5` ON `geoip_ip_to_country` (`country_code`);" with params "[]" DEBUG:south:south execute "CREATE TABLE `geoip_country_to_region` (`country_code` varchar(2) NOT NULL PRIMARY KEY, `region_id` integer NULL, `country_name` varchar(255) NOT NULL, `continent` varchar(2) NOT NULL);" with params "[]" DEBUG:south:south execute "CREATE TABLE `geoip_ip_to_country` (`id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY, `ip_start` numeric(12, 0) NOT NULL, `ip_end` numeric(12, 0) NOT NULL, `country_code` varchar(2) NOT NULL);" with params "[]" DEBUG:south:south execute "CREATE TABLE `geoip_regions` (`id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY, `name` varchar(255) NOT NULL, `priority` integer NOT NULL, `throttle` integer NOT NULL);" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_country_to_region` ADD CONSTRAINT `region_id_refs_id_1969eae5` FOREIGN KEY (`region_id`) REFERENCES `geoip_regions` (`id`);" with params "[]" DEBUG:south:south execute "CREATE INDEX `geoip_country_to_region_9574fce` ON `geoip_country_to_region` (`region_id`);" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_ip_to_country` ADD CONSTRAINT `country_code_refs_country_code_139fcdf6` FOREIGN KEY (`country_code`) REFERENCES `geoip_country_to_region` (`country_code`);" with params "[]" DEBUG:south:south execute "CREATE INDEX `geoip_ip_to_country_2d3827c5` ON `geoip_ip_to_country` (`country_code`);" with params "[]" > geoip:0002_auto__add_field_region_fallback DEBUG:south:south execute "ALTER TABLE `geoip_regions` ADD COLUMN `fallback_id` integer NULL;" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_regions` ADD CONSTRAINT `fallback_id_refs_id_19401993` FOREIGN KEY (`fallback_id`) REFERENCES `geoip_regions` (`id`);" with params "[]" DEBUG:south:south execute "CREATE INDEX `geoip_regions_1d7cd63e` ON `geoip_regions` (`fallback_id`);" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_regions` ADD COLUMN `fallback_id` integer NULL;" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_regions` ;" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_regions` MODIFY `fallback_id` integer NULL;;" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_regions` ALTER COLUMN `fallback_id` DROP DEFAULT;" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_regions` ADD CONSTRAINT `fallback_id_refs_id_19401993` FOREIGN KEY (`fallback_id`) REFERENCES `geoip_regions` (`id`);" with params "[]" DEBUG:south:south execute "CREATE INDEX `geoip_regions_1d7cd63e` ON `geoip_regions` (`fallback_id`);" with params "[]" > geoip:0003_auto__add_field_region_prevent_global_fallback DEBUG:south:south execute "ALTER TABLE `geoip_regions` ADD COLUMN `prevent_global_fallback` bool NOT NULL DEFAULT False;" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_regions` ADD COLUMN `prevent_global_fallback` bool NOT NULL DEFAULT False;" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_regions` ;" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_regions` MODIFY `prevent_global_fallback` bool NOT NULL;;" with params "[]" DEBUG:south:south execute "ALTER TABLE `geoip_regions` ALTER COLUMN `prevent_global_fallback` DROP DEFAULT;" with params "[]" - Loading initial data for geoip. No fixtures found. :bhearsum, Now I get a working page at https://tuxedo.stage.mozilla.com/admin/mirror/mirror/119/ - see http://d.pr/i/4mny , can you confirm it works for you and then let me know what the next steps are to get this dialed in? Thanks!
Whiteboard: [pending dev feedback] → [pending testing]
I'm not going to be able to fully test this right away (I was put on a high priority project last week, and am at the RelEng work week this week), but it appears there's still data missing from the staging database. For example, there are no regions or countries or IP blocks configured: https://tuxedo.stage.mozilla.com/admin/geoip/region/ vs. https://bounceradmin.mozilla.com/admin/geoip/region/ https://tuxedo.stage.mozilla.com/admin/geoip/country/ vs. https://bounceradmin.mozilla.com/admin/geoip/country/ https://tuxedo.stage.mozilla.com/admin/geoip/ipblock/ vs. https://bounceradmin.mozilla.com/admin/geoip/ipblock/
It appears we need to resurrect this - Bouncer is not dead after all, and I'm told we need to fix bug 398366 with some urgency. Ben, can you dive back into testing, or nominate someone else who is available to do so?
Blocks: 398366
Depends on: 786820
Priority: -- → P2
Whiteboard: [pending testing] → [triaged 20120831][waiting][pending bouncer cluster]
Assignee: bburton → cturra
RelEng would like to get the database updated to match prod, so they can test more effectively. Could we please get a prod dump on here, cturra?
:laura - we need a security review run on any production database dumps. do you have any prior bugs approved for this? or are you aware of any personally identifiable information that might be in the database?
This dump is safe.. However, we are just a couple of days from having the new dev environment up - can we focus on that instead (it already has the latest dump in its new db)
There's no PII in the dump. I'm fine with waiting as Corey suggests - sorry about that, I didn't realize the stage setup from this bug was also being replaced.
stage environment built in new bouncer cluster complete. bouncer: https://download.allizom.org tuxedo: https://bounceradmin.allizom.org
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
:cturra - Can I get an account on the stage server? I've tested the dev server and I find it to be good work. Once I test stage We'll ask :bhearsum to take a look as well. We should be able to push this soon.
:brandon - i have created a user `brandonsavage` for you in stage. it has been given the same password as we set in dev for you. when you want code pushed to stage, please open another bug for that as a "push request" *note: only dev is auto updated.
Thanks. I've confirmed that stage is set up properly and all seems to be working. :bhearsum, please let me know if you need any help running your tests.
My brief test looks mostly good. The requests work: ➜ balrog git:(db-migration) curl -I http://download.allizom.org/\?product\=firefox-15.0\&os\=osx\&lang\=en-US HTTP/1.1 301 Moved Permanently Server: Apache X-Backend-Server: bouncer1.stage.webapp.phx1.mozilla.com Content-Type: text/html; charset=iso-8859-1 Date: Tue, 25 Sep 2012 13:07:55 GMT Location: https://download.allizom.org/?product=firefox-15.0&os=osx&lang=en-US Transfer-Encoding: chunked Connection: Keep-Alive ➜ balrog git:(db-migration) curl -I https://download.allizom.org/\?product\=firefox-15.0\&os\=osx\&lang\=en-US HTTP/1.1 302 Found Server: Apache X-Backend-Server: bouncer1.stage.webapp.phx1.mozilla.com Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private Content-Type: text/html; charset=UTF-8 Date: Tue, 25 Sep 2012 13:08:23 GMT Location: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/15.0/mac/en-US/Firefox%2015.0.dmg Pragma: no-cache Transfer-Encoding: chunked Connection: Keep-Alive But production doesn't 301 to https like that: ➜ balrog git:(db-migration) curl -I http://download.mozilla.org/\?product\=firefox-15.0\&os\=osx\&lang\=en-US HTTP/1.1 302 Found Server: Apache X-Backend-Server: pp-app-dist03 Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, private Content-Type: text/html; charset=UTF-8 Date: Tue, 25 Sep 2012 13:07:34 GMT Location: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/15.0/mac/en-US/Firefox%2015.0.dmg Pragma: no-cache Transfer-Encoding: chunked Connection: Keep-Alive Set-Cookie: dmo=10.8.84.212.1348578454323655; path=/; expires=Wed, 25-Sep-13 13:07:34 GMT X-Powered-By: PHP/5.1.6 Are you hoping to have me test the specific features from bug 613620, too?
Oh, if download.allizom.org is staging, I can't do any additional tests until the new code is pushed there anyways.
Everything should be pushed up to staging. :cturra, can you confirm?
:brandon - i have pushed the latest code to stage for you. in future, pls open a push request ticket to webops to have all future pushes to stage completed.
:bhearsum The code should be up on staging now, as :cturra says he's updated it. It would be helpful if you could test the features of bug 613620 so that someone besides me has tested it.
(In reply to Brandon Savage [:brandon] from comment #57) > :bhearsum The code should be up on staging now, as :cturra says he's updated > it. It would be helpful if you could test the features of bug 613620 so that > someone besides me has tested it. To be clear, the only part of bug 613620 that I'm part of is the modified global fallback. I'm unable to test other parts of it. I'm not even sure how much we care about in the CDN world, either.... In any case, I should be able to test the global fallback later this week or early next week - it takes a bit of time to get the IP blocks and tests setup, so I can't do it right away.
Depends on: 794567
OK, I tested the global fallback. Modifications to the staging environment that I made were: * Make releng-mirror01 an active mirror * Remove download-cdnetworks.mozilla.net and Mozilla CDN's from the Build Network region * Check the "Prevent global fallback" option for the Build Network region. * Made a bunch of IP block changes, resulting in this set-up: Ip start Ip end Region ******************************************************************************* 1073075717 (63.245.214.5) 1073075793 (63.245.214.81) US 1073075794 (63.245.214.82) 1073075794 (63.245.214.82) Release Engineering 1073075795 (63.245.214.83) 1073077467 (63.245.220.119) US 1073077468 (63.245.220.220) 1073077468 (63.245.220.220) Release Engineering 1073077469 (63.245.220.221) 1073077825 (63.245.222.65) US 1073077826 (63.245.222.66) 1073077826 (63.245.222.66) Release Engineering 1073077827 (63.245.222.67) 1073091397 (63.246.19.69) US Hide NATs for build network were discovered by running 'curl http://ip.cow.org' on a machine in each of these network classes: * SCL3 (.build): 63.245.214.82 (1073075794) * MTV1: 63.245.220.220 (1073077468) * SCL1 (all) and SCL3 (.winbuild): 63.245.222.66 (1073077826) However, I'm getting 404s. Is there a sentry instance updating stage with uptake data? If not, I can't test global fallback at all.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Depends on: 795068
I'm going to close this bug out, as there appears to be no work here to do. Getting sentry up and working for stage is being handled in bug 795068. I've I'm mistaken and there's more to be done here, please let me know. Thanks!
Status: REOPENED → RESOLVED
Closed: 13 years ago → 13 years ago
Resolution: --- → FIXED
Whiteboard: [triaged 20120831][waiting][pending bouncer cluster] → [triaged 20120831]
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.