Closed Bug 685169 Opened 15 years ago Closed 14 years ago

create new dev environment for next gen aus service

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: arich, Assigned: bhourigan)

References

Details

bhearsum and I had a long conversation about a new dev vm/machine that he and nthomas need to perform R&D for the next generation of the AUS service. With the idea that all services managed by IT should have a permanent dev environment going forward, this ticket is to help track what that dev environment should look like in order to continue functioning in the long term. I think what this technically boils down to is: * a vm on the mpt vlan (so other people in the company can get to it later if need be, like dm-ausstage01) * a read only nfs mount of 10.253.0.11:/vol/aus2/ onto this machine * mysql * apache * virtualenv so they can install some python libs The workflow is planned to go like this: * information from nightly builds are sent to dm-ausstage01, which also NFS mounts 10.253.0.11:/vol/aus2/ * a process on the dev machine will periodically scan the /opt/aus2/incoming/2/ directory (the partition mentioned above that's nfs mounted on both dm-ausstage01 and this dev host) for new files. * that process will parse all of the files in the /opt/aus2/incoming/2/ and if the files validate: * will check to see if the record already exists in the mysql database * if not, insert record * if so, ignore record * allow http(s?) connections to obtain some data from the dev server In addition, there will be release builds that are manually triggered by running the pushsnip script on dm-ausstage01 which currently rsyncs /opt/aus2/snippets/staging/<releasename> out to other aus servers but will be modified to also kick off a database import on this dev box. The import will follow the same format as above, but look in /opt/aus2/snippets/staging/<releasename> instead of /opt/aus2/incoming/2/. I recommended that any additional cron-based imports not be run while a manual push is running * they will only import new data, not any of the historical data on 10.253.0.11:/vol/aus2/ * currently 10.253.0.11:/vol/aus2/ is about 25G which includes a lot data for old releases (won't be imported) * They will age off nightly data in the db older than a week, like they do in AUS2 * release data in the db will not be aged * they expect the vm to be needed for active development for at least 6mo * there may be a need to have people outside of releng but within moco access this server in the future (hence the desire for the mpt vlan instead of the build vlan) Some of the pertinent questions at this point seem to be: 1) can that netapp share take any more IO load (perhaps jakem is the right person to answer this one) 2) should the db be split out from the dev host, and if so, where should that db reside (where do we have capacity on established in infra)? 3) Likely this should be a vm. Since it needs to be on the mpt vlan, have access to the netapp share, and will be run by IT, what vmware server should it reside on? Or conversely, what does IT recommend for a platform if not a vm. 4) What level of sudo access should be granted, and to whom (considering that this is active development). I recommend that some members from corey's team, spec ops, and releng get together next week at the all hands to hash things out.
(In reply to Amy Rich [:arich] from comment #0) > * allow http(s?) connections to obtain some data from the dev server To flush this out a bit more, the connections will mostly be Firefox installations pinging for updates. I'm sure we'll be loading some URLs manually from time to time, too. We should probably use https here, since that's what will be used in staging/production. A cert signed with our own root should be fine. > In addition, there will be release builds that are manually triggered by > running the pushsnip script on dm-ausstage01 which currently rsyncs > /opt/aus2/snippets/staging/<releasename> out to other aus servers but will > be modified to also kick off a database import on this dev box. The import > will follow the same format as above, but look in > /opt/aus2/snippets/staging/<releasename> instead of /opt/aus2/incoming/2/. > I recommended that any additional cron-based imports not be run while a > manual push is running Because of load, or something else? There's no collisions between release and nightly data to worry about. > * there may be a need to have people outside of releng but within moco > access this server in the future (hence the desire for the mpt vlan instead > of the build vlan) To be clear, I don't expect anyone outside of IT/RelEng to need ssh access to the machine, but at some point we may want to allow others to point their Firefoxes at it (which would be http/https only). > 4) What level of sudo access should be granted, and to whom (considering > that this is active development). We're not close to having a thing that's manageable by IT, so I would _really_ prefer to have full root at this time. I don't want to be blocked by IT if things go wrong, or need updating.
Assignee: server-ops-releng → arich
Nick and I chatted with Corey about this on Friday. Here's a couple of additions that came out of that: * This is going to be a "proper" development environment, meaning that RelEng will have read-only access to it, and IT will manage the machine fully, including automatically pulling from source control. * This will probably be a pair of seamicro (?) machines Regarding pulling from source control, we don't have a final repository set-up for this yet. Can we point it at https://hg.mozilla.org/users/nthomas_mozilla.com/aus3-proto/ for now, and change it later (possibly to a git repository). Corey, I _think_ you said the ETA was end of this week, but I'm not 100% sure.
Assignee: arich → server-ops
Component: Server Operations: RelEng → Server Operations
QA Contact: zandr → cshields
Assignee: server-ops → cshields
adm, dev, and stage nodes are all built out last week. bhearsum and nthomas were both on PTO and unavailable to talk about software requirements and needs outside of the webheads and mysql. Filed bug 688972 to get network flows opened up for the webheads and mysql. Once that is done we will be ready to push out code.
Awesome, we really appreciate the fast movement here. I'm waiting for Nick to get back so we can decide where the permanent code repository is going to be (bug 687476). We should also chat today so that I can figure out how I'll need to modify the nightly snippet importer to work in this world.
Depends on: 687476
If you could start by filling out any documentation around how it works (like the flow for a snippet importer), that would help out a lot. https://mana.mozilla.org/wiki/display/websites/aus3.mozilla.org
Component: Server Operations → Server Operations: Web Operations
Depends on: 689289
Depends on: 689297
Just for clarification here, this is likely "aus4" since aus3.mozilla.org is an existing hostname and something we used when we migrated to the two-CA cert for AUS.
(In reply to Corey Shields [:cshields] from comment #5) > If you could start by filling out any documentation around how it works > (like the flow for a snippet importer), that would help out a lot. > > https://mana.mozilla.org/wiki/display/websites/aus3.mozilla.org I updated this page last week.
(In reply to Ben Hearsum [:bhearsum] from comment #7) > (In reply to Corey Shields [:cshields] from comment #5) > > If you could start by filling out any documentation around how it works > > (like the flow for a snippet importer), that would help out a lot. > > > > https://mana.mozilla.org/wiki/display/websites/aus3.mozilla.org > > I updated this page last week. thanks, I noticed that.. Zimbra has had us tied this week but I think (and hope) we are done with that now, so once I get caught up with the last 2 days of work that has been pushed aside I'll get back on this.
I spent some time a couple of weeks ago trying to figure out how to best deploy and configure the new application. Through chatting with rhelmer, catlee, and cshields as well as some experimentation I _think_ I've got it figured out. I want to run things by IT before I get too far, though. So, here's what I have (please bear with me, I'm still new to this area): * The code repository contains all the necessary dependencies in vendor/, except for binary ones, which I understand are typically installed with an RPM. * I've got a wsgi runner for the client-facing application ("client.wsgi"), which adds the necessary directories to sys.path * client.wsgi looks to /etc/aus/client.ini for configuration. The -dist version of this file looks as follows: [database] ;Database to be used by AUS applications, in URI format ;dburi=sqlite:////var/www/aus/update.db ;dburi=mysql://user:password@host/database [logging] ;Where to put the application log. No rotation is done on this file. logfile=/var/log/aus.log To sanity check the set-up, I tried this with a local apache with the following configuration: <VirtualHost *> ServerName localhost DocumentRoot /var/www/aus WSGIDaemonProcess aus user=bhearsum group=bhearsum threads=5 WSGIScriptAlias / /var/www/aus/client.wsgi <Directory /var/www/aus> WSGIProcessGroup aus WSGIApplicationGroup %{GLOBAL} Order deny,allow Allow from all </Directory> </VirtualHost> ...which worked fine after I cloned the repo into /var/www/aus and creating /etc/aus/client.ini At some point in the future we'll have a second wsgi app for the administration interface, I'm not sure if that matters at all to you right now, though. I'd also like to chat with somebody about authentication for the administration interface, but bugzilla probably isn't the right place for that.
Corey and I chatted on the phone about this morning. Here's a recap: - He is happy with the wsgi script, and using /etc/aus for configuration - He is happy with using HTTP Auth for authentication on the admin-app side, but advised me to also speak with infrasec about it. - Once I have the wsgi script and related changes checked into the repository (https://github.com/mozilla/balrog), I can ping him and he'll try to get it going in the dev env. This probably won't be until Monday at the earliest.
(In reply to Ben Hearsum [:bhearsum] from comment #10) > - Once I have the wsgi script and related changes checked into the > repository (https://github.com/mozilla/balrog), I can ping him and he'll try > to get it going in the dev env. This part is ready. Corey, here's a list of the things I know of that need to happen on the server. To set-up the application: - Clone git://github.com/mozilla/balrog.git somewhere - Create /etc/aus/client.ini based on the dist file (client.ini-dist in the clone). This config should have read-only access to the DB. - Setup the web server with the client-facing application (client.wsgi in the clone). It must be run with a Python 2.6 installation with SQLAlchemy (0.7.x), simplejson, and jinja2 To create the initial nightly rule, and set-up automatic nightly imports: - Run the following SQL on the database: INSERT INTO rules (rule_id, priority, mapping, throttle, update_type, product, channel, data_version) VALUES (1, 100, 'Firefox-mozilla-central-nightly-latest', 100, 'minor', 'Firefox', 'nightly', 1); - Setup a cronjob that runs the following at 9am every morning: /path/to/balrog/scripts/import-nightly-snippets.sh python-executable path-to-snippet-dir dburi-with-write-access /path/to/balrog/directory -- python-executable must be Python 2.6 w/ sqlalchemy + simplejson (eg, the same one used in the application) -- path-to-snippet-dir could be /opt/aus2/incoming/2/Firefox, depending on where the production snippets are mounted The aforementioned cronjob is a stop-gap measure until we have the basic admin application ready, and the build machines talking to it. This is probably a few weeks away. Corey, let me know if you have any questions about the above, or if I can help in any way.
Assignee: cshields → bhourigan
Can the configuration file (client.ini) go in the document root as opposed to /etc/aus?
(In reply to Brian Hourigan [:digi] from comment #12) > Can the configuration file (client.ini) go in the document root as opposed > to /etc/aus? We've had problems on various webdev projects where someone doesn't secure the docroot and this ends up readable, so I suggested this to bhearsum (it's how we do it on Socorro - default config in the docroot, but overrides with sensitive data in /etc). The idea is that IT only has to care about the specific values they care about, and that file can be controlled by puppet and not be gitignored because it's in the clone dir etc. I'm not arguing for/against specifically, just giving context since I think I'm the one that pushed this idea :)
I was involved in this design decision with rhelmer for socorro, and I think it is nicer for puppet to manage specific configurations like that, rather than it be involved with the push process. This way pushing code is one thing, changing configuration is a different thing. Configuration is managed by puppet, code is managed however the regular push procedures are done...
Ok - no problem. It will stay in /etc. :) Stay tuned for further updates.
(In reply to Brian Hourigan [:digi] from comment #15) > Ok - no problem. It will stay in /etc. :) Stay tuned for further updates. since the config file will be unique for dev, stage, and prod - push them out in webapp/manifests/aus/[dev.pp stage.pp prod.pp]
As promised got the dev environment setup today. I am able to interact with client.wsgi but I'm getting an exception. I think this has something to do with the code. Traceback (from /var/log/aus.log) is as follows: [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] mod_wsgi (pid=14938): Target WSGI script '/data/www/aus4-dev.allizom.org/client.wsgi' cannot be loaded as Python module. [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] mod_wsgi (pid=14938): Exception occurred processing WSGI script '/data/www/aus4-dev.allizom.org/client.wsgi'. [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] Traceback (most recent call last): [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/data/www/aus4-dev.allizom.org/client.wsgi", line 23, in <module> [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] AUS.setDb(cfg.getDburi()) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/data/www/aus4-dev.allizom.org/auslib/AUS.py", line 17, in setDb [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] self.db = AUSDatabase(dbname) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/data/www/aus4-dev.allizom.org/auslib/db.py", line 694, in __init__ [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] self.setDburi(dburi) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/data/www/aus4-dev.allizom.org/auslib/db.py", line 704, in setDburi [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] self.metadata.create_all() [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/schema.py", line 2434, in create_all [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] tables=tables) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 2228, in _run_visitor [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] conn._run_visitor(visitorcallable, element, **kwargs) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1898, in _run_visitor [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] **kwargs).traverse_single(element) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/visitors.py", line 86, in traverse_single [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] return meth(obj, **kw) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/ddl.py", line 65, in visit_metadata [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] self.traverse_single(table, create_ok=True) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/visitors.py", line 86, in traverse_single [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] return meth(obj, **kw) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/ddl.py", line 82, in visit_table [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] self.connection.execute(schema.CreateTable(table)) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1399, in execute [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] params) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1484, in _execute_ddl [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] compiled = ddl.compile(dialect=dialect) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/expression.py", line 1655, in compile [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] return self._compiler(dialect, bind=bind, **kw) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/schema.py", line 2771, in _compiler [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] return dialect.ddl_compiler(dialect, self, **kw) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 699, in __init__ [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] self.string = self.process(self.statement) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 718, in process [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] return obj._compiler_dispatch(self, **kwargs) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/visitors.py", line 59, in _compiler_dispatch [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] return getter(visitor)(self, **kw) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/compiler.py", line 1269, in visit_create_table [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] not first_pk [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/dialects/mysql/base.py", line 1369, in get_column_specification [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] self.dialect.type_compiler.process(column.type) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 764, in process [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] return type_._compiler_dispatch(self) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/visitors.py", line 59, in _compiler_dispatch [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] return getter(visitor)(self, **kw) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/compiler.py", line 1587, in visit_string [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] return self.visit_VARCHAR(type_) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] File "/usr/lib/python2.6/site-packages/sqlalchemy/dialects/mysql/base.py", line 1656, in visit_VARCHAR [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] self.dialect.name) [Fri Nov 04 12:50:32 2011] [error] [client 10.8.33.248] InvalidRequestError: VARCHAR requires a length on dialect mysql
It's set to pull from git every 5 minutes, the web heads will pull the content every ~15. So you should see an average of 10min delay for code pushes. Please also let me know the path to the snippit mount on the netapp, I will get that set and I think this is done.
(In reply to Brian Hourigan [:digi] from comment #18) > It's set to pull from git every 5 minutes, the web heads will pull the > content every ~15. So you should see an average of 10min delay for code > pushes. Great! > Please also let me know the path to the snippit mount on the netapp, I will > get that set and I think this is done. Should be /vol/aus2. One last thing: Corey mentioned that we could have read-only access in order to help us debug things by looking at web server logs, app logs, etc. without needing to involve you. Is this set-up?
I was able to reproduce the MySQL issue locally. I just pushed a fix.
(In reply to Ben Hearsum [:bhearsum] from comment #19) > One last thing: Corey mentioned that we could have read-only access in order > to help us debug things by looking at web server logs, app logs, etc. > without needing to involve you. Is this set-up? bhearsum and nthomas should already have access (using their ssh keys in ldap) on the dev nodes as outlined in mana.. (note that the stage nodes are not setup yet) https://mana.mozilla.org/wiki/display/websites/aus4.mozilla.org you will need to vpn through mpt-vpn or khan (not sure which has routes off the top of my head)
(In reply to Corey Shields [:cshields] from comment #21) > (In reply to Ben Hearsum [:bhearsum] from comment #19) > > One last thing: Corey mentioned that we could have read-only access in order > > to help us debug things by looking at web server logs, app logs, etc. > > without needing to involve you. Is this set-up? > > bhearsum and nthomas should already have access (using their ssh keys in > ldap) on the dev nodes as outlined in mana.. (note that the stage nodes are > not setup yet) > > https://mana.mozilla.org/wiki/display/websites/aus4.mozilla.org > > you will need to vpn through mpt-vpn or khan (not sure which has routes off > the top of my head) Oh, I just tried through mpt-vpn and it worked, thanks!
Ben do you know the hostname of a server that mounts the NFS volume? Need to find out which shelf this mount is on.
(In reply to Brian Hourigan [:digi] from comment #23) > Ben do you know the hostname of a server that mounts the NFS volume? Need to > find out which shelf this mount is on. dm-ausstage01 is one of the hosts that mounts it.
I've mounted /vol/pio_aus2 on the phoenix netapp as /opt on these hosts. I see that dm-ausstage01 has an rsync to copy data over. If you need some directories added to this sync let me know and we can get them copied over I would prefer using a local NFS mount to a Phoenix netapp vs going over the VPN to MPT. Please let me know if there is anything I've missed in the setup of this environment.
(In reply to Brian Hourigan [:digi] from comment #25) > I've mounted /vol/pio_aus2 on the phoenix netapp as /opt on these hosts. I > see that dm-ausstage01 has an rsync to copy data over. If you need some > directories added to this sync let me know and we can get them copied over > I would prefer using a local NFS mount to a Phoenix netapp vs going over the > VPN to MPT. I can't think of a reason why we need to directly mount the MPT netapp, so the current set-up sounds fine. > Please let me know if there is anything I've missed in the setup of this > environment. Oh, one thing I did notice yesterday is that /var/log/httpd is inaccessible to me. Can that get fixed? I'm going to do a full check and test to make sure that the cronjob is working, and that clients can update, ASAP. (I intended to sooner, but got pulled into some high priority work.) Thanks for your work here!
I poked around more today and have some questions/issues: - I can't seem to access either of the dev machines on port 80 or 443 - the connection simply times out eventually. If I use a couple of port forwards through mpt-vpn, I can successfully talk to the server. - When I try a request to the server, I get a 403 error: [bhearsum@node267.seamicro.phx1 ~]$ telnet localhost 80 Trying ::1... Connected to localhost. Escape character is '^]'. GET /update/3/Firefox/3.6.12/20101026210630/WINNT_x86-msvc/ru/beta/unteo/huteon/htueo/update.xml <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access /update/3/Firefox/3.6.12/20101026210630/WINNT_x86-msvc/ru/beta/unteo/huteon/htueo/update.xml on this server.</p> </body></html> Connection closed by foreign host. - The Confluence page talks about a public interface (https://mana.mozilla.org/wiki/display/websites/aus4.mozilla.org#aus4mozillaorg-Publicinterfaces), can that be activated soon?
Unrelated to my previous comment, I fixed the application not to try to create tables in the wsgi app. Could you run the following from the repo root to get the tables created? python scripts/manage-db.py --db [dburi with CREATE permissions] create
Hi Ben You are seeing forbidden during your HTTP test because you aren't using HTTP/1.1 and your request is in the context of the _default_ VirtualHost (the RedHat test page) If you send a HTTP/1.1 request with a Host of aus4-dev.allizom.org you should get a rewrite to HTTPS (which indicates you're at least hitting the right VirtualHost) You can see my HTTP request here: http://digi.pastebin.mozilla.org/1379390 Then the openssl s_client to simulate an SSL connection: http://digi.pastebin.mozilla.org/1379392 I can't ascertain as to whether the response of this request is valid but it seems to be working in some fashion. So as you know the dev[1,2] hosts are behind a vpn in Phoenix. If you need to access these machines directly from a public network I will need to talk to netops about that. The public interfaces listed in Mana represents the load balancer IP. I tested from an external (Non-Mozilla) host and it seems to respond: digi:~> telnet 63.245.217.120 443 Trying 63.245.217.120... Connected to aus4-dev.zlb.phx.mozilla.net. Escape character is '^]'. My apologies if I am misunderstanding your problem. If you can please give me some more information (where you're connecting from) and we will try to debug this issue. I've ran the manage-db.py as requested. I don't mind running these but I imagine there will be times in the future where you want to run this as well. You should have permission to run this. If not let me know and I will fix it. You can find the database URI in client.ini. [root@node267.seamicro.phx1 aus4-dev.allizom.org]# python scripts/manage-db.py --db mysql://XXXXX/aus4_dev create [root@node267.seamicro.phx1 aus4-dev.allizom.org]#
It seems like everything in comment #26 was only an issue because of my mis-understanding of the architecture. https://aus4-dev.allizom.org/ is working exactly as expected in terms of request/response. However, the SSL cert looks like to be invalid. I have the Mozilla Root Certificate, but Firefox still complains that "The certificate is only valid for aus4-dev.zlb.phx.mozilla.net". Because the test requests are coming from the Update Service inside of Firefox, there's no way to click through these errors. Is it possible to change that certificate to match the domain? I just successfully used the credentials in /etc/aus/client.ini to make a change to the database, so that's great - I won't need to ping you for other such changes. However, the client.wsgi interface only require read-only access to the database. Should that config file be changed to use a read-only account to the database?
I've updated the SSL certificate to use the common name of 'aus4-dev.allizom.org', sorry about the confusion. I've changed the MySQL VIP to use the RO IP address. It's committed to puppet but will take ~30 min to deploy to the nodes.
Confirmed the RO VIP has deployed to the nodes
Yup, that part looks good to me, too! I'm seeing a couple of other issues now: Intermittently, the application loses the MySQL server, resulting in an exception like this: ERROR:auslib.client.base:Exception on /update/3/Firefox/10.0a1/20111108031146/Linux_x86_64-gcc3/en-US/nightly/Linux 3.0.0-12-generic (GTK 2.24.6)/default/default/update.xml [GET] Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/flask/app.py", line 1292, in wsgi_app response = self.full_dispatch_request() File "/usr/lib/python2.6/site-packages/flask/app.py", line 1062, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/lib/python2.6/site-packages/flask/app.py", line 1060, in full_dispatch_request rv = self.dispatch_request() File "/usr/lib/python2.6/site-packages/flask/app.py", line 1047, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/lib/python2.6/site-packages/flask/views.py", line 56, in view return self.dispatch_request(*args, **kwargs) File "/usr/lib/python2.6/site-packages/flask/views.py", line 112, in dispatch_request return meth(*args, **kwargs) File "/data/www/aus4-dev.allizom.org/auslib/client/views/client.py", line 50, in get query = self.getQueryFromURL(queryVersion, url) File "/data/www/aus4-dev.allizom.org/auslib/client/views/client.py", line 42, in getQueryFromURL query['name'] = AUS.identifyRequest(query) File "/data/www/aus4-dev.allizom.org/auslib/AUS.py", line 27, in identifyRequest for release in self.releases.getReleases(product=updateQuery['product'], version=updateQuery['version']): File "/data/www/aus4-dev.allizom.org/auslib/db.py", line 549, in getReleases rows = self.select(where=where, limit=limit) File "/data/www/aus4-dev.allizom.org/auslib/db.py", line 20, in convertRows for row in fn(*args, **kwargs): File "/data/www/aus4-dev.allizom.org/auslib/db.py", line 186, in select return query.execute().fetchall() File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/expression.py", line 2667, in execute return e._execute_clauseelement(self, multiparams, params) File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 2298, in _execute_clauseelement return connection._execute_clauseelement(elem, multiparams, params) File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1532, in _execute_clauseelement compiled_sql, distilled_params File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_context context) File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1633, in _execute_context context) File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 330, in do_execute cursor.execute(statement, parameters) File "/usr/lib/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute self.errorhandler(self, exc, value) File "/usr/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue OperationalError: (OperationalError) (2006, 'MySQL server has gone away') 'SELECT releases.name, releases.product, releases.version, releases.data, releases.data_version \nFROM releases \nWHERE releases.product = %s AND releases.version = %s' (u'Firefox', u'10.0a1') When I don't get that, I get a 200 OK that the client doesn't seem to like. That's probably an application issue though.
Catlee just told me that I probably need to set a timeout in SQLAlchemy to fix the ISAs, I'll give that a try.
Depends on: 701689
Depends on: 701865
bug 701689 seems to have helped with the mysql server going away, but not fixed it completely. I need to look into that more next week. When I tried to put data into the system I hit bug 701865. After working around that, I managed to get a proper response from the server (yay!). For example, from: https://aus4-dev.allizom.org/update/3/Firefox/9.0a1/20110906030844/Linux_x86_64-gcc3/en-US/nightly/uheont/hueton/htneo/update.xml?force=1 However, when Firefox tries to use it to update, it still seems to complain about the Cert. It says "Certificate issuer is not built-in.", which seems like it's complaining about it being self signed. I'm going to look into ways to work around this next week: possible through some prefs to disable that check, or possibly by switching to http for the dev environment, maybe something else.
Spoke with Ben on IRC - all good.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
For posterity: > bug 701689 seems to have helped with the mysql server going away, but not > fixed it completely. I need to look into that more next week. I haven't been able to repro this this week....but will continue to track the issue in the aforementioned bug. > When I tried to put data into the system I hit bug 701865. After working > around that, I managed to get a proper response from the server (yay!). For > example, from: > https://aus4-dev.allizom.org/update/3/Firefox/9.0a1/20110906030844/ > Linux_x86_64-gcc3/en-US/nightly/uheont/hueton/htneo/update.xml?force=1 > > However, when Firefox tries to use it to update, it still seems to complain > about the Cert. It says "Certificate issuer is not built-in.", which seems > like it's complaining about it being self signed. I'm going to look into > ways to work around this next week: possible through some prefs to disable > that check, or possibly by switching to http for the dev environment, maybe > something else. This error can be worked around by setting app.update.cert.requireBuiltIn to false in about:config.
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.