1200349 - Production stackato apps giving intermittent 404s for pages and static assets

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Ioana Chiorean]

Attachment: Screenshot 2015-08-31 21.36.31.png

https://oneanddone.mozilla.org/en-US/ is returning either 404 either a basic html page
See screenshot attached.

Comment 1

[Bob Silverberg [:bsilverberg]]

It does occasionally render fully for me, but I can confirm that I am also getting intermittent 404 errors and missing css as well.

Comment 2

[Bob Silverberg [:bsilverberg]]

I have received a few error emails from the site which might shed light on the problem. The text of one such email is attached below.

The error message included is "OperationalError: (1130, "Host '10.22.93.33' is not allowed to connect to this MySQL server")"

Full email below:

Internal Server Error: /en-US/
Traceback (most recent call last):
  File "/app/python/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/ActivePython-2.7/lib/python2.7/site-packages/newrelic-1.11.0.55/newrelic/api/object_wrapper.py", line 216, in __call__
    self._nr_instance, args, kwargs)
  File "/opt/ActivePython-2.7/lib/python2.7/site-packages/newrelic-1.11.0.55/newrelic/hooks/framework_django.py", line 475, in wrapper
    return wrapped(*args, **kwargs)
  File "/app/python/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "./oneanddone/base/views.py", line 36, in dispatch
    return super(HomeView, self).dispatch(request, *args, **kwargs)
  File "/app/python/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/app/python/lib/python2.7/site-packages/django_filters/views.py", line 63, in get
    object_list=self.object_list)
  File "./oneanddone/base/views.py", line 23, in get_context_data
    ctx = super(HomeView, self).get_context_data(*args, **kwargs)
  File "/app/python/lib/python2.7/site-packages/django/views/generic/list.py", line 120, in get_context_data
    paginator, page, queryset, is_paginated = self.paginate_queryset(queryset, page_size)
  File "/app/python/lib/python2.7/site-packages/django/views/generic/list.py", line 64, in paginate_queryset
    page = paginator.page(page_number)
  File "/app/python/lib/python2.7/site-packages/django/core/paginator.py", line 50, in page
    number = self.validate_number(number)
  File "/app/python/lib/python2.7/site-packages/django/core/paginator.py", line 39, in validate_number
    if number > self.num_pages:
  File "/app/python/lib/python2.7/site-packages/django/core/paginator.py", line 86, in _get_num_pages
    if self.count == 0 and not self.allow_empty_first_page:
  File "/app/python/lib/python2.7/site-packages/django/core/paginator.py", line 72, in _get_count
    self._count = self.object_list.count()
  File "/app/python/lib/python2.7/site-packages/django/db/models/query.py", line 338, in count
    return self.query.get_count(using=self.db)
  File "/app/python/lib/python2.7/site-packages/django/db/models/sql/query.py", line 436, in get_count
    number = obj.get_aggregation(using=using)[None]
  File "/app/python/lib/python2.7/site-packages/django/db/models/sql/query.py", line 402, in get_aggregation
    result = query.get_compiler(using).execute_sql(SINGLE)
  File "/app/python/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 785, in execute_sql
    cursor = self.connection.cursor()
  File "/app/python/lib/python2.7/site-packages/django/db/backends/__init__.py", line 167, in cursor
    cursor = utils.CursorWrapper(self._cursor(), self)
  File "/app/python/lib/python2.7/site-packages/django/db/backends/__init__.py", line 138, in _cursor
    self.ensure_connection()
  File "/app/python/lib/python2.7/site-packages/django/db/backends/__init__.py", line 133, in ensure_connection
    self.connect()
  File "/app/python/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/app/python/lib/python2.7/site-packages/django/db/backends/__init__.py", line 133, in ensure_connection
    self.connect()
  File "/app/python/lib/python2.7/site-packages/django/db/backends/__init__.py", line 122, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/app/python/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 472, in get_new_connection
    conn = Database.connect(**conn_params)
  File "/opt/ActivePython-2.7/lib/python2.7/site-packages/newrelic-1.11.0.55/newrelic/hooks/database_dbapi2.py", line 84, in __call__
    return ConnectionWrapper(self.__connect(*args, **kwargs),
  File "/opt/ActivePython-2.7/lib/python2.7/site-packages/newrelic-1.11.0.55/newrelic/api/object_wrapper.py", line 216, in __call__
    self._nr_instance, args, kwargs)
  File "/opt/ActivePython-2.7/lib/python2.7/site-packages/newrelic-1.11.0.55/newrelic/api/function_trace.py", line 93, in literal_wrapper
    return wrapped(*args, **kwargs)
  File "/app/python/lib/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/app/python/lib/python2.7/site-packages/MySQLdb/connections.py", line 193, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (1130, "Host '10.22.93.33' is not allowed to connect to this MySQL server")

Request repr(): 
<WSGIRequest
path:/en-US/,
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{'_ga': 'GA1.2.1631471024.1440102589',
 '_gat': '1',
 'anoncsrf': '15pVxapPZiZLUF6q5ZsRNChhFq5ZldJb',
 'optimizelyBuckets': '%7B%7D',
 'optimizelyEndUserId': 'oeu1440447549384r0.3152076066268954',
 'optimizelySegments': '%7B%22246002457%22%3A%22campaign%22%2C%22246073290%22%3A%22ff%22%2C%22245984388%22%3A%22false%22%2C%22246073289%22%3A%22none%22%2C%222000810488%22%3A%22false%22%2C%222017550344%22%3A%22ff%22%2C%221994990450%22%3A%22none%22%2C%222011280991%22%3A%22referral%22%7D'},
META:{'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'HTTP_ACCEPT_ENCODING': 'gzip, deflate',
 'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.5',
 'HTTP_CACHE_CONTROL': 'no-cache',
 'HTTP_CONNECTION': 'keep-alive',
 'HTTP_COOKIE': '_ga=GA1.2.1631471024.1440102589; optimizelySegments=%7B%22246002457%22%3A%22campaign%22%2C%22246073290%22%3A%22ff%22%2C%22245984388%22%3A%22false%22%2C%22246073289%22%3A%22none%22%2C%222000810488%22%3A%22false%22%2C%222017550344%22%3A%22ff%22%2C%221994990450%22%3A%22none%22%2C%222011280991%22%3A%22referral%22%7D; optimizelyEndUserId=oeu1440447549384r0.3152076066268954; optimizelyBuckets=%7B%7D; anoncsrf=15pVxapPZiZLUF6q5ZsRNChhFq5ZldJb; _gat=1',
 'HTTP_DNT': '1',
 'HTTP_HOST': 'oneanddone.mozilla.org',
 'HTTP_PRAGMA': 'no-cache',
 'HTTP_SSLCLIENTCERTSTATUS': 'NoClientCert',
 'HTTP_SSLCLIENTCIPHER': 'SSL_DHE_RSA_WITH_AES_128_CBC_SHA, version=TLSv1.2, bits=128',
 'HTTP_SSLSESSIONID': '2194DE3E22806D49A6518045CFD1BEF26455F9AE0DE17D339B310D87E96CE01B',
 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0',
 'HTTP_X_CLUSTER_CLIENT_IP': '63.245.221.32',
 'HTTP_X_FORWARDED_FOR': '63.245.221.32,10.22.93.208',
 'HTTP_X_FORWARDED_PORT': '34010',
 'HTTP_X_FORWARDED_PROTO': 'https',
 'PATH_INFO': u'/en-US/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '10.22.93.31',
 'REQUEST_METHOD': 'GET',
 'REQUEST_URI': '/en-US/',
 'SCRIPT_NAME': u'',
 'SERVER_NAME': '851b69b161d03fde062244b505d8ad00',
 'SERVER_PORT': '3000',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'uwsgi.node': '851b69b161d03fde062244b505d8ad00',
 'uwsgi.version': '1.4.9',
 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x3323420>,
 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
 'wsgi.input': <newrelic.api.web_transaction.WSGIInputWrapper object at 0x34d2c10>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0)}>

Comment 3

[Bob Silverberg [:bsilverberg]]

I am seeing similar errors but with different IP addresses, e.g.,

OperationalError: (2003, "Can't connect to MySQL server on '10.22.93.37' (111)")
OperationalError: (1130, "Host '10.22.93.32' is not allowed to connect to this MySQL server")

Comment 4

[Josh Mize [:jgmize]]

we are seeing the same issues (unable to connect to db, intermittent 404s on either pages or static assets like css) on nucleus.mozilla.org, which started at the same time.

Comment 5

[Josh Mize [:jgmize]]

(In reply to Josh Mize [:jgmize] from comment #4)
> we are seeing the same issues (unable to connect to db, intermittent 404s on
> either pages or static assets like css) on nucleus.mozilla.org, which
> started at the same time.

To clarify: these are related because both apps are currently on the same stackato cluster.

Comment 6

[Josh Mize [:jgmize]]

I just deleted and re-pushed the nucleus app, which restored service for me. This is usually a last resort for me, as it's not entirely without risk, but I've needed to do this several times in the past when there were similar issues with stackato. The steps are

1. Make sure stackato.yml is linked to correct environment:

$ ls -l stackato.yml
lrwxrwxrwx 1 jgmize jgmize 17 Apr 17 15:25 stackato.yml -> stackato.yml.prod

2. log in to the prod stackato cluster and switch to the correct group

$ stackato target https://api.paas.mozilla.org
$ stackato login
$ stackato group nucleus

3. Get the current environment variables

$ stackato env

4. Delete the app, making absolutely sure *not* to delete the provisioned DB:

$ stackato delete
Provisioned service [nucleus-db] detected would you like to delete it ?:  [yN]: n
Deleting application [nucleus] ... OK

5. Re-create the app

$ stackato push

6. verify that the default environment variables match what you saw in step 3, or put in the correct values if not. Also verify that it binds to the correct DB (all this worked for me, but YMMV)

Comment 7

[Bob Silverberg [:bsilverberg]]

That worked to fix oneanddone as well, jgmize. Thanks for that!