1193879 - [traceback] exceptions:TypeError: 'NoneType' object is not iterable resulting from product details update

Status: RESOLVED FIXED

(www.mozilla.org :: Bedrock, defect)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

I couldn't reproduce this manually with an IE 6/Windows XP VM via Sauce Labs, but it's showing up in New Relic[0] a few times, today:

exceptions:TypeError: 'NoneType' object is not iterable on /es-ES/firefox/installer-help/

According to NR, it stems from a GET to https://www.mozilla.org/es-ES/firefox/installer-help/ with user-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

Traceback (most recent call last):
File "/data/www/www.mozilla.org-django/venv/lib/python2.7/site-packages/newrelic-2.50.0.39/newrelic/api/web_transaction.py", line 1329, in _nr_wsgi_application_wrapper_
File "/data/www/www.mozilla.org-django/venv/lib/python2.7/site-packages/whitenoise/base.py", line 62, in __call__
File "/data/www/www.mozilla.org-django/venv/lib/python2.7/site-packages/newrelic-2.50.0.39/newrelic/api/web_transaction.py", line 1215, in _nr_wsgi_application_wrapper_
File "/data/www/www.mozilla.org-django/venv/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 187, in __call__
File "/data/www/www.mozilla.org-django/venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
File "/data/www/www.mozilla.org-django/venv/lib/python2.7/site-packages/newrelic-2.50.0.39/newrelic/hooks/framework_django.py", line 499, in wrapper
File "/data/www/www.mozilla.org-django/bedrock/bedrock/firefox/views.py", line 157, in installer_help
File "/data/www/www.mozilla.org-django/bedrock/lib/l10n_utils/__init__.py", line 101, in render
File "/data/www/www.mozilla.org-django/venv/lib/python2.7/site-packages/django/shortcuts.py", line 50, in render
File "/data/www/www.mozilla.org-django/venv/lib/python2.7/site-packages/django/template/loader.py", line 178, in render_to_string
File "/data/www/www.mozilla.org-django/venv/lib/python2.7/site-packages/jingo/__init__.py", line 195, in render
File "/data/www/www.mozilla.org-django/venv/lib/python2.7/site-packages/newrelic-2.50.0.39/newrelic/api/function_trace.py", line 98, in dynamic_wrapper
File "/data/www/www.mozilla.org-django/venv/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
File "/data/www/www.mozilla.org-django/venv/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
File "/data/www/www.mozilla.org-django/bedrock/bedrock/firefox/templates/firefox/installer-help.html", line 5, in top-level template code
File "/data/www/www.mozilla.org-django/bedrock/bedrock/firefox/templates/firefox/base-resp.html", line 7, in top-level template code
File "/data/www/www.mozilla.org-django/bedrock/bedrock/base/templates/base-resp.html", line 141, in top-level template code
File "/data/www/www.mozilla.org-django/bedrock/bedrock/firefox/templates/firefox/installer-help.html", line 66, in block "content"
File "/data/www/www.mozilla.org-django/bedrock/bedrock/firefox/helpers.py", line 83, in download_firefox

[0]New Relic: https://rpm.newrelic.com/accounts/263620/applications/2639284/traced_errors/39c6d9-d2ccea20-4119-11e5-871a-c81f66b8ceca

Comment 1

[Josh Mize [:jgmize]]

These errors were the result of a race condition resulting from non-atomic updates of the product_details json files via rsync coinciding with a cache miss. We are currently working on transitioning away from this update methodology to an atomic update.

Comment 2

[Josh Mize [:jgmize]]

We moved away from rsync and file-based product details to a database-backed update system last year in an effort to make these updates atomic and eliminate this issue. We've also iterated on this new product-details storage and update mechanism multiple times in efforts to eliminate this problem over the past several months, but we saw a spike of these errors in each bedrock deployment in production earlier today. When I say "these errors", I mean we saw exactly the same kind of tracebacks as in the original comment, for similar reasons, even though the underlying data storage and update mechanism have changed completely. 

In the EU cluster  errors were reported from ~2:25 - ~2:35 PDT, and in the US cluster we saw errors from ~2:29 - ~2:43 PDT. There was a concentrated peak of errors in each case, and the EU deployment was recovering as the US deployment started seeing a high percentage of errors, and the route53 traffic policy was effective in routing the traffic away from each cluster during those peaks, which reduced the end-user impact, but there were ~2-3 minutes of overlap where both deployments were returning errors for a large percentage of requests. Note that this "large percentage" applies only to cache miss requests: Cloudflare continued to serve requests from cache during this time, minimizing the end-user impact as we currently have a cache hit ratio of 93%.

However effective the mitigations were at the infrastructure level, we still need to prioritize eliminating the root cause of this problem. While the database updates themselves may be atomic, the cache updates are clearly not. I think we should consider eliminating the DB + cache update solution we put in place and go back to filesystem storage: now that we deploy using containers instead of rsync this could be an atomic update from each container's perspective. What do you think, :pmac?

Comment 3

[Josh Mize [:jgmize]]

Attachment: bedrock PR 4062

Comment 4

[Josh Mize [:jgmize]]

:pmac backported https://github.com/mozilla/django-product-details/pull/56 and https://github.com/mozilla/django-product-details/pull/57 in https://github.com/mozilla/django-product-details/commits/django-1.7-backport and created https://github.com/mozilla/bedrock/pull/4062 to upgraded bedrock to that branch. In my comment #2 I mistakenly thought that we had already installed those fixes in bedrock and we had a new, different problem with an unknown root cause, instead of what was actually known bugs that we just needed to backport for compatibility with the current version of Django bedrock is using. Those fixes went to production on 2016-04-13.