1167924 - Aggregated metrics dashboard is down

Status: RESOLVED WONTFIX

(support.mozilla.org :: Localization, task)

Description

[vesper]

https://support.mozilla.org/kb/dashboard/metrics/aggregated is resolving to the error page:

"An Error Occurred

Oh, no! It looks like an unexpected error occurred. We've already notified the site administrators. Please try again now, or in a few minutes."

Comment 1

[Michael Cooper [:mythmon]]

This seems to be working now. Looking at errormill, I don't see anything that matches the time you reported this. In the future, knowing when the error occurred, event approximately is really useful.

If this pops again, please re-open this bug, but for now I'm going to close it out.

Comment 2

[vesper]

It's 05:46 AM PST time, and it's there again (thanks to Tonnes for noticing).

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

It's working for me, so I'm puzzled.

Comment 4

[Michael Cooper [:mythmon]]

Bingo. This is the error in error mill: https://errormill.mozilla.org/support/support/group/386211/

I don't really know what this is all about, but I know that I've seen some issues with openssl tracebacks lately that I've been meaning to look into. I suspect that updating some libraries will help here.


TypeError: initializer for ctype 'EVP_PKEY * *' must be a pointer to same type, not cdata 'EVP_PKEY * *'

Stacktrace (most recent call last):

  File "django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "newrelic/hooks/framework_django.py", line 497, in wrapper
    return wrapped(*args, **kwargs)
  File "django/views/decorators/http.py", line 41, in inner
    return func(request, *args, **kwargs)
  File "kitsune/dashboards/views.py", line 193, in aggregated_metrics
    locales = get_locales_by_visit(today - timedelta(days=30), today)
  File "kitsune/dashboards/utils.py", line 101, in get_locales_by_visit
    results = visitors_by_locale(start_date, end_date)
  File "kitsune/sumo/googleanalytics.py", line 82, in visitors_by_locale
    request = _build_request()
  File "kitsune/sumo/googleanalytics.py", line 47, in _build_request
    service = build('analytics', 'v3', request)
  File "oauth2client/util.py", line 120, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "apiclient/discovery.py", line 193, in build
    resp, content = http.request(requested_url)
  File "oauth2client/util.py", line 120, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "oauth2client/client.py", line 405, in new_request
    self._refresh(request_orig)
  File "oauth2client/client.py", line 573, in _refresh
    self._do_refresh_request(http_request)
  File "oauth2client/client.py", line 597, in _do_refresh_request
    body = self._generate_refresh_request_body()
  File "oauth2client/client.py", line 729, in _generate_refresh_request_body
    assertion = self._generate_assertion()
  File "oauth2client/client.py", line 835, in _generate_assertion
    Signer.from_string(private_key, self.private_key_password), payload)
  File "oauth2client/crypt.py", line 124, in from_string
    pkey = crypto.load_pkcs12(key, password).get_privatekey()
  File "OpenSSL/crypto.py", line 2223, in load_pkcs12
    parse_result = _lib.PKCS12_parse(p12, passphrase, pkey, cert, cacerts)

Comment 5

[Ton [:Tonnes]]

For info: another short outage a few minutes before writing this, now back again.

Comment 6

[vesper]

Cleaning up. The new platform will have its own set of metrics and dashboards.