926704 - Add-ons MXR is dead

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Kris Maglione [:kmag]]

When I try to search the add-ons MXR, I get no results and the following message:

No files in search index, please complain to the webmaster [cite: emptyglimpsetarget]

We rely heavily on the add-ons MXR for a lot of things, including critical security issues, so we need this fixed as soon as possible.

Thanks.

Comment 1

[Ashish Vijayaram [:ashish]]

How do I replicate this?

Comment 2

[Phil Ringnalda (:philor)]

The easiest way is to just load https://mxr.mozilla.org/addons/source/ and notice that the only file listed is https://mxr.mozilla.org/addons/source/last-processed, but alternately, https://mxr.mozilla.org/addons/search?string=foo

Comment 4

[Ashish Vijayaram [:ashish]]

Okay, I'm not an AMO Admin so I can't access any of that. Let me see what I can find in logs...

Comment 5

[Ashish Vijayaram [:ashish]]

I had a quick chat with :philor and he says this can wait till the AM, when I'll put this back in the queue.

Comment 6

[Kendall Libby [:fubar] (he/him)]

The weekly MXR cron threw out this error:

addons: Starting Sun Oct 13 22:00:01 PDT 2013
addons: Has changes, running xref and search...
Updating addons...
Traceback (most recent call last):
  File "latest_addon_extractor.py", line 29, in <module>
    with utils.db('read') as cursor:
  File "/data/mxr-data/amo-code/bin/utils.py", line 91, in db
    return mysql.connect(**db_config(name))
  File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (1045, "Access denied for user 'mxr'@'10.8.70.202' (using password: YES)")

I suspect it may be the same issue that hit bonsai.

Comment 7

[Kendall Libby [:fubar] (he/him)]

< fubar> sheeri: did addons db change this weekend? c.f. bug 926704
< sheeri> fubar: nope. And AFAIK the only thing we changed permissions on was 
                metrics, that was last week.
< jason> our databases should not have changed this past weekend :)
< fubar> jason: hrm. mxr queries db-zlb-ro.addons.phx1 and it started failing
< jason> fubar: that seems like the old ro db vip. we need to update mxr to use 
               our new vip
< fubar> ah, that's easy, then
< jason> db.ro.mktns.mktdb.services.phx1.mozilla.com
< jason> we most likely need network flows to this db vip from mxr

Comment 9

[Kendall Libby [:fubar] (he/him)]

Updated to point at db.ro.mktns.mktdb.services.phx1.mozilla.com and now it's returning

07:30 mxr-processor1.private.scl3#./update-src.pl addons
Traceback (most recent call last):
  File "latest_addon_extractor.py", line 29, in <module>
    with utils.db('read') as cursor:
  File "/data/mxr-data/amo-code/bin/utils.py", line 92, in db
    return mysql.connect(**db_config(name))
  File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (1045, "Access denied for user 'mxr'@'10.32.126.20' (using password: YES)")

I'm guessing that the mxr user wasn't carried over from the old db. :jason/:oremj, any thoughts?

Comment 10

[Jason Thomas [:jason]]

I've updated the mxr user credentials in mysql. Please try again and let me know if you continue to have issues.

Comment 11

[Florent Fayolle]

> I've updated the mxr user credentials in mysql. Please try again and let me know if you continue to have issues.

It still doesn't work for me (I have the same message). I can contact you by mail to give you my login (which is an email address).

Am I the only one still having the issue?

Florent

Comment 12

[Kendall Libby [:fubar] (he/him)]

It's not just you! The cron job which updates the search data was wedged; I've kicked it and it's running now. I'll investigate why it got hung up and probably re-run the whole process again.

Comment 13

[Florent Fayolle]

> I'll investigate why it got hung up and probably re-run the whole process again.

I can see some results now. Is it fixed? (or is it still indexing?)

Florent

Comment 14

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Seems to be working for me now.

Honza

Comment 15

[Kendall Libby [:fubar] (he/him)]

Add-ons are back to being indexed and searchable; also, timeouts on the load balancer have been increased (c.f. bug 920806).