Closed Bug 543970 Opened 16 years ago Closed 16 years ago

API Search is broken in production

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: wenzel, Assigned: justdave)

Details

https://services.addons.mozilla.org/en-US/firefox/api/1.5/search/adblock returns: "<error>The search system is temporarily unavailable.</error>" since yesterday's push. CCing oremj for the server side of things, dd for the code.
was just in the process of filing this myself... :) 23:38:02 <@nagios> [09] samo.acelb.sj:services.addons.mozilla.org - String all is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable 01:39:19 <@nagios> [30] samo.acelb.sj:services.addons.mozilla.org - String all is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable 03:39:20 <@nagios> [39] samo.acelb.sj:services.addons.mozilla.org - String all is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable Verified in a browser that searching within the browser UI fails. I ignored this the first time because I thought the upgrade was still in progress. I suddenly realized this hadn't stopped paging and the upgrade's long since done.
problem is on IT end...
Assignee: nobody → server-ops
Component: API → Server Operations: Web Content Push
Product: addons.mozilla.org → mozilla.org
QA Contact: api → mrz
Version: unspecified → other
Nobody svn upped services.a.m.o. [root@mradm02 services.addons.mozilla.org]# svn up U . Fetching external item into 'site' A site/app/locale/sv_SE/LC_MESSAGES/django.po U site/app/locale/sv_SE/LC_MESSAGES/messages.mo U site/app/locale/sv_SE/LC_MESSAGES/messages.po U site/app/locale/uk/LC_MESSAGES/messages.mo U site/app/locale/uk/LC_MESSAGES/messages.po U site/app/locale/sq/LC_MESSAGES/messages.mo U site/app/locale/sq/LC_MESSAGES/messages.po U site/app/locale/da/LC_MESSAGES/messages.mo U site/app/locale/da/LC_MESSAGES/da.txt U site/app/locale/da/LC_MESSAGES/messages.po U site/app/locale/sr/LC_MESSAGES/messages.mo U site/app/locale/sr/LC_MESSAGES/messages.po U site/app/locale/fa/LC_MESSAGES/messages.mo U site/app/locale/fa/LC_MESSAGES/messages.po U site/app/locale/de/LC_MESSAGES/messages.mo U site/app/locale/de/LC_MESSAGES/messages.po U site/app/locale/ja/LC_MESSAGES/messages.mo U site/app/locale/ja/LC_MESSAGES/messages.po U site/app/locale/he/LC_MESSAGES/messages.mo U site/app/locale/he/LC_MESSAGES/messages.po A site/app/locale/es_ES/LC_MESSAGES/django.po U site/app/locale/es_ES/LC_MESSAGES/messages.mo U site/app/locale/es_ES/LC_MESSAGES/messages.po A site/app/locale/es_ES/LC_MESSAGES/django.mo U site/app/locale/fi/LC_MESSAGES/messages.mo U site/app/locale/fi/LC_MESSAGES/messages.po U site/app/locale/pt_BR/LC_MESSAGES/messages.mo U site/app/locale/pt_BR/LC_MESSAGES/messages.po A site/app/locale/fr/pages/validation.thtml A site/app/locale/fr/pages/docs A site/app/locale/fr/pages/docs/gettingstarted.thtml A site/app/locale/fr/pages/docs/policies A site/app/locale/fr/pages/docs/policies/contact.thtml A site/app/locale/fr/pages/docs/policies/submission.thtml A site/app/locale/fr/pages/docs/policies/maintenance.thtml A site/app/locale/fr/pages/docs/policies/recommended.thtml A site/app/locale/fr/pages/docs/policies/reviews.thtml A site/app/locale/fr/pages/statistics_help.thtml A site/app/locale/fr/pages/compatibility_firstrun.thtml A site/app/locale/fr/pages/compatibility_user_tips.thtml A site/app/locale/fr/pages/developer_faq.thtml A site/app/locale/fr/pages/fashion_faq.thtml A site/app/locale/fr/pages/distribution.thtml A site/app/locale/fr/pages/reviewguide.thtml U site/app/locale/fr/LC_MESSAGES/messages.mo U site/app/locale/fr/LC_MESSAGES/messages.po A site/app/locale/nl/pages/docs/policies/submission.thtml U site/app/locale/nl/LC_MESSAGES/messages.mo U site/app/locale/nl/LC_MESSAGES/messages.po U site/app/locale/pl/pages/validation.thtml U site/app/locale/pl/pages/docs/policies/submission.thtml U site/app/locale/pl/pages/docs/case-studies/cooliris.thtml U site/app/locale/pl/LC_MESSAGES/messages.mo U site/app/locale/pl/LC_MESSAGES/messages.po U site/app/locale/hu/LC_MESSAGES/messages.mo U site/app/locale/hu/LC_MESSAGES/messages.po U site/app/locale/keys.pot A site/app/locale/vi/LC_MESSAGES/django.po U site/app/locale/vi/LC_MESSAGES/messages.mo U site/app/locale/vi/LC_MESSAGES/messages.po A site/app/locale/sr-Latn/LC_MESSAGES/django.po U site/app/locale/sr-Latn/LC_MESSAGES/messages.po U site/app/locale/ro/LC_MESSAGES/messages.mo U site/app/locale/ro/LC_MESSAGES/messages.po U site/app/locale/ca/LC_MESSAGES/messages.mo U site/app/locale/ca/LC_MESSAGES/messages.po U site/app/locale/tr/LC_MESSAGES/messages.mo U site/app/locale/tr/LC_MESSAGES/messages.po U site/app/locale/ru/pages/about.thtml U site/app/locale/ru/pages/nomination.thtml A site/app/locale/ru/pages/collector_faq.thtml A site/app/locale/ru/pages/compatibility_developer_tips.thtml U site/app/locale/ru/pages/sandbox.thtml U site/app/locale/ru/pages/error404.thtml A site/app/locale/ru/pages/collector_firstrun.thtml A site/app/locale/ru/pages/statistics_help.thtml U site/app/locale/ru/pages/submission_help.thtml A site/app/locale/ru/pages/collector_features.thtml A site/app/locale/ru/pages/collector.thtml A site/app/locale/ru/pages/compatibility_user_tips.thtml A site/app/locale/ru/pages/faq.thtml U site/app/locale/ru/LC_MESSAGES/messages.mo U site/app/locale/ru/LC_MESSAGES/messages.po A site/app/locale/af/LC_MESSAGES/django.po U site/app/locale/af/LC_MESSAGES/messages.po A site/app/locale/pt_PT/pages/about.thtml A site/app/locale/pt_PT/LC_MESSAGES/django.po U site/app/locale/pt_PT/LC_MESSAGES/messages.mo U site/app/locale/pt_PT/LC_MESSAGES/messages.po A site/app/locale/pt_PT/LC_MESSAGES/django.mo U site/app/locale/zh_TW/LC_MESSAGES/messages.mo U site/app/locale/zh_TW/LC_MESSAGES/messages.po A site/app/locale/id/LC_MESSAGES/django.po U site/app/locale/id/LC_MESSAGES/messages.mo U site/app/locale/id/LC_MESSAGES/messages.po U site/app/locale/el/LC_MESSAGES/messages.mo U site/app/locale/el/LC_MESSAGES/messages.po U site/app/locale/zh_CN/LC_MESSAGES/messages.mo U site/app/locale/zh_CN/LC_MESSAGES/messages.po U site/app/locale/ar/LC_MESSAGES/messages.mo U site/app/locale/ar/LC_MESSAGES/messages.po U site/app/locale/ga_IE/LC_MESSAGES/messages.mo U site/app/locale/ga_IE/LC_MESSAGES/messages.po U site/app/locale/cs/LC_MESSAGES/messages.mo U site/app/locale/cs/LC_MESSAGES/messages.po U site/app/locale/fy_NL/LC_MESSAGES/messages.po U site/app/locale/ko/LC_MESSAGES/messages.mo U site/app/locale/ko/LC_MESSAGES/messages.po U site/app/locale/en_US/pages/nomination.thtml U site/app/locale/en_US/LC_MESSAGES/messages.mo U site/app/locale/en_US/LC_MESSAGES/messages.po U site/app/locale/eu/LC_MESSAGES/messages.mo U site/app/locale/eu/LC_MESSAGES/messages.po A site/app/locale/mn/LC_MESSAGES/django.po U site/app/locale/mn/LC_MESSAGES/messages.mo U site/app/locale/mn/LC_MESSAGES/messages.po U site/app/locale/cy/LC_MESSAGES/messages.mo U site/app/locale/cy/LC_MESSAGES/messages.po U site/app/locale/it/LC_MESSAGES/messages.mo U site/app/locale/it/LC_MESSAGES/messages.po U site/app/locale/sk/LC_MESSAGES/messages.mo U site/app/locale/sk/LC_MESSAGES/messages.po U site/app/webroot/services/install.php U site/app/webroot/services/pfs.php U site/app/webroot/css/admin.css U site/app/webroot/css/editors.css D site/app/webroot/css/amo2009/admin-django.css D site/app/webroot/css/amo2009/admin-mozilla.css A site/app/webroot/css/amo2009/zamboni A site/app/webroot/css/amo2009/zamboni/admin-django.css A site/app/webroot/css/amo2009/zamboni/tabs.css A site/app/webroot/css/amo2009/zamboni/translations A site/app/webroot/css/amo2009/zamboni/translations/trans.css A site/app/webroot/css/amo2009/zamboni/admin-mozilla.css A site/app/webroot/css/amo2009/editors.css U site/app/webroot/.htaccess U site/app/webroot/js/editors.js A site/app/webroot/js/zamboni A site/app/webroot/js/zamboni/translations A site/app/webroot/js/zamboni/translations/trans.js A site/app/webroot/js/zamboni/tabs.js A site/app/webroot/js/zamboni/underscore-min.js A site/app/webroot/js/zamboni/init.js A site/app/webroot/js/admin.js U site/app/webroot/js/amo2009/addons.js U site/app/webroot/js/amo2009/amo2009.min.js U site/app/webroot/js/amo2009/global.js D site/app/tests/selenium/Login.html A site/app/tests/selenium/Reviews A site/app/tests/selenium/Admin A site/app/tests/selenium/Admin/amoadmin.html A site/app/tests/selenium/Admin/usermanager.html A site/app/tests/selenium/Personas A site/app/tests/selenium/Personas/AMO_persona_TS.htm A site/app/tests/selenium/Personas/AMO_persona_sort.htm A site/app/tests/selenium/Personas/AMO_persona_detail_compare.htm A site/app/tests/selenium/Personas/AMO_persona_detail.htm U site/app/tests/selenium/search2.html U site/app/tests/selenium/amo-selenium.html A site/app/tests/selenium/App Views A site/app/tests/selenium/Buttons A site/app/tests/selenium/User Profile A site/app/tests/selenium/User Profile/edituser.html U site/app/tests/selenium/searchapi.html U site/app/tests/selenium/search.html A site/app/tests/selenium/Collections A site/app/tests/selenium/Collections/AMO_collections_TS.htm A site/app/tests/selenium/Collections/AMO_category_landing_collections.htm A site/app/tests/selenium/Accounts A site/app/tests/selenium/Accounts/Login-VerifyLoginFailsWithNoOrIncorrectEmail.html A site/app/tests/selenium/Accounts/Login-VerifySimpleLoginToAMOFirefox.html A site/app/tests/selenium/Accounts/Login-VerifyLoginFailsWithNoOrIncorrectPassword.html A site/app/tests/selenium/Accounts/Login-VerifyLoginFailsWithNoCredentials.html A site/app/tests/selenium/Search & Sort A site/app/tests/selenium/Developer A site/app/tests/selenium/Categories A site/app/tests/selenium/Categories/AMO_dictionaries_TS.htm A site/app/tests/selenium/Categories/AMO_categoryCommonLayout.htm A site/app/tests/selenium/Categories/AMO_category_sort_all_addons.htm A site/app/tests/selenium/Categories/AMO_category_landing_dictionaries.htm A site/app/tests/selenium/Categories/AMO_plugins_TS.htm A site/app/tests/selenium/Categories/AMO_category_landing_plugins.htm A site/app/tests/selenium/Categories/AMO_category_sort_top_ranked.htm A site/app/tests/selenium/Categories/AMO_categories_TS.htm A site/app/tests/selenium/Tags U site/app/models/addon.php U site/app/config/constants.php U site/app/config/revisions.php A site/app/config/migrations/61000-new-index.sql A site/app/config/migrations/60947-src-stats.sql U site/app/controllers/admin_controller.php U site/app/controllers/components/validation.php U site/app/controllers/components/stats.php U site/app/controllers/components/developers.php U site/app/controllers/components/editors.php U site/app/controllers/components/l10n.php U site/app/controllers/components/amo.php U site/app/controllers/components/hub.php U site/app/controllers/components/email.php U site/app/controllers/files_controller.php U site/app/controllers/developers_controller.php U site/app/controllers/addons_controller.php U site/app/controllers/versions_controller.php U site/app/controllers/editors_controller.php A site/app/views/admin/features_category.thtml U site/app/views/layouts/amo2009.thtml U site/app/views/users/info.thtml U site/app/views/helpers/addons_html.php U site/app/views/files/view.thtml U site/app/views/developers/discuss.thtml U site/app/views/developers/versions_edit.thtml U site/app/views/developers/dashboard.thtml U site/app/views/addons/policy.thtml U site/app/views/addons/display.thtml U site/app/views/addons/category_landing.thtml U site/app/views/versions/update_info.thtml D site/app/views/editors/featured.thtml U site/app/views/editors/performance_table.thtml U site/app/views/editors/reviews_queue.thtml U site/app/views/editors/motd_admin.thtml U site/app/views/editors/summary.thtml U site/app/views/editors/email/pending/public_plain.thtml U site/app/views/editors/email/pending/sandbox_plain.thtml U site/app/views/editors/email/inforequest_plain.thtml U site/app/views/editors/email/nominated/public_plain.thtml U site/app/views/editors/email/nominated/sandbox_plain.thtml U site/app/views/editors/review.thtml U site/app/views/editors/reviewlog.thtml U site/app/views/editors/logs.thtml U site/app/views/editors/queue.thtml U site/app/views/editors/performance_charts.thtml U site/app/views/elements/amo2009/visit_listed_button.thtml A site/app/views/elements/amo2009/editors A site/app/views/elements/amo2009/editors/navbar.thtml A site/app/views/elements/amo2009/editors/performance_header.thtml A site/app/views/elements/amo2009/editors/queue_search.thtml A site/app/views/elements/amo2009/editors/queue_tabs.thtml A site/app/views/elements/amo2009/editors/motd.thtml U site/app/views/elements/amo2009/visit_listed_warning.thtml D site/app/views/elements/developers/editorsmenu.thtml D site/app/views/elements/developers/performanceheader.thtml D site/app/views/elements/developers/editorsqueue.thtml D site/app/views/elements/developers/review_motd.thtml U site/app/views/elements/developers/adminmenu.thtml U site/app/views/elements/developers/editors_comment_add_form.thtml A site/app/views/elements/headers/page_header_editors.thtml U site/app/views/dev_hub/api_reference.thtml U site/app/views/collections/detail.thtml U site/vendors/sphinx/addonsSearch.php Updated external to revision 61591. Fetching external item into 'bin' U bin/import-personas.py Updated external to revision 61591. Fetching external item into 'site/vendors/product-details' Updated external to revision 59694. Fetching external item into 'bin/schematic' Updated external to revision 59694. Fetching external item into 'site/vendors/zxtm-api' Updated external to revision 59694. Fetching external item into 'site/vendors/phorms' Updated external to revision 59694. Fetching external item into 'site/vendors/fizzypop' Updated external to revision 59694. Fetching external item into 'site/vendors/l10n-completeness-check' Updated external to revision 59694. Fetching external item into 'site/vendors/gearman-php' Updated external to revision 59694. Fetching external item into 'site/vendors/sphinx/api' Updated external to revision 1822. Updated to revision 61656. 06:00:17 <@nagios> samo.acelb.sj:services.addons.mozilla.org - String all is OK: HTTP OK HTTP/1.1 200 OK - 0.804 second response time
Assignee: server-ops → justdave
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
Thanks to wenzel for the idea to check that... 05:58:05 < wenzel> I am wondering if it's as easy as svn upping services.amo
FWIW, now we have included https://litmus.mozilla.org/show_test.cgi?id=7845 in our post-push tests, which we'll run every time we have an AMO push. Verified FIXED.
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.