Closed Bug 1013321 Opened 10 years ago Closed 10 years ago

Create an admin UI for the supersearch fields list

Categories

(Socorro :: General, task)

Product:
Socorro
Component:
General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: adrian, Assigned: adrian)

References

Details

(Whiteboard: [migration needed]) 

1. Move the hard-coded list of fields into a database (elasticsearch).
2. Create middleware services to manipulate that list.
3. Create an admin UI to see and modify that list.
Pull request: https://github.com/mozilla/socorro/pull/2105

Migration needed! Please run the following command after pushing the code: 

python scripts/setup_supersearch.py

That will take the base data in scripts/data/supersearch_fields.json and put it into elasticsearch, in the correct index. It ensures the Super Search service will not stop working. 

Some options will possibly be needed in that command, I will add them after I have done the migration in stage.
Whiteboard: [migration needed]
Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/7960713def5a344fb259486c0f5c642301bb1402
Fixes bug 1013321 - Admin UI for the master list of Super Search Fields. r=peterbe
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
For the migration, from the socorro directory, run: 

$PYTHON scripts/setup_supersearch_app.py

The correct configuration should be picked up automatically (at least it worked automatically on stage, I hope it will be the same on stage. If it doesn't, you should simply pass the correct host for elasticsearch, use --help or ping me to find out where to pass it).
There is a bug with the few booleans: a false value (unchecked box) never makes it to the middleware. It is thus impossible to turn something to false at the moment, which is kind of problematic (though should not impact release as the default values are already what we use at the moment in production).
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: --- → 89
No, that's a different issue being tracked at bug 1025944.
Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/a6e34bf7bab0f9af6fe83656eb7fb8a4edd2222b
Fixes bug 1013321 - Fixed problem with turning supersearch fields values to false.

https://github.com/mozilla/socorro/commit/0afae911e4a0f9aea2dfc08df28038934eecce9a
Merge pull request #2127 from AdrianGaudebert/1013321-fix-supersearch-field-update

Fixes bug 1013321 - Fixed problem with turning supersearch fields values...
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/59106985d6859e2c16cfd4a3facbffacac54316e
Bug 1013321 - Ensure permissions are correct in the SuperSearchUnredacted model. r=peterbe

https://github.com/mozilla/socorro/commit/b0068e84550f80863e06400bf722164644b7f7b1
Merge pull request #2129 from AdrianGaudebert/1013321-fix-api-documentation

Bug 1013321 - Added verification when building permissions for the Super...
Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/4164ddcd98cc4fd85cf77f56d7becb995b8a4fc8
Revert "Bug 1013321 - Ensure permissions are correct in the SuperSearchUnredacted model. r=peterbe"

This reverts commit 59106985d6859e2c16cfd4a3facbffacac54316e.

https://github.com/mozilla/socorro/commit/9d3b044961a76a5e78bac18864b7a036923d51ca
Bug 1013321 - Ensured permissions are correct before sending them to the middleware. r=peterbe
Target Milestone: 89 → 90
`. /etc/socorro/socorrorc application/scripts/setup_supersearch_app.py` did not create the necessary index.
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217] Traceback (most recent call last):
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/web/application.py", line 237, in process
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     return self.handle()
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/web/application.py", line 228, in handle
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     return self._delegate(fn, self.fvars, args)
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/web/application.py", line 394, in _delegate
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     return handle_class(f)
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/web/application.py", line 385, in handle_class
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     return tocall(*args)
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/application/socorro/middleware/middleware_app.py", line 502, in GET
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     all_services=self.all_services
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/application/socorro/external/elasticsearch/supersearch.py", line 54, in __init__
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     self.all_fields = self.get_fields()
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/application/socorro/external/elasticsearch/supersearch.py", line 335, in get_fields
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     count = search.count()  # Total number of results.
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/elasticutils/__init__.py", line 1306, in count
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     return self[:0].raw()['hits']['total']
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/elasticutils/__init__.py", line 1286, in raw
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     doc_type=self.get_doctypes())
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/pyelasticsearch/client.py", line 96, in decorate
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     return func(*args, query_params=query_params, **kwargs)
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/pyelasticsearch/client.py", line 590, in search
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     return self._search_or_count('_search', query, **kwargs)
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/pyelasticsearch/client.py", line 567, in _search_or_count
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     query_params=query_params)
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/pyelasticsearch/client.py", line 254, in send_request
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     self._raise_exception(resp, prepped_response)
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]   File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/pyelasticsearch/client.py", line 269, in _raise_exception
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217]     raise error_class(response.status_code, error_message)
[Mon Jun 23 15:38:20 2014] [error] [client 10.8.81.217] ElasticHttpNotFoundError: (404, u'IndexMissingException[[socorro] missing]')
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
welllllll that was the wrong command. Peter helped me figure out:

$ . /etc/socorro/socorrorc
$ $PYTHON application/scripts/setup_supersearch_app.py

which yields the following new error:
Traceback (most recent call last):
  File "application/scripts/setup_supersearch_app.py", line 100, in <module>
    generic_app.main(SetupSuperSearchApp)
  File "/data/socorro/application/socorro/app/generic_app.py", line 214, in main
    config_manager_cls
  File "/data/socorro/application/socorro/app/generic_app.py", line 315, in _do_main
    return_code = fix_exit_code(instance.main())
  File "application/scripts/setup_supersearch_app.py", line 58, in main
    index_creator.create_index('socorro', None)
  File "/data/socorro/application/socorro/external/elasticsearch/crashstorage.py", line 274, in create_index
    settings=es_settings
  File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/pyelasticsearch/client.py", line 96, in decorate
    return func(*args, query_params=query_params, **kwargs)
  File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/pyelasticsearch/client.py", line 754, in create_index
    query_params=query_params)
  File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/pyelasticsearch/client.py", line 238, in send_request
    **({'data': request_body} if body else {}))
  File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/requests/sessions.py", line 387, in put
    return self.request('PUT', url, data=data, **kwargs)
  File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, **kwargs)
  File "/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/requests/adapters.py", line 327, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /socorro (Caused by <class 'socket.error'>: [Errno 111] Connection refused)


which looks a bit like the error in bug 1026781
Because it wasn't picking up the elasticsearch_urls from common_elasticsearch.ini I ran it like this::



[pbengtsson@sp-admin01.phx1 socorro]$  $PYTHON application/scripts/setup_supersearch_app.py --resource.elasticsearch.elasticsearch_urls=http://socorro-es.webapp.phx1.mozilla.com:9200
2014-06-23 17:24:53,505 INFO - MainThread -  - MainThread - app_name: setup-supersearch
2014-06-23 17:24:53,505 INFO - MainThread -  - MainThread - app_version: 1.0
2014-06-23 17:24:53,506 INFO - MainThread -  - MainThread - current configuration:
2014-06-23 17:24:53,506 INFO - MainThread -  - MainThread - application: <class '__main__.SetupSuperSearchApp'>
2014-06-23 17:24:53,506 INFO - MainThread -  - MainThread - elasticsearch.backoff_delays: [10, 30, 60, 120, 300]
2014-06-23 17:24:53,506 INFO - MainThread -  - MainThread - elasticsearch.elasticsearch_base_settings: /data/socorro/application/socorro/external/elasticsearch/mappings/socorro_index_settings.json
2014-06-23 17:24:53,506 INFO - MainThread -  - MainThread - elasticsearch.elasticsearch_class: <class 'socorro.external.elasticsearch.connection_context.ConnectionContext'>
2014-06-23 17:24:53,506 INFO - MainThread -  - MainThread - elasticsearch.elasticsearch_default_index: socorro
2014-06-23 17:24:53,507 INFO - MainThread -  - MainThread - elasticsearch.elasticsearch_doctype: crash_reports
2014-06-23 17:24:53,507 INFO - MainThread -  - MainThread - elasticsearch.elasticsearch_emails_index: socorro_emails
2014-06-23 17:24:53,507 INFO - MainThread -  - MainThread - elasticsearch.elasticsearch_emails_index_settings: /data/socorro/application/socorro/external/elasticsearch/mappings/socorro_emails_index_settings.json
2014-06-23 17:24:53,507 INFO - MainThread -  - MainThread - elasticsearch.elasticsearch_index: socorro%Y%W
2014-06-23 17:24:53,507 INFO - MainThread -  - MainThread - elasticsearch.elasticsearch_timeout: 30
2014-06-23 17:24:53,507 INFO - MainThread -  - MainThread - elasticsearch.elasticsearch_urls: ['http://socorro-es.webapp.phx1.mozilla.com:9200']
2014-06-23 17:24:53,507 INFO - MainThread -  - MainThread - elasticsearch.forbidden_keys: url, email, user_id, exploitability,json_dump.sensitive,upload_file_minidump_flash1.json_dump.sensitive,upload_file_minidump_flash2.json_dump.sensitive,upload_file_minidump_browser.json_dump.sensitive
2014-06-23 17:24:53,507 INFO - MainThread -  - MainThread - elasticsearch.index_creator_class: <class 'socorro.external.elasticsearch.crashstorage.ElasticSearchCrashStorage'>
2014-06-23 17:24:53,507 INFO - MainThread -  - MainThread - elasticsearch.redactor_class: <class 'socorro.external.crashstorage_base.Redactor'>
2014-06-23 17:24:53,507 INFO - MainThread -  - MainThread - elasticsearch.transaction_executor_class: <class 'socorro.database.transaction_executor.TransactionExecutorWithLimitedBackoff'>
2014-06-23 17:24:53,508 INFO - MainThread -  - MainThread - elasticsearch.wait_log_interval: 10
2014-06-23 17:24:53,508 INFO - MainThread -  - MainThread - logger: <socorro.app.generic_app.LoggerWrapper object at 0x2118d10>
2014-06-23 17:24:53,508 INFO - MainThread -  - MainThread - logging.stderr_error_logging_level: 10
2014-06-23 17:24:53,508 INFO - MainThread -  - MainThread - logging.stderr_line_format_string: {asctime} {levelname} - {threadName} - {message}
2014-06-23 17:24:53,508 INFO - MainThread -  - MainThread - logging.syslog_error_logging_level: 40
2014-06-23 17:24:53,508 INFO - MainThread -  - MainThread - logging.syslog_facility_string: user
2014-06-23 17:24:53,508 INFO - MainThread -  - MainThread - logging.syslog_host: localhost
2014-06-23 17:24:53,508 INFO - MainThread -  - MainThread - logging.syslog_line_format_string: setup-supersearch (pid {process}): {asctime} {levelname} - {threadName} - {message}
2014-06-23 17:24:53,508 INFO - MainThread -  - MainThread - logging.syslog_port: 514
2014-06-23 17:24:53,509 INFO - MainThread -  - MainThread - resource.elasticsearch.backoff_delays: [10, 30, 60, 120, 300]
2014-06-23 17:24:53,509 INFO - MainThread -  - MainThread - resource.elasticsearch.elasticsearch_base_settings: /data/socorro/application/socorro/external/elasticsearch/mappings/socorro_index_settings.json
2014-06-23 17:24:53,509 INFO - MainThread -  - MainThread - resource.elasticsearch.elasticsearch_default_index: socorro
2014-06-23 17:24:53,509 INFO - MainThread -  - MainThread - resource.elasticsearch.elasticsearch_doctype: crash_reports
2014-06-23 17:24:53,509 INFO - MainThread -  - MainThread - resource.elasticsearch.elasticsearch_emails_index: socorro_emails
2014-06-23 17:24:53,509 INFO - MainThread -  - MainThread - resource.elasticsearch.elasticsearch_emails_index_settings: /data/socorro/application/socorro/external/elasticsearch/mappings/socorro_emails_index_settings.json
2014-06-23 17:24:53,509 INFO - MainThread -  - MainThread - resource.elasticsearch.elasticsearch_index: socorro%Y%W
2014-06-23 17:24:53,509 INFO - MainThread -  - MainThread - resource.elasticsearch.elasticsearch_timeout: 30
2014-06-23 17:24:53,509 INFO - MainThread -  - MainThread - resource.elasticsearch.elasticsearch_urls: ['http://socorro-es.webapp.phx1.mozilla.com:9200']
2014-06-23 17:24:53,509 INFO - MainThread -  - MainThread - resource.elasticsearch.transaction_executor_class: <class 'socorro.database.transaction_executor.TransactionExecutorWithLimitedBackoff'>
2014-06-23 17:24:53,510 INFO - MainThread -  - MainThread - resource.elasticsearch.wait_log_interval: 10
2014-06-23 17:24:53,510 INFO - MainThread -  - MainThread - resource.logging.stderr_error_logging_level: 10
2014-06-23 17:24:53,510 INFO - MainThread -  - MainThread - resource.logging.stderr_line_format_string: {asctime} {levelname} - {threadName} - {message}
2014-06-23 17:24:53,510 INFO - MainThread -  - MainThread - resource.logging.syslog_error_logging_level: 40
2014-06-23 17:24:53,510 INFO - MainThread -  - MainThread - resource.logging.syslog_facility_string: user
2014-06-23 17:24:53,510 INFO - MainThread -  - MainThread - resource.logging.syslog_host: localhost
2014-06-23 17:24:53,510 INFO - MainThread -  - MainThread - resource.logging.syslog_line_format_string: setup-supersearch (pid {process}): {asctime} {levelname} - {threadName} - {message}
2014-06-23 17:24:53,510 INFO - MainThread -  - MainThread - resource.logging.syslog_port: 514
2014-06-23 17:24:53,510 INFO - MainThread -  - MainThread - resource.redactor.forbidden_keys: url, email, user_id, exploitability,json_dump.sensitive,upload_file_minidump_flash1.json_dump.sensitive,upload_file_minidump_flash2.json_dump.sensitive,upload_file_minidump_browser.json_dump.sensitive
2014-06-23 17:24:53,510 INFO - MainThread -  - MainThread - resource.redactor.redactor_class: <class 'socorro.external.crashstorage_base.Redactor'>
2014-06-23 17:24:53,511 INFO - MainThread -  - MainThread - supersearch_fields_file: /data/socorro/application/scripts/data/supersearch_fields.json
2014-06-23 17:24:53,769 INFO - MainThread -  - MainThread - created new elasticsearch index: socorro
2014-06-23 17:24:54,145 INFO - MainThread -  - MainThread - done.
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Target Milestone: 90 → 91
QA verified on prod -- automation is passing again
Blocks: 1029257
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.