809004 - Deploy an http cache for elastic search on stage.

Status: RESOLVED INCOMPLETE

(Cloud Services :: Operations: Marketplace, task)

Description

[Alexis Metaireau (:alexis)]

https://github.com/ametaireau/httpcache is the code of an HTTP proxy handling cache for elasticsearch. We now need to deploy this alongside each webhead, in order to test if it speed things-up or not.

This means:

- Starting the httpcache proxy with the appropriate ports and adresses (I would make it listen from the localhost:9200 to elasticsearch:9200. Note that it needs to be indicated the location of memcache as well.
- Changing the host and port in the configuration of zamboni

And that should be it. You also can specify it the address to send statsd information, which could be helpful to gather statistics as well.

Comment 1

[Jason Thomas [:jason]]

I have deployed httpcache on stage and listening on localhost:9200. I configured stage zamboni to to point to httpcache however it is causing search pages to error. I have reverted the settings change for now. Please ping me on irc to troubleshoot.

[[e-mail]1 ~]# curl http://localhost:9200/_cluster/health?pretty=true
{
  "cluster_name" : "addons-stage-elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 35,
  "active_shards" : 70,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0


This is what I see in the logs after hitting https://addons.allizom.org/en-US/firefox/search/?q=computer&cat=1%2C0&appver=18.0&platform=mac
Jan 31 11:03:41 web1.stage.addons.phx1.mozilla.com: [][] z.es:ERROR {'filter': {'and': [{'term': {'app': 1}}, {'term': {'is_disabled': False}}, {'in': {'status': (8, 9, 4)}}, {'in': {'platform': (3, 1)}}, {'term': {'type': 1}}]}, 'query': {'bool': {'should': [{'text': {'summary': {'query': u'computer', 'boost': 0.80000000000000004, 'type': 'phrase'}}}, {'fuzzy': {'app_slug': {'boost': 2, 'prefix_length': 4, 'value': u'computer'}}}, {'text': {'name_english': {'query': u'computer', 'boost': 2.5, 'analyzer': 'english'}}}, {'prefix': {'app_slug': {'boost': 1.5, 'value': u'computer'}}}, {'text': {'app_slug': {'query': u'computer', 'boost': 4, 'type': 'phrase'}}}, {'text': {'authors': {'query': u'computer', 'boost': 4, 'type': 'phrase'}}}, {'text': {'summary_english': {'query': u'computer', 'boost': 0.59999999999999998, 'analyzer': 'english', 'type': 'phrase'}}}, {'fuzzy': {'name': {'boost': 2, 'prefix_length': 4, 'value': u'computer'}}}, {'prefix': {'slug': {'boost': 1.5, 'value': u'computer'}}}, {'fuzzy': {'authors': {'boost': 2, 'prefix_length': 4, 'value': u'computer'}}}, {'text': {'slug': {'query': u'computer', 'boost': 4, 'type': 'phrase'}}}, {'text': {'description_english': {'query': u'computer', 'boost': 0.10000000000000001, 'analyzer': 'english', 'type': 'phrase'}}}, {'fuzzy': {'slug': {'boost': 2, 'prefix_length': 4, 'value': u'computer'}}}, {'text': {'name': {'query': u'computer', 'boost': 4, 'type': 'phrase'}}}, {'prefix': {'name': {'boost': 1.5, 'value': u'computer'}}}, {'text': {'description': {'query': u'computer', 'boost': 0.29999999999999999, 'type': 'phrase'}}}, {'prefix': {'authors': {'boost': 1.5, 'value': u'computer'}}}]}}, 'facets': {'platforms': {'terms': {'field': 'platform'}}, 'appversions': {'terms': {'field': 'appversion.1.max'}}, 'categories': {'terms': {'field': 'category', 'size': 200}}, 'tags': {'terms': {'field': 'tag'}}}, 'size': 20, 'fields': ['id']} :/data/www/addons.allizom.org/zamboni/apps/amo/search.py:200
Jan 31 11:03:41 web1.stage.addons.phx1.mozilla.com: [][] z.es:ERROR Elasticsearch error:  :/data/www/addons.allizom.org/zamboni/apps/search/middleware.py:16

strace on the httpcache process:

accept(4, {sa_family=AF_INET, sin_port=htons(53267), sin_addr=inet_addr("127.0.0.1")}, [16]) = 5
fcntl(5, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
accept(4, 0x7fff47934c60, [16])         = -1 EAGAIN (Resource temporarily unavailable)
fcntl(5, F_GETFL)                       = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
recvfrom(5, "GET /addons_stage/addons/_search HTTP/1.1\r\nHost: 127.0.0.1:9200\r\nAccept-Encoding: identity\r\nContent-Length: 1825\r\n\r\n{\"filter\": {\"and\": [{\"term\": {\"app\": 1}}, {\"term\": {\"is_disabled\": false}}, {\"in\": {\"status\": [8, 9, 4]}}, {\"in\": {\"platform\": [3, 1]}}, {\"term\": {\"type\": 1}}]}, \"query\": {\"bool\": {\"should\": [{\"text\": {\"summary\": {\"query\": \"computer\", \"boost\": 0.80000000000000004, \"type\": \"phrase\"}}}, {\"fuzzy\": {\"app_slug\": {\"boost\": 2, \"prefix_length\": 4, \"value\": \"computer\"}}}, {\"text\": {\"name_english\": {\"query\": \"computer\", \"boost\": 2.5, \"analyzer\": \"english\"}}}, {\"prefix\": {\"app_slug\": {\"boost\": 1.5, \"value\": \"computer\"}}}, {\"text\": {\"app_slug\": {\"query\": \"computer\", \"boost\": 4, \"type\": \"phrase\"}}}, {\"text\": {\"authors\": {\"query\": \"computer\", \"boost\": 4, \"type\": \"phrase\"}}}, {\"text\": {\"summary_english\": {\"query\": \"computer\", \"boost\": 0.59999999999999998, \"analyzer\": \"english\", \"type\": \"phrase\"}}}, {\"fuzzy\": {\"name\": {\"boost\": 2, \"prefix_length\": 4, \"value\": \"computer\"}}}, {\"prefix\": {\"slug\": {\"boost\": 1.5, \"v"..., 8192, 0, NULL, NULL) = 1941
getsockname(5, {sa_family=AF_INET, sin_port=htons(9200), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
getpeername(5, {sa_family=AF_INET, sin_port=htons(53267), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
write(2, "<socket fileno=5 sock=127.0.0.1:9200 peer=127.0.0.1:53267>: Unexpected Content-Length\n", 86) = 86
sendto(5, "HTTP/1.0 400 Bad Request\r\nConnection: close\r\nContent-length: 0\r\n\r\n", 66, 0, NULL, 0) = 66
close(5)                                = 0

Comment 2

[Jason Thomas [:jason]]

As per @alexis we will not be proceeding with this.