669965 - Fixing 500 Internal Error when using ElasticSearch for search

Status: VERIFIED FIXED

(Socorro :: General, task)

Description

[[DEACTIVATED] Adrian Gaudebert]

As for bug 665251, Powerfuzzer found some 500 errors when using ElasticSearch for the search in the middleware. 

Todo:
* Re-run powerfuzzer on the refactored code (see bug 669961 )
* Attach here the result
* Send patches
* Run tests with QA to confirm ES is ready to be turned on in prod

Comment 1

[[DEACTIVATED] Adrian Gaudebert]

Attachment: 500 Internal Errors on khan when using ES

Problem know, to be fixed soon.

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

This is now fixed by another bug, right?

Comment 3

[[DEACTIVATED] Adrian Gaudebert]

Not yet, it still has to be r+ed, pushed on prod and tested. But yes, the fix is part of the patch you can find on bug 669961.

Comment 4

[[DEACTIVATED] Adrian Gaudebert]

Fixed in commit r3275, by the patch on bug 669961. To be QAed.

Comment 5

[Matt Brandt [:mbrandt]]

I'm not sure if what I'm seeing is due to the patch not having landed on dev yet but I'm still seeing the 500s:

https://crash-stats-dev.allizom.org/query/query?build_id=on&do_query=1&query_search=<script>var+pf_687474703a2f2f616761756465626572742e6b68616e2e6d6f7a696c6c612e6f72672f71756572792f7175657279_71756572795f736561726368=new+Boolean();</script>&hang_type=hang&query_type=on&product=on&process_type=plugin&platform=on&reason=on&version=on&range_unit=on&date=07/07/2011+16:21:00&query=Find&range_value=1

https://crash-stats-dev.allizom.org/query/query?build_id=on&do_query=1&query_search=<script>var+pf_687474703a2f2f616761756465626572742e6b68616e2e6d6f7a696c6c612e6f72672f71756572792f7175657279_71756572795f736561726368=new+Boolean();</script>&hang_type=hang&query_type=on&product=on&process_type=plugin&platform=on&reason=on&version=on&range_unit=on&date=07/07/2011+16:21:00&query=on&range_value=1

https://crash-stats-dev.allizom.org/query/query?build_id=on&do_query=1&query_search=<script>var+pf_687474703a2f2f616761756465626572742e6b68616e2e6d6f7a696c6c612e6f72672f71756572792f7175657279_71756572795f736561726368=new+Boolean();</script>&hang_type=hang&query_type=on&product=on&process_type=plugin&platform=on&reason=on&version=on&range_unit=on&date=07/07/2011&query=on&range_value=1

https://crash-stats-dev.allizom.org/query/query?build_id=on&do_query=1&query_search=<script>var+pf_687474703a2f2f616761756465626572742e6b68616e2e6d6f7a696c6c612e6f72672f71756572792f7175657279_71756572795f736561726368=new+Boolean();</script>&hang_type=hang&query_type=on&product=on&process_type=plugin&platform=on&reason=on&version=on&range_unit=on&date=07/07/2011&query=on&range_value=14

Comment 6

[[DEACTIVATED] Adrian Gaudebert]

dev is using Postgres, so it's weird we see 500 Errors here. I don't know if it's up-to-date though...

Comment 7

[Robert Helmer [:rhelmer]]

(In reply to comment #6)
> dev is using Postgres, so it's weird we see 500 Errors here. I don't know if
> it's up-to-date though...

-dev pulls trunk automatically, so unless that's not working it should always be up-to-date.

Comment 8

[[DEACTIVATED] Adrian Gaudebert]

Waiting for dev to be able to access the ES instance, and then it should be ready for testing. Btw dev was up-to-date but not the config, which caused the 500 problem.

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #8)
> Waiting for dev to be able to access the ES instance, and then it should be
> ready for testing. Btw dev was up-to-date but not the config, which caused
> the 500 problem.

Are we still waiting on this?  Is there a dependency we can link to, if so?

Comment 10

[[DEACTIVATED] Adrian Gaudebert]

dev should be ready for testing as soon as the build is finished. Previous Errors where due to bugs fixed in bug 669961.

Comment 11

[Stephen Donner [:stephend] Not actively reading bugmail]

https://crash-stats-dev.allizom.org/query/query?build_id=on&do_query=1&query_search=signature&hang_type=hang&query_type=on&product=on&process_type=plugin&platform=on&reason=on&version=on&range_unit=on&date=07/26/2011+12:01:08&query=<script>var+pf_68747470733a2f2f63726173682d73746174732d6465762e616c6c697a6f6d2e6f72672f71756572792f7175657279_7175657279=new+Boolean();</script>&range_value=1

https://crash-stats-dev.allizom.org/query/query?build_id=on&do_query=1&query_search=signature&hang_type=hang&query_type=on&product=on&process_type=plugin&platform=on&reason=on&version=on&range_unit=on&date=07/26/2011+12:01:09&query=<script>var+pf_68747470733a2f2f63726173682d73746174732d6465762e616c6c697a6f6d2e6f72672f71756572792f7175657279_7175657279=new+Boolean();</script>&range_value=1

https://crash-stats-dev.allizom.org/query/query?build_id=on&do_query=1&query_search=signature&hang_type=hang&query_type=on&product=on&process_type=plugin&platform=on&reason=on&version=on&range_unit=on&date=07/26/2011&query=<script>var+pf_68747470733a2f2f63726173682d73746174732d6465762e616c6c697a6f6d2e6f72672f71756572792f7175657279_7175657279=new+Boolean();</script>&range_value=14

https://crash-stats-dev.allizom.org/query/query?build_id=on&do_query=1&query_search=signature&hang_type=hang&query_type=on&product=on&process_type=plugin&platform=on&reason=on&version=on&range_unit=on&date=07/26/2011&query=<script>var+pf_68747470733a2f2f63726173682d73746174732d6465762e616c6c697a6f6d2e6f72672f71756572792f7175657279_7175657279=new+Boolean();</script>&range_value=1

Adrian asked that I reopen and paste the above problematic URLs.

Comment 12

[[DEACTIVATED] Adrian Gaudebert]

Attachment: Securing search_mode and dates parameters

* Securing the search_mode so it can only be one of the authorized values ;
* Securing the date fields to make sure it's always a datetime.

Comment 13

[K Lars Lohn [:lars] [:klohn]]

Comment on attachment 548540 [details] [diff] [review]
Securing search_mode and dates parameters

instead of:
     if not authorized_modes.count(args["search_mode"]):
         args["search_mode"] = "default"
how about:
     if args["search_mode"] not in authorized_modes:
         args["search_mod"] = "default"

and drop the "pass" in the date exception handler.

Comment 14

[[DEACTIVATED] Adrian Gaudebert]

Attachment: Securing search_mode and dates parameters

Comment 15

[K Lars Lohn [:lars] [:klohn]]

Comment on attachment 548545 [details] [diff] [review]
Securing search_mode and dates parameters

we'll take up the "to be moved into a config file" comments at some later date.

Comment 16

[[DEACTIVATED] Adrian Gaudebert]

Fixed with commit r3304. Can be tested on dev.

Comment 17

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; I spun off bug 674420 -- not sure in which milestone it should be.