1465782 - investigate slow/failing supersearch requests

Status: RESOLVED INCOMPLETE

(Socorro :: General, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Over the weekend, we had several ES failures in prod. Brian did a write up here:

https://docs.google.com/document/d/1W_w0rdZoGNJd7hxmCL5DcZyvL_7nUJ9DBJAkQOz3Stg/edit?ts=5b0f2253#

In that write up, he identified a query that fails because it's ES-intensive and takes too long to run:

/api/SuperSearch/?signature=~mozilla%3A%3AReentrancyGuard%3A%3AReentrancyGuard%3Cmozilla%3A%3AVectorBase%3Cjs%3A%3AGCHelperState+*%2C0%2Cjs%3A%3ASystemAllocPolicy%2Cjs%3A%3AVector%3Cjs%3A%3AGCHelperState+*%2C0%2Cjs%3A%3ASystemAllocPolicy%3E+%3E+%3E(mozilla%3A%3AVectorBase%3Cjs%3A%3AGCHelperState+*%2C0%2Cjs%3A%3ASystemAllocPolicy%2Cjs%3A%3AVector%3Cjs%3A%3AGCHelperState+*%2C0%2Cjs%3A%3ASystemAllocPolicy%3E+%3E+%26%2Cmozilla%3A%3Adetail%3A%3AGuardObjectNotifier+const+%26)&date=%3E%3D2017-11-30&date=%3C%3D2018-05-28&_histogram.date=release_channel&_histogram_interval=1d&_results_number=0

This bug covers looking into that and figuring out what to do.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

The query string translates to this:

* _histogram.date = release_channel
* _histogram_interval = 1d
* _results_number = 0
* date <= 2018-05-28
* date >= 2017-11-30
* signature ~ mozilla::ReentrancyGuard::ReentrancyGuard<mozilla::VectorBase<js::GCHelperState *,0,js::SystemAllocPolicy,js::Vector<js::GCHelperState *,0,js::SystemAllocPolicy> > >(mozilla::VectorBase<js::GCHelperState *,0,js::SystemAllocPolicy,js::Vector<js::GCHelperState *,0,js::SystemAllocPolicy> > &,mozilla::detail::GuardObjectNotifier const &)

Looking at the query a bit, that's a curious looking signature. Socorro's signature generation wouldn't produce a signature like that. Maybe this is a query coming from other system to see if there's anything in Socorro that matches?

I'll look into what that SuperSearch query translates to for Elasticsearch.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I don't remember looking into this further.

I talked with Brian and he and I don't know of current failing/slowness issues. We fixed bug #1497353 where we stop indexing fields with not_analyzed. We fixed bug #1624345 where we stopped saving fields that weren't specified in the schema.

I'm going to mark this INCOMPLETE. If it comes up again, we'll write up a new bug.