979357 - Report/monitor on incoming crash metadata which is not indexed in supersearch

Status: RESOLVED FIXED

(Socorro :: Backend, task, P1)

Description

[Benjamin Smedberg]

We are adding new metadata to the crash reports fairly regularly. Currently people have to remember to file a bug to get the new metadata key added to supersearch.

It would be awesome if we could detect during processing that a report contains a metadata key which we don't currently index in supersearch and have a daily or weekly report of the top unexpected metadata so that we can get those added to supersearch.

Comment 1

[Peter Bengtsson [:peterbe]]

Is this related to the whitelist of keys? They're currently only used for the API but we talked about building a universal list of whitelist keys and put them into a database instead of being hardcoded in python code.

Will building such a tool (that can be used for API and for Supersearch) solve the problem Benjamin raises for the Supersearch?

Comment 2

[Benjamin Smedberg]

This is about supersearch indexing. That may be related to the whitelist, but I'm not sure.

Comment 3

[Robert Kaiser]

(In reply to Peter Bengtsson [:peterbe] from comment #1)
> Will building such a tool (that can be used for API and for Supersearch)
> solve the problem Benjamin raises for the Supersearch?

I think so, yes.

Comment 4

[[DEACTIVATED] Adrian Gaudebert]

(In reply to Peter Bengtsson [:peterbe] from comment #1)
> Is this related to the whitelist of keys? They're currently only used for
> the API but we talked about building a universal list of whitelist keys and
> put them into a database instead of being hardcoded in python code.

No, there are different lists of fields in the middleware for supersearch, and there are not related to the API whitelist (and supersearch is not exposed in the public API yet anyway). 

> Will building such a tool (that can be used for API and for Supersearch)
> solve the problem Benjamin raises for the Supersearch?

It won't be that simple, we would need to change a lot of supersearch code, but ultimately that would probably solve our problem here. 

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #0)
> It would be awesome if we could detect during processing that a report
> contains a metadata key which we don't currently index in supersearch and
> have a daily or weekly report of the top unexpected metadata so that we can
> get those added to supersearch.

I have a few ideas to build such a thing. We have a mapping in our code base, we could pull the mapping of the last index every week and run a sort of diff. Not sure how to expose it though? A page in the webapp? An email sent to some of us?

Comment 5

[Robert Kaiser]

(In reply to Adrian Gaudebert [:adrian] from comment #4)
> No, there are different lists of fields in the middleware for supersearch,
> and there are not related to the API whitelist (and supersearch is not
> exposed in the public API yet anyway). 

I do not think we should have multiple different whitelists in the future. It's bad enough that we do right now.

Comment 6

[Peter Bengtsson [:peterbe]]

Whatever we do...

* let's only make 1 list of whitelisted keys

* the API would be one of many consumers

* another consumer would be the report index page


Adrian, 
How about if I write a tool that replaces the `API_WHITELIST` lists into a database object thing and then an internal API around that. Perhaps then you can see how the Supersearch can leverage that.

Comment 7

[[DEACTIVATED] Adrian Gaudebert]

So, I created a new admin UI for the master list of fields (see tracker bug 1013306), and I have now added a page that shows the missing fields. It parses the mappings of the last 3 weeks, creates a list of all fields, and does a diff with the list of known fields. You can then see that list of missing fields, and you have a link to quickly create any missing field.

Pull request is here: https://github.com/mozilla/socorro/pull/2139

Comment 8

[[DEACTIVATED] Adrian Gaudebert]

Attachment: missing_fields.png

Comment 9

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/3d48d97d6b2f05f6bf18e6e4fe313fc782321331
Fixes bug 979357 - Created a report to show missing fields in elasticsearch. r=peterbe