823060 - [research] Use datetime instead of ints in ES mappings

Status: RESOLVED FIXED

(support.mozilla.org :: Code Quality, task, P2)

Description

[Ricky Rosario [:rrosario, :r1cky]]

I found this TODO in the code:

        # TODO: Sphinx stores created and updated as seconds since the
        # epoch, so we convert them to that format here so that the
        # search view works correctly. When we ditch Sphinx, we should
        # see if it's faster to filter on ints or whether we should
        # switch them to dates.


 Let's either remove the TODO or come up with plan to do it if it makes sense

Comment 1

[Ricky Rosario [:rrosario, :r1cky]]

Word on the street is that mythmon is the expert here.

Comment 2

[Ricky Rosario [:rrosario, :r1cky]]

Link to the TODO:
https://github.com/mozilla/kitsune/blob/master/apps/questions/models.py#L362

Comment 3

[Michael Cooper [:mythmon]]

To my knowledge, ES internally stores dates and datetimes as milliseconds since Jan 1st 1970. When communicating with ES, it accepts dates in various string formats, such as YYYY-MM-DD HH:MM:SS.MS. pyes and pyelasticsearch both handle the translation from python datetime objects to these strings, and back.

In Fjord, we just shoved datetime objects at ES, and it worked well, and we could use histograms. I think it would be beneficial to do the same thing here. It will likely enable better time based searching, and it's better to let someone else do the conversion, like pyes. 

I have no idea how this would affect performance, but I think that having the right semantic type is a good thing. I think we should switch to using datetimes.

Sources:
https://github.com/aparo/pyes/blob/master/pyes/es.py#L67
https://github.com/rhec/pyelasticsearch/blob/master/pyelasticsearch/client.py#L825
https://github.com/mozilla/fjord/blob/master/fjord/feedback/models.py#L89
http://www.elasticsearch.org/guide/reference/mapping/date-format.html

Comment 4

[Ricky Rosario [:rrosario, :r1cky]]

Thanks mythmon! Sounds like we just want to file a bug to convert to using proper datetimes and close this out.

Comment 5

[Michael Cooper [:mythmon]]

Filed implementation bug 828010.