1111893 - Find a way to add new buchets without requiring a full reindex

Status: VERIFIED FIXED

(Marketplace Graveyard :: API, defect, P2)

Description

[Mathieu Pillard [:mat]]

When we make a ES query and we have some buchets to consider, here is what we do:
- Find out which features are not present in the profile that was sent
- For each of those features, add a must: thatfeature=false to the ES query

This works great, it will return all apps not using the features that the client does not have. 

However, if we add a new feature and an app has not been reindexed yet, it will not be returned, because in the index, there is no value for the new feature for that app, so it won't match the query part that requires newfeature to be false.

This means that currently, when adding a new feature, once the push is done, any request with that new feature absent from the profile won't return *any* app.

To work around this, we'll probably need to reindex *before* the actual push to webheads, but we need a better solution, either by reworking the query or by finding a way to add a default value to features in ES.

Comment 1

[Mathieu Pillard [:mat]]

On IRC :robhudson found out that we could use ES "missing" filter : http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-missing-filter.html

For instance:
 u'should': [{u'term': {u'features.has_hardware_1g_ram': False}},
             {u'missing': {u'field': u'features.has_hardware_1g_ram'}}]}},

Should return apps that have the feature 'has_hardware_1g_ram' set to false or just absent.

An alternative I proposed would be to rework the query and how we set things in the index to only add true or null (instead of false) values, and then use only missing queries, since they match null and absent just the same.

Comment 3

[Mathieu Pillard [:mat]]

Fixed in https://github.com/mozilla/zamboni/commit/8603c7dcdf213c86bd5b25a89872ce918750b50d

STR:
- Test that region exclusions still work correctly
- Test that technical requirements like 1GB RAM still work correctly (recent version of Marketplace app on Firefox OS only)

Comment 4

[Madalin Cotetiu]

Verified as fixed in FF38(Win7) in marketplace-dev.allizom.org
Postfix screencast:  http://screencast.com/t/Hor8t2Ex
Also region exclusion is still working.
Closing.