1392199 - Make sure that similarity based recommendations can happen in a reasonable time

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement)

Description

[Alessio Placitelli [:Dexter]]

Once the theoretical work on the "similarity" model is complete, we need to make sure that we're able to perform real-time suggestion with it. In particular, we need to consider:

- TAAR needs to scale up to the whole user-base;
- Each recommendation must take a very tiny amount of time (it can't take seconds).

Comment 1

[mlopatka]

@Mauro, what is the threshold in terms of milliseconds that we must accompish in order to move a module into TAAR.

Comment 2

[Mauro Doglio [:mdoglio]]

It's hard to tell an exact number. In a perfect world that number is a function of

1 - number of concurrent users which drives
  1.a number of web nodes that we can afford to have and
  1.b number of web workers we can run concurrently on a web node. This depends on how much the code is cpu vs io bound, it needs to be fine tuned once our code is stable.
For example, if our code let us serve 3-4 parallel requests per web node  and it takes ~1s to complete a request, then we need ~10 web nodes to serve 30-40 requests per second.

2 - how long is acceptable for the user to wait for recommendations. For web pages the golden standard is 200ms end to end. Here we are talking about a web service, so we may be a bit more flexible but still we need to take into account the hbase connection, etc.

I guess what I'm trying to say is it's all about trade-offs. I'll try to run a load test on the dev environment to get some insights and then we can start to talk about what kind of SLA we want for taar, how much time we want to spend on engineering vs paying a higher bill, etc.

Comment 3

[Mauro Doglio [:mdoglio]]

As an example, AMO avg response time is ~115ms and they have on avg (last 24h) 23 request/s

Comment 4

[Alessio Placitelli [:Dexter]]

This happened here: https://github.com/mozilla/taar/pull/35

It was reviewed and merge, closing this bug.