Closed Bug 1290433 Opened 8 years ago Closed 8 years ago

Write evaluation (performance testing) function for add-on recommendations for quantitative evaluation of recommendations

Categories

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Product:
Cloud Services Graveyard
Component:
Metrics: Pipeline
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mlopatka, Assigned: mlopatka)

References

Details

A separate notebook to examine different ways of evaluating the performance of recommended add-ons for new users and users with few add-ons.
Points: --- → 1
Depends on: 1287727
Priority: -- → P3
Assignee: nobody → mlopatka
Points: 1 → 2
Priority: P3 → P2
Summary: Write evaluation (performance testing) notebook for add-on recommendations by CLLR and masked add-on list prediction → Write evaluation (performance testing) function for add-on recommendations for quantitative evaluation of recommendations
Cumulative Gain (CG) added to score-dist_rank notebook and as a GIST. I decided not include the position of a result in the consideration of recommender score.

https://gist.github.com/mlopatka/de804ad9662f408f411b365ffcab0e7e
cLLR added to score-dist_rank notebook and as a GIST.
some test cases are included to get an intuition of the cLLR value ranges.
Lower cLLR is better because it is a COST function. cLLR over 1 is never returned because it means baseline performance is worse than chance 50/50 and weighing the outcome is meaningless.

https://gist.github.com/mlopatka/a9e270187629016f6f1144d75d2dc9aa
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: Cloud Services → Cloud Services Graveyard
You need to log in before you can comment on or make changes to this bug.