Provide clear engineering guidelines on inventory projection error and it's dependency on beta sample size

RESOLVED FIXED

Status

Content Services Graveyard
Tiles: Data Processing
P1
normal
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: maxim zhilyaev, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: .?)

(Reporter)

Description

3 years ago
Going through Bug# 1135738 discussions, I realized that I do not have a good answer on how to estimate error limits for inventory projections we make for targeted tiles based on beta sample (or a part of it).  

Since, our targeting inventory projections are potentially customer facing, I believe it is important to:
1) know when and if our projections are unreliable
2) if we do provide projections, we should supply a confidence interval in which our projection may vary.

I am unclear on the business impact that wrong projection may cause and would like jterry to clarify.
(Reporter)

Comment 1

3 years ago
Justin, could you explain how inventory projections are used by us and by clients.
Flags: needinfo?(jterry)
(Reporter)

Updated

3 years ago
Iteration: 39.2 - 23 Mar → ---

Comment 2

3 years ago
Some of these have been noted here: https://bugzilla.mozilla.org/show_bug.cgi?id=1136977

Since we're starting off with site groupings first, what's required is
1. Domain uniques and impressions estimations: we want to project the estimated audience size based on a domain or grouping of domains
2. Segmentation grouping uniques and impressions estimations: we want to project the estimated audience size based on a category the client wishes to target. The target category should provide which domains are included in them.
  - by IAB categories: combinations of 1st and 2nd levels
  - by custom groupings  
3. Filters
  - release channels: beta and release
  - date ranges: default (day, week, month, year), custom

This would need to be a repeatable report, hence the user interface on bug 1136977.

Comment 3

3 years ago
maksik is looking more for how accurate do we need to be as opposed to what we need to report.

I would guess that we can be pretty coarse in accuracy given the beta audience, and we'll need to caveat it to clients as we project things out for them. But once we get live release data for how a given ad group performs, we'll have much better numbers to provide.
(Reporter)

Comment 4

3 years ago
Kevin's requirements are actually very useful.  For the segments he defines we need to be able to tell how wrong our audience/inventory estimations could be and within which confidence interval.

Kevin, I need few data points from you:
- the minimal audience size to require accurate estimates
- the absolute maximal error of estimation that partner will tolerate: is it 1%, 10%, 25% or may 50%
- is this error level different for audience vs. impression estimations

Also, for bug 1144815, we need to have some idea of a domain of sites being used by advertizes. 
Is it possible to get list of sites classified into IAB, or get some data from IAB that contains some examples.
(Reporter)

Comment 5

3 years ago
Engineering guidelines are provided in the attached document: https://docs.google.com/document/d/1J6f2QXIgGdAzMPUgXMs2KB27821xqJXU07KWlnBbr9U

The low size limit for audiences which size we can predict with reasonable accuracy from full beta is 400 people.  Which is roughly 45K US audience.  I will suggest on Bug# 1135738 to run full sample, and will pass to business for comments.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Updated

3 years ago
Flags: needinfo?(jterry)
You need to log in before you can comment on or make changes to this bug.