1434991 - Add categorical thresholds of page views to client_count_daily

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Datasets: Client Count, enhancement, P1)

Description

[Mark Reid [:mreid]]

Adding several threshold values for page views will give a useful means of counting based on different usage levels. 

Implementing them as categories (rather than each individual observed value) will allow us to sensibly combine various groups while not exploding the cardinality of rows in the dataset.

As a start, let's add the following categories for the sum of 
scalar_parent_browser_engagement_total_uri_count
for a given day.

0 <= x <= 5
5 < x <= 20
20 < x <= 50
50 < x <= 100
100 < x <= 250
x > 250

References:
https://metrics.mozilla.com/protected/shong/%5bSu%5d%5bAdHoc%5dPageViewsDistributions.html

Comment 1

[Mark Reid [:mreid]]

Peter, do these look like good values to start with?

Comment 2

[Peter Dolanjski [:pdol]]

(In reply to Mark Reid [:mreid] from comment #1)
> Peter, do these look like good values to start with?

Yes, these look good to me.  Thanks.

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/telemetry-batch-view/pull/370

Comment 4

[Daniel Thorn [:relud]]

My test run of the client_count_daily job can't find the column scalar_parent_browser_engagement_total_uri_count in main_summary to create bucketed_uri_count.

columns found in main_summary/v4/:

 aborts_content
 aborts_gmplugin
 aborts_plugin
 active_addons_count
 active_experiment_branch
 active_experiment_id
 app_build_id
 app_display_version
 app_name
 app_version
 channel
 city
 client_id
 client_submission_date
 country
 crash_submit_attempt_content
 crash_submit_attempt_main
 crash_submit_attempt_plugin
 crash_submit_success_content
 crash_submit_success_main
 crash_submit_success_plugin
 crashes_detected_content
 crashes_detected_gmplugin
 crashes_detected_plugin
 default_search_engine
 default_search_engine_data_name
 devtools_toolbox_opened_count
 distribution_id
 document_id
 e10s_cohort
 e10s_enabled
 env_build_arch
 env_build_id
 env_build_version
 flash_version
 install_year
 is_default_browser
 locale
 loop_activity_counter
 normalized_channel
 os
 os_service_pack_major
 os_service_pack_minor
 os_version
 places_bookmarks_count
 places_pages_count
 plugin_hangs
 popup_notification_stats
 profile_creation_date
 push_api_notification_received
 reason
 sample_id
 search_counts
 submission_date
 submission_date_s3
 subsession_length
 subsession_start_date
 sync_configured
 sync_count_desktop
 sync_count_mobile
 timestamp
 timezone_offset
 vendor
 web_notification_shown
 windows_build_number
 windows_ubr

Comment 5

[Daniel Thorn [:relud]]

I was missing mergeSchema=true, the above issue is fixed now

Comment 6

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/telemetry-airflow/pull/227

Comment 7

[Daniel Thorn [:relud]]

this is blocked by bug 1438927 I will raise it back to p1 when that bug is resolved.

Comment 8

[Daniel Thorn [:relud]]

back filled this change to 20160312, which is currently as far back as main_summary goes