1378989 - Churn job from mozetl should retry on exceptions

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

The churn code currently fails with the following exception.

```
INFO:mozetl.churn.churn:Starting week from 20170618 to 20170624
ERROR:mozetl.churn.churn:Exception for 20170618
Traceback (most recent call last):
  File "/mnt/analyses/python_mozetl/mozetl/churn/churn.py", line 692, in main
    process_week(main_df, week_start_date, bucket, prefix)
  File "/mnt/analyses/python_mozetl/mozetl/churn/churn.py", line 609, in process_week
    result_df = compute_churn_week(df, week_start)
  File "/mnt/analyses/python_mozetl/mozetl/churn/churn.py", line 424, in compute_churn_week
    .select(source_columns)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 861, in select
    jdf = self._jdf.select(self._jcols(*cols))
  File "/usr/lib/spark/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py", line 1133, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 69, in deco
    raise AnalysisException(s.split(': ', 1)[1], stackTrace)
AnalysisException: u"cannot resolve '`total_uri_count`' given input columns: [scalar_parent_gfx_advanced_layers_failure_id, scalar_gpu_telemetry_discarded_keyed_scalar_actions, timezone_offset, env_build_arch, scalar_parent_services_sync_fxa_verification_method, input_event_response_coalesced_ms_content_above_2500, experiments, scalar_parent_browser_engagement_max_concurrent_window_count, telemetry_enabled, gc_max_pause_ms_main_above_150, crashes_detected_content, scalar_parent_browser_engagement_navigation_searchbar, scalar_parent_telemetry_discarded_keyed_accumulations, scalar_content_browser_usage_graphite, histogram_content_ssl_tls13_intolerance_reason_pre, web_notification_shown, os_service_pack_major, popup_notification_stats, devtools_toolbox_opened_count, os, histogram_parent_http_pageload_is_ssl, gc_max_pause_ms_content_above_2500, scalar_parent_webrtc_nicer_turn_403s, scalar_parent_storage_sync_api_usage_extensions_using, crash_submit_success_content, app_build_id, os_version, scalar_parent_devtools_grid_showInfiniteLines_checked, scalar_parent_devtools_copy_xpath_opened, scalar_parent_devtools_copy_full_css_selector_opened, scalar_content_telemetry_discarded_keyed_scalar_actions, cycle_collector_max_pause_content_above_2500, ghost_windows_main_above_1, histogram_content_http_channel_disposition, session_restored, histogram_content_input_event_response_coalesced_ms, scalar_parent_browser_engagement_tab_open_event_count, shutdown_kill, subsession_counter, input_event_response_coalesced_ms_content_above_150, client_submission_date, user_prefs, profile_creation_date, scalar_parent_telemetry_discarded_child_events, sync_count_mobile, env_build_version, country, active_experiment_branch, histogram_content_ssl_handshake_version, histogram_parent_cycle_collector_max_pause, plugins_infobar_allow, gfx_compositor, scalar_content_telemetry_discarded_keyed_accumulations, scalar_parent_browser_engagement_restored_pinned_tabs_count, total_time, windows_ubr, scalar_parent_browser_engagement_total_uri_count, scalar_gpu_browser_usage_graphite, histogram_parent_ghost_windows, aborts_plugin, histogram_parent_ssl_tls13_intolerance_reason_pre, flash_version, ghost_windows_content_above_1, e10s_cohort, histogram_parent_http_channel_disposition, scalar_parent_webrtc_nicer_turn_438s, scalar_parent_navigator_storage_estimate_count, scalar_parent_devtools_grid_showGridAreasOverlay_checked, city, places_bookmarks_count, gc_max_pause_ms_content_above_150, crash_submit_attempt_main, scalar_parent_browser_usage_graphite, scalar_parent_browser_engagement_max_concurrent_tab_count, scalar_parent_preferences_prefs_file_was_invalid, is_default_browser, cycle_collector_max_pause_content_above_150, subsession_start_date, scalar_parent_telemetry_discarded_accumulations, scalar_gpu_telemetry_discarded_keyed_accumulations, default_search_engine, input_event_response_coalesced_ms_main_above_2500, scalar_parent_browser_engagement_unique_domains_count, histogram_parent_ssl_handshake_result, first_paint, plugin_hangs, default_search_engine_data_name, plugins_notification_shown, scalar_parent_storage_sync_api_usage_items_stored, scalar_parent_devtools_grid_showGridLineNumbers_checked, scalar_parent_devtools_toolbar_eyedropper_opened, cycle_collector_max_pause_main_above_2500, scalar_content_telemetry_discarded_child_events, scalar_parent_telemetry_discarded_keyed_scalar_actions, scalar_content_navigator_storage_persist_count, app_display_version, histogram_content_ssl_handshake_result, cycle_collector_max_pause_main_above_150, push_api_notify, e10s_enabled, ssl_handshake_result_success, gc_max_pause_ms_main_above_250, install_year, push_api_notification_received, profile_subsession_counter, sample_id, scalar_parent_webrtc_nicer_turn_401s, submission_date_s3, is_wow64, search_counts, creation_date, document_id, scalar_parent_browser_engagement_navigation_about_newtab, scalar_content_webrtc_nicer_turn_438s, default_search_engine_data_load_path, scalar_content_navigator_storage_estimate_count, cycle_collector_max_pause_main_above_250, scalar_parent_devtools_copy_unique_css_selector_opened, scalar_parent_webrtc_nicer_stun_retransmits, crash_submit_success_main, vendor, main, default_search_engine_data_origin, scalar_parent_preferences_read_user_js, active_theme, sync_configured, histogram_parent_gc_max_pause_ms, scalar_parent_navigator_storage_persist_count, timestamp, scalar_gpu_telemetry_discarded_scalar_actions, crash_submit_attempt_content, crashes_detected_gmplugin, crash_submit_success_plugin, scalar_content_webrtc_nicer_turn_403s, plugins_infobar_block, normalized_channel, scalar_gpu_telemetry_discarded_child_events, scalar_content_webrtc_nicer_turn_401s, histogram_parent_cert_validation_success_by_ca, scalar_parent_browser_engagement_navigation_urlbar, scalar_content_webrtc_nicer_stun_retransmits, attribution, crash_submit_attempt_plugin, gc_max_pause_ms_content_above_250, plugins_infobar_shown, app_name, histogram_content_gc_max_pause_ms_2, histogram_content_cycle_collector_max_pause, memory_mb, env_build_id, gc_max_pause_ms_main_above_2500, client_id, channel, ssl_handshake_result_failure, loop_activity_counter, aborts_content, search_cohort, locale, quantum_ready, histogram_parent_input_event_response_coalesced_ms, windows_build_number, histogram_content_ghost_windows, scalar_parent_telemetry_os_shutting_down, plugins_infobar_dismissed, input_event_response_coalesced_ms_main_above_250, histogram_parent_ssl_tls12_intolerance_reason_pre, scalar_content_telemetry_discarded_scalar_actions, histogram_content_http_pageload_is_ssl, histogram_parent_ssl_handshake_version, scalar_parent_dom_contentprocess_troubled_due_to_memory, crashes_detected_plugin, input_event_response_coalesced_ms_main_above_150, plugins_notification_user_action, ssl_handshake_result, histogram_content_ssl_tls12_intolerance_reason_pre, scalar_parent_browser_engagement_unfiltered_uri_count, os_service_pack_minor, scalar_parent_aushelper_websense_reg_version, sync_count_desktop, scalar_content_telemetry_discarded_accumulations, scalar_parent_devtools_grid_gridinspector_opened, input_event_response_coalesced_ms_content_above_250, submission_date, default_search_engine_data_submission_url, events, histogram_content_gc_max_pause_ms, places_pages_count, cycle_collector_max_pause_content_above_250, scalar_parent_a11y_instantiators, scalar_parent_browser_engagement_navigation_about_home, scalar_parent_telemetry_discarded_scalar_actions, scalar_parent_preferences_created_new_user_prefs_file, histogram_content_cert_validation_success_by_ca, addon_compatibility_check_enabled, scalar_parent_browser_engagement_window_open_event_count, blocklist_enabled, app_version, scalar_parent_devtools_rules_gridinspector_opened, active_experiment_id, scalar_parent_browser_engagement_navigation_contextmenu, active_addons_count, aborts_gmplugin, histogram_parent_gc_max_pause_ms_2, scalar_parent_security_pkcs11_modules_loaded, active_ticks, distribution_id, reason, scalar_parent_storage_sync_api_usage_storage_consumed, scalar_gpu_telemetry_discarded_accumulations, active_addons, subsession_length];;\n'Project [app_version#24, attribution#69, channel#2, client_id#1, country#4, default_search_engine#54, distribution_id#16, locale#31, normalized_channel#3, profile_creation_date#13L, submission_date_s3#209, subsession_length#15L, subsession_start_date#14, sync_configured#18, sync_count_desktop#19, sync_count_mobile#20, timestamp#25L, 'total_uri_count, 'unique_domains_count]\n+- Filter (subsession_start_date#14 < 2017-06-25)\n   +- Filter (subsession_start_date#14 >= 2017-06-18)\n      +- Filter (submission_date_s3#209 <= 20170704)\n         +- Filter (submission_date_s3#209 >= 20170618)\n            +- Relation[document_id#0,client_id#1,channel#2,normalized_channel#3,country#4,city#5,os#6,os_version#7,os_service_pack_major#8L,os_service_pack_minor#9L,windows_build_number#10L,windows_ubr#11L,install_year#12L,profile_creation_date#13L,subsession_start_date#14,subsession_length#15L,distribution_id#16,submission_date#17,sync_configured#18,sync_count_desktop#19,sync_count_mobile#20,app_build_id#21,app_display_version#22,app_name#23,... 187 more fields] parquet\n"
```

This should cause airflow to retry the job. A try-except block prevents the error state from propagating to the job scheduler. [1] 


[1] https://github.com/mozilla/python_mozetl/blob/master/mozetl/churn/churn.py#L693-L696

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Bug 1378989 - remove catching of exceptions in churn job #65