Airflow task probe_scraper.mozilla_schema_generator failed for exec_date 05.21.24
Categories
(Data Platform and Tools :: General, defect)
Tracking
(Not tracked)
People
(Reporter: linh, Assigned: ascholtz)
References
Details
(Whiteboard: [airflow-triage])
Attachments
(1 file)
Airflow task probe_scraper.mozilla_schema_generator failed for exec_date 05.21.24
Log extract:
failed to execute due to an issue with a Kubernetes pod
|
||
Comment 1•1 year ago
•
|
||
I'm able to see logs for this DAG, and it doesn't look like a k8s execution issue to me. Here's the relevant excerpt:
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] allowing incompatible changes in the following documents:
...
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] --- /app/validate_schema_evolution/af5d54b56312486ffd3452fa81e9021482e62f0a_base-revision/firefox-desktop.user-characteristics.1.txt
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] +++ /app/validate_schema_evolution/d4bd2ab26da4c24a6793a8802fb929496e795730_generated-schemas/firefox-desktop.user-characteristics.1.txt
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] @@ -1,19 +1,2 @@
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] root.additional_properties STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.android_sdk_version STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.app_build STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.app_channel STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.app_display_version STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.architecture STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.build_date STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.client_id STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.device_manufacturer STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.device_model STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.first_run_date STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.locale STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.os STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.os_version STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.session_count INT64
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.session_id STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.telemetry_sdk_build STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.client_info.windows_build_number INT64
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] root.document_id STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] @@ -122,11 +105,2 @@
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] root.normalized_os_version STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.ping_info.end_time STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.ping_info.experiments.[].key STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.ping_info.experiments.[].value.branch STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.ping_info.experiments.[].value.extra.enrollment_id STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.ping_info.experiments.[].value.extra.type STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.ping_info.ping_type STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.ping_info.reason STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.ping_info.seq INT64
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] -root.ping_info.start_time STRING
[2024-05-21, 20:31:23 UTC] {pod_manager.py:466} INFO - [base] root.sample_id INT64
[2024-05-21, 20:31:24 UTC] {pod_manager.py:483} INFO - [base] found incompatible changes
Glancing at upstream I see https://github.com/mozilla/gecko-dev/commit/8430881ac73e9be2b340a5b9b572adaf00104c06 which points at bug #1897506 as the change whose purpose seems pretty explicitly to drop some columns from the ping. This is something that is unfortunately not supported by our Data Warehouse but I thought there was some schemas generation logic to keep fields even when they are removed upstream to avoid issues like this.
Until we resolve this we won't be able to update BQ schemas for any products. There are a couple of ways to proceed here but restoring the columns so that MSG can propagate other schemas updates is probably the first step.
|
Assignee | |
Comment 2•1 year ago
|
||
|
||
Comment 3•1 year ago
|
||
If it's helpful, you can completely wipe the data for this ping in its entirety. It's only been a few dozen volunteer mozillians submitting the data; and now that we've dropped the extra columns we'll need to get new volunteer data anyway.
|
||
Comment 4•1 year ago
|
||
I have merged Anna's PR and restarted schema generator which succeeded.
The fix makes schema generator keep info sections in the schema if they have been removed at some point. I think it makes sense for it to work this way so we're not hit by this again with other pings.
|
||
Comment 5•1 year ago
|
||
looks like this fix is causing issues when going from include_info_sections: true -> include_info_sections: false. The new pings will be sent without the info sections but because we don't update the schema based on the history, the schema validation fails. One fix would be to adjust the minimum ping to include ping_info and client_info columns in the minimum schema but allow them to be optional. Looking for feedback on other ways to fix this.
|
||
Comment 6•1 year ago
|
||
Hey Wil, was this problem solved?
Did you end up asking for feedback on this? (maybe we could grab folks this week :) )
|
|
||
Comment 7•1 year ago
|
||
Just talked to folks at the workweek yesterday and we're going to go with the option I described in the above comment. Work will happen at some point in the near future.
|
|
||
Comment 8•1 year ago
|
||
We talked about this during the last Platform WG meeting. There was a concern that making these fields optional in the schema would allow the pattern of going from include_info_sections: true -> include_info_sections: false. Then we would end up with the data we don't want to collect (although historical). :Wil - do you think this is a valid concern?
:whd suggested that we could special case this and delete/recreate the ping tables. We would also need to temporarily add this ping to incompatibility allowlist in schema generator and revert the fix mentioned above. Choosing this route means we would not be supporting changing include_info_sections: true -> include_info_sections: false. In the future such changes would require either manual intervention as this one or definition of a new ping.
|
|
||
Comment 9•1 year ago
|
||
Yeah I think that's a valid concern. I would also prefer to not even have the info sections as optional on those pings to make it abundantly clear to consumers that the pings do not contain that information. I'm fine with not supporting a transition from include_info_sections: true -> include_info_sections: false for now. At least to unblock the collection here.
:whd, What is the process for deleting the tables above? I can make the schema generator changes (allowlist and revert) but we probably want to coordinate the deletion with the merging of those changes.
|
|
||
Comment 10•1 year ago
|
||
The process is somewhat described on https://docs.google.com/document/d/1a89FH0jjByAjKkl3BxNtLAgqDtZr2HTp9550t1ZyBww/edit#heading=h.khq4461hjk49. Depending on the volume / frequency of pings we can either be somewhat cavalier or do this in a way that guarantees we don't wind up with unrouteable messages in the pipeline. Assuming we can be reasonably certain that no pings will land during a small window, we can:
- land allowlist PR to allow the incompatible schema change
- run MSG to update
generated-schemaswith the change - this will automatically kick of a BQ deploy to stage, which will automatically detect that a create/recreate is required and ping DSRE to approve
- since there is deletion protection this won't actually succeed even if manually approved (https://mozilla-hub.atlassian.net/browse/DSRE-125), so DSRE would manually delete the live and stable tables in stage and re-run the stage pipeline, which will then create the new tables
- the same would be done for prod: the tables will be deleted manually just before the prod BQ schema deploy
- revert the allowlist change
The above is contingent on being sure no messages are being decoded/routed using the old schema to the table with the new schema which I believe is true given they're all going to PBE now. So I think we can do the somewhat cavalier approach above without updating the gcp-ingestion code to explicitly drop the messages while the change is rolling out. Since DSRE will be responsible for table deletes we should indeed coordinate the changes, but I wouldn't expect it to make more than an hour or so from PR merge to land the new tables. When we delete the source tables we'll lose access to the historical data so if that's a concern we should back those up somewhere before making the change.
|
|
||
Comment 11•1 year ago
•
|
||
We're rolling this out now. I've put a backup of the old stable table in shared-nonprod that is inaccessible for general analysis and will be deleted on Sept. 30 2024. If we end up needing this backup you'll have to ask DSRE to put a table ACL on it to make it accessible or copy it into a per-workgroup analysis dataset.
$ bq --project_id moz-fx-data-shar-nonprod-efed show firefox_desktop_stable.user_characteristics_v1_backup
Table moz-fx-data-shar-nonprod-efed:firefox_desktop_stable.user_characteristics_v1_backup
Last modified Schema Total Rows Total Bytes Expiration Time Partitioning Clustered Fields Total Logical Bytes Total Physical Bytes Labels
----------------- ------------------------------------------------------------------------------ ------------ ------------- ----------------- ----------------------------------- ------------------------------- --------------------- ---------------------- ----------------------------------------
28 Jun 10:28:59 |- additional_properties: string 109 10165962 30 Sep 00:00:00 DAY (field: submission_timestamp) normalized_channel, sample_id 10165962 include_client_id:false
+- client_info: record schema_id:glean_ping_1
| |- android_sdk_version: string schemas_build_id:202406280831_c83e6fd7
| |- app_build: string
| |- app_channel: string
| |- app_display_version: string
| |- architecture: string
| |- build_date: string
| |- client_id: string
| |- device_manufacturer: string
| |- device_model: string
| |- first_run_date: string
| |- locale: string
| |- os: string
| |- os_version: string
| |- session_count: integer
| |- session_id: string
| |- telemetry_sdk_build: string
| |- windows_build_number: integer
|- document_id: string
+- events: record (repeated)
| |- category: string
| +- extra: record (repeated)
| | |- key: string
| | |- value: string
| |- name: string
| |- timestamp: integer
+- metadata: record
| +- geo: record
| | |- city: string
| | |- country: string
| | |- db_version: string
| | |- subdivision1: string
| | |- subdivision2: string
| +- header: record
| | |- date: string
| | |- dnt: string
| | |- x_debug_id: string
| | |- x_foxsec_ip_reputation: string
| | |- x_lb_tags: string
| | |- x_pingsender_version: string
| | |- x_source_tags: string
| | |- x_telemetry_agent: string
| +- isp: record
| | |- db_version: string
| | |- name: string
| | |- organization: string
| +- user_agent: record
| | |- browser: string
| | |- os: string
| | |- version: string
+- metrics: record
| +- labeled_counter: record
| | +- glean_error_invalid_label: record (repeated)
| | | |- key: string
| | | |- value: integer
| | +- glean_error_invalid_overflow: record (repeated)
| | | |- key: string
| | | |- value: integer
| | +- glean_error_invalid_state: record (repeated)
| | | |- key: string
| | | |- value: integer
| | +- glean_error_invalid_value: record (repeated)
| | | |- key: string
| | | |- value: integer
| +- quantity: record
| | |- characteristics_submission_schema: integer
| | |- characteristics_color_depth: integer
| | |- characteristics_color_gamut: integer
| | |- characteristics_color_scheme: integer
| | |- characteristics_max_touch_points: integer
| | |- characteristics_prefers_contrast: integer
| | |- characteristics_processor_count: integer
| | |- characteristics_screen_height: integer
| | |- characteristics_screen_width: integer
| | |- characteristics_target_frame_rate: integer
| | |- characteristics_font_default_modified: integer
| | |- characteristics_font_minimum_size_modified: integer
| | |- characteristics_font_name_list_cursive_modified: integer
| | |- characteristics_font_name_list_monospace_modified: integer
| | |- characteristics_font_name_list_sans_serif_modified: integer
| | |- characteristics_font_name_list_serif_modified: integer
| | |- characteristics_font_name_monospace_modified: integer
| | |- characteristics_font_name_sans_serif_modified: integer
| | |- characteristics_font_name_serif_modified: integer
| | |- characteristics_font_size_monospace_modified: integer
| | |- characteristics_font_size_variable_modified: integer
| | |- characteristics_prefs_network_cookie_cookiebehavior: integer
| | |- characteristics_color_accentcolor: integer
| | |- characteristics_color_accentcolortext: integer
| | |- characteristics_color_canvas: integer
| | |- characteristics_color_canvastext: integer
| | |- characteristics_color_highlight: integer
| | |- characteristics_color_highlighttext: integer
| | |- characteristics_color_selecteditem: integer
| | |- characteristics_color_selecteditemtext: integer
| | |- characteristics_avail_height: integer
| | |- characteristics_avail_width: integer
| | |- characteristics_orientation_angle: integer
| | |- characteristics_pixel_depth: integer
| | |- characteristics_screen_orientation: integer
| | |- characteristics_zoom_count: integer
| | |- characteristics_inner_height: integer
| | |- characteristics_inner_width: integer
| | |- characteristics_outer_height: integer
| | |- characteristics_outer_width: integer
| | |- characteristics_posx: integer
| | |- characteristics_posy: integer
| | |- characteristics_size_mode: integer
| | |- characteristics_audio_fingerprint: integer
| | |- characteristics_pixel_ratio: integer
| | |- characteristics_any_pointer_type: integer
| | |- characteristics_motion_decimals: integer
| | |- characteristics_motion_freq: integer
| | |- characteristics_orientation_decimals: integer
| | |- characteristics_orientation_freq: integer
| | |- characteristics_orientationabs_decimals: integer
| | |- characteristics_orientationabs_freq: integer
| | |- characteristics_pointer_height: integer
| | |- characteristics_pointer_tiltx: integer
| | |- characteristics_pointer_tilty: integer
| | |- characteristics_pointer_twist: integer
| | |- characteristics_pointer_type: integer
| | |- characteristics_pointer_width: integer
| +- string: record
| | |- glean_client_annotation_experimentation_id: string
| | |- characteristics_prefs_intl_accept_languages: string
| | |- characteristics_timezone: string
| | |- characteristics_system_locale: string
| | |- characteristics_font_default_default_group: string
| | |- characteristics_font_default_western: string
| | |- characteristics_font_minimum_size_default_group: string
| | |- characteristics_font_minimum_size_western: string
| | |- characteristics_font_name_monospace_default_group: string
| | |- characteristics_font_name_monospace_western: string
| | |- characteristics_font_name_sans_serif_default_group: string
| | |- characteristics_font_name_sans_serif_western: string
| | |- characteristics_font_name_serif_default_group: string
| | |- characteristics_font_name_serif_western: string
| | |- characteristics_font_size_monospace_default_group: string
| | |- characteristics_font_size_monospace_western: string
| | |- characteristics_font_size_variable_default_group: string
| | |- characteristics_font_size_variable_western: string
| | |- characteristics_scalings: string
| | |- characteristics_changed_media_prefs: string
| | |- characteristics_intl_locale: string
| | |- characteristics_languages: string
| | |- characteristics_text_anti_aliasing: string
| | |- characteristics_keyboard_layout: string
| | |- characteristics_pointer_pressure: string
| | |- characteristics_pointer_tangentinal_pressure: string
| | |- characteristics_touch_rotation_angle: string
| +- uuid: record
| | |- characteristics_client_identifier: string
| +- boolean: record
| | |- characteristics_inverted_colors: boolean
| | |- characteristics_prefers_reduced_motion: boolean
| | |- characteristics_prefers_reduced_transparency: boolean
| | |- characteristics_video_dynamic_range: boolean
| | |- characteristics_prefs_media_eme_enabled: boolean
| | |- characteristics_prefs_zoom_text_only: boolean
| | |- characteristics_prefs_block_popups: boolean
| | |- characteristics_prefs_general_autoscroll: boolean
| | |- characteristics_prefs_general_smoothscroll: boolean
| | |- characteristics_prefs_overlay_scrollbars: boolean
| | |- characteristics_prefs_privacy_donottrackheader_enabled: boolean
| | |- characteristics_prefs_privacy_globalprivacycontrol_enabled: boolean
| | |- characteristics_font_name_list_emoji_modified: boolean
| | |- characteristics_prefs_browser_display_use_document_fonts: boolean
| | |- characteristics_use_document_colors: boolean
| +- text2: record
| | |- characteristics_missing_fonts: string
| | |- characteristics_canvasdata1: string
| | |- characteristics_canvasdata10: string
| | |- characteristics_canvasdata11_webgl: string
| | |- characteristics_canvasdata12_fingerprintjs1: string
| | |- characteristics_canvasdata13_fingerprintjs2: string
| | |- characteristics_canvasdata2: string
| | |- characteristics_canvasdata3: string
| | |- characteristics_canvasdata4: string
| | |- characteristics_canvasdata5: string
| | |- characteristics_canvasdata6: string
| | |- characteristics_canvasdata7: string
| | |- characteristics_canvasdata8: string
| | |- characteristics_canvasdata9: string
| | |- characteristics_audio_devices: string
| | |- characteristics_media_devices: string
| | |- characteristics_voices: string
| | |- characteristics_math_ops: string
| | |- characteristics_media_capabilities: string
| | |- characteristics_errors: string
| | |- characteristics_ice_foundations: string
| | |- characteristics_js_errors: string
| +- string_list: record
| | |- characteristics_gamepads: string (repeated)
|- normalized_app_name: string
|- normalized_channel: string
|- normalized_country_code: string
|- normalized_os: string
|- normalized_os_version: string
+- ping_info: record
| |- end_time: string
| +- experiments: record (repeated)
| | |- key: string
| | +- value: record
| | | |- branch: string
| | | +- extra: record
| | | | |- enrollment_id: string
| | | | |- type: string
| |- ping_type: string
| |- reason: string
| |- seq: integer
| |- start_time: string
|- sample_id: integer
|- submission_timestamp: timestamp
|
|
||
Comment 12•1 year ago
|
||
The new user-characteristics ping table is available with the reduced schema and we don't plan on supporting this conversion going forward.
Description
•