1852117 - "Adding a new field to Clients Daily" cookbook no longer works

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Frank Bertsch [:frank]]

Cookbook is here: https://mozilla.github.io/bigquery-etl/cookbooks/common_workflows/#example-add-a-new-field-to-clients_daily

The first issue is that clients_last_seen and friends are not being dryrun. Given that, we need to use bqetl query schema update telemetry_derived.clients_daily_v6 --update-downstream --ignore-dryrun-skip

However even when I ran with that option, I didn't see updates to clients_last_seen. At this point I manually update the schemas.

Comment 1

[Mike Williams [:mwilliams]]

Investigated this with :ascholtz, and it appears that the incorrect docs being followed initially caused the changes to be ignored subsequently. If the docs had been correct (i.e., with --ignore-dryrun-skip), then the changes would be caught (just tested this) and updated downstream. But running without the flag led to the changes being ignored downstream and also no longer new, so subsequent runs with the flag ignored them (also tested this).

tl;dr -- a fix to the docs should be all that is needed

Comment 2

[Mike Williams [:mwilliams]]

:frank indicated that clients_last_seen_joined was missing and was causing errors, so I will create a new bug to investigate that. Docs update was merged though here https://github.com/mozilla/bigquery-etl/pull/4270