1646436 - Need to rotate Airflow instances to pick up python module changes

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, task, P1)

Description

[Jeff Klukas [:klukas] (UTC-4)]

Follow-up to https://bugzilla.mozilla.org/show_bug.cgi?id=1646115 from yesterday.

The new task was failing, so I filed and merged https://github.com/mozilla/telemetry-airflow/pull/1041 yesterday with changes, but those new changes are not being picked up.

:hwoo - Can you rotate Airflow instances again to get this DAG to pick up the latest code?

If you have any advice about how to better approach these kinds of changes, please do let me know.

Comment 1

[Harold Woo]

Are we sure this is a code changes not being picked up issue?

The failure mode yesterday was that the UI was showing "DAG fxa_export_to_amplitude seems to be missing." when clicking the dag. I'm not seeing this today.

https://workflow.telemetry.mozilla.org/tree?dag_id=fxa_export_to_amplitude.fxa_email_click_amplitude_export&execution_date=2020-06-16T07%3A30%3A00%2B00%3A00

I'm seeing errors in recreate_view, and create_temporary_table:
ERROR - tag name expected
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/airflow/models/taskinstance.py", line 940, in _run_raw_task
self.render_templates(context=context)
File "/usr/local/lib/python2.7/site-packages/airflow/models/taskinstance.py", line 1319, in render_templates
self.task.render_template_fields(context)
File "/usr/local/lib/python2.7/site-packages/airflow/models/baseoperator.py", line 708, in render_template_fields
self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())
File "/usr/local/lib/python2.7/site-packages/airflow/models/baseoperator.py", line 715, in _do_render_template_fields
rendered_content = self.render_template(content, context, jinja_env, seen_oids)
File "/usr/local/lib/python2.7/site-packages/airflow/models/baseoperator.py", line 744, in render_template
return jinja_env.from_string(content).render(**context)
File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 880, in from_string
return cls.from_code(self, self.compile(source), globals, None)
File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 591, in compile
self.handle_exception(exc_info, source_hint=source_hint)
File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "<unknown>", line 120, in template
TemplateSyntaxError: tag name expected

ERROR - BigQuery job failed. Final error was: {u'reason': u'notFound', u'message': u'Not found: Table moz-fx-data-shared-prod:firefox_accounts.fxa_amplitude_email_clicks was not found in location US'}. The job was: {u'status': {u'state': u'DONE', u'errors': [{u'reason': u'notFound', u'message': u'Not found: Table moz-fx-data-shared-prod:firefox_accounts.fxa_amplitude_email_clicks was not found in location US'}], u'errorResult': {u'reason': u'notFound', u'message': u'Not found: Table moz-fx-data-shared-prod:firefox_accounts.fxa_amplitude_email_clicks was not found in location US'}}, u'kind': u'bigquery#job', u'statistics': {u'endTime': u'1592335591853', u'creationTime': u'1592335591803', u'startTime': u'1592335591853'}, u'jobReference': {u'projectId': u'moz-fx-data-derived-datasets', u'location': u'US', u'jobId': u'job_HGdmxKzz_Qb58qJCimpLMHwOVmbX'}, u'etag': u'IGYy3Pl7KaBjDTWWBPflbA==', u'selfLink': u'https://bigquery.googleapis.com/bigquery/v2/projects/moz-fx-data-derived-datasets/jobs/job_HGdmxKzz_Qb58qJCimpLMHwOVmbX?location=US', u'configuration': {u'query': {u'useLegacySql': False, u'destinationTable': {u'projectId': u'moz-fx-data-derived-datasets', u'tableId': u'fxa_amplitude_email_clicks_20200615', u'datasetId': u'firefox_accounts'}, u'priority': u'INTERACTIVE', u'writeDisposition': u'WRITE_EMPTY', u'allowLargeResults': False, u'createDisposition': u'CREATE_IF_NEEDED', u'query': u'SELECT * EXCEPT (submission_timestamp) FROM moz-fx-data-shared-prod.firefox_accounts.fxa_amplitude_email_clicks WHERE DATE(submission_timestamp) = "2020-06-15"'}, u'jobType': u'QUERY'}, u'id': u'moz-fx-data-derived-datasets:US.job_HGdmxKzz_Qb58qJCimpLMHwOVmbX', u'user_email': u'[e-mail]'}

Comment 2

[Harold Woo]

Rolled webapp anyway. Let me know if this helps.

I don't have context into this issue but the PR https://github.com/mozilla/telemetry-airflow/pull/1041/files given only seems to have added owner email args and switched ordering of recreate_view >> wait_for_data, which was already reflected in the UI prior. There are probably concurrent code changes in other repos I'm unaware of.

Comment 3

[Jeff Klukas [:klukas] (UTC-4)]

The DAG pulls in the view query via github url; I did indeed change the view definition in bigquery-etl, but managed to not completely solve the issue there. I knew this strategy was brittle, so I'm going to follow up with rethinking how to do this.

For the time being, I now see the corrected dependency order, etc. so this worked correctly and we can mark as fixed.

Comment 4

[Harold Woo]

Just to be clear, rolling the webapp in this scenario did not fix the issue. The dependency order was already this way prior.