1744727 - Airflow DAG probe_scraper tasks lookml_generator & lookml_generator_staging failing on 2021-12-06

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Krzysztof Ignasiak [:kik]]

Airflow DAG probe_scraper tasks lookml_generator & lookml_generator_staging failing on 2021-12-06.

Log from lookml_generator_staging:

{pod_launcher.py:149}} INFO - + lookml-generator namespaces --custom-namespaces /app/lookml-generator/custom-namespaces.yaml --generated-sql-uri https://github.com/mozilla/bigquery-etl/archive/generated-sql.tar.gz --app-listings-uri https://probeinfo.telemetry.mozilla.org/v2/glean/app-listings --disallowlist /app/lookml-generator/namespaces-disallowlist.yaml
[2021-12-07 04:37:15,810] {{pod_launcher.py:149}} INFO - ~ ~ ~
[2021-12-07 04:37:22,415] {{pod_launcher.py:149}} INFO - + lookml-generator lookml --namespaces namespaces.yaml --target-dir looker-hub
[2021-12-07 04:39:44,446] {{pod_launcher.py:149}} INFO - Traceback (most recent call last):
[2021-12-07 04:39:44,447] {{pod_launcher.py:149}} INFO -   File "/usr/local/bin/lookml-generator", line 33, in <module>
[2021-12-07 04:39:44,447] {{pod_launcher.py:149}} INFO -     sys.exit(load_entry_point('lookml-generator', 'console_scripts', 'lookml-generator')())
[2021-12-07 04:39:44,447] {{pod_launcher.py:149}} INFO -   File "/app/lookml-generator/generator/__main__.py", line 9, in main
[2021-12-07 04:39:44,447] {{pod_launcher.py:149}} INFO -     cli("generator")
[2021-12-07 04:39:44,447] {{pod_launcher.py:149}} INFO -   File "/app/lookml-generator/generator/__init__.py", line 37, in cli
[2021-12-07 04:39:44,448] {{pod_launcher.py:149}} INFO -     group(prog_name=prog_name)
[2021-12-07 04:39:44,448] {{pod_launcher.py:149}} INFO -   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
[2021-12-07 04:39:44,448] {{pod_launcher.py:149}} INFO -     return self.main(*args, **kwargs)
[2021-12-07 04:39:44,448] {{pod_launcher.py:149}} INFO -   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
[2021-12-07 04:39:44,448] {{pod_launcher.py:149}} INFO -     rv = self.invoke(ctx)
[2021-12-07 04:39:44,448] {{pod_launcher.py:149}} INFO -   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
[2021-12-07 04:39:44,449] {{pod_launcher.py:149}} INFO -     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2021-12-07 04:39:44,449] {{pod_launcher.py:149}} INFO -   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
[2021-12-07 04:39:44,449] {{pod_launcher.py:149}} INFO -     return ctx.invoke(self.callback, **ctx.params)
[2021-12-07 04:39:44,449] {{pod_launcher.py:149}} INFO -   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
[2021-12-07 04:39:44,449] {{pod_launcher.py:149}} INFO -     return __callback(*args, **kwargs)
[2021-12-07 04:39:44,449] {{pod_launcher.py:149}} INFO -   File "/app/lookml-generator/generator/lookml.py", line 188, in lookml
[2021-12-07 04:39:44,450] {{pod_launcher.py:149}} INFO -     return _lookml(namespaces, glean_apps, target_dir)
[2021-12-07 04:39:44,450] {{pod_launcher.py:149}} INFO -   File "/app/lookml-generator/generator/lookml.py", line 152, in _lookml
[2021-12-07 04:39:44,450] {{pod_launcher.py:149}} INFO -     for explore_path in _generate_explores(
[2021-12-07 04:39:44,450] {{pod_launcher.py:149}} INFO -   File "/app/lookml-generator/generator/lookml.py", line 61, in _generate_explores
[2021-12-07 04:39:44,450] {{pod_launcher.py:149}} INFO -     "explores": explore.to_lookml(client, v1_name, namespace_data),
[2021-12-07 04:39:44,451] {{pod_launcher.py:149}} INFO -   File "/app/lookml-generator/generator/explores/explore.py", line 65, in to_lookml
[2021-12-07 04:39:44,451] {{pod_launcher.py:149}} INFO -     new_lookml = self._to_lookml(client, v1_name, data)
[2021-12-07 04:39:44,451] {{pod_launcher.py:149}} INFO -   File "/app/lookml-generator/generator/explores/operational_monitoring_explore.py", line 76, in _to_lookml
[2021-12-07 04:39:44,451] {{pod_launcher.py:149}} INFO -     {f"{base_view_name}.{dimension['name']}": dimension["default"]}
[2021-12-07 04:39:44,451] {{pod_launcher.py:149}} INFO - KeyError: 'default'

Looks like some sort of configuration is missing an attribute default.

This is a link to the specific line in lookml-generator project:
https://github.com/mozilla/lookml-generator/blob/43e8e44c7a7c99d1a800c80750a8fd433a1e704e/generator/explores/operational_monitoring_explore.py#L76

Seems like this line executed the above code:
https://github.com/mozilla/lookml-generator/blob/main/generator/lookml.py#L61

I couldn't quickly figure out where that data actually originates.

Perhaps it's related to this PR: https://github.com/mozilla/lookml-generator/pull/318?

---

lookml_generator failed with the same error.

Comment 1

[Krzysztof Ignasiak [:kik]]

I believe this has been resolved by this change: https://github.com/mozilla/lookml-generator/pull/322

Comment 2

[Krzysztof Ignasiak [:kik]]

Anna confirmed this can be closed.