1798269 - The Airflow DAG probe_scrapper.mozilla_schema_generator failing since 2022-10-28

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P1)

Description

[Krzysztof Ignasiak [:kik]]

The Airflow DAG probe_scrapper.mozilla_schema_generator failing since 2022-10-28

Airflow task log link:
https://workflow.telemetry.mozilla.org/log?dag_id=probe_scraper&task_id=mozilla_schema_generator&execution_date=2022-10-28T00%3A00%3A00%2B00%3A00

Airflow task log:

[2022-10-31 08:33:20,138] {{pod_launcher.py:149}} INFO - + mozilla-schema-generator generate-main-ping --mps-branch main --out-dir ./telemetry
[2022-10-31 08:33:46,821] {{pod_launcher.py:149}} INFO - Traceback (most recent call last):
[2022-10-31 08:33:46,821] {{pod_launcher.py:149}} INFO -   File "/usr/local/bin/mozilla-schema-generator", line 33, in <module>
[2022-10-31 08:33:46,821] {{pod_launcher.py:149}} INFO -     sys.exit(load_entry_point('mozilla-schema-generator', 'console_scripts', 'mozilla-schema-generator')())
[2022-10-31 08:33:46,821] {{pod_launcher.py:149}} INFO -   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in __call__
[2022-10-31 08:33:46,822] {{pod_launcher.py:149}} INFO -     return self.main(*args, **kwargs)
[2022-10-31 08:33:46,822] {{pod_launcher.py:149}} INFO -   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1062, in main
[2022-10-31 08:33:46,822] {{pod_launcher.py:149}} INFO -     rv = self.invoke(ctx)
[2022-10-31 08:33:46,822] {{pod_launcher.py:149}} INFO -   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1668, in invoke
[2022-10-31 08:33:46,822] {{pod_launcher.py:149}} INFO -     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2022-10-31 08:33:46,822] {{pod_launcher.py:149}} INFO -   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
[2022-10-31 08:33:46,822] {{pod_launcher.py:149}} INFO -     return ctx.invoke(self.callback, **ctx.params)
[2022-10-31 08:33:46,823] {{pod_launcher.py:149}} INFO -   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 763, in invoke
[2022-10-31 08:33:46,823] {{pod_launcher.py:149}} INFO -     return __callback(*args, **kwargs)
[2022-10-31 08:33:46,823] {{pod_launcher.py:149}} INFO -   File "/app/mozilla-schema-generator/mozilla_schema_generator/__main__.py", line 92, in generate_main_ping
[2022-10-31 08:33:46,823] {{pod_launcher.py:149}} INFO -     schemas = schema_generator.generate_schema(config, split=False)
[2022-10-31 08:33:46,823] {{pod_launcher.py:149}} INFO -   File "/app/mozilla-schema-generator/mozilla_schema_generator/generic_ping.py", line 92, in generate_schema
[2022-10-31 08:33:46,823] {{pod_launcher.py:149}} INFO -     "Schema must be smaller or equal max_size {}".format(max_size)
[2022-10-31 08:33:46,823] {{pod_launcher.py:149}} INFO - mozilla_schema_generator.schema.SchemaException: Schema must be smaller or equal max_size 11000
[2022-10-31 08:33:48,839] {{pod_launcher.py:198}} INFO - Event: schema-generator-1.c1fd0a8001664aca9636981e4f150a34 had an event of type Failed

Comment 2

[Frank Bertsch [:frank]]

The new increased max size is deployed, and the generator ran successfully.