1834587 - Airflow task `probe_scraper.mozilla_schema_generator` failing for 2023-05-22

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Sean Rose [:srose]]

+ mozilla-schema-generator generate-main-ping --mps-branch main --out-dir ./telemetry
Traceback (most recent call last):
  File "/usr/local/bin/mozilla-schema-generator", line 33, in <module>
    sys.exit(load_entry_point('mozilla-schema-generator', 'console_scripts', 'mozilla-schema-generator')())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/app/mozilla-schema-generator/mozilla_schema_generator/__main__.py", line 92, in generate_main_ping
    schemas = schema_generator.generate_schema(config, split=False)
  File "/app/mozilla-schema-generator/mozilla_schema_generator/generic_ping.py", line 52, in generate_schema
    env = self.get_env()
  File "/app/mozilla-schema-generator/mozilla_schema_generator/common_ping.py", line 83, in get_env
    env_property = json.loads("{" + self._get_json_str(self.env_url) + "}")
  File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 406 column 9 (char 13682)

Comment 1

[Sean Rose [:srose]]

It looks like this was caused by mozilla-pipeline-schemas PR 773, and it seems like mozilla_schema_generator/common_ping.py isn't accounting for the possibility that there may be inclusion expressions like @COMMON_ATTRIBUTION_1_JSON@ in mozilla-pipeline-schemas/templates/include/telemetry/environment.1.schema.json.

Comment 2

[BMO Github Automation]

Attachment: [mozilla-services/mozilla-pipeline-schemas] Bug 1834587 remove common attribute from environment schema (#775)

Comment 3

[Leli [:Leli]]

fixed and rerun successfully