1503464 - Language Packs should be more resilient to hg failures

Status: NEW

(Firefox Build System :: General, defect)

Description

[Justin Wood (:Callek)]

For deved 64.0b5 we hit an intermittent hg issue (pushlog)

19:06:24     INFO -  z:/task_1540837714/build/src/obj-firefox/_virtualenvs/init/Scripts/python.exe -m mozbuild.action.langpack_manifest --locales th --min-app-ver 64.0 --max-app-ver 64.* --app-name "Firefox Developer Edition" --l10n-basedir "z:/task_1540837714/build/l10n" --defines z:/task_1540837714/build/l10n/th/toolkit/defines.inc z:/task_1540837714/build/l10n/th/browser/defines.inc  --langpack-eid "[e-mail]" --input ../../dist/xpi-stage/locale-th
19:06:26    ERROR -  Traceback (most recent call last):
19:06:26     INFO -    File "c:\mozilla-build\python\Lib\runpy.py", line 174, in _run_module_as_main
19:06:26     INFO -      "__main__", fname, loader, pkg_name)
19:06:26     INFO -    File "c:\mozilla-build\python\Lib\runpy.py", line 72, in _run_code
19:06:26     INFO -      exec code in run_globals
19:06:26     INFO -    File "z:\task_1540837714\build\src\python\mozbuild\mozbuild\action\langpack_manifest.py", line 490, in <module>
19:06:26     INFO -      main(sys.argv[1:])
19:06:26     INFO -    File "z:\task_1540837714\build\src\python\mozbuild\mozbuild\action\langpack_manifest.py", line 484, in main
19:06:26     INFO -      chrome_entries
19:06:26     INFO -    File "z:\task_1540837714\build\src\python\mozbuild\mozbuild\action\langpack_manifest.py", line 431, in create_webmanifest
19:06:26     INFO -      'version': get_timestamp_for_locale(os.path.join(l10n_basedir, loc)),
19:06:26     INFO -    File "z:\task_1540837714\build\src\python\mozbuild\mozbuild\action\langpack_manifest.py", line 100, in get_timestamp_for_locale
19:06:26     INFO -      dt = get_dt_from_hg(path)
19:06:26     INFO -    File "z:\task_1540837714\build\src\python\mozbuild\mozbuild\action\langpack_manifest.py", line 72, in get_dt_from_hg
19:06:26     INFO -      data = response.json()
19:06:26     INFO -    File "z:\task_1540837714\build\src\third_party\python\requests\requests\models.py", line 808, in json
19:06:26     INFO -      return complexjson.loads(self.text, **kwargs)
19:06:26     INFO -    File "c:\mozilla-build\python\Lib\json\__init__.py", line 339, in loads
19:06:26     INFO -      return _default_decoder.decode(s)
19:06:26     INFO -    File "c:\mozilla-build\python\Lib\json\decoder.py", line 364, in decode
19:06:26     INFO -      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
19:06:26     INFO -    File "c:\mozilla-build\python\Lib\json\decoder.py", line 382, in raw_decode
19:06:26     INFO -      raise ValueError("No JSON object could be decoded")
19:06:26     INFO -  ValueError: No JSON object could be decoded
19:06:26     INFO -  z:/task_1540837714/build/src/toolkit/locales/l10n.mk:213: recipe for target 'package-langpack-th' failed
19:06:26     INFO -  mozmake.exe[1]: *** [package-langpack-th] Error 1

Which the code in question doesn't seem to do any sort of retrying:

https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/langpack_manifest.py#54

Including checking the response status code before trying to parse it as json.

Comment 1

[Gregory Szorc [:gps]]

Networks are intrinsically unreliable and even network services that are 100% "available" will experience intermittent failures. Even if hg.mozilla.org (or github.com or any other network service for that matter) is available, we will experience random failures.

The failing code is performing an HTTP request without retry logic.

All robust code performing network requests should support retry.

This code should be changed to support retrying failing requests. Proper retries of failing network requests involve increasing (often exponential) backoffs and adding jitter to sleep times. https://dxr.mozilla.org/mozilla-central/search?q=retrier&case=true demonstrates some examples of code in tree that does this.