1687360 - Intermittent decision task TypeError: getresponse() got an unexpected keyword argument 'buffering'

Status: RESOLVED FIXED

(Release Engineering :: Firefox-CI Administration, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=327038553&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PTurb92fR9a81Z5xn91MwA/runs/0/artifacts/public/logs/live_backing.log

---

jcristau pointed to https://github.com/urllib3/urllib3/commit/4c65505f1dd420a15eb7bbcd1fc84f4cee701234 for the 'buffering' code removal.

[task 2021-01-18T16:43:59.091Z] attempt 2/4
[task 2021-01-18T16:43:59.091Z] retry: calling find_decision_task with args: ({'project': 'mozilla-central', 'target_tasks_method': 'nightly_macosx', 'release_enable_partner_attribution': False, 'release_eta': '', 'release_type': 'nightly', 'head_repository': 'https://hg.mozilla.org/mozilla-central', 'optimize_strategies': None, 'build_date': 1610987637, 'moz_build_date': '20210118163357', 'pushdate': 1610987637, 'try_mode': None, 'filters': ['target_tasks_method'], 'existing_tasks': {}, 'tasks_for': 'cron', 'optimize_target_tasks': True, 'release_product': None, 'release_partners': [], 'hg_branch': 'default', 'message': '', 'try_options': None, 'owner': '[e-mail]', 'release_enable_partner_repack': False, 'test_manifest_loader': 'default', 'head_rev': '1926d7641900f10134d1591637fa1bc90815c2cf', 'try_task_config': {}, 'level': '3', 'version': '86.0a1', 'phabricator_diff': None, 'base_repository': 'https://hg.mozilla.org/mozilla-unified', 'release_enable_emefree': False, 'signoff_urls': {}, 'next_version': None, 'release_partner_build_number': 1, 'do_not_optimize': [], 'head_ref': '1926d7641900f10134d1591637fa1bc90815c2cf', 'pushlog_id': '38114', 'app_version': '86.0a1', 'build_number': 1, 'release_partner_config': {}, 'required_signoffs': []}, GraphConfig(_config={'trust-domain': 'gecko', 'scriptworker': {'scope-prefix': 'project:releng'}, 'project-repo-param-prefix': '', 'partner-urls': {'release-partner-repack': {'by-release-product': {'default': None, 'firefox': {'by-release-type': {'beta|release.*': {'by-release-level': {'production': '[e-mail]:mozilla-partners/repack-manifests.git', 'staging': '[e-mail]:moz-releng-automation-stage/repack-manifests.git'}}, 'default': None, 'esr.*': {'by-release-level': {'production': '[e-mail]:mozilla-partners/esr-repack-manifests.git', 'staging': '[e-mail]:moz-releng-automation-stage/esr-repack-manifests.git'}}}}}}, 'release-eme-free-repack': {'by-release-product': {'default': None, 'firefox': {'by-release-type': {'beta|release.*': {'by-release-level': {'production': '[e-mail]:mozilla-partners/mozilla-EME-free-manifest.git', 'staging': '[e-mail]:moz-releng-automation-stage/mozilla-EME-free-manifest.git'}}, 'default': None}}}}, 'release-partner-attribution': {'by-release-product': {'default': None, 'firefox': {'by-release-type': {'beta|release.*': {'by-release-level': {'production': '[e-mail]:mozilla-partners/repack-manifests.git', 'staging': '[e-mail]:moz-releng-automation-stage/repack-manifests.git'}}, 'default': None}}}}}, 'index': {'products': ['firefox', 'fennec', 'mobile', 'static-analysis', 'devedition', 'source', 'system-symbols', 'geckodriver']}, 'merge-automation': {'behaviors': {'bump-central': {'fetch-version-from': 'browser/config/version.txt', 'version-files': [{'new-suffix': 'a1', 'filename': 'config/milestone.txt', 'version-bump': 'major'}, {'new-suffix': 'a1', 'filename': 'browser/config/version.txt', 'version-bump': 'major'}, {'new-suffix': 'a1', 'filename': 'browser/config/version_display.txt', 'version-bump': 'major'}], 'replacements': [['xpcom/components/Module.h', 'static const unsigned int kVersion = {current_major_version};', 'static const unsigned int kVersion = {next_major_version};'], ['services/sync/modules/constants.js', 'WEAVE_VERSION: "1.{current_weave_version}.0"', 'WEAVE_VERSION: "1.{next_weave_version}.0"']], 'merge-old-head': False, 'end-tag': 'FIREFOX_NIGHTLY_{major_version}_END', 'to-repo': 'https://hg.mozilla.org/mozilla-central', 'to-branch': 'central'}, 'central-to-beta': {'to-branch': 'beta', 'version-files': [{'new-suffix': '', 'filename': 'config/milestone.txt'}, {'new-suffix': '', 'filename': 'browser/config/version.txt'}, {'new-suffix': 'b1', 'filename': 'browser/config/version_display.txt'}], 'from-branch': 'central', 'replacements': [['browser/config/mozconfigs/linux32/l10n-mozconfig', 'ac_add_options --with-branding=browser/branding/nightly', 'ac_add_options --enable-official-branding'], ['browser/config/mozconfigs/linux64/l10n-mozconfig', 'ac_add_options --with-branding=browser/branding/nightly', 'ac_add_options --enable-official-branding'], ['browser/config/mozconfigs/win32/l10n-mozconfig', 'ac_add_options --with-branding=browser/branding/nightly', 'ac_add_options --enable-official-branding'], ['browser/config/mozconfigs/win64/l10n-mozconfig', 'ac_add_options --with-branding=browser/branding/nightly', 'ac_add_options --enable-official-branding'], ['browser/config/mozconfigs/win64-aarch64/l10n-mozconfig', 'ac_add_options --with-branding=browser/branding/nightly', 'ac_add_options --enable-official-branding'], ['browser/config/mozconfigs/macosx64/l10n-mozconfig', 'ac_add_options --with-branding=browser/branding/nightly', 'ac_add_options --enable-official-branding'], ['build/mozconfig.common', 'MOZ_REQUIRE_SIGNING=${MOZ_REQUIRE_SIGNING-0}', 'MOZ_REQUIRE_SIGNING=${MOZ_REQUIRE_SIGNING-1}'], ['build/mozconfig.common', '# Disable enforcing that add-ons are signed by the trusted root', '# Enable enforcing that add-ons are signed by the trusted root']], 'end-tag': 'FIREFOX_BETA_{major_version}_END', 'to-repo': 'https://hg.mozilla.org/releases/mozilla-beta', 'fetch-version-from': 'browser/config/version.txt', 'from-repo': 'https://hg.mozilla.org/mozilla-central', 'merge-old-head': True, 'base-tag': 'FIREFOX_BETA_{major_version}_BASE'}, 'beta-to-release': {'to-branch': 'release', 'version-files': [{'new-suffix': '', 'filename': 'browser/config/version_display.txt'}], 'from-branch': 'beta', 'replacements': [], 'end-tag': 'FIREFOX_RELEASE_{major_version}_END', 'to-repo': 'https://hg.mozilla.org/releases/mozilla-release', 'fetch-version-from': 'browser/config/version.txt', 'from-repo': 'https://hg.mozilla.org/releases/mozilla-beta', 'merge-old-head': True, 'base-tag': 'FIREFOX_RELEASE_{major_version}_BASE'}, 'bump-esr': {'fetch-version-from': 'browser/config/version.txt', 'version-files': [{'filename': 'config/milestone.txt', 'version-bump': 'minor'}, {'filename': 'browser/config/version.txt', 'version-bump': 'minor'}, {'filename': 'browser/config/version_display.txt', 'version-bump': 'minor'}], 'replacements': [], 'merge-old-head': False, 'to-repo': 'https://hg.mozilla.org/releases/mozilla-esr78', 'to-branch': 'esr78'}, 'release-to-esr': {'to-branch': 'esr78', 'version-files': [{'new-suffix': 'esr', 'filename': 'browser/config/version_display.txt'}], 'from-branch': 'release', 'replacements': [['build/mozconfig.common', '# Enable enforcing that add-ons are signed by the trusted root', '# Disable enforcing that add-ons are signed by the trusted root'], ['build/mozconfig.common', 'MOZ_REQUIRE_SIGNING=${MOZ_REQUIRE_SIGNING-1}', 'MOZ_REQUIRE_SIGNING=${MOZ_REQUIRE_SIGNING-0}']], 'to-repo': 'https://hg.mozilla.org/releases/mozilla-esr78', 'fetch-version-from': 'browser/config/version.txt', 'from-repo': 'https://hg.mozilla.org/releases/mozilla-release', 'merge-old-head': False, 'base-tag': 'FIREFOX_ESR_{major_version}_BASE'}}}, 'release-promotion': {'flavors': {'ship_firefox_rc': {'product': 'firefox', 'is-rc': True, 'partial-updates': True, 'target-tasks-method': 'ship_desktop'}, 'promote_firefox_partner_attribution': {'product': 'firefox', 'rebuild-kinds': ['release-partner-attribution', 'release-partner-attribution-beetmover'], 'target-tasks-method': 'promote_desktop'}, 'promote_firefox_partner_repack': {'product': 'firefox', 'rebuild-kinds': ['release-partner-repack', 'release-partner-repack-chunking-dummy', 'release-partner-repack-signing', 'release-partner-repack-notarization-part-1', 'release-partner-repack-notarization-poller', 'release-partner-repack-repackage', 'release-partner-repack-repackage-signing', 'release-partner-repack-beetmover'], 'target-tasks-method': 'promote_desktop'}, 'promote_firefox': {'product': 'firefox', 'partial-updates': True, 'target-tasks-method': 'promote_desktop'}, 'push_devedition': {'product': 'devedition', 'partial-updates': True, 'target-tasks-method': 'push_desktop'}, 'promote_firefox_rc': {'product': 'firefox', 'is-rc': True, 'partial-updates': True, 'target-tasks-method': 'promote_desktop'}, 'ship_firefox': {'product': 'firefox', 'version-bump': True, 'partial-updates': True, 'target-tasks-method': 'ship_desktop'}, 'push_firefox': {'product': 'firefox', 'partial-updates': True, 'target-tasks-method': 'push_desktop'}, 'promote_devedition': {'product': 'devedition', 'partial-updates': True, 'target-tasks-method': 'promote_desktop'}, 'ship_devedition': {'product': 'devedition', 'version-bump': True, 'partial-updates': True, 'target-tasks-method': 'ship_desktop'}}, 'products': ['devedition', 'fennec', 'firefox']}, 'task-priority': {'by-project': {'default': 'very-low', 'mozilla-central': 'medium', 'mozilla-release': 'highest', 'mozilla-beta': 'high', 'autoland': 'low', 'mozilla-esr.*': 'very-high', 'mozilla-inbound': 'low'}}, 'product-dir': 'browser', 'workers': {'aliases': {'mac-signing': {'provisioner': 'scriptworker-prov-v1', 'os': 'macosx', 'implementation': 'scriptworker-signing', 'worker-type': {'by-release-level': {'production': 'signing-mac-v1', 'staging': 'depsigning-mac-v1'}}}, 'linux-signing': {'provisioner': 'scriptworker-k8s', 'os': 'linux', 'implementation': 'scriptworker-signing', 'worker-type': {'by-release-level': {'production': '{trust-domain}-3-signing', 'staging': '{trust-domain}-t-signing'}}}, 'b-linux.*': {'provisioner': '{trust-domain}-{level}', 'os': 'linux', 'implementation': 'docker-worker', 'worker-type': '{alias}'}, 't-linux-xlarge-pgo': {'provisioner': {'by-level': {'default': '{trust-domain}-t', '3': '{trust-domain}-{level}'}}, 'os': 'linux', 'implementation': 'docker-worker', 'worker-type': 't-linux-xlarge'}, 't-linux(-large|-xlarge|-xlarge-source|-metal)': {'provisioner': '{trust-domain}-t', 'os': 'linux', 'implementation': 'docker-worker', 'worker-type': '{alias}'}, 'mac-depsigning': {'provisioner': 'scriptworker-prov-v1', 'os': 'macosx', 'implementation': 'scriptworker-signing', 'worker-type': 'depsigning-mac-v1'}, 'linux-depsigning': {'provisioner': 'scriptworker-k8s', 'os': 'linux', 'implementation': 'scriptworker-signing', 'worker-type': '{trust-domain}-t-signing'}, 'balrog': {'provisioner': 'scriptworker-k8s', 'os': 'scriptworker', 'implementation': 'balrog', 'worker-type': {'by-release-level': {'production': '{trust-domain}-3-balrog', 'staging': '{trust-domain}-1-balrog'}}}, 't-osx-1014': {'provisioner': 'releng-hardware', 'os': 'macosx', 'implementation': 'generic-worker', 'worker-type': 'gecko-{alias}'}, 'succeed': {'provisioner': 'built-in', 'os': 'none', 'implementation': 'succeed', 'worker-type': 'succeed'}, 't-win10-64(-hw|-ref-hw|-1803-hw)': {'provisioner': 'releng-hardware', 'os': 'windows', 'implementation': 'generic-worker', 'worker-type': 'gecko-{alias}'}, 't-bitbar-gw.*': {'provisioner': 'proj-autophone', 'os': 'linux-bitbar', 'implementation': 'generic-worker', 'worker-type': 'gecko-{alias}'}, 'beetmover': {'provisioner': 'scriptworker-k8s', 'os': 'scriptworker', 'implementation': 'beetmover', 'worker-type': {'by-release-level': {'production': '{trust-domain}-3-beetmover', 'staging': '{trust-domain}-1-beetmover'}}}, 'images': {'provisioner': '{trust-domain}-{level}', 'os': 'linux', 'implementation': 'docker-worker', 'worker-type': '{alias}'}, 'tree': {'provisioner': 'scriptworker-k8s', 'os': 'scriptworker', 'implementation': 'treescript', 'worker-type': {'by-release-level': {'production': '{trust-domain}-3-tree', 'staging': '{trust-domain}-1-tree'}}}, 't-linux-talos': {'provisioner': 'releng-hardware', 'os': 'linux', 'implementation': 'generic-worker', 'worker-type': 'gecko-{alias}'}, 't-win10-64(|-gpu-s|-source)': {'provisioner': '{trust-domain}-t', 'os': 'windows', 'implementation': 'generic-worker', 'worker-type': '{alias}'}, 't-win7-32(|-gpu)': {'provisioner': '{trust-domain}-t', 'os': 'windows', 'implementation': 'generic-worker', 'worker-type': '{alias}'}, 'bouncer': {'provisioner': 'scriptworker-k8s', 'os': 'scriptworker', 'implementation': 'bouncer', 'worker-type': {'by-release-level': {'production': '{trust-domain}-3-bouncer', 'staging': '{trust-domain}-1-bouncer'}}}, 'mac-notarization-poller': {'provisioner': 'scriptworker-prov-v1', 'os': 'macosx', 'implementation': 'notarization-poller', 'worker-type': 'mac-notarization-poller'}, 'tree-dev': {'provisioner': 'scriptworker-k8s', 'os': 'scriptworker', 'implementation': 'treescript', 'worker-type': '{trust-domain}-1-tree-dev'}, 't-osx-1014-power': {'provisioner': 'releng-hardware', 'os': 'macosx', 'implementation': 'generic-worker', 'worker-type': 'gecko-{alias}'}, 'misc': {'provisioner': '{trust-domain}-t', 'os': 'linux', 'implementation': 'docker-worker', 'worker-type': 'misc'}, 'b-osx-1015': {'provisioner': 'releng-hardware', 'os': 'macosx', 'implementation': 'generic-worker', 'worker-type': {'by-level': {'default': 'gecko-1-b-osx-1015', '3': 'gecko-3-b-osx-1015'}}}, 'b-win2012': {'provisioner': '{trust-domain}-{level}', 'os': 'windows', 'implementation': 'generic-worker', 'worker-type': '{alias}'}, 't-win64-aarch64-laptop': {'provisioner': 'bitbar', 'os': 'windows', 'implementation': 'generic-worker', 'worker-type': 'gecko-{alias}'}}}, 'taskgraph': {}, 'mac-notarization': {'mac-entitlements': {'by-platform': {'macosx64.*': {'by-release-level': {'default': 'security/mac/hardenedruntime/developer.entitlements.xml', 'production': 'security/mac/hardenedruntime/production.entitlements.xml'}}, 'default': ''}}, 'mac-behavior': {'by-project': {'default': 'mac_sign_and_pkg', 'mozilla-(central|beta|release|esr.*)': {'by-shippable': {'default': 'mac_sign_and_pkg', 'true': 'mac_notarize'}}}}}, 'try': {'ridealong-builds': {'linux': ['sm-arm-sim-linux32'], 'win32': ['sm-plain-win32', 'sm-compacting-win32'], 'linux64': ['sm-plain-linux64', 'sm-nojit-linux64', 'sm-nonunified-linux64', 'sm-arm-sim-linux32', 'sm-arm64-sim-linux64', 'sm-compacting-linux64', 'sm-rootanalysis-linux64', 'sm-package-linux64', 'sm-tsan-linux64', 'sm-asan-linux64', 'sm-mozjs-sys-linux64', 'sm-msan-linux64', 'sm-fuzzing-linux64', 'sm-rust-bindings-linux64']}}, 'treeherder': {'group-names': {'Btime-live-fenix-wr': 'Browsertime performance tests on Firefox with webrender enabled', 'ms-stage': 'Autograph-stage MAR signing test', 'Btime-Prof-wr': 'Browsertime performance tests on Firefox with Gecko Profiling, webrender enabled', 'X': 'Xpcshell tests', 'Aries': 'Aries Device Image', 'Btime-ChR': 'Browsertime performance tests on Google Chrome Release', 'Rap-fennec': 'Raptor performance tests on Fennec', 'l10n-bump': 'L10n Bumper', 'Flatpak': 'Flatpak image generation', 'VP': 'VideoPuppeteer tests', 'M-V-1proc': 'Mochitests on Valgrind without e10s', 'Rap': 'Raptor performance tests on Firefox', 'js': 'JavaScript checks', 'Btime-live': 'Browsertime performance tests on Firefox', 'cs': 'Checksum signing', 'Btime-fennec': 'Browsertime performance tests on Fennec', 'Rel': 'Release promotion', 'M-gli': 'Mochitests with WebGL over IPC', 'W-1proc': 'Web platform tests without e10s', 'Rap-fennec-1proc': 'Raptor performance tests on Fennec without e10s', 'Btime-live-fenix': 'Browsertime performance tests on Firefox', 'T-Prof-1proc': 'Talos performance tests on Firefox with Gecko Profiling and without e10s', 'SM': 'Spidermonkey builds', 'Btime-Cr': 'Browsertime performance tests on Google Chromium', 'Rap-live': 'Raptor performance tests on Firefox with live sites', 'Btime-Prof-fennec': 'Browsertime performance tests on Fennec with Gecko Profiling', 'BM': 'Beetmover', 'Btime-wr-1proc': 'Browsertime performance tests on Firefox without e10s, with webrender enabled', 'remote': 'Remote debugging protocol tests', 'pydep': 'python dependency update', 'W-fis': 'Web platform tests with fission enabled', 'M': 'Mochitests', 'Searchfox': 'Searchfox builds', 'Btime': 'Browsertime performance tests on Firefox', 'WR': 'WebRender standalone', 'vismet': 'Visual Metrics Analysis', 'T-Prof-fis': 'Talos performance tests on Firefox with fission and Gecko Profiling', 'W-b': 'Web platform tests (backlog)', 'TL': 'Toolchain builds for Linux 64-bits', 'R': 'Reftests', 'TL32': 'Toolchain builds for Linux 32-bits', 'Rap-wr': 'Raptor performance tests on Firefox', 'W-b-fis': 'Web platform tests (backlog) with fission enabled', 'M-fis-xorig': 'Mochitests with cross-origin and fission enabled', 'Btime-P-wr': 'Browsertime power tests on Firefox/Geckoview with webrender enabled', 'T-1proc': 'Talos performance tests without e10s', 'SY': 'Are we slim yet tests by TaskCluster', 'pub': 'APK publishing', 'TW64': 'Toolchain builds for Windows 64-bits', 'Rap-Prof-1proc': 'Raptor performance tests on Firefox with Gecko Profiling and without e10s', 'misc': 'Misc checks', 'T-swr': 'Talos performance tests with software webrender enabled', 'M-spi-1proc': 'Mochitests with socket process without e10s', 'T-gli': 'Talos performance tests with WebGL over IPC', 'Static-Analysis': 'Full tree static-analysis', 'perftest': 'Performance tests', 'T-Prof-gli': 'Talos performance tests with WebGL over IPC and Gecko Profiling', 'Deb9': 'Packages for Debian 9', 'Btime-fenix': 'Browsertime performance tests on Fenix', 'M-1proc': 'Mochitests without e10s', 'L10n': 'Localised Repacks', 'Btime-Prof': 'Browsertime performance tests on Firefox with Gecko Profiling', 'condprof': 'Conditioned Profile Builder', 'perftest-http3': 'Performance tests with HTTP/3', 'fuzzing': 'Fuzzing checks', 'rs': 'Repackage signing', 'TM': 'Toolchain builds for OSX', 'js-bench-v8': 'JavaScript shell benchmarks with Google V8', 'TPS': 'Sync tests', 'Btime-Prof-fennec-1proc': 'Browsertime performance tests on Fennec without e10s with Gecko Profiling', 'W': 'Web platform tests', 'TMW': 'Toolchain builds for Windows MinGW', 'Btime-live-wr': 'Browsertime performance tests on Android with webrender enabled', 'GhS': 'GitHub Synchronization', 'BMcs': 'Beetmover checksums', 'MSIs': 'Signing of Repacked installers of MSIs', 'Fxfn-r': 'Firefox functional tests (remote)', 'WMC32': 'MinGW-Clang builds for Windows 32-bits', 'Btime-fis': 'Browsertime performance tests on Firefox with fission enabled', 'Rap-Prof': 'Raptor performance tests on Firefox with Gecko Profiling', 'MSI': 'Repack installers into MSIs', 'py2': 'Python 2 unit tests', 'Nexus 5-L': 'Nexus 5-L Device Image', 'M-spi-nw-1proc': 'Mochitests with networking on socket process without e10s', 'pedantic': 'pedantic checks', 'WMC64': 'MinGW-Clang builds for Windows 64-bits', 'doc': 'Documentation', 'M-spi-nw': 'Mochitests with networking on socket process', 'node': 'Node tests', 'Btime-P-refbrow-wr': 'Browsertime power tests on reference browser with webrender enabled', 'Btime-Prof-1proc': 'Browsertime performance tests on Firefox with Gecko Profiling and without e10s', 'py': 'Python checks', 'rust': 'Rust checks', 'Btime-wr': 'Browsertime performance tests on Android with webrender enabled', 'cram': 'Cram tests', 'T': 'Talos performance tests', 'p': 'Partial generation', 'Fxfn-l': 'Firefox functional tests (local)', 'Deb8-32': 'Packages for Debian 8 32-bits', 'js-bench-sm': 'JavaScript shell benchmarks with Spidermonkey', 'M-spi': 'Mochitests with socket process', 'Btime-P-fenix-wr': 'Browsertime power tests on Fenix with webrender enabled', 'Btime-fennec-1proc': 'Browsertime performance tests on Fennec without e10s', 'Btime-Prof-refbrow': 'Browsertime performance tests on the reference browser with Gecko Profiling', 'cpp': 'C/C++ checks', 'SS': 'Shadow scheduler', 'Btime-P-1proc-wr': 'Browsertime power tests on Fennec with webrender enabled', 'SY-fis': 'Are we slim yet tests by TaskCluster, fission enabled', 'Rap-1proc': 'Raptor performance tests on Firefox without e10s', 'BMcslang': 'Beetmover checksums for language packs', 'Rap-refbrow': 'Raptor performance tests on the reference browser', 'A': 'Android Gradle tests', 'test-info': 'Test manifest skip/fail information', 'Btime-refbrow': 'Browsertime performance tests on the reference browser', 'iris': 'Iris testing suite', 'css': 'Checksum signing for source', 'R-fis': 'Reftests with fission enabled', 'Btime-1proc': 'Browsertime performance tests on Firefox without e10s', 'ms': 'Complete MAR signing', 'Rap-refbrow-wr': 'Raptor performance tests on the reference browser', 'Gd': 'Geckodriver', 'TW32': 'Toolchain builds for Windows 32-bits', 'Snap': 'Snap image generation', 'Tss-fis': 'Talos performance tests, Stylo sequential with fission enabled', 'Deb8': 'Packages for Debian 8', 'BMR': 'Beetmover repackages', 'Rap-Cr': 'Raptor performance tests on Google Chromium', 'py3': 'Python 3 unit tests', 'Rap-fenix': 'Raptor performance tests on Fenix', 'Rap-fis': 'Raptor performance tests on Firefox with fission enabled', 'ps': 'Partials signing', 'T-Prof': 'Talos performance tests on Firefox with Gecko Profiling', 'java': 'Java checks', 'Btime-fenix-wr': 'Browsertime performance tests on Fenix with webrender enabled', 'M-fis': 'Mochitests with fission enabled', 'R-1proc': 'Reftests without e10s', 'Rap-P-1proc': 'Raptor power tests on Firefox without e10s', 'Wgpu': 'WebGPU standalone', 'Tss': 'Talos performance tests, Stylo sequential', 'Btime-Prof-fenix': 'Browsertime performance tests on Fenix with Gecko Profiling', 'c-Up': 'Balrog submission of complete updates', 'L10n-Rpk': 'Localized Repackaged Repacks', 'tt': 'Telemetry tests', 'tt-1proc': 'Telemetry tests without e10s', 'R-swr': 'Reftests with software webrender enabled', 'Rap-ChR': 'Raptor performance tests on Google Chrome Release', 'Deb10': 'Packages for Debian 10', 'Btime-P-refbrow': 'Browsertime power tests on reference browser', 'Btime-live-ChR': 'Browsertime performance tests on Firefox', 'Btime-P-fenix': 'Browsertime power tests on Fenix', 'Bpgo': 'Profile-guided optimization builds', 'Btime-refbrow-wr': 'Browsertime performance tests on the reference browser with webrender enabled', 'text': 'Check on texts', 'X-fis': 'Xpcshell tests with fission enabled', 'Tss-1proc': 'Talos performance tests with Stylo sequential without e10s', 'BMcss': 'Beetmover checksums for source', 'X-spi-nw': 'Xpcshell tests with networking on socket process', 'UV': 'Update verify', 'Fxfn-l-1proc': 'Firefox functional tests (local) without e10s', 'X-1proc': 'Xpcshell tests, without e10s', 'Fxfn-r-1proc': 'Firefox functional tests (remote) without e10s', 'langpack': 'Langpack sigatures and uploads', 'Rap-P': 'Raptor power tests on Firefox', 'Fetch': 'Fetch and store content', 'Btime-P': 'Browsertime power tests on Firefox/Geckoview', 'Btime-P-1proc': 'Browsertime power tests on Fennec', 'T-fis': 'Talos performance tests with fission enabled', 'I': 'Docker Image Builds'}}}, root_dir='taskcluster/ci')), kwargs: {}, attempt #2
[task 2021-01-18T16:43:59.093Z] Resetting dropped connection: firefox-ci-tc.services.mozilla.com
[task 2021-01-18T16:43:59.137Z] "GET /api/index/v1/task/gecko.v2.mozilla-central.revision.1926d7641900f10134d1591637fa1bc90815c2cf.taskgraph.decision HTTP/1.1" 200 220
[task 2021-01-18T16:43:59.175Z] "GET /api/queue/v1/task/IGNGPEwFTOKAb-EZWdVNtg/artifacts/public/full-task-graph.json HTTP/1.1" 303 122
[task 2021-01-18T16:43:59.177Z] Starting new HTTPS connection (1): firefoxci.taskcluster-artifacts.net
[task 2021-01-18T16:43:59.383Z] "GET /IGNGPEwFTOKAb-EZWdVNtg/0/public/full-task-graph.json HTTP/1.1" 200 147689354
[task 2021-01-18T16:44:04.570Z] "GET /api/queue/v1/task/IGNGPEwFTOKAb-EZWdVNtg/artifacts/public/label-to-taskid.json HTTP/1.1" 303 122
[task 2021-01-18T16:44:04.722Z] "GET /IGNGPEwFTOKAb-EZWdVNtg/0/public/label-to-taskid.json HTTP/1.1" 200 292813
[task 2021-01-18T16:44:05.085Z] Connecting to https://aus5.mozilla.org/api/v1/releases?names_only=True&product=Firefox&name_prefix=Firefox-mozilla-central-nightly-2
[task 2021-01-18T16:44:05.085Z] attempt 1/10
[task 2021-01-18T16:44:05.086Z] Starting new HTTPS connection (1): aus5.mozilla.org
[task 2021-01-18T16:44:09.214Z] Traceback (most recent call last):
[task 2021-01-18T16:44:09.214Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/packages/urllib3/connectionpool.py", line 376, in _make_request
[task 2021-01-18T16:44:09.215Z]     httplib_response = conn.getresponse(buffering=True)
[task 2021-01-18T16:44:09.215Z] TypeError: getresponse() got an unexpected keyword argument 'buffering'
[task 2021-01-18T16:44:09.215Z] 
[task 2021-01-18T16:44:09.215Z] During handling of the above exception, another exception occurred:
[task 2021-01-18T16:44:09.215Z] 
[task 2021-01-18T16:44:09.215Z] Traceback (most recent call last):
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/packages/urllib3/connectionpool.py", line 378, in _make_request
[task 2021-01-18T16:44:09.215Z]     httplib_response = conn.getresponse()
[task 2021-01-18T16:44:09.215Z]   File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse
[task 2021-01-18T16:44:09.215Z]     response.begin()
[task 2021-01-18T16:44:09.215Z]   File "/usr/lib/python3.5/http/client.py", line 297, in begin
[task 2021-01-18T16:44:09.215Z]     version, status, reason = self._read_status()
[task 2021-01-18T16:44:09.215Z]   File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
[task 2021-01-18T16:44:09.215Z]     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
[task 2021-01-18T16:44:09.215Z]   File "/usr/lib/python3.5/socket.py", line 575, in readinto
[task 2021-01-18T16:44:09.215Z]     return self._sock.recv_into(b)
[task 2021-01-18T16:44:09.215Z]   File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
[task 2021-01-18T16:44:09.215Z]     return self.read(nbytes, buffer)
[task 2021-01-18T16:44:09.215Z]   File "/usr/lib/python3.5/ssl.py", line 791, in read
[task 2021-01-18T16:44:09.215Z]     return self._sslobj.read(len, buffer)
[task 2021-01-18T16:44:09.215Z]   File "/usr/lib/python3.5/ssl.py", line 575, in read
[task 2021-01-18T16:44:09.215Z]     v = self._sslobj.read(len, buffer)
[task 2021-01-18T16:44:09.215Z] socket.timeout: The read operation timed out
[task 2021-01-18T16:44:09.215Z] 
[task 2021-01-18T16:44:09.215Z] During handling of the above exception, another exception occurred:
[task 2021-01-18T16:44:09.215Z] 
[task 2021-01-18T16:44:09.215Z] Traceback (most recent call last):
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/adapters.py", line 376, in send
[task 2021-01-18T16:44:09.215Z]     timeout=timeout
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/packages/urllib3/connectionpool.py", line 609, in urlopen
[task 2021-01-18T16:44:09.215Z]     _stacktrace=sys.exc_info()[2])
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/packages/urllib3/util/retry.py", line 247, in increment
[task 2021-01-18T16:44:09.215Z]     raise six.reraise(type(error), error, _stacktrace)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/packages/urllib3/packages/six.py", line 310, in reraise
[task 2021-01-18T16:44:09.215Z]     raise value
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
[task 2021-01-18T16:44:09.215Z]     body=body, headers=headers)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/packages/urllib3/connectionpool.py", line 380, in _make_request
[task 2021-01-18T16:44:09.215Z]     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/packages/urllib3/connectionpool.py", line 308, in _raise_timeout
[task 2021-01-18T16:44:09.215Z]     raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
[task 2021-01-18T16:44:09.215Z] requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='aus5.mozilla.org', port=443): Read timed out. (read timeout=4)
[task 2021-01-18T16:44:09.215Z] 
[task 2021-01-18T16:44:09.215Z] During handling of the above exception, another exception occurred:
[task 2021-01-18T16:44:09.215Z] 
[task 2021-01-18T16:44:09.215Z] Traceback (most recent call last):
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 324, in taskgraph_decision
[task 2021-01-18T16:44:09.215Z]     ret = taskgraph.decision.taskgraph_decision(options)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 222, in taskgraph_decision
[task 2021-01-18T16:44:09.215Z]     set_decision_indexes(decision_task_id, tgg.parameters, tgg.graph_config)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 155, in parameters
[task 2021-01-18T16:44:09.215Z]     return self._run_until("parameters")
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 417, in _run_until
[task 2021-01-18T16:44:09.215Z]     k, v = next(self._run)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 251, in _run
[task 2021-01-18T16:44:09.215Z]     parameters = self._parameters(graph_config)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 208, in <lambda>
[task 2021-01-18T16:44:09.215Z]     lambda graph_config: get_decision_parameters(graph_config, options)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 388, in get_decision_parameters
[task 2021-01-18T16:44:09.215Z]     parameters["release_history"] = populate_release_history("Firefox", project)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/partials.py", line 199, in populate_release_history
[task 2021-01-18T16:44:09.215Z]     product, branch, maxbuilds=maxbuilds, maxsearch=maxsearch
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/partials.py", line 232, in _populate_nightly_history
[task 2021-01-18T16:44:09.215Z]     last_releases = get_sorted_releases(product, branch)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/partials.py", line 153, in get_sorted_releases
[task 2021-01-18T16:44:09.215Z]     req = _retry_on_http_errors(url=url, verify=True, params=params, errors=[500])
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/partials.py", line 124, in _retry_on_http_errors
[task 2021-01-18T16:44:09.215Z]     req = requests.get(url, verify=verify, params=params, timeout=4)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/api.py", line 67, in get
[task 2021-01-18T16:44:09.215Z]     return request('get', url, params=params, **kwargs)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/api.py", line 53, in request
[task 2021-01-18T16:44:09.215Z]     return session.request(method=method, url=url, **kwargs)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/sessions.py", line 468, in request
[task 2021-01-18T16:44:09.215Z]     resp = self.send(prep, **send_kwargs)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/sessions.py", line 576, in send
[task 2021-01-18T16:44:09.215Z]     r = adapter.send(request, **kwargs)
[task 2021-01-18T16:44:09.215Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/adapters.py", line 449, in send
[task 2021-01-18T16:44:09.215Z]     raise ReadTimeout(e, request=request)

Comment 1

[Julien Cristau [:jcristau]]

The TypeError is a red herring, the actual issue seems to be that querying balrog took longer than the 4s timeout.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The 10am UTC desktop Nightly cron task failed for the same reason. Can this be fixed, or migitated by increasing the timeout threshold?

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Tested locally, takes 4.709 for my machine too. Bumping that timeout should fix for now, while has this changed recently. Either there's more data to retrieve there or something else is slowlying Balrog in serving the data. CC'ing Ben too.

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1687360 - bump partials Balrog info retrievel timeout. r=#releng

Comment 5

[[e-mail] (:bhearsum)]

I opened https://github.com/mozilla-releng/balrog/issues/1737 for improving the balrog side.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2de991b48f37
bump partials Balrog info retrievel timeout. r=releng-reviewers,jmaher

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Clearing the NI for now, patch is on autoland, hopefully this fixes things in the short term.
For longer term: https://github.com/mozilla-releng/balrog/issues/1737 is tracking it on the Balrog side while bug 1687520 tracks the apparent red herring in urllib.

Comment 8

[[e-mail] (:bhearsum)]

https://github.com/mozilla-releng/balrog/pull/1738 greatly improves the performance of the endpoint in question. Requests will take 1/4 to 1/2 the amount of time they currently do after it's deployed to prod tomorrow. This, along with the timeout bump, should keep us in a good state for quite awhile. We eventually need to address point #1 in https://github.com/mozilla-releng/balrog/issues/1737 to stop this from happening again, though.

Comment 9

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/2de991b48f37

Comment 10

[Intermittent Failures Robot]

2 failures in 3488 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• gecko-decision: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687360&startday=2021-01-18&endday=2021-01-24&tree=all