1687520 - investigate buffering traceback for requests

Status: NEW

(Release Engineering :: Release Automation, defect)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

In bug 1687360 comment 0 the following issue is described:

[task 2021-01-18T16:43:59.091Z] attempt 2/4
[task 2021-01-18T16:43:59.091Z] retry: calling find_decision_task with args: ({'project': 'mozilla-central', 'target_tasks_method': 'nightly_macosx', 'release_enable_partner_attribution': False, 'release_eta': '', 'release_type': 'nightly', 'head_repository': 'https://hg.mozilla.org/mozilla-central', 'optimize_strategies': None, 'build_date': 1610987637, 'moz_build_date': '20210118163357', 'pushdate': 1610987637, 'try_mode': None, 'filters': ['target_tasks_method'], 'existing_tasks': {}, 'tasks_for': 'cron', 'optimize_target_tasks': True, 'release_product': None, 'release_partners': [], 'hg_branch': 'default', 'message': '', 'try_options': None, 'owner': '[e-mail]', 'release_enable_partner_repack': False, 'test_manifest_loader': 'default', 'head_rev': '1926d7641900f10134d1591637fa1bc90815c2cf', 'try_task_config': {}, 'level': '3', 'version': '86.0a1', 'phabricator_diff': None, 'base_repository': 'https://hg.mozilla.org/mozilla-unified', 'release_enable_emefree': False, 'signoff_urls': {}, 'next_version': None, 'release_partner_build_number': 1, 'do_not_optimize': [], 'head_ref': '1926d7641900f10134d1591637fa1bc90815c2cf', 'pushlog_id': '38114', 'app_version': '86.0a1', 'build_number': 1, 'release_partner_config': {}, 'required_signoffs': []}, GraphConfig(_config={'trust-domain': 'gecko', 'scriptworker': {'scope-prefix': 'project:releng'}, 'project-repo-param-prefix': '', 'partner-urls': {'release-partner-repack': {'by-release-product': {'default': None, 'firefox': {'by-release-type': {'beta|release.*': {'by-release-level': {'production': '[e-mail]:mozilla-partners/repack-manifests.git', 'staging': '[e-mail]:moz-releng-automation-stage/repack-manifests.git'}}, 'default': None, 'esr.*': {'by-release-level': {'production': '[e-mail]:mozilla-partners/esr-repack-manifests.git', 'staging': '[e-mail]:moz-releng-automation-stage/esr-repack-manifests.git'}}}}}}, 'release-eme-free-repack': {'by-release-product': {'default': None, 'firefox': {'by-release-type': {'beta|release.*': {'by-release-level': {'production': '[e-mail]:mozilla-partners/mozilla-EME-free-manifest.git', 'staging': '[e-mail]:moz-releng-automation-stage/mozilla-EME-free-manifest.git'}}, 'default': None}}}}, 'release-partner-attribution': {'by-release-product': {'default': None, 'firefox': {'by-release-type': {'beta|release.*': {'by-release-level': {'production': '[e-mail]:mozilla-partners/repack-manifests.git', 'staging': '[e-mail]:moz-releng-automation-stage/repack-manifests.git'}}, 'default': None}}}}}, 'index': {'products': ['firefox', 'fennec', 'mobile', 'static-analysis', 'devedition', 'source', 'system-symbols', 'geckodriver']}, 'merge-automation': {'behaviors': {'bump-central': {'fetch-version-from': 'browser/config/version.txt', 'version-files': [{'new-suffix': 'a1', 'filename': 'config/milestone.txt', 'version-bump': 'major'}, {'new-suffix': 'a1', 'filename': 'browser/config/version.txt', 'version-bump': 'major'}, {'new-suffix': 'a1', 'filename': 'browser/config/version_display.txt', 'version-bump': 'major'}], 'replacements': [['xpcom/components/Module.h', 'static const unsigned int kVersion = {current_major_version};', 'static const unsigned int kVersion = {next_major_version};'], ['services/sync/modules/constants.js', 'WEAVE_VERSION: "1.{current_weave_version}.0"', 'WEAVE_VERSION: "1.{next_weave_version}.0"']], 'merge-old-head': False, 'end-tag': 'FIREFOX_NIGHTLY_{major_version}_END', 'to-repo': 'https://hg.mozilla.org/mozilla-central', 'to-branch': 'central'}, 'central-to-beta': {'to-branch': 'beta', 'version-files': [{'new-suffix': '', 'filename': 'config/milestone.txt'}, {'new-suffix': '', 'filename': 'browser/config/version.txt'}, {'new-suffix': 'b1', 'filename': 'browser/config/version_display.txt'}], 'from-branch': 'central', 'replacements': [['browser/config/mozconfigs/linux32/l10n-mozconfig', 'ac_add_options --with-branding=browser/branding/nightly', 'ac_add_options --enable-official-branding'], ['browser/config/mozconfigs/linux64/l10n-mozconfig', 'ac_add_options --with-branding=browser/branding/nightly', 'ac_add_options --enable-official-branding'], ['browser/config/mozconfigs/win32/l10n-mozconfig', 'ac_add_options --with-branding=browser/branding/nightly', 'ac_add_options --enable-official-branding'], ['browser/config/mozconfigs/win64/l10n-mozconfig', 'ac_add_options --with-branding=browser/branding/nightly', 'ac_add_options --enable-official-branding'], ['browser/config/mozconfigs/win64-aarch64/l10n-mozconfig', 'ac_add_options --with-branding=browser/branding/nightly', 'ac_add_options --enable-official-branding'], ['browser/config/mozconfigs/macosx64/l10n-mozconfig', 'ac_add_options --with-branding=browser/branding/nightly', 'ac_add_options --enable-official-branding'], ['build/mozconfig.common', 'MOZ_REQUIRE_SIGNING=${MOZ_REQUIRE_SIGNING-0}', 'MOZ_REQUIRE_SIGNING=${MOZ_REQUIRE_SIGNING-1}'], ['build/mozconfig.common', '# Disable enforcing that add-ons are signed by the trusted root', '# Enable enforcing that add-ons are signed by the trusted root']], 'end-tag': 'FIREFOX_BETA_{major_version}_END', 'to-repo': 'https://hg.mozilla.org/releases/mozilla-beta', 'fetch-version-from': 'browser/config/version.txt', 'from-repo': 'https://hg.mozilla.org/mozilla-central', 'merge-old-head': True, 'base-tag': 'FIREFOX_BETA_{major_version}_BASE'}, 'beta-to-release': {'to-branch': 'release', 'version-files': [{'new-suffix': '', 'filename': 'browser/config/version_display.txt'}], 'from-branch': 'beta', 'replacements': [], 'end-tag': 'FIREFOX_RELEASE_{major_version}_END', 'to-repo': 'https://hg.mozilla.org/releases/mozilla-release', 'fetch-version-from': 'browser/config/version.txt', 'from-repo': 'https://hg.mozilla.org/releases/mozilla-beta', 'merge-old-head': True, 'base-tag': 'FIREFOX_RELEASE_{major_version}_BASE'}, 'bump-esr': {'fetch-version-from': 'browser/config/version.txt', 'version-files': [{'filename': 'config/milestone.txt', 'version-bump': 'minor'}, {'filename': 'browser/config/version.txt', 'version-bump': 'minor'}, {'filename': 'browser/config/version_display.txt', 'version-bump': 'minor'}], 'replacements': [], 'merge-old-head': False, 'to-repo': 'https://hg.mozilla.org/releases/mozilla-esr78', 'to-branch': 'esr78'}, 'release-to-esr': {'to-branch': 'esr78', 'version-files': [{'new-suffix': 'esr', 'filename': 'browser/config/version_display.txt'}], 'from-branch': 'release', 'replacements': [['build/mozconfig.common', '# Enable enforcing that add-ons are signed by the trusted root', '# Disable enforcing that add-ons are signed by the trusted root'], ['build/mozconfig.common', 'MOZ_REQUIRE_SIGNING=${MOZ_REQUIRE_SIGNING-1}', 'MOZ_REQUIRE_SIGNING=${MOZ_REQUIRE_SIGNING-0}']], 'to-repo': 'https://hg.mozilla.org/releases/mozilla-esr78', 'fetch-version-from': 'browser/config/version.txt', 'from-repo': 'https://hg.mozilla.org/releases/mozilla-release', 'merge-old-head': False, 'base-tag': 'FIREFOX_ESR_{major_version}_BASE'}}}, 'release-promotion': {'flavors': {'ship_firefox_rc': {'product': 'firefox', 'is-rc': True, 'partial-updates': True, 'target-tasks-method': 'ship_desktop'}, 'promote_firefox_partner_attribution': {'product': 'firefox', 'rebuild-kinds': ['release-partner-attribution', 'release-partner-attribution-beetmover'], 'target-tasks-method': 'promote_desktop'}, 'promote_firefox_partner_repack': {'product': 'firefox', 'rebuild-kinds': ['release-partner-repack', 'release-partner-repack-chunking-dummy', 'release-partner-repack-signing', 'release-partner-repack-notarization-part-1', 'release-partner-repack-notarization-poller', 'release-partner-repack-repackage', 'release-partner-repack-repackage-signing', 'release-partner-repack-beetmover'], 'target-tasks-method': 'promote_desktop'}, 'promote_firefox': {'product': 'firefox', 'partial-updates': True, 'target-tasks-method': 'promote_desktop'}, 'push_devedition': {'product': 'devedition', 'partial-updates': True, 'target-tasks-method': 'push_desktop'}, 'promote_firefox_rc': {'product': 'firefox', 'is-rc': True, 'partial-updates': True, 'target-tasks-method': 'promote_desktop'}, 'ship_firefox': {'product': 'firefox', 'version-bump': True, 'partial-updates': True, 'target-tasks-method': 'ship_desktop'}, 'push_firefox': {'product': 'firefox', 'partial-updates': True, 'target-tasks-method': 'push_desktop'}, 'promote_devedition': {'product': 'devedition', 'partial-updates': True, 'target-tasks-method': 'promote_desktop'}, 'ship_devedition': {'product': 'devedition', 'version-bump': True, 'partial-updates': True, 'target-tasks-method': 'ship_desktop'}}, 'products': ['devedition', 'fennec', 'firefox']}, 'task-priority': {'by-project': {'default': 'very-low', 'mozilla-central': 'medium', 'mozilla-release': 'highest', 'mozilla-beta': 'high', 'autoland': 'low', 'mozilla-esr.*': 'very-high', 'mozilla-inbound': 'low'}}, 'product-dir': 'browser', 'workers': {'aliases': {'mac-signing': {'provisioner': 'scriptworker-prov-v1', 'os': 'macosx', 'implementation': 'scriptworker-signing', 'worker-type': {'by-release-level': {'production': 'signing-mac-v1', 'staging': 'depsigning-mac-v1'}}}, 'linux-signing': {'provisioner': 'scriptworker-k8s', 'os': 'linux', 'implementation': 'scriptworker-signing', 'worker-type': {'by-release-level': {'production': '{trust-domain}-3-signing', 'staging': '{trust-domain}-t-signing'}}}, 'b-linux.*': {'provisioner': '{trust-domain}-{level}', 'os': 'linux', 'implementation': 'docker-worker', 'worker-type': '{alias}'}, 't-linux-xlarge-pgo': {'provisioner': {'by-level': {'default': '{trust-domain}-t', '3': '{trust-domain}-{level}'}}, 'os': 'linux', 'implementation': 'docker-worker', 'worker-type': 't-linux-xlarge'}, 't-linux(-large|-xlarge|-xlarge-source|-metal)': {'provisioner': '{trust-domain}-t', 'os': 'linux', 'implementation': 'docker-worker', 'worker-type': '{alias}'}, 'mac-depsigning': {'provisioner': 'scriptworker-prov-v1', 'os': 'macosx', 'implementation': 'scriptworker-signing', 'worker-type': 'depsigning-mac-v1'}, 'linux-depsigning': {'provisioner': 'scriptworker-k8s', 'os': 'linux', 'implementation': 'scriptworker-signing', 'worker-type': '{trust-domain}-t-signing'}, 'balrog': {'provisioner': 'scriptworker-k8s', 'os': 'scriptworker', 'implementation': 'balrog', 'worker-type': {'by-release-level': {'production': '{trust-domain}-3-balrog', 'staging': '{trust-domain}-1-balrog'}}}, 't-osx-1014': {'provisioner': 'releng-hardware', 'os': 'macosx', 'implementation': 'generic-worker', 'worker-type': 'gecko-{alias}'}, 'succeed': {'provisioner': 'built-in', 'os': 'none', 'implementation': 'succeed', 'worker-type': 'succeed'}, 't-win10-64(-hw|-ref-hw|-1803-hw)': {'provisioner': 'releng-hardware', 'os': 'windows', 'implementation': 'generic-worker', 'worker-type': 'gecko-{alias}'}, 't-bitbar-gw.*': {'provisioner': 'proj-autophone', 'os': 'linux-bitbar', 'implementation': 'generic-worker', 'worker-type': 'gecko-{alias}'}, 'beetmover': {'provisioner': 'scriptworker-k8s', 'os': 'scriptworker', 'implementation': 'beetmover', 'worker-type': {'by-release-level': {'production': '{trust-domain}-3-beetmover', 'staging': '{trust-domain}-1-beetmover'}}}, 'images': {'provisioner': '{trust-domain}-{level}', 'os': 'linux', 'implementation': 'docker-worker', 'worker-type': '{alias}'}, 'tree': {'provisioner': 'scriptworker-k8s', 'os': 'scriptworker', 'implementation': 'treescript', 'worker-type': {'by-release-level': {'production': '{trust-domain}-3-tree', 'staging': '{trust-domain}-1-tree'}}}, 't-linux-talos': {'provisioner': 'releng-hardware', 'os': 'linux', 'implementation': 'generic-worker', 'worker-type': 'gecko-{alias}'}, 't-win10-64(|-gpu-s|-source)': {'provisioner': '{trust-domain}-t', 'os': 'windows', 'implementation': 'generic-worker', 'worker-type': '{alias}'}, 't-win7-32(|-gpu)': {'provisioner': '{trust-domain}-t', 'os': 'windows', 'implementation': 'generic-worker', 'worker-type': '{alias}'}, 'bouncer': {'provisioner': 'scriptworker-k8s', 'os': 'scriptworker', 'implementation': 'bouncer', 'worker-type': {'by-release-level': {'production': '{trust-domain}-3-bouncer', 'staging': '{trust-domain}-1-bouncer'}}}, 'mac-notarization-poller': {'provisioner': 'scriptworker-prov-v1', 'os': 'macosx', 'implementation': 'notarization-poller', 'worker-type': 'mac-notarization-poller'}, 'tree-dev': {'provisioner': 'scriptworker-k8s', 'os': 'scriptworker', 'implementation': 'treescript', 'worker-type': '{trust-domain}-1-tree-dev'}, 't-osx-1014-power': {'provisioner': 'releng-hardware', 'os': 'macosx', 'implementation': 'generic-worker', 'worker-type': 'gecko-{alias}'}, 'misc': {'provisioner': '{trust-domain}-t', 'os': 'linux', 'implementation': 'docker-worker', 'worker-type': 'misc'}, 'b-osx-1015': {'provisioner': 'releng-hardware', 'os': 'macosx', 'implementation': 'generic-worker', 'worker-type': {'by-level': {'default': 'gecko-1-b-osx-1015', '3': 'gecko-3-b-osx-1015'}}}, 'b-win2012': {'provisioner': '{trust-domain}-{level}', 'os': 'windows', 'implementation': 'generic-worker', 'worker-type': '{alias}'}, 't-win64-aarch64-laptop': {'provisioner': 'bitbar', 'os': 'windows', 'implementation': 'generic-worker', 'worker-type': 'gecko-{alias}'}}}, 'taskgraph': {}, 'mac-notarization': {'mac-entitlements': {'by-platform': {'macosx64.*': {'by-release-level': {'default': 'security/mac/hardenedruntime/developer.entitlements.xml', 'production': 'security/mac/hardenedruntime/production.entitlements.xml'}}, 'default': ''}}, 'mac-behavior': {'by-project': {'default': 'mac_sign_and_pkg', 'mozilla-(central|beta|release|esr.*)': {'by-shippable': {'default': 'mac_sign_and_pkg', 'true': 'mac_notarize'}}}}}, 'try': {'ridealong-builds': {'linux': ['sm-arm-sim-linux32'], 'win32': ['sm-plain-win32', 'sm-compacting-win32'], 'linux64': ['sm-plain-linux64', 'sm-nojit-linux64', 'sm-nonunified-linux64', 'sm-arm-sim-linux32', 'sm-arm64-sim-linux64', 'sm-compacting-linux64', 'sm-rootanalysis-linux64', 'sm-package-linux64', 'sm-tsan-linux64', 'sm-asan-linux64', 'sm-mozjs-sys-linux64', 'sm-msan-linux64', 'sm-fuzzing-linux64', 'sm-rust-bindings-linux64']}}, 'treeherder': {'group-names': {'Btime-live-fenix-wr': 'Browsertime performance tests on Firefox with webrender enabled', 'ms-stage': 'Autograph-stage MAR signing test', 'Btime-Prof-wr': 'Browsertime performance tests on Firefox with Gecko Profiling, webrender enabled', 'X': 'Xpcshell tests', 'Aries': 'Aries Device Image', 'Btime-ChR': 'Browsertime performance tests on Google Chrome Release', 'Rap-fennec': 'Raptor performance tests on Fennec', 'l10n-bump': 'L10n Bumper', 'Flatpak': 'Flatpak image generation', 'VP': 'VideoPuppeteer tests', 'M-V-1proc': 'Mochitests on Valgrind without e10s', 'Rap': 'Raptor performance tests on Firefox', 'js': 'JavaScript checks', 'Btime-live': 'Browsertime performance tests on Firefox', 'cs': 'Checksum signing', 'Btime-fennec': 'Browsertime performance tests on Fennec', 'Rel': 'Release promotion', 'M-gli': 'Mochitests with WebGL over IPC', 'W-1proc': 'Web platform tests without e10s', 'Rap-fennec-1proc': 'Raptor performance tests on Fennec without e10s', 'Btime-live-fenix': 'Browsertime performance tests on Firefox', 'T-Prof-1proc': 'Talos performance tests on Firefox with Gecko Profiling and without e10s', 'SM': 'Spidermonkey builds', 'Btime-Cr': 'Browsertime performance tests on Google Chromium', 'Rap-live': 'Raptor performance tests on Firefox with live sites', 'Btime-Prof-fennec': 'Browsertime performance tests on Fennec with Gecko Profiling', 'BM': 'Beetmover', 'Btime-wr-1proc': 'Browsertime performance tests on Firefox without e10s, with webrender enabled', 'remote': 'Remote debugging protocol tests', 'pydep': 'python dependency update', 'W-fis': 'Web platform tests with fission enabled', 'M': 'Mochitests', 'Searchfox': 'Searchfox builds', 'Btime': 'Browsertime performance tests on Firefox', 'WR': 'WebRender standalone', 'vismet': 'Visual Metrics Analysis', 'T-Prof-fis': 'Talos performance tests on Firefox with fission and Gecko Profiling', 'W-b': 'Web platform tests (backlog)', 'TL': 'Toolchain builds for Linux 64-bits', 'R': 'Reftests', 'TL32': 'Toolchain builds for Linux 32-bits', 'Rap-wr': 'Raptor performance tests on Firefox', 'W-b-fis': 'Web platform tests (backlog) with fission enabled', 'M-fis-xorig': 'Mochitests with cross-origin and fission enabled', 'Btime-P-wr': 'Browsertime power tests on Firefox/Geckoview with webrender enabled', 'T-1proc': 'Talos performance tests without e10s', 'SY': 'Are we slim yet tests by TaskCluster', 'pub': 'APK publishing', 'TW64': 'Toolchain builds for Windows 64-bits', 'Rap-Prof-1proc': 'Raptor performance tests on Firefox with Gecko Profiling and without e10s', 'misc': 'Misc checks', 'T-swr': 'Talos performance tests with software webrender enabled', 'M-spi-1proc': 'Mochitests with socket process without e10s', 'T-gli': 'Talos performance tests with WebGL over IPC', 'Static-Analysis': 'Full tree static-analysis', 'perftest': 'Performance tests', 'T-Prof-gli': 'Talos performance tests with WebGL over IPC and Gecko Profiling', 'Deb9': 'Packages for Debian 9', 'Btime-fenix': 'Browsertime performance tests on Fenix', 'M-1proc': 'Mochitests without e10s', 'L10n': 'Localised Repacks', 'Btime-Prof': 'Browsertime performance tests on Firefox with Gecko Profiling', 'condprof': 'Conditioned Profile Builder', 'perftest-http3': 'Performance tests with HTTP/3', 'fuzzing': 'Fuzzing checks', 'rs': 'Repackage signing', 'TM': 'Toolchain builds for OSX', 'js-bench-v8': 'JavaScript shell benchmarks with Google V8', 'TPS': 'Sync tests', 'Btime-Prof-fennec-1proc': 'Browsertime performance tests on Fennec without e10s with Gecko Profiling', 'W': 'Web platform tests', 'TMW': 'Toolchain builds for Windows MinGW', 'Btime-live-wr': 'Browsertime performance tests on Android with webrender enabled', 'GhS': 'GitHub Synchronization', 'BMcs': 'Beetmover checksums', 'MSIs': 'Signing of Repacked installers of MSIs', 'Fxfn-r': 'Firefox functional tests (remote)', 'WMC32': 'MinGW-Clang builds for Windows 32-bits', 'Btime-fis': 'Browsertime performance tests on Firefox with fission enabled', 'Rap-Prof': 'Raptor performance tests on Firefox with Gecko Profiling', 'MSI': 'Repack installers into MSIs', 'py2': 'Python 2 unit tests', 'Nexus 5-L': 'Nexus 5-L Device Image', 'M-spi-nw-1proc': 'Mochitests with networking on socket process without e10s', 'pedantic': 'pedantic checks', 'WMC64': 'MinGW-Clang builds for Windows 64-bits', 'doc': 'Documentation', 'M-spi-nw': 'Mochitests with networking on socket process', 'node': 'Node tests', 'Btime-P-refbrow-wr': 'Browsertime power tests on reference browser with webrender enabled', 'Btime-Prof-1proc': 'Browsertime performance tests on Firefox with Gecko Profiling and without e10s', 'py': 'Python checks', 'rust': 'Rust checks', 'Btime-wr': 'Browsertime performance tests on Android with webrender enabled', 'cram': 'Cram tests', 'T': 'Talos performance tests', 'p': 'Partial generation', 'Fxfn-l': 'Firefox functional tests (local)', 'Deb8-32': 'Packages for Debian 8 32-bits', 'js-bench-sm': 'JavaScript shell benchmarks with Spidermonkey', 'M-spi': 'Mochitests with socket process', 'Btime-P-fenix-wr': 'Browsertime power tests on Fenix with webrender enabled', 'Btime-fennec-1proc': 'Browsertime performance tests on Fennec without e10s', 'Btime-Prof-refbrow': 'Browsertime performance tests on the reference browser with Gecko Profiling', 'cpp': 'C/C++ checks', 'SS': 'Shadow scheduler', 'Btime-P-1proc-wr': 'Browsertime power tests on Fennec with webrender enabled', 'SY-fis': 'Are we slim yet tests by TaskCluster, fission enabled', 'Rap-1proc': 'Raptor performance tests on Firefox without e10s', 'BMcslang': 'Beetmover checksums for language packs', 'Rap-refbrow': 'Raptor performance tests on the reference browser', 'A': 'Android Gradle tests', 'test-info': 'Test manifest skip/fail information', 'Btime-refbrow': 'Browsertime performance tests on the reference browser', 'iris': 'Iris testing suite', 'css': 'Checksum signing for source', 'R-fis': 'Reftests with fission enabled', 'Btime-1proc': 'Browsertime performance tests on Firefox without e10s', 'ms': 'Complete MAR signing', 'Rap-refbrow-wr': 'Raptor performance tests on the reference browser', 'Gd': 'Geckodriver', 'TW32': 'Toolchain builds for Windows 32-bits', 'Snap': 'Snap image generation', 'Tss-fis': 'Talos performance tests, Stylo sequential with fission enabled', 'Deb8': 'Packages for Debian 8', 'BMR': 'Beetmover repackages', 'Rap-Cr': 'Raptor performance tests on Google Chromium', 'py3': 'Python 3 unit tests', 'Rap-fenix': 'Raptor performance tests on Fenix', 'Rap-fis': 'Raptor performance tests on Firefox with fission enabled', 'ps': 'Partials signing', 'T-Prof': 'Talos performance tests on Firefox with Gecko Profiling', 'java': 'Java checks', 'Btime-fenix-wr': 'Browsertime performance tests on Fenix with webrender enabled', 'M-fis': 'Mochitests with fission enabled', 'R-1proc': 'Reftests without e10s', 'Rap-P-1proc': 'Raptor power tests on Firefox without e10s', 'Wgpu': 'WebGPU standalone', 'Tss': 'Talos performance tests, Stylo sequential', 'Btime-Prof-fenix': 'Browsertime performance tests on Fenix with Gecko Profiling', 'c-Up': 'Balrog submission of complete updates', 'L10n-Rpk': 'Localized Repackaged Repacks', 'tt': 'Telemetry tests', 'tt-1proc': 'Telemetry tests without e10s', 'R-swr': 'Reftests with software webrender enabled', 'Rap-ChR': 'Raptor performance tests on Google Chrome Release', 'Deb10': 'Packages for Debian 10', 'Btime-P-refbrow': 'Browsertime power tests on reference browser', 'Btime-live-ChR': 'Browsertime performance tests on Firefox', 'Btime-P-fenix': 'Browsertime power tests on Fenix', 'Bpgo': 'Profile-guided optimization builds', 'Btime-refbrow-wr': 'Browsertime performance tests on the reference browser with webrender enabled', 'text': 'Check on texts', 'X-fis': 'Xpcshell tests with fission enabled', 'Tss-1proc': 'Talos performance tests with Stylo sequential without e10s', 'BMcss': 'Beetmover checksums for source', 'X-spi-nw': 'Xpcshell tests with networking on socket process', 'UV': 'Update verify', 'Fxfn-l-1proc': 'Firefox functional tests (local) without e10s', 'X-1proc': 'Xpcshell tests, without e10s', 'Fxfn-r-1proc': 'Firefox functional tests (remote) without e10s', 'langpack': 'Langpack sigatures and uploads', 'Rap-P': 'Raptor power tests on Firefox', 'Fetch': 'Fetch and store content', 'Btime-P': 'Browsertime power tests on Firefox/Geckoview', 'Btime-P-1proc': 'Browsertime power tests on Fennec', 'T-fis': 'Talos performance tests with fission enabled', 'I': 'Docker Image Builds'}}}, root_dir='taskcluster/ci')), kwargs: {}, attempt #2
[task 2021-01-18T16:43:59.093Z] Resetting dropped connection: firefox-ci-tc.services.mozilla.com
[task 2021-01-18T16:43:59.137Z] "GET /api/index/v1/task/gecko.v2.mozilla-central.revision.1926d7641900f10134d1591637fa1bc90815c2cf.taskgraph.decision HTTP/1.1" 200 220
[task 2021-01-18T16:43:59.175Z] "GET /api/queue/v1/task/IGNGPEwFTOKAb-EZWdVNtg/artifacts/public/full-task-graph.json HTTP/1.1" 303 122
[task 2021-01-18T16:43:59.177Z] Starting new HTTPS connection (1): firefoxci.taskcluster-artifacts.net
[task 2021-01-18T16:43:59.383Z] "GET /IGNGPEwFTOKAb-EZWdVNtg/0/public/full-task-graph.json HTTP/1.1" 200 147689354
[task 2021-01-18T16:44:04.570Z] "GET /api/queue/v1/task/IGNGPEwFTOKAb-EZWdVNtg/artifacts/public/label-to-taskid.json HTTP/1.1" 303 122
[task 2021-01-18T16:44:04.722Z] "GET /IGNGPEwFTOKAb-EZWdVNtg/0/public/label-to-taskid.json HTTP/1.1" 200 292813
[task 2021-01-18T16:44:05.085Z] Connecting to https://aus5.mozilla.org/api/v1/releases?names_only=True&product=Firefox&name_prefix=Firefox-mozilla-central-nightly-2
[task 2021-01-18T16:44:05.085Z] attempt 1/10
[task 2021-01-18T16:44:05.086Z] Starting new HTTPS connection (1): aus5.mozilla.org
[task 2021-01-18T16:44:09.214Z] Traceback (most recent call last):
[task 2021-01-18T16:44:09.214Z]   File "/builds/worker/checkouts/gecko/third_party/python/requests/requests/packages/urllib3/connectionpool.py", line 376, in _make_request
[task 2021-01-18T16:44:09.215Z]     httplib_response = conn.getresponse(buffering=True)
[task 2021-01-18T16:44:09.215Z] TypeError: getresponse() got an unexpected keyword argument 'buffering'

While the chasing errors was somewhere down the line in a timeout to access Balrog, this error in the traceback was initially considered a red herring. However, that's indicative of a real underlying issue that we should look into at some point.

This bug is to track that.

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

More potential context via Aryx and jcristau from Matrix:

• https://stackoverflow.com/questions/33792582/typeerror-getresponse-got-an-unexpected-keyword-argument-buffering/38579824 and
• https://github.com/urllib3/urllib3/commit/4c65505f1dd420a15eb7bbcd1fc84f4cee701234