Closed Bug 1541807 Opened 5 years ago Closed 5 years ago

Perma beta ckbouncer Uncaught exception: Traceback (most recent call last): after Finished check-bouncer step (failed)

Categories

(Release Engineering :: Applications: MozharnessCore, defect, P5)

defect

Tracking

(firefox-esr60 unaffected, firefox66 unaffected, firefox67 fixed, firefox68 fixed)

RESOLVED FIXED
Tracking Status
firefox-esr60 --- unaffected
firefox66 --- unaffected
firefox67 --- fixed
firefox68 --- fixed

People

(Reporter: intermittent-bug-filer, Unassigned)

References

(Regression)

Details

(Keywords: regression)

#[markdown(off)]
Filed by: csabou [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=238063406&repo=mozilla-beta

https://queue.taskcluster.net/v1/task/BmDsYCIMS-GrlDDcy_VKHw/runs/0/artifacts/public/logs/live_backing.log

[vcs 2019-04-04T07:05:18.138Z] adding remote bookmark release
[vcs 2019-04-04T07:05:18.537Z] (setting sparse config to profile build/sparse-profiles/mozharness)
[vcs 2019-04-04T07:05:18.541Z] 0 files added, 0 files dropped, 0 files conflicting
[vcs 2019-04-04T07:05:18.541Z] (sparse refresh complete)
[vcs 2019-04-04T07:05:20.512Z] warning: sparse profile 'build/sparse-profiles/mozharness' not found in rev 000000000000 - ignoring it
[vcs 2019-04-04T07:05:21.978Z]
[vcs 2019-04-04T07:05:22.343Z] updating [ ] 100/6657
[vcs 2019-04-04T07:05:22.502Z]
[vcs 2019-04-04T07:05:22.502Z] 6657 files updated, 0 files merged, 0 files removed, 0 files unresolved
[vcs 2019-04-04T07:05:22.539Z] updated to 3e71adc4ab1dfcbabeb559613924b755391e08d1
[vcs 2019-04-04T07:05:22.542Z] PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["m4.4xlarge"], "lowerIsBetter": true, "name": "clone", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 98.9665880203247}, {"extraOptions": ["m4.4xlarge"], "lowerIsBetter": true, "name": "sparse_update_config", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 0.0037109851837158203}, {"extraOptions": ["m4.4xlarge"], "lowerIsBetter": true, "name": "update_sparse", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 3.997907876968384}, {"extraOptions": ["m4.4xlarge"], "lowerIsBetter": true, "name": "overall", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 103.90866494178772}, {"extraOptions": ["m4.4xlarge"], "lowerIsBetter": true, "name": "overall_clone", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 103.90866494178772}, {"extraOptions": ["m4.4xlarge"], "lowerIsBetter": true, "name": "overall_clone_fullcheckout", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 103.90866494178772}]}
[vcs 2019-04-04T07:05:22.686Z] TinderboxPrint:<a href=https://hg.mozilla.org/releases/mozilla-beta/rev/3e71adc4ab1dfcbabeb559613924b755391e08d1 title='Built from mozilla-beta revision 3e71adc4ab1dfcbabeb559613924b755391e08d1'>3e71adc4ab1dfcbabeb559613924b755391e08d1</a>
[task 2019-04-04T07:05:22.686Z] executing ['bash', '-cx', 'cd $GECKO_PATH && ./mach python testing/mozharness/scripts/release/bouncer_check.py --config releases/bouncer_firefox_beta.py --product-field=LATEST_FIREFOX_RELEASED_DEVEL_VERSION --products-url=https://product-details.mozilla.org/1.0/firefox_versions.json']
[task 2019-04-04T07:05:22.688Z] + cd /builds/worker/checkouts/gecko
[task 2019-04-04T07:05:22.688Z] + ./mach python testing/mozharness/scripts/release/bouncer_check.py --config releases/bouncer_firefox_beta.py --product-field=LATEST_FIREFOX_RELEASED_DEVEL_VERSION --products-url=https://product-details.mozilla.org/1.0/firefox_versions.json
[task 2019-04-04T07:05:23.200Z] New python executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
[task 2019-04-04T07:05:23.200Z] Also creating executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
[task 2019-04-04T07:05:24.607Z] Installing setuptools, pip, wheel...done.
[task 2019-04-04T07:05:25.593Z] running build_ext
[task 2019-04-04T07:05:25.593Z] building 'psutil._psutil_linux' extension
[task 2019-04-04T07:05:25.593Z] creating build
[task 2019-04-04T07:05:25.593Z] creating build/temp.linux-x86_64-2.7
[task 2019-04-04T07:05:25.593Z] creating build/temp.linux-x86_64-2.7/psutil
[task 2019-04-04T07:05:25.593Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2019-04-04T07:05:25.593Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2019-04-04T07:05:25.593Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
[task 2019-04-04T07:05:25.593Z] creating build/lib.linux-x86_64-2.7
[task 2019-04-04T07:05:25.593Z] creating build/lib.linux-x86_64-2.7/psutil
[task 2019-04-04T07:05:25.593Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so
[task 2019-04-04T07:05:25.593Z] building 'psutil._psutil_posix' extension
[task 2019-04-04T07:05:25.593Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2019-04-04T07:05:25.593Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2019-04-04T07:05:25.593Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2019-04-04T07:05:25.593Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2019-04-04T07:05:25.593Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2019-04-04T07:05:25.593Z]
[task 2019-04-04T07:05:25.596Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2019-04-04T07:05:25.695Z] 07:05:25 INFO - ConsoleLogger online at 20190404 07:05:25Z in /builds/worker/checkouts/gecko
[task 2019-04-04T07:05:25.695Z] 07:05:25 INFO - retry: Calling _urlopen with args: (), kwargs: {'url': 'https://product-details.mozilla.org/1.0/firefox_versions.json'}, attempt #1
[task 2019-04-04T07:05:25.807Z] 07:05:25 INFO - Set Firefox version 67.0b7
[task 2019-04-04T07:05:25.807Z] 07:05:25 INFO - Run as testing/mozharness/scripts/release/bouncer_check.py --config releases/bouncer_firefox_beta.py --product-field=LATEST_FIREFOX_RELEASED_DEVEL_VERSION --products-url=https://product-details.mozilla.org/1.0/firefox_versions.json
[task 2019-04-04T07:05:25.807Z] 07:05:25 INFO - Dumping config to /builds/worker/checkouts/gecko/logs/localconfig.json.
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - {'append_to_log': False,
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'base_work_dir': '/builds/worker/checkouts/gecko',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'cdn_urls': ('download-installer.cdn.mozilla.net',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'download.cdn.mozilla.net',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'download.mozilla.org',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'archive.mozilla.org'),
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'config_files': ('releases/bouncer_firefox_beta.py',),
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'locales': ('en-US', 'de', 'it', 'zh-TW'),
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'log_level': 'info',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'log_to_console': True,
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'opt_config_files': (),
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'parallelization': 20,
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'partials': {'releases-dir': {'check_uptake': True,
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'platforms': ('linux',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'linux64',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'osx',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'win',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'win64',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'win64-aarch64'),
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'product-name': 'Firefox-%(version)s-Partial-%(prev_version)s'}},
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'product_field': 'LATEST_FIREFOX_RELEASED_DEVEL_VERSION',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'products': {'complete-mar': {'check_uptake': True,
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'platforms': ('linux',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'linux64',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'osx',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'win',
[task 2019-04-04T07:05:25.810Z] 07:05:25 INFO - 'win64',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'win64-aarch64'),
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'product-name': 'Firefox-%(version)s-Complete'},
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'installer': {'check_uptake': True,
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'platforms': ('linux',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'linux64',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'osx',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'win',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'win64',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'win64-aarch64'),
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'product-name': 'Firefox-%(version)s'},
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'installer-ssl': {'check_uptake': True,
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'platforms': ('linux',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'linux64',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'osx',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'win',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'win64',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'win64-aarch64'),
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'product-name': 'Firefox-%(version)s-SSL'},
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'msi': {'check_uptake': True,
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'platforms': ('win', 'win64'),
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'product-name': 'Firefox-%(version)s-msi-SSL'},
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'stub-installer': {'check_uptake': True,
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'platforms': ('win',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'win64',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'win64-aarch64'),
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'product-name': 'Firefox-%(version)s-stub'}},
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'products_url': 'https://product-details.mozilla.org/1.0/firefox_versions.json',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'version': u'67.0b7',
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'volatile_config': {'actions': None, 'add_actions': None, 'no_actions': None},
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - 'work_dir': 'build'}
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - [mozharness: 2019-04-04 07:05:25.811720Z] Running check-bouncer step.
[task 2019-04-04T07:05:25.811Z] 07:05:25 INFO - Running main action method: check_bouncer
[task 2019-04-04T07:05:25.870Z] 07:05:25 INFO - [mozharness: 2019-04-04 07:05:25.870844Z] Finished check-bouncer step (failed)
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - Uncaught exception: Traceback (most recent call last):
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - File "/builds/worker/checkouts/gecko/testing/mozharness/mozharness/base/script.py", line 2089, in run
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - self.run_action(action)
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - File "/builds/worker/checkouts/gecko/testing/mozharness/mozharness/base/script.py", line 2028, in run_action
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - self._possibly_run_method(method_name, error_if_missing=True)
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - File "/builds/worker/checkouts/gecko/testing/mozharness/mozharness/base/script.py", line 1983, in _possibly_run_method
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - return getattr(self, method_name)()
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - File "testing/mozharness/scripts/release/bouncer_check.py", line 166, in check_bouncer
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - for url in self.get_urls():
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - File "testing/mozharness/scripts/release/bouncer_check.py", line 131, in get_urls
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - bouncer_prefix=self.config["bouncer_prefix"],
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - KeyError: 'bouncer_prefix'
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - Running post_fatal callback...
[task 2019-04-04T07:05:25.871Z] 07:05:25 FATAL - Exiting -1
[taskcluster 2019-04-04 07:05:26.242Z] === Task Finished ===
[taskcluster 2019-04-04 07:05:26.242Z] Unsuccessful task run with exit code: 255 completed in 126.449 seconds

Flags: needinfo?(mozilla)
Summary: Intermittent beta ckbouncer Uncaught exception: Traceback (most recent call last): after Finished check-bouncer step (failed) → Perma beta ckbouncer Uncaught exception: Traceback (most recent call last): after Finished check-bouncer step (failed)

This was caused by, and fixed in Bug 1533589.

Status: NEW → RESOLVED
Closed: 5 years ago
Flags: needinfo?(mozilla)
Regressed by: 1533589
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.