1663449 - Intermittent raptor-perftest Critical: Could not get the conditioned profile | raptor-main Critical: TEST-UNEXPECTED-FAIL: could not initialize the raptor test runner

Reporter

Description

•

4 years ago

treeherder

Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=314999481&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Yi-_WmALTsqK63eMltcMnw/runs/0/artifacts/public/logs/live_backing.log

[task 2020-09-07T13:55:59.012Z] 13:55:59     INFO -  raptor-main Info: raptor-start
[task 2020-09-07T13:55:59.013Z] 13:55:59     INFO -  raptor-main Info: received command line arguments: Namespace(activity=None, app='firefox', binary='/home/cltbld/tasks/task_1599486904/build/application/firefox/firefox', browser_cycles=None, browsertime=False, browsertime_browsertimejs=None, browsertime_chromedriver=None, browsertime_ffmpeg=None, browsertime_geckodriver=None, browsertime_no_ffwindowrecorder=False, browsertime_node=None, browsertime_video=False, chimera=False, cold=True, conditioned_profile_scenario='settled', cpu_test=False, debug_mode=False, device_name=None, disable_perf_tuning=False, e10s=True, enable_fission=False, enable_webrender=False, extra_prefs={'extensions.logging.enabled': True}, gecko_profile=False, gecko_profile_entries=None, gecko_profile_interval=None, gecko_profile_threads=None, host='127.0.0.1', installerpath=None, intent=None, is_release_build=False, live_sites=False, log_errorsummary=None, log_grouped=None, log_html=None, log_mach=None, log_mach_buffer=None, log_mach_level=None, log_mach_screenshot=None, log_mach_verbose=None, log_raw=None, log_raw_level=None, log_tbpl=None, log_tbpl_buffer=None, log_tbpl_compact=None, log_tbpl_level='debug', log_unittest=None, log_xunit=None, memory_test=False, no_conditioned_profile=False, noinstall=False, obj_path=None, page_cycles=None, page_timeout=None, post_startup_delay=30000, power_test=False, project='mozilla-central', run_local=False, symbols_path='https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Wk7ZuBzRQj-pcZngAQ3qVg/artifacts/public/build/target.crashreporter-symbols.zip', test='raptor-tp6-reddit-firefox', test_url_params=None, verbose=False)
[task 2020-09-07T13:55:59.013Z] 13:55:59     INFO -  raptor-manifest Info: /home/cltbld/tasks/task_1599486904/build/tests/raptor/raptor/raptor.ini
[task 2020-09-07T13:55:59.029Z] 13:55:59     INFO -  raptor-manifest Info: configuring settings for test raptor-tp6-reddit-firefox
[task 2020-09-07T13:55:59.029Z] 13:55:59     INFO -  raptor-main Info: raptor tests scheduled to run:
[task 2020-09-07T13:55:59.029Z] 13:55:59     INFO -  raptor-main Info: raptor-tp6-reddit-firefox-cold
[task 2020-09-07T13:55:59.029Z] 13:55:59     INFO -  raptor-perftest Info: Using a conditioned profile.
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion application_buildid: 20200907094801
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion application_changeset: 493274bb362f2c701849bc7cc7425bcba53f35be
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion application_display_name: Firefox Nightly
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion application_name: Firefox
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion application_remotingname: firefox
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion application_repository: https://hg.mozilla.org/integration/autoland
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion application_vendor: Mozilla
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion application_version: 82.0a1
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion platform_buildid: 20200907094801
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion platform_changeset: 493274bb362f2c701849bc7cc7425bcba53f35be
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion platform_repository: https://hg.mozilla.org/integration/autoland
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion platform_version: 82.0a1
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  raptor-perftest Info: Browser name: Firefox
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  raptor-perftest Info: Browser version: 82.0a1
[task 2020-09-07T13:55:59.030Z] 13:55:59     INFO -  mozversion application_buildid: 20200907094801
[task 2020-09-07T13:55:59.031Z] 13:55:59     INFO -  mozversion application_changeset: 493274bb362f2c701849bc7cc7425bcba53f35be
[task 2020-09-07T13:55:59.031Z] 13:55:59     INFO -  mozversion application_display_name: Firefox Nightly
[task 2020-09-07T13:55:59.031Z] 13:55:59     INFO -  mozversion application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
[task 2020-09-07T13:55:59.031Z] 13:55:59     INFO -  mozversion application_name: Firefox
[task 2020-09-07T13:55:59.031Z] 13:55:59     INFO -  mozversion application_remotingname: firefox
[task 2020-09-07T13:55:59.031Z] 13:55:59     INFO -  mozversion application_repository: https://hg.mozilla.org/integration/autoland
[task 2020-09-07T13:55:59.031Z] 13:55:59     INFO -  mozversion application_vendor: Mozilla
[task 2020-09-07T13:55:59.031Z] 13:55:59     INFO -  mozversion application_version: 82.0a1
[task 2020-09-07T13:55:59.031Z] 13:55:59     INFO -  mozversion platform_buildid: 20200907094801
[task 2020-09-07T13:55:59.032Z] 13:55:59     INFO -  mozversion platform_changeset: 493274bb362f2c701849bc7cc7425bcba53f35be
[task 2020-09-07T13:55:59.032Z] 13:55:59     INFO -  mozversion platform_repository: https://hg.mozilla.org/integration/autoland
[task 2020-09-07T13:55:59.032Z] 13:55:59     INFO -  mozversion platform_version: 82.0a1
[task 2020-09-07T13:55:59.032Z] 13:55:59     INFO -  raptor-perftest Info: Browser name: Firefox
[task 2020-09-07T13:55:59.032Z] 13:55:59     INFO -  raptor-perftest Info: Browser version: 82.0a1
[task 2020-09-07T13:55:59.032Z] 13:55:59     INFO -  raptor-perftest Info: Post startup delay set to 1000 ms
[task 2020-09-07T13:55:59.032Z] 13:55:59     INFO -  raptor-perftest Info: main raptor init, config is: {'verbose': False, 'e10s': True, 'app': 'firefox', 'symbols_path': 'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Wk7ZuBzRQj-pcZngAQ3qVg/artifacts/public/build/target.crashreporter-symbols.zip', 'power_test': False, 'run_local': False, 'chimera': False, 'cold': True, 'gecko_profile_interval': None, 'binary': '/home/cltbld/tasks/task_1599486904/build/application/firefox/firefox', 'enable_control_server_wait': False, 'no_conditioned_profile': False, 'platform': 'linux', 'conditioned_profile_scenario': 'settled', 'extra_prefs': {'extensions.logging.enabled': True}, 'gecko_profile': False, 'gecko_profile_entries': None, 'disable_perf_tuning': False, 'host': '127.0.0.1', 'enable_fission': False, 'memory_test': False, 'enable_webrender': False, 'obj_path': None, 'live_sites': False, 'device_name': None, 'project': 'mozilla-central', 'cpu_test': False, 'is_release_build': False, 'using_condprof': True, 'processor': 'x86_64'}
[task 2020-09-07T13:55:59.032Z] 13:55:59     INFO -  raptor-perftest Info: Making temp_download_dir from inside get_conditioned_profile /tmp/tmpwTqjmX
[task 2020-09-07T13:55:59.032Z] 13:55:59     INFO -  raptor-perftest Info: Platform used: linux64
[task 2020-09-07T13:55:59.032Z] 13:55:59     INFO -  raptor-perftest Info: Getting profile from project mozilla-central
[task 2020-09-07T13:55:59.032Z] 13:55:59     INFO -  condprof INFO | Getting conditioned profile with arguments: {'repo': 'mozilla-central', 'platform': 'linux64', 'task_id': None, 'customization': 'default', 'scenario': 'settled'}
[task 2020-09-07T13:55:59.032Z] 13:55:59     INFO -  condprof INFO | Getting headers at https://firefox-ci-tc.services.mozilla.com
[task 2020-09-07T13:56:39.264Z] 13:56:39     INFO -  condprof INFO | Failed, retrying
[task 2020-09-07T13:57:24.310Z] 13:57:24     INFO -  condprof INFO | Getting headers at https://firefox-ci-tc.services.mozilla.com
[task 2020-09-07T13:57:59.504Z] 13:57:59     INFO -  condprof INFO | Failed, retrying
[task 2020-09-07T13:59:07.041Z] 13:59:07     INFO -  condprof INFO | Getting headers at https://firefox-ci-tc.services.mozilla.com
[task 2020-09-07T13:59:42.253Z] 13:59:42     INFO -  condprof INFO | Failed, retrying
[task 2020-09-07T14:01:23.601Z] 14:01:23     INFO -  condprof ERROR | All attempt failed
[task 2020-09-07T14:01:23.601Z] 14:01:23 CRITICAL -  raptor-perftest Critical: Could not get the conditioned profile
[task 2020-09-07T14:01:23.601Z] 14:01:23    ERROR -  Traceback (most recent call last):
[task 2020-09-07T14:01:23.601Z] 14:01:23     INFO -    File "/home/cltbld/tasks/task_1599486904/build/tests/raptor/raptor/perftest.py", line 278, in get_conditioned_profile
[task 2020-09-07T14:01:23.602Z] 14:01:23     INFO -      repo=repo
[task 2020-09-07T14:01:23.602Z] 14:01:23     INFO -    File "/home/cltbld/tasks/task_1599486904/build/venv/lib/python2.7/site-packages/condprof/client.py", line 151, in get_profile
[task 2020-09-07T14:01:23.602Z] 14:01:23     INFO -      _check_service(TC_SERVICE)
[task 2020-09-07T14:01:23.602Z] 14:01:23     INFO -    File "/home/cltbld/tasks/task_1599486904/build/venv/lib/python2.7/site-packages/condprof/client.py", line 69, in _check_service
[task 2020-09-07T14:01:23.603Z] 14:01:23     INFO -      raise ServiceUnreachableError(url)
[task 2020-09-07T14:01:23.603Z] 14:01:23     INFO -  ServiceUnreachableError: https://firefox-ci-tc.services.mozilla.com
[task 2020-09-07T14:01:23.603Z] 14:01:23    ERROR -  Traceback (most recent call last):
[task 2020-09-07T14:01:23.604Z] 14:01:23     INFO -    File "/home/cltbld/tasks/task_1599486904/build/tests/raptor/raptor/raptor.py", line 140, in main
[task 2020-09-07T14:01:23.604Z] 14:01:23     INFO -      verbose=args.verbose
[task 2020-09-07T14:01:23.604Z] 14:01:23     INFO -    File "/home/cltbld/tasks/task_1599486904/build/tests/raptor/raptor/webextension/desktop.py", line 25, in __init__
[task 2020-09-07T14:01:23.604Z] 14:01:23     INFO -      super(WebExtensionDesktop, self).__init__(*args, **kwargs)
[task 2020-09-07T14:01:23.605Z] 14:01:23     INFO -    File "/home/cltbld/tasks/task_1599486904/build/tests/raptor/raptor/webextension/base.py", line 37, in __init__
[task 2020-09-07T14:01:23.605Z] 14:01:23     INFO -      super(WebExtension, self).__init__(*args, **kwargs)
[task 2020-09-07T14:01:23.605Z] 14:01:23     INFO -    File "/home/cltbld/tasks/task_1599486904/build/tests/raptor/raptor/perftest.py", line 210, in __init__
[task 2020-09-07T14:01:23.605Z] 14:01:23     INFO -      self.build_browser_profile()
[task 2020-09-07T14:01:23.605Z] 14:01:23     INFO -    File "/home/cltbld/tasks/task_1599486904/build/tests/raptor/raptor/webextension/base.py", line 168, in build_browser_profile
[task 2020-09-07T14:01:23.606Z] 14:01:23     INFO -      super(WebExtension, self).build_browser_profile()
[task 2020-09-07T14:01:23.606Z] 14:01:23     INFO -    File "/home/cltbld/tasks/task_1599486904/build/tests/raptor/raptor/perftest.py", line 322, in build_browser_profile
[task 2020-09-07T14:01:23.606Z] 14:01:23     INFO -      self.profile_class, profile=self.get_conditioned_profile()
[task 2020-09-07T14:01:23.606Z] 14:01:23     INFO -    File "/home/cltbld/tasks/task_1599486904/build/tests/raptor/raptor/perftest.py", line 278, in get_conditioned_profile
[task 2020-09-07T14:01:23.607Z] 14:01:23     INFO -      repo=repo
[task 2020-09-07T14:01:23.607Z] 14:01:23     INFO -    File "/home/cltbld/tasks/task_1599486904/build/venv/lib/python2.7/site-packages/condprof/client.py", line 151, in get_profile
[task 2020-09-07T14:01:23.607Z] 14:01:23     INFO -      _check_service(TC_SERVICE)
[task 2020-09-07T14:01:23.607Z] 14:01:23     INFO -    File "/home/cltbld/tasks/task_1599486904/build/venv/lib/python2.7/site-packages/condprof/client.py", line 69, in _check_service
[task 2020-09-07T14:01:23.607Z] 14:01:23     INFO -      raise ServiceUnreachableError(url)
[task 2020-09-07T14:01:23.607Z] 14:01:23     INFO -  ServiceUnreachableError: https://firefox-ci-tc.services.mozilla.com
[task 2020-09-07T14:01:23.608Z] 14:01:23    ERROR -  raptor-main Critical: TEST-UNEXPECTED-FAIL: could not initialize the raptor test runner
[task 2020-09-07T14:01:23.668Z] 14:01:23    ERROR - Return code: 1
[task 2020-09-07T14:01:23.668Z] 14:01:23  WARNING - setting return code to 1
[task 2020-09-07T14:01:23.668Z] 14:01:23     INFO - Copying Raptor results to upload dir:
[task 2020-09-07T14:01:23.668Z] 14:01:23     INFO - /home/cltbld/tasks/task_1599486904/build/blobber_upload_dir/perfherder-data.json
[task 2020-09-07T14:01:23.669Z] 14:01:23     INFO - Copying raptor results from /home/cltbld/tasks/task_1599486904/build/raptor.json to /home/cltbld/tasks/task_1599486904/build/blobber_upload_dir/perfherder-data.json
[task 2020-09-07T14:01:23.669Z] 14:01:23 CRITICAL - Error copying results /home/cltbld/tasks/task_1599486904/build/raptor.json to upload dir /home/cltbld/tasks/task_1599486904/build/blobber_upload_dir/perfherder-data.json
[task 2020-09-07T14:01:23.669Z] 14:01:23     INFO - [Errno 2] No such file or directory: u'/home/cltbld/tasks/task_1599486904/build/raptor.json'
[task 2020-09-07T14:01:23.670Z] 14:01:23     INFO - Running post-action listener: _package_coverage_data
[task 2020-09-07T14:01:23.670Z] 14:01:23     INFO - Running post-action listener: _resource_record_post_action
[task 2020-09-07T14:01:23.670Z] 14:01:23     INFO - Running post-action listener: process_java_coverage_data
[task 2020-09-07T14:01:23.670Z] 14:01:23     INFO - Running post-action listener: stop_device
[task 2020-09-07T14:01:23.670Z] 14:01:23     INFO - [mozharness: 2020-09-07 14:01:23.670656Z] Finished run-tests step (success)
[task 2020-09-07T14:01:23.670Z] 14:01:23     INFO - Running post-run listener: _resource_record_post_run
[task 2020-09-07T14:01:23.721Z] 14:01:23     INFO - Total resource usage - Wall time: 332s; CPU: 0.0%; Read bytes: 1343488; Write bytes: 288686592; Read time: 276; Write time: 27848
[task 2020-09-07T14:01:23.721Z] 14:01:23     INFO - TinderboxPrint: CPU usage<br/>0.5%
[task 2020-09-07T14:01:23.721Z] 14:01:23     INFO - TinderboxPrint: I/O read bytes / time<br/>1,343,488 / 276
[task 2020-09-07T14:01:23.721Z] 14:01:23     INFO - TinderboxPrint: I/O write bytes / time<br/>288,686,592 / 27,848
[task 2020-09-07T14:01:23.721Z] 14:01:23     INFO - TinderboxPrint: CPU idle<br/>2,633.6 (99.5%)
[task 2020-09-07T14:01:23.721Z] 14:01:23     INFO - TinderboxPrint: Swap in / out<br/>0 / 0
[task 2020-09-07T14:01:23.722Z] 14:01:23     INFO - install-chromium-distribution - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2020-09-07T14:01:23.722Z] 14:01:23     INFO - install - Wall time: 7s; CPU: 13.0%; Read bytes: 180224; Write bytes: 34676736; Read time: 8; Write time: 4016
[task 2020-09-07T14:01:23.726Z] 14:01:23     INFO - run-tests - Wall time: 325s; CPU: 0.0%; Read bytes: 1163264; Write bytes: 254009856; Read time: 268; Write time: 23832
[task 2020-09-07T14:01:23.799Z] 14:01:23  WARNING - returning nonzero exit status 1
[taskcluster 2020-09-07T14:01:23.817Z]    Exit Code: 1
[taskcluster 2020-09-07T14:01:23.817Z]    User Time: 37.944s
[taskcluster 2020-09-07T14:01:23.817Z]  Kernel Time: 3.488s
[taskcluster 2020-09-07T14:01:23.817Z]    Wall Time: 6m18.753805301s
[taskcluster 2020-09-07T14:01:23.817Z]       Result: FAILED
[taskcluster 2020-09-07T14:01:23.817Z] === Task Finished ===
[taskcluster 2020-09-07T14:01:23.817Z] Task Duration: 6m18.75454989s
[taskcluster 2020-09-07T14:01:38.913Z] Uploading artifact public/logs/localconfig.json from file logs/localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2021-09-07T09:50:56.638Z
[taskcluster 2020-09-07T14:01:39.151Z] Uploading artifact public/test_info/resource-usage.json from file build/blobber_upload_dir/resource-usage.json with content encoding "gzip", mime type "application/json" and expiry 2021-09-07T09:50:56.638Z
[taskcluster:error] exit status 1

Greg Mierzwinski [:sparky]

Comment 1

•

4 years ago

:tarek, this looks like an intermittent failure in the condprof server or in taskcluster. ni?ed :jlorenzo too in case he has any thoughts.

Flags: needinfo?(tarek)

Flags: needinfo?(jlorenzo)

Johan Lorenzo [:jlorenzo]

Comment 2

•

4 years ago

I read the code and I don't see anything wrong with it. My first guess there was a network blip when trying to request https://firefox-ci-tc.services.mozilla.com. I'm surprised exponential backoff didn't work either. We waited 4 minutes between the first and the last request and it didn't come back.

I also see the bug occurred 10 times last week. That makes me wonder if one of the items that serve that URL is configured to not accept HTTP HEAD requests. Would it sound plausible to you, :hassan?

Flags: needinfo?(jlorenzo) → needinfo?(helfi92)

Andreea Pavel [:apavel]

Updated

•

4 years ago

Summary: Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: could not initialize the raptor test runner → Intermittent CRITICAL - raptor-perftest Critical: Could not get the conditioned profile | raptor-main Critical: TEST-UNEXPECTED-FAIL: could not initialize the raptor test runner

Andreea Pavel [:apavel]

Updated

•

4 years ago

Summary: Intermittent CRITICAL - raptor-perftest Critical: Could not get the conditioned profile | raptor-main Critical: TEST-UNEXPECTED-FAIL: could not initialize the raptor test runner → Intermittent raptor-perftest Critical: Could not get the conditioned profile | raptor-main Critical: TEST-UNEXPECTED-FAIL: could not initialize the raptor test runner

Hassan Ali (:hassan)

Comment 3

•

4 years ago

That makes me wonder if one of the items that serve that URL is configured to not accept HTTP HEAD requests. Would it sound plausible to you, :hassan?

Do we know if this was an intermittent or is it still occuring?

Flags: needinfo?(helfi92)

Greg Mierzwinski [:sparky]

Comment 4

•

4 years ago

:hassan, this is an intermittent failure that is still happening. Here's the latest failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=8948d4a41f23973ebcbeb27daebc8d281c58f31b&selectedTaskRun=aMtfhomLRnesWM2xo2NnuQ.0

Flags: needinfo?(helfi92)

Hassan Ali (:hassan)

Comment 5

•

4 years ago

:dustin, any idea if this from one of our recent deployments? I'm not sure how the frontend would be causing this.

Flags: needinfo?(helfi92) → needinfo?(dustin)

Dustin J. Mitchell [:dustin] (he/him)

Comment 6

•

4 years ago

Without being able to see the condprof code, it's hard to say what's going on here. What API is it calling? Why is it doing a HEAD request? Many of the URLs on that origin are a REST API, and I don't recall any of our methods defining an HEAD verb. But, other URLs, and in particular / respond with static assets, so I suppose a HEAD would work. And indeed it does:

rubin ~ $ curl -i -I https://firefox-ci-tc.services.mozilla.com/
HTTP/2 200 
server: openresty
date: Thu, 10 Sep 2020 15:32:31 GMT
content-type: text/html; charset=utf-8
content-length: 579
last-modified: Mon, 31 Aug 2020 15:50:40 GMT
etag: "5f4d1c50-243"
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
referrer-policy: origin
content-security-policy: default-src 'none'; connect-src 'self' https: wss:; media-src data:; script-src 'self' 'unsafe-eval' 'unsafe-inline' https:; font-src 'self' data:; img-src 'self' https: data:; style-src https: 'unsafe-inline'; object-src 'none'; frame-ancestors 'self'; frame-src 'self' https://auth.mozilla.auth0.com; base-uri 'none'; form-action 'none'
strict-transport-security: max-age=31536000
accept-ranges: bytes
via: 1.1 google
alt-svc: clear

The name ServiceUnreachableError sounds like it might be a 503. Looking at the graphs of response codes, we do see a sort of "background hum" of 503's, but there's no particular spike in 5xx errors around 14:00:01Z.

Anyway, a bit more information about what's happening would help to diagnose.

Flags: needinfo?(dustin)

Dustin J. Mitchell [:dustin] (he/him)

Comment 7

•

4 years ago

Oh, and the deployment of 36.0.0 was on the 8th, so not related.

Comment hidden (Intermittent Failures Robot)

Greg Mierzwinski [:sparky]

Comment 9

•

4 years ago

The failures seem to have stopped on the 10th, I'll leave this open for now though.

Florin Strugariu [:Bebe]

Assignee

Comment 10

•

4 years ago

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup

Status: NEW → RESOLVED

Closed: 4 years ago

Resolution: --- → INCOMPLETE

Natalia Csoregi [:nataliaCs]

Comment 11

•

3 years ago

https://treeherder.mozilla.org/logviewer?job_id=322960618&repo=mozilla-central&lineNumber=791

Status: RESOLVED → REOPENED

Resolution: INCOMPLETE → ---

Comment hidden (Intermittent Failures Robot)

Razvan Maries

Comment 15

•

3 years ago

There have been 50 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2020-12-12&endday=2020-12-19&tree=trunk&bug=1663449

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=324954652&repo=mozilla-central&lineNumber=822

Greg, could you assign this to someone?

Flags: needinfo?(gmierz2)

Whiteboard: [stockwell needswork:owner]

Comment hidden (Intermittent Failures Robot)

Narcis Beleuzu [:NarcisB]

Comment 20

•

3 years ago

•

Edited

There have been 65 failures in the last 7 days.
All failures occurs on mozilla central, opt:

50 failures on macosx1014-64
10 failures on windows10-64
5 failures on linux64

:sparky, can you please take a look?

Comment hidden (Intermittent Failures Robot)

Noemi Erli[:noemi_erli]

Comment 22

•

3 years ago

We had 63 failures in the past week.
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=325622622&repo=mozilla-central&lineNumber=789

Comment hidden (Intermittent Failures Robot)

Andreea Pavel [:apavel]

Updated

•

3 years ago

Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

Comment hidden (Intermittent Failures Robot)

Andreea Pavel [:apavel]

Comment 26

•

3 years ago

Hi Joel, do you know how this gets disabled?

There are 46 total failures in the last 7 days and 200 in the last 30 on

windows10-64-shippable-qr opt
windows10-64-shippable opt
macosx1014-64-shippable-qr opt (majority of failures)
linux64-shippable-qr opt
linux64-shippable opt

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=325690379&repo=mozilla-central&lineNumber=793

[task 2021-01-03T11:38:19.823Z] 11:38:19 [task 2021-01-03T11:38:19.823Z] 11:38:19 [task 2021-01-03T11:38:19.823Z] 11:38:19 [task 2021-01-03T11:38:19.823Z] 11:38:19 [task 2021-01-03T11:38:19.823Z] 11:38:19 [task 2021-01-03T11:38:19.824Z] 11:38:19 [task 2021-01-03T11:38:19.824Z] 11:38:19 [task 2021-01-03T11:38:19.824Z] 11:38:19 [task 2021-01-03T11:38:19.824Z] 11:38:19 [task 2021-01-03T11:38:19.824Z] 11:38:19 [task 2021-01-03T11:38:19.824Z] 11:38:19 [task 2021-01-03T11:38:19.824Z] 11:38:19 [task 2021-01-03T11:38:19.824Z] 11:38:19 [task 2021-01-03T11:38:19.824Z] 11:38:19 [task 2021-01-03T11:38:19.824Z] 11:38:19 [task 2021-01-03T11:38:19.824Z] 11:38:19 [task 2021-01-03T11:38:19.825Z] 11:38:19 [task 2021-01-03T11:38:19.825Z] 11:38:19 [task 2021-01-03T11:38:19.825Z] 11:38:19 [task 2021-01-03T11:38:19.825Z] 11:38:19 [task 2021-01-03T11:38:19.825Z] 11:38:19 [task 2021-01-03T11:38:19.825Z] 11:38:19 [task 2021-01-03T11:38:19.825Z] 11:38:19 [task 2021-01-03T11:38:19.825Z] 11:38:19 [task 2021-01-03T11:38:19.825Z] 11:38:19 [task 2021-01-03T11:38:19.825Z] 11:38:19 [task 2021-01-03T11:38:19.826Z] 11:38:19 [task 2021-01-03T11:38:19.826Z] 11:38:19 [task 2021-01-03T11:38:19.826Z] 11:38:19 [task 2021-01-03T11:38:19.826Z] 11:38:19 [task 2021-01-03T11:38:19.826Z] 11:38:19 [task 2021-01-03T11:38:19.826Z] 11:38:19 [task 2021-01-03T11:38:19.826Z] 11:38:19 [task 2021-01-03T11:38:19.826Z] 11:38:19 [task 2021-01-03T11:38:19.826Z] 11:38:19 [task 2021-01-03T11:38:19.827Z] 11:38:19 [task 2021-01-03T11:38:19.827Z] 11:38:19 [task 2021-01-03T11:38:19.827Z] 11:38:19 [task 2021-01-03T11:38:19.827Z] 11:38:19 [task 2021-01-03T11:38:19.827Z] 11:38:19 [task 2021-01-03T11:38:19.827Z] 11:38:19 [task 2021-01-03T11:38:19.827Z] 11:38:19 [task 2021-01-03T11:38:19.827Z] 11:38:19 [task 2021-01-03T11:38:19.827Z] 11:38:19 [task 2021-01-03T11:38:27.670Z] 11:38:27 [task 2021-01-03T11:39:12.707Z] 11:39:12 [task 2021-01-03T11:39:20.554Z] 11:39:20 [task 2021-01-03T11:40:28.090Z] 11:40:28 [task 2021-01-03T11:40:35.933Z] 11:40:35 [task 2021-01-03T11:42:17.239Z] 11:42:17 [task 2021-01-03T11:42:17.240Z] 11:42:17 CRITICAL - [task 2021-01-03T11:42:17.240Z] 11:42:17 [task 2021-01-03T11:42:17.240Z] 11:42:17 [task 2021-01-03T11:42:17.240Z] 11:42:17 [task 2021-01-03T11:42:17.240Z] 11:42:17 [task 2021-01-03T11:42:17.240Z] 11:42:17 [task 2021-01-03T11:42:17.240Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.241Z] 11:42:17 [task 2021-01-03T11:42:17.394Z] 11:42:17 INFO - raptor-main Info: raptor-start
INFO - raptor-main Info: received command line arguments: Namespace(activity=None, app='firefox', binary='/Users/cltbld/tasks/task_1609632751/build/application/Firefox Nightly.app/Contents/MacOS/firefox', browser_cycles=None, browsertime=False, browsertime_browsertimejs=None, browsertime_chromedriver=None, browsertime_ffmpeg=None, browsertime_geckodriver=None, browsertime_no_ffwindowrecorder=False, browsertime_node=None, browsertime_video=False, browsertime_vismet_script=None, browsertime_visualmetrics=False, chimera=False, cold=False, conditioned_profile_scenario='settled', cpu_test=False, debug_mode=False, device_name=None, disable_perf_tuning=False, e10s=True, enable_fission=False, enable_webrender=True, environment={}, extra_prefs={'extensions.logging.enabled': True}, gecko_profile=False, gecko_profile_entries=None, gecko_profile_interval=None, gecko_profile_threads=None, host='127.0.0.1', installerpath=None, intent=None, is_release_build=False, live_sites=False, log_errorsummary=None, log_grouped=None, log_html=None, log_mach=None, log_mach_buffer=None, log_mach_level=None, log_mach_screenshot=None, log_mach_verbose=None, log_raw=None, log_raw_level=None, log_tbpl=None, log_tbpl_buffer=None, log_tbpl_compact=None, log_tbpl_level='debug', log_unittest=None, log_xunit=None, memory_test=False, no_conditioned_profile=False, noinstall=False, obj_path=None, page_cycles=None, page_timeout=None, post_startup_delay=30000, power_test=False, project='mozilla-central', run_local=False, symbols_path='https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TvIXMwXlStO3l31vByKbHg/artifacts/public/build/target.crashreporter-symbols.zip', test='raptor-webaudio', test_url_params=None, verbose=False)
INFO - raptor-manifest Info: /Users/cltbld/tasks/task_1609632751/build/tests/raptor/raptor/raptor.ini
INFO - raptor-manifest Info: configuring settings for test raptor-webaudio-firefox
INFO - raptor-main Info: raptor tests scheduled to run:
INFO - raptor-main Info: raptor-webaudio-firefox
INFO - raptor-perftest Info: Using a conditioned profile.
INFO - mozversion application_buildid: 20210103092941
INFO - mozversion application_changeset: 1098ab162ad2352f7623193c1bbb80834d0b5d3a
INFO - mozversion application_display_name: Firefox Nightly
INFO - mozversion application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
INFO - mozversion application_name: Firefox
INFO - mozversion application_remotingname: firefox
INFO - mozversion application_repository: https://hg.mozilla.org/mozilla-central
INFO - mozversion application_vendor: Mozilla
INFO - mozversion application_version: 86.0a1
INFO - mozversion platform_buildid: 20210103092941
INFO - mozversion platform_changeset: 1098ab162ad2352f7623193c1bbb80834d0b5d3a
INFO - mozversion platform_repository: https://hg.mozilla.org/mozilla-central
INFO - mozversion platform_version: 86.0a1
INFO - raptor-perftest Info: Browser name: Firefox
INFO - raptor-perftest Info: Browser version: 86.0a1
INFO - mozversion application_buildid: 20210103092941
INFO - mozversion application_changeset: 1098ab162ad2352f7623193c1bbb80834d0b5d3a
INFO - mozversion application_display_name: Firefox Nightly
INFO - mozversion application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
INFO - mozversion application_name: Firefox
INFO - mozversion application_remotingname: firefox
INFO - mozversion application_repository: https://hg.mozilla.org/mozilla-central
INFO - mozversion application_vendor: Mozilla
INFO - mozversion application_version: 86.0a1
INFO - mozversion platform_buildid: 20210103092941
INFO - mozversion platform_changeset: 1098ab162ad2352f7623193c1bbb80834d0b5d3a
INFO - mozversion platform_repository: https://hg.mozilla.org/mozilla-central
INFO - mozversion platform_version: 86.0a1
INFO - raptor-perftest Info: Browser name: Firefox
INFO - raptor-perftest Info: Browser version: 86.0a1
INFO - raptor-perftest Info: Post startup delay set to 1000 ms
INFO - raptor-perftest Info: main raptor init, config is: {'verbose': False, 'e10s': True, 'app': 'firefox', 'symbols_path': 'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TvIXMwXlStO3l31vByKbHg/artifacts/public/build/target.crashreporter-symbols.zip', 'power_test': False, 'run_local': False, 'chimera': False, 'cold': False, 'gecko_profile_interval': None, 'binary': '/Users/cltbld/tasks/task_1609632751/build/application/Firefox Nightly.app/Contents/MacOS/firefox', 'enable_control_server_wait': False, 'no_conditioned_profile': False, 'environment': {'MOZ_WEBRENDER': '1'}, 'platform': 'mac', 'conditioned_profile_scenario': 'settled', 'extra_prefs': {'extensions.logging.enabled': True}, 'gecko_profile': False, 'gecko_profile_entries': None, 'disable_perf_tuning': False, 'host': '127.0.0.1', 'enable_fission': False, 'memory_test': False, 'enable_webrender': True, 'obj_path': None, 'live_sites': False, 'device_name': None, 'project': 'mozilla-central', 'cpu_test': False, 'is_release_build': False, 'using_condprof': True, 'processor': 'x86_64'}
INFO - raptor-perftest Info: Making temp_download_dir from inside get_conditioned_profile /var/folders/1w/j9dyfbx91ndcv4bm944ys80c000017/T/tmpIy1x1X
INFO - raptor-perftest Info: Platform used: macosx64
INFO - raptor-perftest Info: Getting profile from project mozilla-central
INFO - condprof INFO | Getting conditioned profile with arguments: {'repo': 'mozilla-central', 'platform': 'macosx64', 'task_id': None, 'customization': 'default', 'scenario': 'settled'}
INFO - condprof INFO | Getting headers at https://firefox-ci-tc.services.mozilla.com
INFO - condprof INFO | Failed, retrying
INFO - condprof INFO | Getting headers at https://firefox-ci-tc.services.mozilla.com
INFO - condprof INFO | Failed, retrying
INFO - condprof INFO | Getting headers at https://firefox-ci-tc.services.mozilla.com
INFO - condprof INFO | Failed, retrying
INFO - condprof ERROR | All attempt failed
raptor-perftest Critical: Could not get the conditioned profile
ERROR - Traceback (most recent call last):
INFO - File "/Users/cltbld/tasks/task_1609632751/build/tests/raptor/raptor/perftest.py", line 281, in get_conditioned_profile
INFO - temp_download_dir, platform, profile_scenario, repo=repo
INFO - File "/Users/cltbld/tasks/task_1609632751/build/venv/lib/python2.7/site-packages/condprof/client.py", line 149, in get_profile
INFO - _check_service(TC_SERVICE)
INFO - File "/Users/cltbld/tasks/task_1609632751/build/venv/lib/python2.7/site-packages/condprof/client.py", line 68, in _check_service
INFO - raise ServiceUnreachableError(url)
INFO - ServiceUnreachableError: https://firefox-ci-tc.services.mozilla.com
ERROR - Traceback (most recent call last):
INFO - File "/Users/cltbld/tasks/task_1609632751/build/tests/raptor/raptor/raptor.py", line 142, in main
INFO - verbose=args.verbose,
INFO - File "/Users/cltbld/tasks/task_1609632751/build/tests/raptor/raptor/webextension/desktop.py", line 25, in init
INFO - super(WebExtensionDesktop, self).init(*args, **kwargs)
INFO - File "/Users/cltbld/tasks/task_1609632751/build/tests/raptor/raptor/perftest.py", line 687, in init
INFO - super(PerftestDesktop, self).init(*args, **kwargs)
INFO - File "/Users/cltbld/tasks/task_1609632751/build/tests/raptor/raptor/webextension/base.py", line 37, in init
INFO - super(WebExtension, self).init(*args, **kwargs)
INFO - File "/Users/cltbld/tasks/task_1609632751/build/tests/raptor/raptor/perftest.py", line 216, in init
INFO - self.build_browser_profile()
INFO - File "/Users/cltbld/tasks/task_1609632751/build/tests/raptor/raptor/webextension/base.py", line 172, in build_browser_profile
INFO - super(WebExtension, self).build_browser_profile()
INFO - File "/Users/cltbld/tasks/task_1609632751/build/tests/raptor/raptor/perftest.py", line 323, in build_browser_profile
INFO - self.profile_class, profile=self.get_conditioned_profile()
INFO - File "/Users/cltbld/tasks/task_1609632751/build/tests/raptor/raptor/perftest.py", line 281, in get_conditioned_profile
INFO - temp_download_dir, platform, profile_scenario, repo=repo
INFO - File "/Users/cltbld/tasks/task_1609632751/build/venv/lib/python2.7/site-packages/condprof/client.py", line 149, in get_profile
INFO - _check_service(TC_SERVICE)
INFO - File "/Users/cltbld/tasks/task_1609632751/build/venv/lib/python2.7/site-packages/condprof/client.py", line 68, in _check_service
INFO - raise ServiceUnreachableError(url)
INFO - ServiceUnreachableError: https://firefox-ci-tc.services.mozilla.com
ERROR - raptor-main Critical: TEST-UNEXPECTED-FAIL: could not initialize the raptor test runner
ERROR - Return code: 1

Flags: needinfo?(tarek) → needinfo?(jmaher)

Joel Maher ( :jmaher ) (UTC -8)

Comment 27

•

3 years ago

:sparky has been on pto and will come back tomorrow- lets wait until the end of the week. In general this looks like it is an issue with taskcluster service. From what I can tell we use conditioned profiles all the time, so disabling this requires changing the harness defaults to not use conditioned profiles or turning off all jobs.

:davehunt, do you have other resources to look into this, or can you help make this a priority given that this failure falls into our disable category.

Flags: needinfo?(jmaher) → needinfo?(dave.hunt)

Andreea Pavel [:apavel]

Comment 28

•

3 years ago

Thank you Joel!

Dave Hunt [:davehunt] [he/him] ⌚BST

Comment 29

•

3 years ago

We've had a number of issues with conditioned profiles and disabled them for raptor-browsertime in bug 1677503. We could do the same for raptor-webextension, as this will happen once these remaining tests are migrated to browsertime anyway. This will cause the test signatures to change and will mean there's a risk of a regression slipping through, but if the sheriffs are aware of this they can manually compare the signatures as a one off. We should also file a bug for the TaskCluster service issue, as disabling conditioned profiles just works around the issue here.

:bebe could you assign this to someone on the team?

Flags: needinfo?(dave.hunt) → needinfo?(fstrugariu)

Florin Strugariu [:Bebe]

Assignee

Comment 30

•

3 years ago

Attached file Bug 1663449 - Disable conditioned profiles on raptor webext tests. — Details

Florin Strugariu [:Bebe]

Assignee

Updated

•

3 years ago

Depends on: 1685256

Florin Strugariu [:Bebe]

Assignee

Comment 31

•

3 years ago

•

Edited

Opened disable patch and logged bug.

Will review the alerts to make sure no regressions are missed

Flags: needinfo?(fstrugariu)

Phabricator Automation

Updated

•

3 years ago

Assignee: nobody → fstrugariu

Phabricator Automation

Updated

•

3 years ago

Attachment #9195569 - Attachment description: Bug 1663449 - Dissable cond profiles on raptor webext tests as the cause test failures → Bug 1663449 - Disable conditioned profiles on raptor webext tests.

Greg Mierzwinski [:sparky]

Comment 32

•

3 years ago

Thanks for taking care of this :bebe!

Flags: needinfo?(gmierz2)

Pulsebot

Comment 33

•

3 years ago

Pushed by fstrugariu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1d56d67f98e3
Disable conditioned profiles on raptor webext tests. r=perftest-reviewers,AlexandruIonescu

Cosmin Sabou [:CosminS]

Comment 34

•

3 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/1d56d67f98e3

Status: REOPENED → RESOLVED

Closed: 4 years ago → 3 years ago

status-firefox86: --- → fixed

Resolution: --- → FIXED

Target Milestone: --- → 86 Branch

Comment hidden (Intermittent Failures Robot)

Andreea Pavel [:apavel]

Updated

•

3 years ago

Status: RESOLVED → REOPENED

status-firefox86: fixed → ---

Keywords: leave-open

Resolution: FIXED → ---

Whiteboard: [stockwell disable-recommended] → [stockwell disabled]

Target Milestone: 86 Branch → ---

Comment hidden (Intermittent Failures Robot)

Florin Strugariu [:Bebe]

Assignee

Comment 37

•

3 years ago

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup

Status: REOPENED → RESOLVED

Closed: 3 years ago → 3 years ago

Resolution: --- → INCOMPLETE

BugBot [:suhaib / :marco/ :calixte]

Updated

•

3 years ago

Keywords: leave-open