1603978 - Perma Tier 2 Central condprof.client.ProfileNotFoundError: https://*.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public... | After raptor-mitmproxy Error: Mitmproxy exited with error code 572

Status: RESOLVED DUPLICATE of bug 1587387

(Testing :: Raptor, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: dvarga [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=281197392&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/a648ZMBiS2-E7OFjl_DGaw/runs/0/artifacts/public/logs/live_backing.log

---

[task 2019-12-14T20:46:53.040Z] 20:46:53     INFO -  raptor-control-server Info: shutting down browser (pid: 6260)
[task 2019-12-14T20:46:53.049Z] 20:46:53     INFO -  PID 6260 | console.log: "[raptor-runnerjs] post success"
[task 2019-12-14T20:46:53.049Z] 20:46:53     INFO -  PID 6260 | console.log: "[raptor-runnerjs] post success"
[task 2019-12-14T20:46:53.049Z] 20:46:53     INFO -  PID 6260 | console.log: "[raptor-runnerjs] post success"
[task 2019-12-14T20:46:53.493Z] 20:46:53     INFO -  PID 6260 | console.log: "[raptor-runnerjs] Removed tab: 2"
[task 2019-12-14T20:46:53.493Z] 20:46:53     INFO -  PID 6260 | console.log: "[raptor-runnerjs] posting to control server"
[task 2019-12-14T20:46:53.493Z] 20:46:53     INFO -  PID 6260 | console.log: "[raptor-runnerjs] Removed tab: 2"
[task 2019-12-14T20:46:53.512Z] 20:46:53     INFO -  raptor-control-server Info: received webext_status: Removed tab: 2
[task 2019-12-14T20:46:53.512Z] 20:46:53     INFO -  PID 6260 | console.log: "[raptor-runnerjs] post success"
[task 2019-12-14T20:46:54.871Z] 20:46:54     INFO -  raptor-main Info: begin browser cycle 25 of 25 for test raptor-tp6-docs-firefox-cold
[task 2019-12-14T20:46:54.872Z] 20:46:54     INFO -  raptor-main Info: starting test: raptor-tp6-docs-firefox-cold
[task 2019-12-14T20:46:54.872Z] 20:46:54     INFO -  raptor-main Info: starting raptor test: raptor-tp6-docs-firefox-cold
[task 2019-12-14T20:46:54.872Z] 20:46:54     INFO -  raptor-main Info: test settings: {'expected_browser_cycles': 25, 'alert_threshold': '2.0', 'playback_pageset_manifest': 'mitm4-linux-firefox-google-docs.manifest', 'browser_cycles': '25', 'here': 'C:\\Users\\task_1576343143\\build\\tests\\raptor\\raptor\\tests\\tp6\\desktop', 'expected': 'pass', 'measure': ['fnbpaint', 'fcp', 'dcf', 'loadtime'], 'path': 'C:\\Users\\task_1576343143\\build\\tests\\raptor\\raptor\\tests\\tp6\\desktop\\raptor-tp6-docs-firefox-cold', 'cold': True, 'manifest_relpath': 'tests\\tp6\\desktop\\raptor-tp6-5-cold.ini', 'unit': 'ms', 'alert_on': ['fcp', 'loadtime'], 'name': 'raptor-tp6-docs-firefox-cold', 'playback_recordings': 'google-docs.mp', 'page_cycles': 1, 'apps': 'firefox', 'test_url': 'https://docs.google.com/document/d/1US-07msg12slQtI_xchzYxcKlTs6Fp7WqIc6W5GK5M8/edit?usp=sharing', 'manifest': 'C:\\Users\\task_1576343143\\build\\tests\\raptor\\raptor\\tests\\tp6\\desktop\\raptor-tp6-5-cold.ini', 'ancestor_manifest': 'raptor.ini', 'page_timeout': '60000', 'playback': 'mitmproxy', 'browser_cycle': 25, 'lower_is_better': True, 'type': 'pageload', 'relpath': 'tests\\tp6\\desktop\\raptor-tp6-docs-firefox-cold'}
[task 2019-12-14T20:46:54.872Z] 20:46:54     INFO -  raptor-main Info: raptor config: {'subtest_alert_on': ['fcp', 'loadtime'], 'e10s': True, 'app': 'firefox', 'symbols_path': 'https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Uuh5xQYSRSSA2t4iVQEORQ/artifacts/public/build/target.crashreporter-symbols.zip', 'power_test': False, 'playback_binary_zip': None, 'run_local': False, 'playback_files': ['C:\\Users\\task_1576343143\\testing\\mozproxy\\google-docs.mp'], 'gecko_profile_interval': None, 'binary': 'C:\\Users\\task_1576343143\\build\\application\\firefox\\firefox.exe', 'local_profile_dir': 'c:\\users\\task_1576343143\\appdata\\local\\temp\\tmpumvn1n', 'playback_binary_manifest': 'mitmproxy-rel-bin-4.0.4-{platform}.manifest', 'enable_control_server_wait': False, 'no_conditioned_profile': False, 'platform': 'win', 'playback_version': '4.0.4', 'enable_webrender': False, 'gecko_profile': False, 'gecko_profile_entries': None, 'playback_pageset_manifest': 'C:\\Users\\task_1576343143\\build\\tests\\raptor\\raptor\\playback\\mitm4-linux-firefox-google-docs.manifest', 'playback_pageset_zip': None, 'playback_tool': 'mitmproxy', 'host': '127.0.0.1', 'enable_fission': False, 'memory_test': False, 'extra_prefs': {}, 'obj_path': None, 'cpu_test': False, 'is_release_build': False, 'processor': 'x86_64'}
[task 2019-12-14T20:46:54.872Z] 20:46:54     INFO -  raptor-gen-test-config Info: writing test settings into background js, so webext can get it
[task 2019-12-14T20:46:54.872Z] 20:46:54     INFO -  raptor-gen-test-config Info: finished writing test config to C:\Users\task_1576343143\build\tests\raptor\webext\raptor\auto_gen_test_config.js
[task 2019-12-14T20:46:54.873Z] 20:46:54     INFO -  raptor-main Info: installing webext C:\Users\task_1576343143\build\tests\raptor\raptor\..\webext\raptor
[task 2019-12-14T20:46:54.873Z] 20:46:54     INFO -  raptor-main Info: Making temp_download_dir from inside get_conditioned_profile c:\users\task_1576343143\appdata\local\temp\tmp9i_vza
[task 2019-12-14T20:48:56.386Z] 20:48:56     INFO -  [2019-12-14T20:46:54.866000] Getting https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz
[task 2019-12-14T20:48:56.386Z] 20:48:56     INFO -  [2019-12-14T20:47:15.882000] Cannot find 'https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz'
[task 2019-12-14T20:48:56.386Z] 20:48:56     INFO -  [2019-12-14T20:47:15.882000] Failed to get the profile.
[task 2019-12-14T20:48:56.386Z] 20:48:56    ERROR -  Traceback (most recent call last):
[task 2019-12-14T20:48:56.386Z] 20:48:56     INFO -    File "C:\Users\task_1576343143\build\venv\lib\site-packages\condprof\client.py", line 84, in get_profile
[task 2019-12-14T20:48:56.386Z] 20:48:56     INFO -      raise ProfileNotFoundError(url)
[task 2019-12-14T20:48:56.386Z] 20:48:56     INFO -  ProfileNotFoundError: https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz
[task 2019-12-14T20:48:56.386Z] 20:48:56     INFO -  [2019-12-14T20:47:45.888000] Getting https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz
[task 2019-12-14T20:48:56.387Z] 20:48:56     INFO -  [2019-12-14T20:47:48.908000] Cannot find 'https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz'
[task 2019-12-14T20:48:56.387Z] 20:48:56     INFO -  [2019-12-14T20:47:48.908000] Failed to get the profile.
[task 2019-12-14T20:48:56.387Z] 20:48:56    ERROR -  Traceback (most recent call last):
[task 2019-12-14T20:48:56.387Z] 20:48:56     INFO -    File "C:\Users\task_1576343143\build\venv\lib\site-packages\condprof\client.py", line 84, in get_profile
[task 2019-12-14T20:48:56.387Z] 20:48:56     INFO -      raise ProfileNotFoundError(url)
[task 2019-12-14T20:48:56.387Z] 20:48:56     INFO -  ProfileNotFoundError: https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz
[task 2019-12-14T20:48:56.387Z] 20:48:56     INFO -  [2019-12-14T20:48:18.911000] Getting https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz
[task 2019-12-14T20:48:56.387Z] 20:48:56     INFO -  [2019-12-14T20:48:26.380000] Cannot find 'https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz'
[task 2019-12-14T20:48:56.387Z] 20:48:56     INFO -  [2019-12-14T20:48:26.380000] Failed to get the profile.
[task 2019-12-14T20:48:56.387Z] 20:48:56    ERROR -  Traceback (most recent call last):
[task 2019-12-14T20:48:56.387Z] 20:48:56     INFO -    File "C:\Users\task_1576343143\build\venv\lib\site-packages\condprof\client.py", line 84, in get_profile
[task 2019-12-14T20:48:56.388Z] 20:48:56     INFO -      raise ProfileNotFoundError(url)
[task 2019-12-14T20:48:56.388Z] 20:48:56     INFO -  ProfileNotFoundError: https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz
[task 2019-12-14T20:48:56.388Z] 20:48:56     INFO -  [2019-12-14T20:48:56.383000] All attempt failed
[task 2019-12-14T20:48:56.388Z] 20:48:56    ERROR -  Traceback (most recent call last):
[task 2019-12-14T20:48:56.388Z] 20:48:56     INFO -    File "C:\Users\task_1576343143\build\venv\lib\site-packages\condprof\client.py", line 84, in get_profile
[task 2019-12-14T20:48:56.388Z] 20:48:56     INFO -      raise ProfileNotFoundError(url)
[task 2019-12-14T20:48:56.388Z] 20:48:56     INFO -  ProfileNotFoundError: https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz
[task 2019-12-14T20:48:56.388Z] 20:48:56     INFO -  raptor-mitmproxy Info: Stopping mitmproxy playback, killing process 3816
[task 2019-12-14T20:48:56.406Z] 20:48:56    ERROR -  raptor-mitmproxy Error: Mitmproxy exited with error code 572
[task 2019-12-14T20:48:56.406Z] 20:48:56     INFO -  raptor-mitmproxy Info: Turning off the browser proxy
[task 2019-12-14T20:48:56.406Z] 20:48:56     INFO -  raptor-mitmproxy Info: writing: C:\Users\task_1576343143\build\application\firefox\distribution\policies.json
[task 2019-12-14T20:48:56.407Z] 20:48:56     INFO -  raptor-main Info: removing webext C:\Users\task_1576343143\build\tests\raptor\raptor\..\webext\raptor
[task 2019-12-14T20:48:56.407Z] 20:48:56     INFO -  raptor-control-server Info: shutting down control server
[task 2019-12-14T20:48:56.623Z] 20:48:56     INFO -  raptor-main Info: finished
[task 2019-12-14T20:48:56.633Z] 20:48:56    ERROR -  Traceback (most recent call last):
[task 2019-12-14T20:48:56.633Z] 20:48:56     INFO -    File "C:\Users\task_1576343143\build\tests\raptor\raptor\raptor.py", line 1857, in <module>
[task 2019-12-14T20:48:56.633Z] 20:48:56     INFO -      main()
[task 2019-12-14T20:48:56.633Z] 20:48:56     INFO -    File "C:\Users\task_1576343143\build\tests\raptor\raptor\raptor.py", line 1815, in main
[task 2019-12-14T20:48:56.633Z] 20:48:56     INFO -      success = raptor.run_tests(raptor_test_list, raptor_test_names)
[task 2019-12-14T20:48:56.633Z] 20:48:56     INFO -    File "C:\Users\task_1576343143\build\tests\raptor\raptor\raptor.py", line 285, in run_tests
[task 2019-12-14T20:48:56.634Z] 20:48:56     INFO -      self.run_test(test, timeout=int(test.get('page_timeout')))
[task 2019-12-14T20:48:56.634Z] 20:48:56     INFO -    File "C:\Users\task_1576343143\build\tests\raptor\raptor\raptor.py", line 1143, in run_test
[task 2019-12-14T20:48:56.634Z] 20:48:56     INFO -      self.__run_test_cold(test, timeout)
[task 2019-12-14T20:48:56.634Z] 20:48:56     INFO -    File "C:\Users\task_1576343143\build\tests\raptor\raptor\raptor.py", line 1205, in __run_test_cold
[task 2019-12-14T20:48:56.634Z] 20:48:56     INFO -      self.build_browser_profile()
[task 2019-12-14T20:48:56.634Z] 20:48:56     INFO -    File "C:\Users\task_1576343143\build\tests\raptor\raptor\raptor.py", line 994, in build_browser_profile
[task 2019-12-14T20:48:56.634Z] 20:48:56     INFO -      super(Raptor, self).build_browser_profile()
[task 2019-12-14T20:48:56.634Z] 20:48:56     INFO -    File "C:\Users\task_1576343143\build\tests\raptor\raptor\raptor.py", line 218, in build_browser_profile
[task 2019-12-14T20:48:56.634Z] 20:48:56     INFO -      self.get_conditioned_profile()
[task 2019-12-14T20:48:56.635Z] 20:48:56     INFO -    File "C:\Users\task_1576343143\build\tests\raptor\raptor\raptor.py", line 199, in get_conditioned_profile
[task 2019-12-14T20:48:56.635Z] 20:48:56     INFO -      cond_prof_target_dir = get_profile(temp_download_dir, platform, "settled")
[task 2019-12-14T20:48:56.635Z] 20:48:56     INFO -    File "C:\Users\task_1576343143\build\venv\lib\site-packages\condprof\client.py", line 119, in get_profile
[task 2019-12-14T20:48:56.635Z] 20:48:56     INFO -      raise ProfileNotFoundError(url)
[task 2019-12-14T20:48:56.635Z] 20:48:56     INFO -  condprof.client.ProfileNotFoundError: https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64/artifacts/public/condprof/profile-win64-settled-default.tgz
[task 2019-12-14T20:48:56.667Z] 20:48:56    ERROR - Return code: 1
[task 2019-12-14T20:48:56.667Z] 20:48:56  WARNING - setting return code to 1
[task 2019-12-14T20:48:56.667Z] 20:48:56     INFO - Copying Raptor results to upload dir:
[task 2019-12-14T20:48:56.667Z] 20:48:56     INFO - C:\Users\task_1576343143\build\blobber_upload_dir\perfherder-data.json
[task 2019-12-14T20:48:56.668Z] 20:48:56     INFO - Copying raptor results from C:\Users\task_1576343143\build\raptor.json to C:\Users\task_1576343143\build\blobber_upload_dir\perfherder-data.json
[task 2019-12-14T20:48:56.668Z] 20:48:56 CRITICAL - Error copying results C:\Users\task_1576343143\build\raptor.json to upload dir C:\Users\task_1576343143\build\blobber_upload_dir\perfherder-data.json
[task 2019-12-14T20:48:56.668Z] 20:48:56     INFO - [Errno 2] No such file or directory: u'C:\\Users\\task_1576343143\\build\\raptor.json'
[task 2019-12-14T20:48:56.668Z] 20:48:56     INFO - Running post-action listener: _package_coverage_data
[task 2019-12-14T20:48:56.668Z] 20:48:56     INFO - Running post-action listener: _resource_record_post_action
[task 2019-12-14T20:48:56.669Z] 20:48:56     INFO - Running post-action listener: process_java_coverage_data
[task 2019-12-14T20:48:56.669Z] 20:48:56     INFO - Running post-action listener: stop_device
[task 2019-12-14T20:48:56.669Z] 20:48:56     INFO - [mozharness: 2019-12-14 20:48:56.669000Z] Finished run-tests step (success)
[task 2019-12-14T20:48:56.669Z] 20:48:56     INFO - Running post-run listener: _resource_record_post_run
[task 2019-12-14T20:48:56.770Z] 20:48:56     INFO - Total resource usage - Wall time: 1044s; CPU: 5.0%; Read bytes: 158531584; Write bytes: 1380241408; Read time: 2; Write time: 22
[task 2019-12-14T20:48:56.770Z] 20:48:56     INFO - TinderboxPrint: CPU usage<br/>5.0%
[task 2019-12-14T20:48:56.770Z] 20:48:56     INFO - TinderboxPrint: I/O read bytes / time<br/>158,531,584 / 2
[task 2019-12-14T20:48:56.770Z] 20:48:56     INFO - TinderboxPrint: I/O write bytes / time<br/>1,380,241,408 / 22
[task 2019-12-14T20:48:56.770Z] 20:48:56     INFO - TinderboxPrint: CPU idle<br/>7,935.6 (95.0%)
[task 2019-12-14T20:48:56.770Z] 20:48:56     INFO - TinderboxPrint: CPU user<br/>331.9 (4.0%)
[task 2019-12-14T20:48:56.771Z] 20:48:56     INFO - install - Wall time: 2s; CPU: 13.0%; Read bytes: 77824; Write bytes: 3237376; Read time: 0; Write time: 0
[task 2019-12-14T20:48:56.780Z] 20:48:56     INFO - run-tests - Wall time: 1043s; CPU: 5.0%; Read bytes: 156206080; Write bytes: 1369789952; Read time: 2; Write time: 22
[task 2019-12-14T20:48:56.998Z] 20:48:56  WARNING - returning nonzero exit status 1
[taskcluster 2019-12-14T20:48:57.033Z]    Exit Code: 1
[taskcluster 2019-12-14T20:48:57.033Z]    User Time: 15.625ms
[taskcluster 2019-12-14T20:48:57.033Z]  Kernel Time: 15.625ms
[taskcluster 2019-12-14T20:48:57.033Z]    Wall Time: 18m20.0091134s
[taskcluster 2019-12-14T20:48:57.033Z]       Result: FAILED
[taskcluster 2019-12-14T20:48:57.033Z] === Task Finished ===
[taskcluster 2019-12-14T20:48:57.033Z] Task Duration: 18m20.0091134s
[taskcluster 2019-12-14T20:48:57.175Z] Uploading artifact public/logs/localconfig.json from file logs\localconfig.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2020-12-13T18:05:08.023Z
[taskcluster 2019-12-14T20:48:57.438Z] Uploading artifact public/test_info/mitmproxy.log from file build\blobber_upload_dir\mitmproxy.log with content encoding "gzip", mime type "text/plain" and expiry 2020-12-13T18:05:08.023Z
[taskcluster 2019-12-14T20:48:57.754Z] Uploading artifact public/test_info/resource-usage.json from file build\blobber_upload_dir\resource-usage.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2020-12-13T18:05:08.023Z
[taskcluster 2019-12-14T20:48:57.997Z] Uploading redirect artifact public/logs/live.log to URL https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/a648ZMBiS2-E7OFjl_DGaw/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2020-12-13T18:05:08.023Z
[taskcluster:error] exit status 1

Comment 1

[Intermittent Failures Robot]

35 failures in 198 pushes (0.177 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 35

Platform breakdown:

• windows7-32-shippable: 10
• windows10-64-shippable-qr: 15
• windows10-64-shippable: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603978&startday=2019-12-14&endday=2019-12-14&tree=all

Comment 2

[Intermittent Failures Robot]

35 failures in 4867 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

This is the #40 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 35

Platform breakdown:

• windows7-32-shippable: 10
• windows10-64-shippable-qr: 15
• windows10-64-shippable: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603978&startday=2019-12-09&endday=2019-12-15&tree=all

Comment 3

[Noemi Erli[:noemi_erli]]

Performed some retriggers: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=1c10f669c66038a6410d47d9c59aa7b41c0481b6&tochange=4478cd379335a7e46d158929fa5de72c045f4680&fromchange=db127fef777496f40cd08a26f7f81b3ee8170cf9
Brad, could these failures be triggered by the changes made in Bug 1595092?

Comment 4

[Robert Wood [:rwood]]

Adding Stephen, thanks

Comment 5

[Brad Werth [:bradwerth]]

(In reply to Noemi Erli[:noemi_erli] from comment #3)

Brad, could these failures be triggered by the changes made in Bug 1595092?

That seems very unlikely to me. The patches in Bug 1595092 adjust the behavior of a devtools panel that is only enabled behind an off-by-default pref. I don't see how it could affect profile retrieval on an unrelated test.

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

I've looked, and haven't been able to figure out why this is happening.

The cron for running condprof fires off at 1600 UTC daily; the 404s here don't seem to correlate to an error trying to fetch the profile while it's still being uploaded, as they are timestamped [task 2019-12-14T20:48:56.386Z] 20:48:56.

Perhaps bug 1603427 fixed this? I don't know why the client would suddenly fail (unless it lost connectivity, in which case, shouldn't we see other errors?), either, given it ran fine for 24 cycles.

@tarek - any ideas here, and/or know whom else to ping, to try to see what's happening on the backend? Thanks!

Comment 7

[Tarek Ziadé (:tarek)]

Could be a temp network issue, I can see the file right now, and no new failures

Comment 8

[Cristian Brindusan [:cbrindusan]]

This seems to be happening again on central:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=282227844&repo=mozilla-central&lineNumber=1033

Comment 9

[Intermittent Failures Robot]

90 failures in 767 pushes (0.117 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 90

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 33
• android-hw-p2-8-0-arm7-api-16: 33
• android-hw-p2-8-0-android-aarch64: 24

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603978&startday=2019-12-20&endday=2019-12-20&tree=all

Comment 10

[Intermittent Failures Robot]

180 failures in 4582 pushes (0.039 failures/push) were associated with this bug in the last 7 days.

This is the #3 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 180

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 64
• android-hw-p2-8-0-android-aarch64: 45
• android-hw-g5-7-0-arm7-api-16: 71

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603978&startday=2019-12-16&endday=2019-12-22&tree=all

Comment 11

[Florin Strugariu [:Bebe]]

Robert any suggestions who can take a look over this one?

Comment 12

[Robert Wood [:rwood]]

Yes, Stephen (he is already NI'd above) thanks.

Comment 13

[Bogdan Tara[:bogdan_tara | bogdant]]

This seems to perma fail on Tier 2 on Central: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&fromchange=930ad6def3c7961c82b2af20b66be3351603684f&searchStr=android%2C7.0%2Cmotog5%2Cpgo%2Craptor%2Cperformance%2Ctests%2Con%2Cthe%2Creference%2Cbrowser%2Ctest-android-hw-g5-7-0-arm7-api-16%2Fpgo-raptor-tp6m-1-refbrow-cold-e10s%2Crap-refbrow%28tp6m-c-1%29&tochange=48159e53bfb85d9b22e94ecaaa6590ab4abd9545&selectedJob=282033554

Comment 14

[Stephen Donner [:stephend] Not actively reading bugmail]

I think there are 3 classes of failures/issues:

1. older clients needing uplifts (like mozilla-beta does, in bug 1605237) to both switch INDEX/ROOT_URLs as well as introduce the "retry fetches using try-server" behavior
2. intermittents, such as this one (as originally filed) where we can't reproduce the 404s/503s (https://bugzilla.mozilla.org/show_bug.cgi?id=1603978#c6 etc.)
3. some of our browser variants (like refbrow) are getting pointed to a conditioned profile, where it's either a) a bug that there's no build, but should be, or b) (more likely) a bug that they're not falling back to non-conditioned profiles
4. and I guess a 4th case might be a combination of 1) and 3), where, if they can't get the profile from mozilla-central, they'll try from try

(In reply to Cristian Brindusan [:cbrindusan] from comment #8)

This seems to be happening again on central:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=282227844&repo=mozilla-central&lineNumber=1033

This is https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-p2_aarch64-fennec_aurora/artifacts/public/condprof/profile-p2_aarch64-fennec_aurora-settled-default.tgz 404ing; I don't know if/that we're supporting conditioned-profile builds for p2_aarch64-fennec_aurora (nor refbrow); as commits for bug 1597711 target GeckoView and Fenix, specifically.

Comment 15

[Andreea Pavel [:apavel]]

Stephen, is there someone that can take a look at classes 2-4 you mentioned above? For the first 1 I see Tom already made a patch.

Comment 16

[Andreea Pavel [:apavel]]

Changing the stockwell tag for now.

Comment 17

[Intermittent Failures Robot]

18 failures in 1365 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #45 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 18

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 12
• android-hw-p2-8-0-arm7-api-16: 3
• android-hw-p2-8-0-android-aarch64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1603978&startday=2019-12-23&endday=2019-12-29&tree=all

Comment 18

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Andreea Pavel [:apavel] from comment #15)

Stephen, is there someone that can take a look at classes 2-4 you mentioned above? For the first 1 I see Tom already made a patch.

For 2), actual intermittents, I've got a thread in another bug with Dustin that I need to follow-up on (which I'll do today), seeing if there's additional logging we need to help determine root cause.
For 3), I've taken and have a hopeful fix for bug 1606199; I'm still not 100% sure if refbrow should have a conditioned profile or not, but I'll try to take a closer look again while Tarek is PTO'd.
4) is really a combination of and outshoot from the uplift for mozilla-beta, 1), which is being addressed

Leaving need-info? on to keep me honest.

Comment 19

[Stephen Donner [:stephend] Not actively reading bugmail]

The bug I was thinking of is bug 1605885, which just recently landed a fix to silence errors, once we actually successfully get the profile (as it retries); as Tarek says in that bug, the real issue is bug 1606793, which also just added a fix.

Hopefully with those two fixes, plus the refbrow (bug 1606767) and fennec_aurora (bug 1606199) fixes, this will be sorted out, or at least very close to being!

Comment 20

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Bebe is this related to the intermittents you've been investigating recently?

Comment 21

[Florin Strugariu [:Bebe]]

As the cond profiles issue where fixed in the bugs mentioned by Stephen and the mitm 572 was not an actual error (fixed in Bug 1587387 ) I feel that this bug is not valid anymore