1773889 - Perma Windows 10 x64 2004 WebRender mochitests/xpcshell condprof.client.ProfileNotFoundError: https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64-full/artifacts/public/condprof/profile-win

Status: RESOLVED FIXED

(Testing :: Condprofile, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: smolnar [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=381044574&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RqGOzh5FSwaFE-9xosNTJA/runs/0/artifacts/public/logs/live_backing.log

---

INFO - condprof Could not find an existing archive.
[task 2022-06-12T16:31:28.265Z] 16:31:28     INFO - condprof Content in cache directory C:\Users\task_165504871952763/.condprof-cache: ['profile-win64-full-default.tgz.etag']
[task 2022-06-12T16:31:28.266Z] 16:31:28     INFO - condprof Downloading https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64-full/artifacts/public/condprof/profile-win64-full-default.tgz
[task 2022-06-12T16:31:28.357Z] 16:31:28     INFO - condprof Failed to extract the tarball
[task 2022-06-12T16:31:28.360Z] 16:31:28     INFO - condprof Removing cached file to attempt a new download
[task 2022-06-12T16:31:28.360Z] 16:31:28     INFO - condprof Failed to get the profile.
[task 2022-06-12T16:31:28.361Z] 16:31:28     INFO - condprof Failed, retrying
[task 2022-06-12T16:33:09.608Z] 16:33:09    ERROR - condprof All attempt failed
[task 2022-06-12T16:33:09.610Z] 16:33:09     INFO -  Traceback (most recent call last):
[task 2022-06-12T16:33:09.610Z] 16:33:09     INFO -    File "Z:\task_165504871952763\build\venv\lib\site-packages\condprof\client.py", line 211, in get_profile
[task 2022-06-12T16:33:09.610Z] 16:33:09     INFO -      return _retries(_get_profile, onerror)
[task 2022-06-12T16:33:09.611Z] 16:33:09     INFO -    File "Z:\task_165504871952763\build\venv\lib\site-packages\condprof\client.py", line 120, in _retries
[task 2022-06-12T16:33:09.611Z] 16:33:09     INFO -      raise RetriesError()
[task 2022-06-12T16:33:09.611Z] 16:33:09     INFO -  condprof.client.RetriesError
[task 2022-06-12T16:33:09.611Z] 16:33:09     INFO -  During handling of the above exception, another exception occurred:
[task 2022-06-12T16:33:09.612Z] 16:33:09     INFO -  Traceback (most recent call last):
[task 2022-06-12T16:33:09.612Z] 16:33:09     INFO -    File "Z:\task_165504871952763\build\tests\mochitest\runtests.py", line 3879, in <module>
[task 2022-06-12T16:33:09.612Z] 16:33:09     INFO -      sys.exit(cli())
[task 2022-06-12T16:33:09.613Z] 16:33:09     INFO -    File "Z:\task_165504871952763\build\tests\mochitest\runtests.py", line 3875, in cli
[task 2022-06-12T16:33:09.613Z] 16:33:09     INFO -      return run_test_harness(parser, options)
[task 2022-06-12T16:33:09.613Z] 16:33:09     INFO -    File "Z:\task_165504871952763\build\tests\mochitest\runtests.py", line 3860, in run_test_harness
[task 2022-06-12T16:33:09.613Z] 16:33:09     INFO -      result = runner.runTests(options)
[task 2022-06-12T16:33:09.614Z] 16:33:09     INFO -    File "Z:\task_165504871952763\build\tests\mochitest\runtests.py", line 3223, in runTests
[task 2022-06-12T16:33:09.614Z] 16:33:09     INFO -      res = self.runMochitests(options, tests_in_manifest, manifestToFilter=m)
[task 2022-06-12T16:33:09.614Z] 16:33:09     INFO -    File "Z:\task_165504871952763\build\tests\mochitest\runtests.py", line 2921, in runMochitests
[task 2022-06-12T16:33:09.614Z] 16:33:09     INFO -      result = self.doTests(options, testsToRun, manifestToFilter)
[task 2022-06-12T16:33:09.614Z] 16:33:09     INFO -    File "Z:\task_165504871952763\build\tests\mochitest\runtests.py", line 3357, in doTests
[task 2022-06-12T16:33:09.615Z] 16:33:09     INFO -      self.manifest = self.buildProfile(options)
[task 2022-06-12T16:33:09.615Z] 16:33:09     INFO -    File "Z:\task_165504871952763\build\tests\mochitest\runtests.py", line 2219, in buildProfile
[task 2022-06-12T16:33:09.615Z] 16:33:09     INFO -      options.profilePath = self.downloadConditionedProfile("full")
[task 2022-06-12T16:33:09.615Z] 16:33:09     INFO -    File "Z:\task_165504871952763\build\tests\mochitest\runtests.py", line 2172, in downloadConditionedProfile
[task 2022-06-12T16:33:09.616Z] 16:33:09     INFO -      cond_prof_target_dir = get_profile(
[task 2022-06-12T16:33:09.616Z] 16:33:09     INFO -    File "Z:\task_165504871952763\build\venv\lib\site-packages\condprof\client.py", line 213, in get_profile
[task 2022-06-12T16:33:09.616Z] 16:33:09     INFO -      raise ProfileNotFoundError(url)
[task 2022-06-12T16:33:09.617Z] 16:33:09     INFO -  condprof.client.ProfileNotFoundError: https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-win64-full/artifacts/public/condprof/profile-win64-full-default.tgz
[task 2022-06-12T16:33:09.693Z] 16:33:09    ERROR - Return code: 1
[task 2022-06-12T16:33:09.694Z] 16:33:09    ERROR - No suite end message was emitted by this harness.
[task 2022-06-12T16:33:09.694Z] 16:33:09     INFO - TinderboxPrint: mochitest-mochitest-plain<br/>104/0/0
[task 2022-06-12T16:33:09.694Z] 16:33:09    ERROR - # TBPL FAILURE #
[task 2022-06-12T16:33:09.694Z] 16:33:09  WARNING - setting return code to 2

Comment 1

[Natalia Csoregi [:nataliaCs]]

The task fails even where it has previously passed: https://treeherder.mozilla.org/jobs?repo=autoland&fromchange=a419f8030e7556aa789dc5dfaa43a2ff9f6ad883&searchStr=Windows%2C10%2Cx64%2C2004%2CWebRender%2CShippable%2Ccondprof&group_state=expanded&tochange=f921b60e43ea865beb13dd7a17e3f9a6d6299949

Gregory, could you help us assign this to someone?
Thank you.

Comment 2

[Intermittent Failures Robot]

54 failures in 3336 pushes (0.016 failures/push) were associated with this bug in the last 7 days.

This is the #18 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 49
• mozilla-central: 5

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 49
  • opt: 49
• windows10-64-2004-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1773889&startday=2022-06-06&endday=2022-06-12&tree=all

Comment 3

[Greg Mierzwinski [:sparky]]

:jmaher, is this something you can look into? It's related to mochitest tests using condprof and I'm not sure who owns those.

Comment 4

[Natalia Csoregi [:nataliaCs]]

Backfilling/retriggering the jobs now has them passing, e.g.: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&searchStr=windows%2C10%2Cx64%2C2004%2Cwebrender%2Cshippable%2Copt%2Cmochitests%2Cwith%2Ca%2Cconditioned%2Cprofile%2Cand%2Cfission%2Cenabled%2Ctest-windows10-64-2004-shippable-qr%2Fopt-mochitest-plain-condprof-fis-e10s%2C1&tochange=29227ffebd04ab1312cbf6e65cd601eb2c05f7e1&fromchange=9011b8f9667f8513f0677fa4d250a27399a58952&selectedTaskRun=Mh_OC_o4SpacqFYfP5vUfA.0

Comment 5

[Greg Mierzwinski [:sparky]]

Thanks :nataliacs! So the issue came from this task which failed silently: https://treeherder.mozilla.org/jobs?repo=mozilla-central&selectedTaskRun=NYWEh8ttRyC-AYSjjltR9A.0&searchStr=condprof&revision=960df119139665fb503dd69be79e38337e0093e3

The profile was not produced but the mochitest tests were still trying to pull from it.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I will look into making this job fail, which should help out in tracking failures and maybe fixing the index to be the last known good.

Comment 7

[Intermittent Failures Robot]

25 failures in 593 pushes (0.042 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 5
• autoland: 20

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 15
  • opt: 15
• windows10-64-2004-qr: 10
  • opt: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1773889&startday=2022-06-16&endday=2022-06-16&tree=all

Comment 8

[Intermittent Failures Robot]

25 failures in 3170 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• autoland: 20

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 15
  • opt: 15
• windows10-64-2004-qr: 10
  • opt: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1773889&startday=2022-06-13&endday=2022-06-19&tree=all

Comment 9

[Intermittent Failures Robot]

30 failures in 518 pushes (0.058 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 5
• try: 5
• autoland: 20

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 15
  • opt: 15
• windows10-64-2004-qr: 15
  • opt: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1773889&startday=2022-06-20&endday=2022-06-20&tree=all

Comment 10

[Iulian Moraru]

This is now permafailing on xpcshell jobs: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&searchStr=windows%2C10%2Cx64%2C2004%2Cwebrender%2Copt%2Cxpcshell%2Ctests%2Cwith%2Ca%2Cconditioned%2Cprofile%2Cand%2Cfission%2Cenabled%2Ctest-windows10-64-2004-qr%2Fopt-xpcshell-condprof-fis-e10s%2Cx8&fromchange=f2ae3ed2e501913df3bfc7350bcf5d59343bdc70&tochange=eae358b04c3a1aadbb4de6172e0b80749ac2e58d&selectedTaskRun=cg-ubh4mTGCxUM_McLAKSg.0
This started when the recent changes from Bug 1769154 landed.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1773889 - exit with TBPL_RETRY for failed condprof job. r=sparky!

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

I have a patch up to mark this as TBPL_RETRY so we automatically retry the job. I am happy to adjust to failure if needed.

Comment 13

[Intermittent Failures Robot]

62 failures in 684 pushes (0.091 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 10
• autoland: 47
• try: 5

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 31
  • opt: 31
• windows10-64-2004-qr: 31
  • opt: 31

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1773889&startday=2022-06-22&endday=2022-06-22&tree=all

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/43773fd2a3ad
exit with TBPL_RETRY for failed condprof job. r=sparky

Comment 15

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/43773fd2a3ad

Comment 16

[Intermittent Failures Robot]

81 failures in 640 pushes (0.127 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 65
• mozilla-central: 16

Platform and build breakdown:

• windows10-64-2004-qr: 33
  • opt: 33
• windows10-64-2004-shippable-qr: 48
  • opt: 48

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1773889&startday=2022-06-23&endday=2022-06-23&tree=all

Comment 17

[Intermittent Failures Robot]

173 failures in 3075 pushes (0.056 failures/push) were associated with this bug in the last 7 days.

This is the #5 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 31
• try: 10
• autoland: 132

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 94
  • opt: 94
• windows10-64-2004-qr: 79
  • opt: 79

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1773889&startday=2022-06-20&endday=2022-06-26&tree=all

Comment 18

[Cristian Tuns]

Hello, this issue it's still on, Joel can you please take a look?
Autoland link: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&searchStr=windows%2C10%2Cx64%2C2004%2Cwebrender%2Copt&tochange=01eac14dcf7c2091c9418c0c419e525ad9f1606d&fromchange=81a09589dba32fda0999bfe350e2bb89f05ffcab&selectedTaskRun=N2UeRMR0QFym00WrsyrfBQ.0
Central link: https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&fromchange=5140fba12e4a4c58953227b0ba20efbefbf8086b&searchStr=windows%2C10%2Cx64%2C2004%2Cwebrender%2Cshippable%2Copt%2Cmochitests%2Cwith%2Cconditioned%2Cprofile%2Ctest-windows10-64-2004-shippable-qr%2Fopt-mochitest-plain-condprof%2C1&tochange=8cbe8b85fe01a202b6b6d65a279a0b7ab41fface&selectedTaskRun=VC0GiY2zS4u0CLEqqGZ4Ew.0

Comment 19

[Cosmin Sabou [:CosminS]]

This is now fixed with the latest run of tests: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&searchStr=condprof&fromchange=01eac14dcf7c2091c9418c0c419e525ad9f1606d&selectedTaskRun=e4-Cu2doSJKfoGxCeRiZTg.0
Retriggers also were green. Must've been an infra issue.

Comment 20

[Intermittent Failures Robot]

60 failures in 65 pushes (0.923 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 44
• mozilla-central: 16

Platform and build breakdown:

• windows10-64-2004-qr: 28
  • opt: 28
• windows10-64-2004-shippable-qr: 32
  • opt: 32

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1773889&startday=2022-07-02&endday=2022-07-02&tree=all

Comment 21

[Intermittent Failures Robot]

74 failures in 3312 pushes (0.022 failures/push) were associated with this bug in the last 7 days.

This is the #8 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 58
• mozilla-central: 16

Platform and build breakdown:

• windows10-64-2004-qr: 42
  • opt: 42
• windows10-64-2004-shippable-qr: 32
  • opt: 32

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1773889&startday=2022-06-27&endday=2022-07-03&tree=all

Comment 22

[Joel Maher ( :jmaher ) (UTC -8)]

actually this is a different error- I am testing some stuff on try to see if I can reproduce it and if my fix works.

Comment 23

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1773889 - handing another exception edge case with conditioned profiles. r=sparky!

Comment 24

[BugBot [:suhaib / :marco/ :calixte]]

A patch has been attached on this bug, which was already closed. Filing a separate bug will ensure better tracking. If this was not by mistake and further action is needed, please alert the appropriate party.

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/067434790830
handing another exception edge case with conditioned profiles. r=sparky

Comment 26

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/067434790830