1685285 - Permanent staging release TEST-UNEXPECTED-FAIL: no partial update found for https://stage.balrog.nonprod.cloudops.mozgcp.net/update/3/Firefox/85.0/20201214185926/Linux_x86_64-gcc3/en-CA/beta-localtest/default/default/default/update.xml?force=1

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=325960939&repo=try
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/MSpCJOvaTMCInQX70oELWw/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-01-06T09:57:03.287Z] calling QuitProgressUI
[task 2021-01-06T09:57:04.427Z] Comparing source/firefox with target/firefox...
[task 2021-01-06T09:57:04.427Z] No differences found
[task 2021-01-06T09:57:04.427Z] Using  https://stage.balrog.nonprod.cloudops.mozgcp.net/update/3/Firefox/85.0/20201214185926/Linux_x86_64-gcc3/en-CA/beta-localtest/default/default/default/update.xml?force=1
[task 2021-01-06T09:57:04.429Z] Retrieving 'https://stage.balrog.nonprod.cloudops.mozgcp.net/update/3/Firefox/85.0/20201214185926/Linux_x86_64-gcc3/en-CA/beta-localtest/default/default/default/update.xml?force=1' from cache...
[task 2021-01-06T09:57:04.433Z] Got this response:
[task 2021-01-06T09:57:04.433Z] <?xml version="1.0"?>
[task 2021-01-06T09:57:04.433Z] <updates>
[task 2021-01-06T09:57:04.433Z]     <update actions="showURL" appVersion="86.0" buildID="20210105105446" detailsURL="https://www.mozilla.org/en-CA/firefox/86.0/releasenotes/" displayVersion="86.0 Beta 1" openURL="https://www.mozilla.org/en-CA/firefox/86.0beta/whatsnew/?oldversion=%OLD_VERSION%" type="minor">
[task 2021-01-06T09:57:04.433Z]         <patch type="complete" URL="https://ftp.stage.mozaws.net/pub/firefox/candidates/86.0b1-candidates/build1/update/linux-x86_64/en-CA/firefox-86.0b1.complete.mar" hashFunction="sha512" hashValue="6f8ffed3c95c5911190da1e4926d401668318adbd7c5f4753b8d2baefcea12d39b1ffddc781dc9ea0d1a89a1bc3d5d6e21203119387146cdb00277bcac9e4daf" size="60400461"/>
[task 2021-01-06T09:57:04.433Z]     </update>
[task 2021-01-06T09:57:04.435Z] </updates>
[task 2021-01-06T09:57:04.435Z] 
[task 2021-01-06T09:57:04.440Z] TEST-UNEXPECTED-FAIL: no partial update found for https://stage.balrog.nonprod.cloudops.mozgcp.net/update/3/Firefox/85.0/20201214185926/Linux_x86_64-gcc3/en-CA/beta-localtest/default/default/default/update.xml?force=1
[task 2021-01-06T09:57:04.440Z] TEST-UNEXPECTED-FAIL: [85.0 en-CA partial] download_mars returned non-zero exit code: 1```

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Ben, your 85.0b1 staging release on Monday didn't encounter the issue. Any idea what's wrong with this 86.0b1 staging release?

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #1)

Ben, your 85.0b1 staging release on Monday didn't encounter the issue. Any idea what's wrong with this 86.0b1 staging release?

The buildid being used there doesn't match the one in staging balrog. 85.0b1 en-CA in staging has:

      "en-CA": {
          "appVersion": "85.0",
          "buildID": "20201126101736",

My release used a different set of partials, which is why I didn't hit it.

Comment 3

[Alexandru Michis [:malexandru]]

Geoff, this has appeared once again, could you please take a look at what might've changed?

Comment 4

[Intermittent Failures Robot]

162 failures in 697 pushes (0.232 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• try: 162

Platform and build breakdown:

• linux64-shippable: 16
  • opt: 16
• linux64-devedition: 5
  • opt: 5
• linux32-devedition: 5
  • opt: 5
• linux32-shippable: 16
  • opt: 16
• windows2012-aarch64-devedition: 5
  • opt: 5
• osx-cross-devedition: 27
  • opt: 27
• windows2012-32-devedition: 5
  • opt: 5
• windows2012-aarch64-shippable: 16
  • opt: 16
• windows2012-64-devedition: 5
  • opt: 5
• windows2012-32-shippable: 16
  • opt: 16
• windows2012-64-shippable: 16
  • opt: 16
• osx-shippable: 30
  • opt: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1685285&startday=2022-01-25&endday=2022-01-25&tree=all

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

That got triggered because I had manually requested the release graph for yesterday's central-as-beta simulations.

Comment 6

[Intermittent Failures Robot]

162 failures in 3445 pushes (0.047 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• try: 162

Platform and build breakdown:

• linux64-shippable: 16
  • opt: 16
• linux64-devedition: 5
  • opt: 5
• linux32-devedition: 5
  • opt: 5
• linux32-shippable: 16
  • opt: 16
• windows2012-aarch64-devedition: 5
  • opt: 5
• osx-cross-devedition: 27
  • opt: 27
• windows2012-32-devedition: 5
  • opt: 5
• windows2012-aarch64-shippable: 16
  • opt: 16
• windows2012-64-devedition: 5
  • opt: 5
• windows2012-32-shippable: 16
  • opt: 16
• windows2012-64-shippable: 16
  • opt: 16
• osx-shippable: 30
  • opt: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1685285&startday=2022-01-24&endday=2022-01-30&tree=all

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This is a permanent failure for release graphs added to central-as-beta simulations. Can this be resolved or the testing setup be modified to suppress this?

Comment 8

[Julien Cristau [:jcristau]]

The easiest might be to turn off partials for these graphs (when submitting the release from shipit staging). Would that be acceptable?

We'd lose a bit of coverage for partials generation and downstream tasks, but we know the automatic selection of partials in stage is not reliable, so if we don't want failing UV tasks then no partials might be an ok tradeoff.