1873359 - Intermittent linux64-snap Sel [taskcluster:error] exit status 1 | After cannot find required base "core22"

Status: RESOLVED FIXED

(Firefox Build System :: Third Party Packaging, defect)

Description

[Treeherder Bug Filer]

Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=442231971&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bNJ4RjJqSL-kit9uPXhymg/runs/0/artifacts/public/logs/live_backing.log

---

[taskcluster 2024-01-05T01:05:19.679Z] Executing command 1: './run-task' -- bash -cx 'export TASKCLUSTER_ROOT_DIR=$PWD && cd $MOZ_FETCHES_DIR/ && ./tests.sh'
[setup 2024-01-05T01:05:19.747Z] run-task started in /home/ubuntu/tasks/task_170441667401445
[setup 2024-01-05T01:05:19.757Z] MOZ_FETCHES_DIR is /home/ubuntu/tasks/task_170441667401445/fetches
[fetches 2024-01-05T01:05:19.757Z] fetching artifacts
[fetches 2024-01-05T01:05:19.758Z] executing ['/usr/bin/python3', '-u', '/home/ubuntu/tasks/task_170441667401445/./fetch-content', 'task-artifacts']
[fetches 2024-01-05T01:05:19.999Z] attempt 1/5
[fetches 2024-01-05T01:05:19.999Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aNwhWKDjQtWNedLYBYYGwQ/artifacts/public/build/firefox.snap to /home/ubuntu/tasks/task_170441667401445/fetches/firefox.snap
[fetches 2024-01-05T01:05:19.999Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aNwhWKDjQtWNedLYBYYGwQ/artifacts/public/build/firefox.snap
[fetches 2024-01-05T01:05:20.000Z] /home/ubuntu/tasks/task_170441667401445/./fetch-content:186: DeprecationWarning: cafile, capath and cadefault are deprecated, use a custom context instead.
[fetches 2024-01-05T01:05:20.000Z]   with urllib.request.urlopen(
[fetches 2024-01-05T01:05:20.001Z] attempt 1/5
[fetches 2024-01-05T01:05:20.001Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aNwhWKDjQtWNedLYBYYGwQ/artifacts/public/build/snap-tests.zip to /home/ubuntu/tasks/task_170441667401445/fetches/snap-tests.zip
[fetches 2024-01-05T01:05:20.002Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aNwhWKDjQtWNedLYBYYGwQ/artifacts/public/build/snap-tests.zip
[fetches 2024-01-05T01:05:22.077Z] https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aNwhWKDjQtWNedLYBYYGwQ/artifacts/public/build/snap-tests.zip resolved to 3100 bytes with sha256 b582e7e7c1d8bdfd1f635fc562f2067c7b6832acf7a2e14b201900bb619b6af4 in 2.075s
[fetches 2024-01-05T01:05:22.078Z] Extracting /home/ubuntu/tasks/task_170441667401445/fetches/snap-tests.zip to /home/ubuntu/tasks/task_170441667401445/fetches
[fetches 2024-01-05T01:05:22.113Z] Archive:  /home/ubuntu/tasks/task_170441667401445/fetches/snap-tests.zip
[fetches 2024-01-05T01:05:22.114Z]   inflating: basic_tests.py          
[fetches 2024-01-05T01:05:22.115Z]   inflating: expectations.json.in    
[fetches 2024-01-05T01:05:22.115Z]   inflating: requirements.txt        
[fetches 2024-01-05T01:05:22.115Z]   inflating: tests.sh                
[fetches 2024-01-05T01:05:22.116Z] /home/ubuntu/tasks/task_170441667401445/fetches/snap-tests.zip extracted in 0.038s
[fetches 2024-01-05T01:05:22.116Z] Removing /home/ubuntu/tasks/task_170441667401445/fetches/snap-tests.zip
[fetches 2024-01-05T01:05:45.423Z] https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aNwhWKDjQtWNedLYBYYGwQ/artifacts/public/build/firefox.snap resolved to 982831104 bytes with sha256 f82a238ee5a56ae2bb76de9db266db88e095165faebe53b155939a52ba6c46ee in 25.424s
[fetches 2024-01-05T01:05:45.424Z] Extracting /home/ubuntu/tasks/task_170441667401445/fetches/firefox.snap to /home/ubuntu/tasks/task_170441667401445/fetches
[fetches 2024-01-05T01:05:45.437Z] PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 22.92402491700001, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-01-05T01:05:45.439Z] finished fetching artifacts
[task 2024-01-05T01:05:45.440Z] executing ['bash', '-cx', 'export TASKCLUSTER_ROOT_DIR=$PWD && cd $MOZ_FETCHES_DIR/ && ./tests.sh']
[task 2024-01-05T01:05:45.444Z] + export TASKCLUSTER_ROOT_DIR=/home/ubuntu/tasks/task_170441667401445
[task 2024-01-05T01:05:45.444Z] + TASKCLUSTER_ROOT_DIR=/home/ubuntu/tasks/task_170441667401445
[task 2024-01-05T01:05:45.444Z] + cd /home/ubuntu/tasks/task_170441667401445/fetches/
[task 2024-01-05T01:05:45.444Z] + ./tests.sh
[task 2024-01-05T01:05:45.447Z] + pwd
[task 2024-01-05T01:05:45.447Z] /home/ubuntu/tasks/task_170441667401445/fetches
[task 2024-01-05T01:05:45.447Z] + ARTIFACT_DIR=/home/ubuntu/tasks/task_170441667401445/builds/worker/artifacts/
[task 2024-01-05T01:05:45.447Z] + mkdir -p /home/ubuntu/tasks/task_170441667401445/builds/worker/artifacts/
[task 2024-01-05T01:05:45.451Z] + sudo snap refresh
[task 2024-01-05T01:06:12.855Z] 2024-01-05T01:06:05Z INFO Waiting for automatic snapd restart...
[task 2024-01-05T01:07:02.127Z] error: cannot perform the following tasks:
[task 2024-01-05T01:07:02.127Z] - Mount snap "firefox" (3600) (cannot find required base "core22")
[taskcluster 2024-01-05T01:07:02.144Z]    Exit Code: 1
[taskcluster 2024-01-05T01:07:02.144Z]    User Time: 10.862739s
[taskcluster 2024-01-05T01:07:02.144Z]  Kernel Time: 8.634314s
[taskcluster 2024-01-05T01:07:02.144Z]    Wall Time: 1m42.464563917s
[taskcluster 2024-01-05T01:07:02.144Z]       Result: FAILED
[taskcluster 2024-01-05T01:07:02.144Z] === Task Finished ===
[taskcluster 2024-01-05T01:07:02.144Z] Task Duration: 1m42.470506504s
[taskcluster 2024-01-05T01:07:02.183Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2024-04-04T00:04:10.354Z
[taskcluster:error] exit status 1

Comment 1

[Serban Stanca [:SerbanS]]

Hello Gerard! This seems to be permafailing from 4th of January, but also not in all cases, as it can be seen here. I suspect it could be an upstream issue. Could you please take a look at this?

Thank you!

Comment 2

[:gerard-majax]

This looks like outside of our reach ...

Comment 3

[:gerard-majax]

This looks intermittent, to make it even better.

Comment 4

[:gerard-majax]

Snap store issues ?

Comment 5

[Intermittent Failures Robot]

8 failures in 2274 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8

Platform and build breakdown:

• linux64-snap-stable: 4
  • debug: 2
  • opt: 2
• linux64-snap-beta: 3
  • debug: 2
  • opt: 1
• linux64-snap-esr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1873359&startday=2024-01-01&endday=2024-01-07&tree=all

Comment 6

[Intermittent Failures Robot]

22 failures in 3217 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 22

Platform and build breakdown:

• linux64-snap-beta: 7
  • opt: 4
  • debug: 3
• linux64-snap-nightly: 3
  • opt: 3
• linux64-snap-stable: 5
  • opt: 3
  • debug: 2
• linux64-snap-esr: 7
  • opt: 3
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1873359&startday=2024-01-08&endday=2024-01-14&tree=all

Comment 7

[:gerard-majax]

Andrew, according to https://bugs.launchpad.net/snapd/+bug/2048104/comments/13 there should be no more problem if we start for sure with snapd 2.61.1. Could you rebuild / update the current images, and ensuring before making them available that snap info snapd reports 2.61.1 ?

Comment 8

[Intermittent Failures Robot]

12 failures in 3599 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 12

Platform and build breakdown:

• linux64-snap-esr: 5
  • opt: 5
• linux64-snap-stable: 4
  • opt: 3
  • debug: 1
• linux64-snap-nightly: 1
  • opt: 1
• linux64-snap-beta: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1873359&startday=2024-01-15&endday=2024-01-21&tree=all

Comment 9

[Iulian Moraru]

This started again to permafail on some builds, example from the latest push from central and it's permafailing where it was green before as it can be seen here.

Comment 10

[Intermittent Failures Robot]

16 failures in 789 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 16

Platform and build breakdown:

• linux64-snap-beta: 5
  • opt: 4
  • debug: 1
• linux64-snap-stable: 6
  • opt: 6
• linux64-snap-nightly: 3
  • debug: 3
• linux64-snap-esr: 2
  • debug: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1873359&startday=2024-01-23&endday=2024-01-23&tree=all

Comment 11

[Cosmin Sabou [:CosminS]]

Eventually turned green after a long list of retriggers.

Comment 12

[Intermittent Failures Robot]

17 failures in 845 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 17

Platform and build breakdown:

• linux64-snap-beta: 8
  • debug: 6
  • opt: 2
• linux64-snap-esr: 3
  • debug: 1
  • opt: 2
• linux64-snap-nightly: 6
  • debug: 5
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1873359&startday=2024-01-24&endday=2024-01-24&tree=all

Comment 13

[:gerard-majax]

Latest answer from upstream https://bugs.launchpad.net/snapd/+bug/2048104/comments/15 confirms that as soon as we boot a VM running already snapd 2.61.1 we should be safe. It is my understanding people in charge of our images might not have a few cycles to update it, and the new wayland support is coming (good news), so I will try and see if we can have a snap refresh step that is more reliable to this specific issue.

Comment 14

[Intermittent Failures Robot]

59 failures in 4034 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

This is the #19 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 59

Platform and build breakdown:

• linux64-snap-beta: 22
  • debug: 11
  • opt: 11
• linux64-snap-nightly: 17
  • debug: 15
  • opt: 2
• linux64-snap-esr: 9
  • debug: 4
  • opt: 5
• linux64-snap-stable: 11
  • opt: 7
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1873359&startday=2024-01-22&endday=2024-01-28&tree=all

Comment 15

[:gerard-majax]

Attachment: Bug 1873359 - Add support for install-from-store for tests r?jcristau!

Comment 16

[:gerard-majax]

Attachment: Bug 1873359 - Allow snap refresh failure r?jcristau!

Depends on D199207

Comment 17

[:gerard-majax]

From https://treeherder.mozilla.org/logviewer?job_id=445034881&repo=try&lineNumber=87-108 it looks like this can work

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3d51860ca492
Add support for install-from-store for tests r=jcristau
https://hg.mozilla.org/integration/autoland/rev/1ac69623ae79
Allow snap refresh failure r=jcristau

Comment 19

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/3d51860ca492
https://hg.mozilla.org/mozilla-central/rev/1ac69623ae79

Comment 20

[Intermittent Failures Robot]

8 failures in 3952 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8

Platform and build breakdown:

• linux64-snap-esr: 2
  • debug: 1
  • opt: 1
• linux64-snap-beta: 3
  • opt: 3
• linux64-snap-nightly: 2
  • opt: 2
• linux64-snap-stable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1873359&startday=2024-01-29&endday=2024-02-04&tree=all