1806134 - docker-image-updatebot-3rdp task is failing

Status: RESOLVED FIXED

(Developer Infrastructure :: Mach Vendor & Updatebot, defect)

Description

[Marco Castelluccio [:marco]]

Attachment: live_backing.log

Landing https://phabricator.services.mozilla.com/D162661 triggered the task, and it failed: https://treeherder.mozilla.org/jobs?repo=autoland&revision=6c277442d5aa1e4fe699f37061389295a2d44e3d&selectedTaskRun=GmvGCoYkRayJ5Pzqcgvk0A.0.

I'm attaching the raw log for posterity.

Comment 1

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

I.... have no idea what could be happening. It's failing on poetry install like maybe it can't contact a server to download a package from? But there are no logs. I guess I could maybe add a -v flag if it doesn't fix itself?

Comment 3

[Marco Castelluccio [:marco]]

(In reply to Tom Ritter [:tjr] from comment #1)

I.... have no idea what could be happening. It's failing on poetry install like maybe it can't contact a server to download a package from? But there are no logs. I guess I could maybe add a -v flag if it doesn't fix itself?

Yeah that would help, it looks like it is still happening.

Comment 4

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

Attachment: Bug 1806134: Add verbose flags for debugging r?marco

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/76ec3da6cab6
Add verbose flags for debugging r=marco DONTBUILD

Comment 6

[Cristian Tuns]

If the patch is supposed to fixe the problem, it's not working: https://treeherder.mozilla.org/logviewer?job_id=399952864&repo=autoland

Comment 7

[Intermittent Failures Robot]

17 failures in 618 pushes (0.028 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 17

Platform and build breakdown:

• taskcluster-images: 17
  • opt: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1806134&startday=2022-12-16&endday=2022-12-16&tree=all

Comment 8

[Marco Castelluccio [:marco]]

The patch was trying to add more logging to help figure out the problem, but apparently it didn't help at all, the log is as succinct as it was before. Maybe we could try "-vvv" for debug output?

CCing ahal to see if he has any thoughts.

Comment 9

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/76ec3da6cab6

Comment 10

[Intermittent Failures Robot]

18 failures in 142 pushes (0.127 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• mozilla-central: 2

Platform and build breakdown:

• taskcluster-images: 18
  • opt: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1806134&startday=2022-12-17&endday=2022-12-17&tree=all

Comment 11

[Intermittent Failures Robot]

61 failures in 3558 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 44
• mozilla-central: 17

Platform and build breakdown:

• taskcluster-images: 61
  • opt: 61

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1806134&startday=2022-12-12&endday=2022-12-18&tree=all

Comment 12

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

I am still not sure. I used an interactive task in GCP (external IP was 34.168.214.146) and was able to poetry install everything fine. I also built the Docker image locally and poetry worked fine here too.

Comment 13

[Intermittent Failures Robot]

67 failures in 631 pushes (0.106 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 60
• mozilla-central: 6
• toolchains: 1

Platform and build breakdown:

• taskcluster-images: 67
  • opt: 67

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1806134&startday=2022-12-19&endday=2022-12-19&tree=all

Comment 14

[Marco Castelluccio [:marco]]

Julien, Andrew, could you help?

Comment 15

[Julien Cristau [:jcristau] (back April 22)]

I'm also unable to reproduce the poetry install failure locally. I don't know anything about poetry though so no idea why it would fail in automation.

Comment 16

[Andrew Halberstadt [:ahal]]

The error message is coming out of kaniko:
https://github.com/GoogleContainerTools/kaniko/blob/146ec6a9cd6f87b4a12e8119ded575d5edca35ac/pkg/executor/build.go#L370

I'm not even sure if poetry install is the culprit or not given the lack of an error coming out of it.. Maybe it's just taking too long and hitting some timeout in kaniko? Though poetry -vv is pretty verbose and should produce output pretty much immediately after that "Package operations" log.

Looks like we're pinned to kaniko v1.0.0, whereas the latest is v1.9.1 (13 releases later). Maybe we should try upgrading this in the image builder image.

• /usr/local/bin/poetry install -vv

One thing to try would be adding --no-ansi here in case there's something in the rich output that is tripping up kaniko's command executor.

• python3 -m pip install poetry

Another thing to try would be to pin poetry to whatever version was installed the last time this ran.

Comment 17

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

I'm trying these ideas (no ansi and the last poetry version) in holly - https://treeherder.mozilla.org/jobs?repo=holly

Comment 18

[Julien Cristau [:jcristau] (back April 22)]

I think poetry install is the culprit, because I tried adding more commands after it to setup.sh in https://hg.mozilla.org/try/rev/dc82d15aeb5f93d70c0df5bc5fd5a94a8dc0e37e and those commands didn't run.

Comment 19

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

Attachment: Bug 1806134: Use --no-ansi with Poetry to avoid an unexpected error r?marco

Also, pin poetry at the same time

Comment 20

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

It looks like --no-ansi worked. I also pinned it to the last known good revision (just in case that was the problem also.) I'm out today so please land this once (someone/anyone) approves it.

Comment 21

[June Wilde (she/her) [:jewilde]]

I've been able to reproduce this in try so i've been trying to debug this for a bit this morning and it looks like we're hitting this issue in poetry 1.3.1.
https://github.com/python-poetry/poetry/issues/7184

Confirmed adding -no-ansi to the poetry install command solves the issue in try as well, although pinning to anything pre 1.3.1 should solve the issue as well.
https://treeherder.mozilla.org/jobs?repo=try&revision=f267f1f8d7b2a73ee6fe3464cdfce6f8979b3dbe

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d0ea6be8e3d0
Use --no-ansi with Poetry to avoid an unexpected error r=marco

Comment 23

[Intermittent Failures Robot]

52 failures in 731 pushes (0.071 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 38
• mozilla-central: 14

Platform and build breakdown:

• taskcluster-images: 52
  • opt: 52

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1806134&startday=2022-12-20&endday=2022-12-20&tree=all

Comment 24

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/d0ea6be8e3d0

Comment 25

[Intermittent Failures Robot]

129 failures in 2999 pushes (0.043 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 98
• mozilla-central: 28
• toolchains: 1
• try: 2

Platform and build breakdown:

• taskcluster-images: 129
  • opt: 129

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1806134&startday=2022-12-19&endday=2022-12-25&tree=all