1845272 - Permafailing Rpk ModuleNotFoundError: No module named 'fluent.runtime'

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=423794833&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/NO5IGz39TJyJGhkEGBOLuQ/runs/2/artifacts/public/logs/live_backing.log

---

[task 2023-07-25T11:23:24.102Z] 11:23:24     INFO -  Successfully installed babel-2.12.1 fluent.runtime-0.4.0 fluent.syntax-0.19.0 pytz-2023.3 typing-extensions-4.7.1
[task 2023-07-25T11:23:24.319Z] 11:23:24     INFO -  Error running mach:
[task 2023-07-25T11:23:24.319Z] 11:23:24     INFO -      mach --log-no-times repackage deb --arch x86 --templates browser/installer/linux/app/debian --version 117.0a1 --build-number 1 --release-product None --release-type nightly --input /builds/worker/fetches/target.tar.bz2 --output /builds/worker/workspace/outputs/target.deb
[task 2023-07-25T11:23:24.320Z] 11:23:24     INFO -  The error occurred in code that was called by the mach command. This is either
[task 2023-07-25T11:23:24.320Z] 11:23:24     INFO -  a bug in the called code itself or in the way that mach is calling it.
[task 2023-07-25T11:23:24.320Z] 11:23:24     INFO -  You can invoke ``./mach busted`` to check if this issue is already on file. If it
[task 2023-07-25T11:23:24.320Z] 11:23:24     INFO -  isn't, please use ``./mach busted file repackage`` to report it. If ``./mach busted`` is
[task 2023-07-25T11:23:24.320Z] 11:23:24     INFO -  misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2023-07-25T11:23:24.320Z] 11:23:24     INFO -  If filing a bug, please include the full output of mach, including this error
[task 2023-07-25T11:23:24.320Z] 11:23:24     INFO -  message.
[task 2023-07-25T11:23:24.320Z] 11:23:24     INFO -  The details of the failure are as follows:
[task 2023-07-25T11:23:24.320Z] 11:23:24     INFO -  ModuleNotFoundError: No module named 'redo'
[task 2023-07-25T11:23:24.320Z] 11:23:24     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/mach_commands.py", line 2330, in repackage_deb
[task 2023-07-25T11:23:24.320Z] 11:23:24     INFO -      from mozbuild.repackaging.deb import repackage_deb
[task 2023-07-25T11:23:24.320Z] 11:23:24     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/repackaging/deb.py", line 22, in <module>
[task 2023-07-25T11:23:24.320Z] 11:23:24     INFO -      from redo import retry
[task 2023-07-25T11:23:24.364Z] 11:23:24     INFO - Return code: 1
[task 2023-07-25T11:23:24.364Z] 11:23:24    ERROR - 1 not in success codes: [0]
[task 2023-07-25T11:23:24.364Z] 11:23:24  WARNING - setting return code to 2
[task 2023-07-25T11:23:24.364Z] 11:23:24    FATAL - Halting on failure while running ['/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/common/bin/python', 'mach', '--log-no-times', 'repackage', 'deb', '--arch', 'x86', '--templates', 'browser/installer/linux/app/debian', '--version', '117.0a1', '--build-number', '1', '--release-product', 'None', '--release-type', 'nightly', '--input', '/builds/worker/fetches/target.tar.bz2', '--output', '/builds/worker/workspace/outputs/target.deb']
[task 2023-07-25T11:23:24.364Z] 11:23:24    FATAL - Running post_fatal callback...
[task 2023-07-25T11:23:24.364Z] 11:23:24    FATAL - Exiting 2
[task 2023-07-25T11:23:24.364Z] 11:23:24     INFO - [mozharness: 2023-07-25 11:23:24.364892Z] Finished repackage step (failed)
[taskcluster 2023-07-25 11:23:33.852Z] === Task Finished ===
[taskcluster 2023-07-25 11:23:33.864Z] Artifact "public/build" not found at "/builds/worker/artifacts/": (HTTP code 404) no such container - Could not find the file /builds/worker/artifacts/ in container 149e6fe5728a7f431e2d8f92dd02cef0a50485d1874c9fee0209386faa10e278 
[taskcluster 2023-07-25 11:23:33.879Z] Artifact "public/build/target.deb" not found at "/builds/worker/workspace/outputs/target.deb": (HTTP code 404) no such container - Could not find the file /builds/worker/workspace/outputs/target.deb in container 149e6fe5728a7f431e2d8f92dd02cef0a50485d1874c9fee0209386faa10e278 
[taskcluster 2023-07-25 11:23:34.149Z] Unsuccessful task run with exit code: 2 completed in 159.159 seconds

Comment 1

[Natalia Csoregi [:nataliaCs]]

Alex, do you have time to check if Bug 1695312 has anything to do with this?
Thank you!

Comment 2

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Yep, that's mine. I will fix.

Comment 3

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1845272 - Add missing module 'redo' to `repackage-deb` site r?#build

This was missed in the dependency rework in D180501.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ebe18262bdd8
Add missing module 'redo' to `repackage-deb` site r=ahal

Comment 5

[Greg Tatum [:gregtatum]]

I was getting the ModuleNotFoundError: No module named 'redo' error on ./mach wpt as well, and applying this patch didn't seem to fix it, although I may have just applied it wrong.

Comment 6

[Thomas Wisniewski [:twisniewski]]

FWIW, I'm also getting the error running ./mach wpt, after applying the patch, clobbering, rebuilding and running.

Comment 7

[Greg Tatum [:gregtatum]]

I rolled back to before Bug 1695312 and mach wpt worked for me.

Comment 8

[Iulian Moraru]

This seems to be happening on perftest with startup-<something> on android-hw-a51-11-0-aarch64-shippable opt

failure log for such jobs

Hi Alex! Not sure if this is something that your patch will cover. If you could please take a look at this ^
Thank you!

Comment 9

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/ebe18262bdd8

Comment 10

[Serban Stanca [:SerbanS]]

Hello Alex! Could you please take another look at this? Now it shifted to No module named 'fluent.runtime', as it can be seen here, and I think it's also caused by your changes.

Thank you!

Comment 11

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

(In reply to Greg Tatum [:gregtatum] from comment #5)

I was getting the ModuleNotFoundError: No module named 'redo' error on ./mach wpt as well, and applying this patch didn't seem to fix it, although I may have just applied it wrong.

That should be fixed by https://phabricator.services.mozilla.com/D184519 from Bug 1845365.

(In reply to Iulian Moraru from comment #8)

This seems to be happening on perftest with startup-<something> on android-hw-a51-11-0-aarch64-shippable opt

failure log for such jobs

Hi Alex! Not sure if this is something that your patch will cover. If you could please take a look at this ^
Thank you!

This one's tricky, but I've almost got something that works.

(In reply to Serban Stanca [:SerbanS] from comment #10)

Hello Alex! Could you please take another look at this? Now it shifted to No module named 'fluent.runtime', as it can be seen here, and I think it's also caused by your changes.

Thank you!

Still working on this one, I think my fix for something else in https://phabricator.services.mozilla.com/D184519 may have caused the regression.

Comment 12

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1845272 - Activate the `common` site virtualenv when calling the `mozperftest/runner.py` script directly (outside of a mach command context) r?#build

Since various dependencies were moved out of the 'mach' site, it itself
is no longer sufficient. Activating the common site solves this
problem, since it has everything that was moved out of the mach site.

Comment 13

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

(In reply to Alex Hochheiden [:ahochheiden] from comment #11)

(In reply to Serban Stanca [:SerbanS] from comment #10)

Hello Alex! Could you please take another look at this? Now it shifted to No module named 'fluent.runtime', as it can be seen here, and I think it's also caused by your changes.

Thank you!

Still working on this one, I think my fix for something else in https://phabricator.services.mozilla.com/D184519 may have caused the regression.

I actually can't reproduce this on current central. I think this was just a failure before my fix in the other bug landed. I had a different error locally, but that's because I was missing a different fix as well. I think it should be good now. This latest patch should resolve the last issue introduced in bug 1695312.

Comment 15

[Noemi Erli[:noemi_erli]]

On the latest merge now we have these failures.
Alex, could you please take a look?

Comment 16

[Ryan VanderMeulen [:RyanVM]]

This is blocking the Nightly release graph, which in turn is blocking all Linux nightlies from being updated at the moment. Presumably this is also going to block Beta's release graph next week. We need a fix or backout ASAP given that we're already in the soft freeze for 117.

Comment 17

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

I believe my current patch should fix those. I'm doing a try push right now to confirm and finding somebody to review.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9682d7c6107b
Activate the `common` site virtualenv when calling the `mozperftest/runner.py` script directly (outside of a mach command context) r=ahal

Comment 20

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/9682d7c6107b

Comment 21

[Intermittent Failures Robot]

19 failures in 704 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 19

Platform and build breakdown:

• android-hw-a51-11-0-aarch64-shippable: 15
  • opt: 15
• linux1804-64-shippable: 2
  • opt: 2
• linux64-shippable: 1
  • opt: 1
• linux32-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1845272&startday=2023-07-27&endday=2023-07-27&tree=all

Comment 22

[Sandor Molnar[:smolnar]]

it is still perma-failling

Failure log -> ModuleNotFoundError: No module named 'fluent.runtime'

Comment 23

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1845272 - Set 'site' early before checking for a `sub_command` `site` r?ahal

This will make it so that if there was a potential subcommand, but it
was not a real subcommand, the base command's site is still passed
through and activated.

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/fe8f67887731
Set 'site' early before checking for a `sub_command` `site` r=ahal a=RyanVM

Comment 25

[Ryan VanderMeulen [:RyanVM]]

Looks like that did the trick, thanks!

Comment 26

[Intermittent Failures Robot]

52 failures in 3403 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

This is the #13 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 52

Platform and build breakdown:

• linux32-shippable: 13
  • opt: 13
• linux64-shippable: 7
  • opt: 7
• android-hw-a51-11-0-aarch64-shippable: 21
  • opt: 21
• linux1804-64-shippable: 11
  • opt: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1845272&startday=2023-07-24&endday=2023-07-30&tree=all