1569091 - ModuleNotFoundError: No module named 'encodings'

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Windows OS, defect)

Description

[Alexandru Michis [:malexandru]]

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=258083822&repo=mozilla-central&lineNumber=30

https://taskcluster-artifacts.net/ZlEEfNuBSu2UP_NC6Jciwg/0/public/logs/live_backing.log

[taskcluster 2019-07-24T12:30:17.928Z] [mounts] Creating directory C:\Users\task_1563970793 with permissions 0700
[taskcluster 2019-07-24T12:30:17.928Z] [mounts] Copying C:\downloads\P_kvGa-HTW6K1pRJD3hYWQ to C:\Users\task_1563970793\run-task
[taskcluster 2019-07-24T12:30:18.318Z] Executing command 0: C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\raptor_script.py --cfg mozharness\configs\raptor\windows_config.py --test=raptor-motionmark-htmlsuite --download-symbols ondemand --test=raptor-motionmark-htmlsuite
C:\Users\task_1563970793>C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\raptor_script.py --cfg mozharness\configs\raptor\windows_config.py --test=raptor-motionmark-htmlsuite --download-symbols ondemand --test=raptor-motionmark-htmlsuite
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'
Current thread 0x0000166c (most recent call first):
[taskcluster 2019-07-24T12:30:19.302Z] Exit Code: 3221226505
[taskcluster 2019-07-24T12:30:19.302Z] User Time: 0s
[taskcluster 2019-07-24T12:30:19.302Z] Kernel Time: 46.875ms
[taskcluster 2019-07-24T12:30:19.302Z] Wall Time: 983.2508ms
[taskcluster 2019-07-24T12:30:19.302Z] Result: FAILED
[taskcluster 2019-07-24T12:30:19.305Z] === Task Finished ===
[taskcluster 2019-07-24T12:30:19.305Z] Task Duration: 987.2534ms
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory 'C:\Users\task_1563970793\logs'", reason "file-missing-on-worker" and expiry 2020-07-23T09:57:22.375Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'C:\Users\task_1563970793\logs'
[taskcluster:error] Uploading error artifact public/test_info from file build/blobber_upload_dir with message "Could not read directory 'C:\Users\task_1563970793\build\blobber_upload_dir'", reason "file-missing-on-worker" and expiry 2020-07-23T09:57:22.375Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'C:\Users\task_1563970793\build\blobber_upload_dir'
[taskcluster 2019-07-24T12:30:20.830Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/ZlEEfNuBSu2UP_NC6Jciwg/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2020-07-23T09:57:22.375Z
[taskcluster:error] exit status 3221226505
[taskcluster:error] file-missing-on-worker: Could not read directory 'C:\Users\task_1563970793\logs'
[taskcluster:error] file-missing-on-worker: Could not read directory 'C:\Users\task_1563970793\build\blobber_upload_dir'

Comment 1

[Alexandru Michis [:malexandru]]

These failures seem to start in this push: https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=5585edba8fdb83267722e62241952272c8e24a8f

This might be caused by Bug 1565130 or Bug 1525373, could you please take a look at it?

Comment 2

[Mike Hommey [:glandium]]

Clearly bug 1525373. Yet another set of workers with a broken python 3 install... (same error as https://bugzilla.mozilla.org/show_bug.cgi?id=1545339#c3 )

Comment 3

[Intermittent Failures Robot]

37 failures in 954 pushes (0.039 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 37

Platform breakdown:

• windows10-64-ref-hw-2017: 37

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1569091&startday=2019-07-25&endday=2019-07-25&tree=all

Comment 4

[Rob Thijssen [:grenade (EET/UTC+0300)]]

failure appears to be on releng-hardware/gecko-t-win10-64-ref-hw which i believe uses the new ronin puppet manifests. passing to mark who knows this deployment platform.

Comment 5

[Mark Cornmesser [:markco] OOO 2024/04/15]

These workers are getting Pyhton 3 installed by mozilla-build version 3.2. Which I think is the same for all of the Windows x64 workers. I have contacted Bitbar, who hosts these workers, and ask them to verify that Python 3 has been installed on theses workers.

Comment 6

[Mark Cornmesser [:markco] OOO 2024/04/15]

Bitbar verified that Python3 is installed, and is consistent with the gecko-t-win10-64-hw workers. However, on the hw workers I am unable to import the encoding module, but one the gecko-t-win10-64 workers the encoding module is importable. All the workers are using mozilla-build 3.2.

Rob: I looked through the OCC gecko-t-win10-64 manifest, and I am not seeing any additional Python3 worker besides "DeleteMozillaBuildPython3PythonExe" . Which is addressed in Ronin Puppet. In OCC is there another place where 10-64 workers are having additional modules copied locally?

Comment 7

[Intermittent Failures Robot]

20 failures in 777 pushes (0.026 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 20

Platform breakdown:

• windows10-64-ref-hw-2017: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1569091&startday=2019-07-26&endday=2019-07-26&tree=all

Comment 8

[Intermittent Failures Robot]

89 failures in 4608 pushes (0.019 failures/push) were associated with this bug in the last 7 days.

This is the #11 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 89

Platform breakdown:

• windows10-64-ref-hw-2017: 89

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1569091&startday=2019-07-22&endday=2019-07-28&tree=all

Comment 9

[Rob Thijssen [:grenade (EET/UTC+0300)]]

(In reply to Mark Cornmesser [:markco] from comment #6)

is there another place where 10-64 workers are having additional modules copied locally?

I don't think so.

Comment 10

[Intermittent Failures Robot]

25 failures in 866 pushes (0.029 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 21
• try: 4

Platform breakdown:

• windows10-64-ref-hw-2017: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1569091&startday=2019-08-02&endday=2019-08-02&tree=all

Comment 11

[Intermittent Failures Robot]

21 failures in 231 pushes (0.091 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 21

Platform breakdown:

• windows10-64-ref-hw-2017: 21

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1569091&startday=2019-08-03&endday=2019-08-03&tree=all

Comment 12

[Intermittent Failures Robot]

94 failures in 4439 pushes (0.021 failures/push) were associated with this bug in the last 7 days.

This is the #12 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 90
• try: 4

Platform breakdown:

• windows10-64-ref-hw-2017: 94

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1569091&startday=2019-07-29&endday=2019-08-04&tree=all

Comment 13

[Bogdan Tara[:bogdan_tara | bogdant]]

There have been 91 occurrences in the last 7 days, on windows10-64-ref-hw-2017 opt.

Recent failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=260024534&repo=mozilla-central&lineNumber=34

[taskcluster 2019-08-06T00:26:44.752Z] [mounts] Copying C:\downloads\HlAS4MWiRnqMNglJ7W5CWg to C:\Users\task_1565049514\run-task
[taskcluster 2019-08-06T00:26:45.114Z] Executing command 0: C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\raptor_script.py --cfg mozharness\configs\raptor\windows_config.py --test=raptor-stylebench --download-symbols ondemand --test=raptor-stylebench
C:\Users\task_1565049514>C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\raptor_script.py --cfg mozharness\configs\raptor\windows_config.py --test=raptor-stylebench --download-symbols ondemand --test=raptor-stylebench
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'
Current thread 0x000016bc (most recent call first):
[taskcluster 2019-08-06T00:26:45.549Z] Exit Code: 3221226505
[taskcluster 2019-08-06T00:26:45.549Z] User Time: 0s
[taskcluster 2019-08-06T00:26:45.549Z] Kernel Time: 0s
[taskcluster 2019-08-06T00:26:45.549Z] Wall Time: 433.8667ms
[taskcluster 2019-08-06T00:26:45.549Z] Result: FAILED
[taskcluster 2019-08-06T00:26:45.549Z] === Task Finished ===
[taskcluster 2019-08-06T00:26:45.549Z] Task Duration: 434.8667ms
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory 'C:\Users\task_1565049514\logs'", reason "file-missing-on-worker" and expiry 2020-08-04T22:03:29.522Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'C:\Users\task_1565049514\logs'
[taskcluster:error] Uploading error artifact public/test_info from file build/blobber_upload_dir with message "Could not read directory 'C:\Users\task_1565049514\build\blobber_upload_dir'", reason "file-missing-on-worker" and expiry 2020-08-04T22:03:29.522Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'C:\Users\task_1565049514\build\blobber_upload_dir'
[taskcluster 2019-08-06T00:26:47.043Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/ZagtEtJbTRikHCfO5Mr9hw/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2020-08-04T22:03:29.522Z
[taskcluster:error] exit status 3221226505
[taskcluster:error] file-missing-on-worker: Could not read directory 'C:\Users\task_1565049514\logs'
[taskcluster:error] file-missing-on-worker: Could not read directory 'C:\Users\task_1565049514\build\blobber_upload_dir'

Comment 14

[Intermittent Failures Robot]

15 failures in 858 pushes (0.017 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 15

Platform breakdown:

• windows10-64-ref-hw-2017: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1569091&startday=2019-08-06&endday=2019-08-06&tree=all

Comment 15

[Mark Cornmesser [:markco] OOO 2024/04/15]

I took another look at the configuration this morning. On the datacenter Win 10 hardware I was able to import the encodings module. That datecenter hardware has the same Mozilla-build\python installation and is the same architecture (but different hardware).

Microsoft Windows [Version 10.0.15063]
(c) 2017 Microsoft Corporation. All rights reserved.

C:\mozilla-build\python3>python3.exe
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import encodings

Also the encoding module appears to have the same creation date/time as the other Python 3 modules. Which is the same time as the Mozilla-Build installation.
C:\mozilla-build\python3\Lib>dir
Volume in drive C is Windows
Volume Serial Number is 72C7-8528

Directory of C:\mozilla-build\python3\Lib
...
06/08/2019 03:46 AM <DIR> email
06/08/2019 03:46 AM <DIR> encodings
06/08/2019 03:46 AM <DIR> ensurepip
...

I think my initial test, mentioned in comment 6, was missing the "s" on encodings.

I currently have a request in to Bitbar to run the same import on the ref hardware and take a screenshot. I will attach the screenshot once they get back to me. If it is importable on the ref hardware I would suspect something in the test environment is causing the issue.

Comment 16

[Mark Cornmesser [:markco] OOO 2024/04/15]

Attachment: encodingsimport.png

This is from Bitbar from one of the reference laptops. Python 3 is able to import encodings.

Comment 17

[Cosmin Sabou [:CosminS]]

Mark, this looks to still be happening: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception&revision=2909b0a1eb06cc34ce0a11544e5e6826aba87c06&searchStr=windows%2C10%2Cx64%2C2017%2Cref%2Chw%2Copt%2Craptor%2Cperformance%2Ctests%2Con%2Cfirefox%2Ctest-windows10-64-ref-hw-2017%2Fopt-raptor-tp6-3-firefox-e10s%2Crap%28tp6-3%29&selectedJob=260890066

Also, all failures in the last 7 days are on acer-007, maybe you could do smthg about that machine. https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-08-03&endday=2019-08-10&tree=trunk&bug=1569091

Comment 18

[Intermittent Failures Robot]

77 failures in 4666 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #15 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 77

Platform breakdown:

• windows10-64-ref-hw-2017: 77

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1569091&startday=2019-08-05&endday=2019-08-11&tree=all

Comment 19

[Mark Cornmesser [:markco] OOO 2024/04/15]

I have requested Bitbar to reset and redploy acer-007.

Comment 20

[Mark Cornmesser [:markco] OOO 2024/04/15]

Acer-007 has been re-installed.

Comment 21

[Intermittent Failures Robot]

3 failures in 4839 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 3

Platform breakdown:

• windows10-64-ref-hw-2017: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1569091&startday=2019-08-12&endday=2019-08-18&tree=all

Comment 22

[Mark Cornmesser [:markco] OOO 2024/04/15]

Closing this bug. This was specific to one node that has been redaployed, and the error has not reoccurred since 2019-08-12.