1874758 - Upload-generated-sources build is failing but still green: "ImportError: cannot import name 'Mapping' from 'collections'"

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Markus Stange [:mstange]]

https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=upload-generated-sources

Example log: https://firefoxci.taskcluster-artifacts.net/BqetOBrxRmm0X9VQ80_unw/0/public/logs/live_backing.log

The job is failing halfway through, with ImportError: cannot import name 'Mapping' from 'collections'

However, it is still reporting its status as success, so the treeherder jobs are green and nobody has noticed the failure.

The first mozilla-central push with this failure appears to be https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=shippable%2Cupload-generated-sources&revision=d6f61c448b906c1e68cdc66920d227f008cc2db9

This merge includes bug 1866976, so maybe it's a regression from that? We don't run upload-generated-sources on autoland so I don't know if I can get a smaller regression range.

[...]
[task 2024-01-08T11:22:31.969Z] INFO - MainThread - Queueing "devtools/client/debugger/src/workers/search/node.stub"
[task 2024-01-08T11:22:31.969Z] INFO - MainThread - Queueing "devtools/shared/webconsole/reserved-js-words.js"
[task 2024-01-08T11:22:31.970Z] INFO - MainThread - Queueing "dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast/index.js"
[task 2024-01-08T11:22:31.970Z] INFO - MainThread - Queueing "dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast/setInScopeLines.js"
[task 2024-01-08T11:22:31.970Z] INFO - MainThread - Queueing "dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/breakpoints/breakpointPositions.js"
[task 2024-01-08T11:22:31.970Z] INFO - MainThread - Queueing "dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/breakpoints/index.js"
[task 2024-01-08T11:22:31.971Z] INFO - MainThread - Queueing "dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/breakpoints/modify.js"
[task 2024-01-08T11:22:31.978Z] ERROR - Thread-1 (upload_worker) - Thread encountered exception:
[task 2024-01-08T11:22:31.978Z] Traceback (most recent call last):
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/checkouts/gecko/build/upload_generated_sources.py", line 65, in upload_worker
[task 2024-01-08T11:22:31.978Z]     import boto3
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/boto3/__init__.py", line 16, in <module>
[task 2024-01-08T11:22:31.978Z]     from boto3.session import Session
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/boto3/session.py", line 17, in <module>
[task 2024-01-08T11:22:31.978Z]     import botocore.session
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/botocore/session.py", line 25, in <module>
[task 2024-01-08T11:22:31.978Z]     import botocore.configloader
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/botocore/configloader.py", line 19, in <module>
[task 2024-01-08T11:22:31.978Z]     from botocore.compat import six
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/botocore/compat.py", line 23, in <module>
[task 2024-01-08T11:22:31.978Z]     from botocore.exceptions import MD5UnavailableError
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/botocore/exceptions.py", line 15, in <module>
[task 2024-01-08T11:22:31.978Z]     from botocore.vendored.requests.exceptions import ConnectionError
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/botocore/vendored/requests/__init__.py", line 58, in <module>
[task 2024-01-08T11:22:31.978Z]     from . import utils
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/botocore/vendored/requests/utils.py", line 26, in <module>
[task 2024-01-08T11:22:31.978Z]     from .compat import parse_http_list as _parse_list_header
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/botocore/vendored/requests/compat.py", line 7, in <module>
[task 2024-01-08T11:22:31.978Z]     from .packages import chardet
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/botocore/vendored/requests/packages/__init__.py", line 3, in <module>
[task 2024-01-08T11:22:31.978Z]     from . import urllib3
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/botocore/vendored/requests/packages/urllib3/__init__.py", line 10, in <module>
[task 2024-01-08T11:22:31.978Z]     from .connectionpool import (
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 38, in <module>
[task 2024-01-08T11:22:31.978Z]     from .response import HTTPResponse
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/botocore/vendored/requests/packages/urllib3/response.py", line 9, in <module>
[task 2024-01-08T11:22:31.978Z]     from ._collections import HTTPHeaderDict
[task 2024-01-08T11:22:31.978Z]   File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources/lib/python3.11/site-packages/botocore/vendored/requests/packages/urllib3/_collections.py", line 1, in <module>
[task 2024-01-08T11:22:31.978Z]     from collections import Mapping, MutableMapping
[task 2024-01-08T11:22:31.978Z] ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib/python3.11/collections/__init__.py)
[task 2024-01-08T11:22:31.981Z] INFO - MainThread - Queueing "dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/breakpoints/syncBreakpoint.js"
[task 2024-01-08T11:22:31.981Z] ERROR - MainThread - Worker thread encountered exception, exiting...
[task 2024-01-08T11:22:31.984Z] INFO - MainThread - Finished in 0.580s
[task 2024-01-08T11:22:32.015Z] Creating default state directory: /builds/worker/.mozbuild
[task 2024-01-08T11:22:32.015Z] Creating local state directory: /builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69
[task 2024-01-08T11:22:32.015Z] Site not up-to-date reason: "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/common" does not exist
[task 2024-01-08T11:22:32.015Z] Site not up-to-date reason: "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/upload-generated-sources" does not exist
[taskcluster 2024-01-08 11:22:32.333Z] === Task Finished ===
[taskcluster 2024-01-08 11:22:32.333Z] Successful task run with exit code: 0 completed in 301.17 seconds

Comment 1

[Markus Stange [:mstange]]

Some jobs also contain another error:

[task 2024-01-15T18:04:45.084Z] ERROR - Thread-9 (upload_worker) - Thread encountered exception:
[task 2024-01-15T18:04:45.084Z] Traceback (most recent call last):
[task 2024-01-15T18:04:45.084Z]   File "/builds/worker/checkouts/gecko/build/upload_generated_sources.py", line 67, in upload_worker
[task 2024-01-15T18:04:45.084Z]     session = boto3.session.Session(**session_args)
[task 2024-01-15T18:04:45.084Z]               ^^^^^^^^^^^^^
[task 2024-01-15T18:04:45.084Z] AttributeError: module 'boto3' has no attribute 'session'

Comment 2

[Markus Stange [:mstange]]

I noticed this when I was looking at this profile and double-clicked on mozilla::dom::PContentChild::SendInitializeFamily and didn't get source code in the profiler source view.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1866976

Comment 4

[Alex Hochheiden [:ahochheiden]]

(In reply to Markus Stange [:mstange] from comment #0)

[task 2024-01-08T11:22:31.978Z] ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib/python3.11/collections/__init__.py)

I think that's caused by a Python version bump. Mapping just needs to be imported from collections.abc

eg:

from collections.abc import Mapping

Importing from just collections was deprecated in Python 3.9 and removed in Python 3.10.

As for the boto3 error, I'm not sure what the problem is. I will note that the version of boto3 we're using here (1.4.4) is quite old (Released Jan 16, 2017). We use a relatively newer version for docs (1.33.5) which seems to use the same syntax in upload.py. So if that still works, but this doesn't, maybe updating it will resolve the issue?

Comment 5

[Mike Hommey [:glandium]]

Attachment: Bug 1874758 - Avoid errors during upload_generated_sources.py being ignored.

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1874758 - Upgrade boto3 to latest version for upload-generated-sources.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/20b5b5dd0ce0
Avoid errors during upload_generated_sources.py being ignored. r=firefox-build-system-reviewers,ahochheiden
https://hg.mozilla.org/integration/autoland/rev/6ddec14f722f
Upgrade boto3 to latest version for upload-generated-sources. r=firefox-build-system-reviewers,mach-reviewers,ahochheiden

Comment 8

[Mike Hommey [:glandium]]

Attachment: Bug 1874758 - Avoid errors during upload_generated_sources.py being ignored.

Original Revision: https://phabricator.services.mozilla.com/D198639

Comment 9

[Mike Hommey [:glandium]]

Attachment: Bug 1874758 - Upgrade boto3 to latest version for upload-generated-sources.

Original Revision: https://phabricator.services.mozilla.com/D198640

Comment 10

[Phabricator Automation]

Uplift Approval Request

• Risk associated with taking this patch: Low
• String changes made/needed: N/A
• Needs manual QE test: no
• User impact if declined: Missing generated sources for crashes and profiles
• Explanation of risk level: No change to Firefox itself, and the worst that can happen is that it stays broken
• Code covered by automated testing: no
• Is Android affected?: yes
• Fix verified in Nightly: yes
• Steps to reproduce for manual QE testing: N/A

Comment 11

[Mike Hommey [:glandium]]

Attachment: Bug 1874758 - Avoid errors during upload_generated_sources.py being ignored.

Original Revision: https://phabricator.services.mozilla.com/D198639

Comment 12

[Mike Hommey [:glandium]]

Attachment: Bug 1874758 - Upgrade boto3 to latest version for upload-generated-sources.

Original Revision: https://phabricator.services.mozilla.com/D198640

Comment 13

[Phabricator Automation]

Uplift Approval Request

• Needs manual QE test: no
• Fix verified in Nightly: yes
• Is Android affected?: no
• Code covered by automated testing: no
• Steps to reproduce for manual QE testing: N/A
• User impact if declined: Missing generated sources for crashes and profiles
• Explanation of risk level: No change to Firefox itself, and the worst that can happen is that it stays broken
• String changes made/needed: N/A
• Risk associated with taking this patch: Low

Comment 14

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/20b5b5dd0ce0
https://hg.mozilla.org/mozilla-central/rev/6ddec14f722f

Comment 15

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/3ddcea7c8382
https://hg.mozilla.org/releases/mozilla-release/rev/e8892fca5a1b