1716860 - Intermittent [tier2] TV File "/opt/worker/tasks/task_162386645497752/build/venv/bin/mozinstall", line 8, in <module> | AttributeError: 'module' object has no attribute 'load'

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: abutkovits [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=342990356&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JEXcXOu4Tz-mYZZBcwGsnQ/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JEXcXOu4Tz-mYZZBcwGsnQ/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

[task 2021-06-16T18:50:03.094Z] 18:50:03     INFO - Reading from file tmpfile_stdout
[task 2021-06-16T18:50:03.095Z] 18:50:03     INFO - Output received:
[task 2021-06-16T18:50:03.095Z] 18:50:03     INFO -  Usage: mozinstall [options] installer
[task 2021-06-16T18:50:03.095Z] 18:50:03     INFO -  Options:
[task 2021-06-16T18:50:03.095Z] 18:50:03     INFO -    -h, --help            show this help message and exit
[task 2021-06-16T18:50:03.095Z] 18:50:03     INFO -    -d DEST, --destination=DEST
[task 2021-06-16T18:50:03.095Z] 18:50:03     INFO -                          Directory to install application into. [default:
[task 2021-06-16T18:50:03.095Z] 18:50:03     INFO -                          "/opt/worker/tasks/task_162386645497752"]
[task 2021-06-16T18:50:03.095Z] 18:50:03     INFO -    --app=APP             Application being installed. [default: firefox]
[task 2021-06-16T18:50:03.095Z] 18:50:03     INFO - mkdir: /opt/worker/tasks/task_162386645497752/build/application
[task 2021-06-16T18:50:03.096Z] 18:50:03     INFO - Getting output from command: ['/opt/worker/tasks/task_162386645497752/build/venv/bin/mozinstall', '/opt/worker/tasks/task_162386645497752/installer.dmg', '--destination', '/opt/worker/tasks/task_162386645497752/build/application']
[task 2021-06-16T18:50:03.096Z] 18:50:03     INFO - Copy/paste: /opt/worker/tasks/task_162386645497752/build/venv/bin/mozinstall /opt/worker/tasks/task_162386645497752/installer.dmg --destination /opt/worker/tasks/task_162386645497752/build/application
[task 2021-06-16T18:50:23.221Z] 18:50:23    ERROR - Errors received:
[task 2021-06-16T18:50:23.221Z] 18:50:23     INFO - Reading from file tmpfile_stderr
[task 2021-06-16T18:50:23.222Z] 18:50:23    ERROR -  Traceback (most recent call last):
[task 2021-06-16T18:50:23.222Z] 18:50:23    ERROR -    File "/opt/worker/tasks/task_162386645497752/build/venv/bin/mozinstall", line 8, in <module>
[task 2021-06-16T18:50:23.222Z] 18:50:23    ERROR -      sys.exit(install_cli())
[task 2021-06-16T18:50:23.222Z] 18:50:23    ERROR -    File "/opt/worker/tasks/task_162386645497752/build/venv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 379, in install_cli
[task 2021-06-16T18:50:23.222Z] 18:50:23    ERROR -      binary = get_binary(install_path, app_name=options.app)
[task 2021-06-16T18:50:23.222Z] 18:50:23    ERROR -    File "/opt/worker/tasks/task_162386645497752/build/venv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 80, in get_binary
[task 2021-06-16T18:50:23.222Z] 18:50:23    ERROR -      path, "Contents/MacOS/", _readPlist(plist)["CFBundleExecutable"]
[task 2021-06-16T18:50:23.222Z] 18:50:23    ERROR -    File "/opt/worker/tasks/task_162386645497752/build/venv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 61, in _readPlist
[task 2021-06-16T18:50:23.222Z] 18:50:23    ERROR -      return plistlib.load(fp)
[task 2021-06-16T18:50:23.222Z] 18:50:23    ERROR -  AttributeError: 'module' object has no attribute 'load'
[task 2021-06-16T18:50:23.222Z] 18:50:23    ERROR - Return code: 1
[task 2021-06-16T18:50:23.222Z] 18:50:23  WARNING - setting return code to 3
[task 2021-06-16T18:50:23.223Z] 18:50:23    FATAL - Halting on failure while running ['/opt/worker/tasks/task_162386645497752/build/venv/bin/mozinstall', '/opt/worker/tasks/task_162386645497752/installer.dmg', '--destination', '/opt/worker/tasks/task_162386645497752/build/application']
[task 2021-06-16T18:50:23.223Z] 18:50:23    FATAL - Running post_fatal callback...
[task 2021-06-16T18:50:23.223Z] 18:50:23    FATAL - Exiting 3
[task 2021-06-16T18:50:23.223Z] 18:50:23     INFO - Running post-action listener: _resource_record_post_action
[task 2021-06-16T18:50:23.223Z] 18:50:23     INFO - [mozharness: 2021-06-16 18:50:23.223296Z] Finished install step (failed)
[task 2021-06-16T18:50:23.223Z] 18:50:23     INFO - Running post-run listener: _resource_record_post_run
[task 2021-06-16T18:50:23.328Z] 18:50:23     INFO - instance_metadata.json not found; unable to determine instance type

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::Task Configuration' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[Andreea Pavel [:apavel]]

Hi, Eitan. This showed up when bug 1691813 landed and it fails only on TV so far. Please take a look when you have time.
Thanks!

Comment 3

[Intermittent Failures Robot]

28 failures in 574 pushes (0.049 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 28

Platform and build breakdown:

• macosx1015-64-qr: 28
  • opt: 14
  • debug: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1716860&startday=2021-06-16&endday=2021-06-16&tree=all

Comment 4

[Natalia Csoregi [:nataliaCs]]

[task 2021-06-16T19:00:04.494Z] 19:00:04    ERROR - Errors received:
[task 2021-06-16T19:00:04.494Z] 19:00:04     INFO - Reading from file tmpfile_stderr
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -  Traceback (most recent call last):
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -    File "/opt/worker/tasks/task_162386618319829/build/venv/bin/mozinstall", line 8, in <module>
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -      sys.exit(install_cli())
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -    File "/opt/worker/tasks/task_162386618319829/build/venv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 379, in install_cli
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -      binary = get_binary(install_path, app_name=options.app)
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -    File "/opt/worker/tasks/task_162386618319829/build/venv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 80, in get_binary
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -      path, "Contents/MacOS/", _readPlist(plist)["CFBundleExecutable"]
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -    File "/opt/worker/tasks/task_162386618319829/build/venv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 61, in _readPlist
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -      return plistlib.load(fp)
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -  AttributeError: 'module' object has no attribute 'load'
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR - Return code: 1

Mitchell, AttributeError: 'module' object has no attribute 'load' seems to be an error within testing/mozbase/mozinstall/mozinstall/mozinstall.py so we might have Bug 1714688 as the root cause?

Could you or Surajeet take a look at this?

Thank you.

Comment 5

[Surajeet Das]

(In reply to Natalia Csoregi [:nataliaCs] from comment #4)

[task 2021-06-16T19:00:04.494Z] 19:00:04    ERROR - Errors received:
[task 2021-06-16T19:00:04.494Z] 19:00:04     INFO - Reading from file tmpfile_stderr
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -  Traceback (most recent call last):
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -    File "/opt/worker/tasks/task_162386618319829/build/venv/bin/mozinstall", line 8, in <module>
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -      sys.exit(install_cli())
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -    File "/opt/worker/tasks/task_162386618319829/build/venv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 379, in install_cli
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -      binary = get_binary(install_path, app_name=options.app)
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -    File "/opt/worker/tasks/task_162386618319829/build/venv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 80, in get_binary
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -      path, "Contents/MacOS/", _readPlist(plist)["CFBundleExecutable"]
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -    File "/opt/worker/tasks/task_162386618319829/build/venv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 61, in _readPlist
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -      return plistlib.load(fp)
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR -  AttributeError: 'module' object has no attribute 'load'
[task 2021-06-16T19:00:04.495Z] 19:00:04    ERROR - Return code: 1

Mitchell, AttributeError: 'module' object has no attribute 'load' seems to be an error within testing/mozbase/mozinstall/mozinstall/mozinstall.py so we might have Bug 1714688 as the root cause?

Could you or Surajeet take a look at this?

Thank you.

Python2 is being used I suppose, which is causing this.

Comment 6

[Surajeet Das]

The patch regarding removing the Python2 support for mozinstall.py might be causing this issue, i am not sure. Revoking this might be one thing to do .

Comment 7

[Mike Hommey [:glandium]]

[taskcluster 2021-06-16T18:48:00.140Z] Executing command 1: /usr/local/bin/python3 run-task -- /usr/local/bin/python2 -u 'mozharness/scripts/desktop_unittest.py' --cfg 'mozharness/configs/unittests/mac_unittest.py' --verify '--setpref=media.peerconnection.mtransport_process=false' '--setpref=network.process.enabled=false' --enable-webrender '--setpref=layers.d3d11.enable-blacklist=false' --download-symbols ondemand

These tasks are still running python2.

Comment 8

[Mike Hommey [:glandium]]

They are covered by bug 1714406.

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

Yeah, this is caused by 74fd8bb17de0.
Let's back it out until these tasks are running with Python 3.
(This was my mistake, didn't see that mozinstall.py was used in a Python 2 task.)
Thanks!

Comment 10

[Mitchell Hentges [:mhentges] 🦀]

(NI-ing Natalia, but if there's a specific individual I should be NI-ing about this backout please redirect 😄)

Comment 11

[Natalia Csoregi [:nataliaCs]]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #10)

(NI-ing Natalia, but if there's a specific individual I should be NI-ing about this backout please redirect 😄)

Backed out changeset 74fd8bb17de0 (bug 1714688) for causing Bug 1716860.

backout: https://hg.mozilla.org/integration/autoland/rev/5bb44754f8119fefe1206bd0bf5e614b6452288d

Comment 12

[Intermittent Failures Robot]

40 failures in 774 pushes (0.052 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 40

Platform and build breakdown:

• macosx1015-64-qr: 40
  • debug: 20
  • opt: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1716860&startday=2021-06-17&endday=2021-06-17&tree=all

Comment 13

[Intermittent Failures Robot]

68 failures in 2888 pushes (0.024 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 68

Platform and build breakdown:

• macosx1015-64-qr: 68
  • opt: 34
  • debug: 34

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1716860&startday=2021-06-14&endday=2021-06-20&tree=all