1649266 - 79.0b1 release-generate-checksums failures blocking releases

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P1)

Description

[(PTO June 19-July 7) Ryan VanderMeulen]

These failures appeared on both the Desktop and DevEdition 79.0b1 graphs and are blocking the releases at the moment.

https://firefoxci.taskcluster-artifacts.net/E07TV_RoSZ23ON94YFd0ZA/0/public/logs/live_backing.log

[task 2020-06-29T20:00:46.537Z] 20:00:46     INFO - [mozharness: 2020-06-29 20:00:46.537812Z] Finished collect-individual-checksums step (failed)
[task 2020-06-29T20:00:46.538Z] 20:00:46    FATAL - Uncaught exception: Traceback (most recent call last):
[task 2020-06-29T20:00:46.538Z] 20:00:46    FATAL -   File "/builds/worker/checkouts/gecko/testing/mozharness/mozharness/base/script.py", line 2132, in run
[task 2020-06-29T20:00:46.538Z] 20:00:46    FATAL -     self.run_action(action)
[task 2020-06-29T20:00:46.538Z] 20:00:46    FATAL -   File "/builds/worker/checkouts/gecko/testing/mozharness/mozharness/base/script.py", line 2071, in run_action
[task 2020-06-29T20:00:46.538Z] 20:00:46    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
[task 2020-06-29T20:00:46.538Z] 20:00:46    FATAL -   File "/builds/worker/checkouts/gecko/testing/mozharness/mozharness/base/script.py", line 2026, in _possibly_run_method
[task 2020-06-29T20:00:46.538Z] 20:00:46    FATAL -     return getattr(self, method_name)()
[task 2020-06-29T20:00:46.538Z] 20:00:46    FATAL -   File "/builds/worker/checkouts/gecko/testing/mozharness/scripts/release/generate-checksums.py", line 166, in collect_individual_checksums
[task 2020-06-29T20:00:46.538Z] 20:00:46    FATAL -     for f, info in parse_checksums_file(c).iteritems():
[task 2020-06-29T20:00:46.538Z] 20:00:46    FATAL - AttributeError: 'dict' object has no attribute 'iteritems'
[task 2020-06-29T20:00:46.538Z] 20:00:46    FATAL - Running post_fatal callback...
[task 2020-06-29T20:00:46.538Z] 20:00:46    FATAL - Exiting -1

Comment 1

[(PTO June 19-July 7) Ryan VanderMeulen]

Looks like we switched from Python 2.7.9 to 3.6.3 during the 79 Nightly cycle.

Comment 2

[(PTO June 19-July 7) Ryan VanderMeulen]

I'm wondering if this is a regression from bug 1621960.

Comment 3

[Ricky Stewart]

Yeah, that checks out:

/builds/worker/checkouts/gecko/mach python /builds/worker/checkouts/gecko/testing/mozharness/scripts/release/generate-checksums.py

I don't know what script this is so I can't write the patch myself, but you can do python2 mach python (instead of a bare mach python) to force running with Python 2.

Comment 4

[Rail Aliiev [:rail]]

Something like https://gist.github.com/rail/be91c15728eef26e64c93c5681c34b8f should fix the issue. Or we can use six to make iteritems() works in py2 and py3 efficiently.

Comment 5

[Rail Aliiev [:rail]]

... or https://gist.github.com/rail/724f85c34009fe8d248d9a698a23ca25

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Yeah, the specific py3 error here I think has a straightforward solution.
I'm running generate-checksums.py locally here, and I'm able to reproduce this specific problem and fix it. However, parts of the script aren't able to be run locally (looks like boto expects the current machine to be an AWS machine? Unsure). Therefore, fully testing this and ensuring py3-compatibility is tough.

I'll post a patch, but I've got two questions:

1. Is there a way I can test this in CI without uplifting? I suppose that "staging releases" will make this possible, eh?
2. I won't have full confidence in this patch - if we want a guaranteed short-term fix, that would be running this script with python2 instead (as Ricky suggested)

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1649266: generate-checksums.py should be compatible with py3 r?rail

Adds iter() changes and str conversions with six to ensure generate-checksums
will work with both py2 and py3.

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

However, parts of the script aren't able to be run locally (looks like boto expects the current machine to be an AWS machine? Unsure

This had less severity than I thought - boto attempts to determine its credentials at some point, and gracefully fails when it realizes that it's not an AWS machine.

The patch I posted works locally with python 3. I'm relatively confident that it will work in CI, I'll chat with RyanVM again shortly here to find out which direction we want to move.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/08366bf6c3b2
generate-checksums.py should be compatible with py3 r=rail

Comment 10

[(PTO June 19-July 7) Ryan VanderMeulen]

https://hg.mozilla.org/releases/mozilla-beta/rev/81e954c63322

Comment 11

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/08366bf6c3b2