Closed Bug 1455061 Opened 6 years ago Closed 6 years ago

upgrade releng docker images to latest python + python dependency versions

Categories

(Release Engineering :: General, enhancement, P1)

enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: bhearsum)

References

Details

(Whiteboard: [keep-open][leave-open])

Attachments

(2 files)

Per our new Python Standards (https://docs.google.com/document/d/1tS0xfE9sOCCJOBr2J3nY2zo0nn9B0fj0MLzXe704n4g/edit#), we must keep these up to date with the latest versions. Build & test machinery is not in scope, but we have a number of release-related in-tree Docker images that need to be updated:
final verify
update verify
funsize update generator
firefox snap
periodic updates
bouncer check
beetmover
This worked fine with yesterday's push phase. I'm also removing the "beet-mover" docker image, which is unused with in-tree releases.

Using update verify will bring these up to date to Python 2.7.14.
Attachment #8969062 - Flags: review?(rail)
Comment on attachment 8969062 [details] [diff] [review]
switch bouncer check and final verify to update verify image

Review of attachment 8969062 [details] [diff] [review]:
-----------------------------------------------------------------

I think Mihai was going to land a conflicting (clean up mostly), but in any case LGTM.
Attachment #8969062 - Flags: review?(rail) → review+
Attachment #8969062 - Flags: checked-in+
Pushed by bhearsum@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d9ef9d29fd77
upgrade releng docker images to latest python + python dependency versions. r=rail
Whiteboard: [keep-open][leave-open]
Blocks: 1450035
Johan, I've been looking into how to upgrade the firefox-snap docker image to something with a newer Python, but it looks like it's based on snapcore/snapcraft, which is pinned to ubuntu:xenial - which only has Python 2.7.12. I couldn't find any newer snapcraft docker image that we could try. Do you have any ideas about how we can get the firefox-snap tasks using an up-to-date Python?
Flags: needinfo?(jlorenzo)
QA Contact: catlee
Not much to say, it seems to work. I ran it through a staging release: https://tools.taskcluster.net/groups/fHJ22-gARDu8Sz9-X3aNnQ/tasks/CBu45HVWRjSQbW8zZVka8g/runs/0/artifacts

We should probably backport this to esr60, but it can ride with the next merge to Beta for beta/release.
Attachment #8984252 - Flags: review?(sfraser)
Comment on attachment 8984252 [details] [diff] [review]
upgrade update verify + funsize update generator images

Review of attachment 8984252 [details] [diff] [review]:
-----------------------------------------------------------------

Nice work, thank you
Attachment #8984252 - Flags: review?(sfraser) → review+
Attachment #8984252 - Flags: checked-in+
Pushed by bhearsum@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2ed797e0a24e
upgrade update verify and funsize update generator docker images to newer ubuntu to get latest python. r=sfraser
Keywords: leave-open
I backported the update verify + funsize changes to esr60: https://hg.mozilla.org/releases/mozilla-esr60/rev/c7187146116fe90f3e98a7d7657fe5501faad472

As far as I know, the only thing left to do here is deal with the snapcraft images.
(In reply to Ben Hearsum (:bhearsum) from comment #5)
> Johan, I've been looking into how to upgrade the firefox-snap docker image
> to something with a newer Python, but it looks like it's based on
> snapcore/snapcraft, which is pinned to ubuntu:xenial - which only has Python
> 2.7.12. I couldn't find any newer snapcraft docker image that we could try.
> Do you have any ideas about how we can get the firefox-snap tasks using an
> up-to-date Python?

We switched to the snapcraft docker image in bug 1444898. Before then, we used to build the image ourselves by crafting from the Ubuntu base image. Sadly the Ubuntu repos don't always have the most recent version of Snap. Sometimes, the Snap Store server breaks with earlier versions of the client (called snapcraft). That's why we want to run on the most recent version of snapcraft.

The Dockerfile lives at [1]. Like you said, it's pinned to xenial (Ubuntu 16.04). There may be a bump of the Python package in 16.04.5 scheduled on August 2nd, 2018.

If we really want an update, we can go back to building our own docker image, but I'd prefer to stick to what Canonical provides. Thus, I don't think there's a lot we can act on. Let's wait August to see if Python got bumped.


Aside from that, all other docker images got updated. We can close this bug. Please reopen if you think we should update the snap docker image before next August. 

[1] https://github.com/snapcore/snapcraft/blob/master/docker/stable.Dockerfile
Status: NEW → RESOLVED
Closed: 6 years ago
Depends on: 1444898
Flags: needinfo?(jlorenzo)
Resolution: --- → FIXED
(In reply to Johan Lorenzo [:jlorenzo] - On PTO - Back on July 19th from comment #11)
> (In reply to Ben Hearsum (:bhearsum) from comment #5)
> > Johan, I've been looking into how to upgrade the firefox-snap docker image
> > to something with a newer Python, but it looks like it's based on
> > snapcore/snapcraft, which is pinned to ubuntu:xenial - which only has Python
> > 2.7.12. I couldn't find any newer snapcraft docker image that we could try.
> > Do you have any ideas about how we can get the firefox-snap tasks using an
> > up-to-date Python?
> 
> We switched to the snapcraft docker image in bug 1444898. Before then, we
> used to build the image ourselves by crafting from the Ubuntu base image.
> Sadly the Ubuntu repos don't always have the most recent version of Snap.
> Sometimes, the Snap Store server breaks with earlier versions of the client
> (called snapcraft). That's why we want to run on the most recent version of
> snapcraft.
> 
> The Dockerfile lives at [1]. Like you said, it's pinned to xenial (Ubuntu
> 16.04). There may be a bump of the Python package in 16.04.5 scheduled on
> August 2nd, 2018.
> 
> If we really want an update, we can go back to building our own docker
> image, but I'd prefer to stick to what Canonical provides. Thus, I don't
> think there's a lot we can act on. Let's wait August to see if Python got
> bumped.
> 
> 
> Aside from that, all other docker images got updated. We can close this bug.
> Please reopen if you think we should update the snap docker image before
> next August. 
> 
> [1]
> https://github.com/snapcore/snapcraft/blob/master/docker/stable.Dockerfile

It sounds like without investing a ton of work into maintaining our own image, the best we can do is rely on Ubuntu LTS. It feels like that's probably the right trade off here, so let's go with that. cc'ing catlee in case he feels differently.

Thanks for detailed explanation, Johan.
You need to log in before you can comment on or make changes to this bug.