Closed Bug 1524370 Opened 6 years ago Closed 6 years ago

Deploy python3 Balrog

Categories

(Cloud Services :: Operations: Miscellaneous, task)

Product:
Cloud Services
Component:
Operations: Miscellaneous
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: catlee, Assigned: oremj)

References

Details

We now have python3 images for Balrog being produced and published to dockerhub. These are named latest-py3 or master-$HASH-py3.

https://hub.docker.com/r/mozilla/balrog/tags

The rough plan for getting these into prod is:

  1. Switch over the dev and stage environments to use the py3 images.

  2. Do some sanity checking of dev/stage to make sure they're behaving properly

  3. Set up a canary py3 deployment in production. Watch that it behaves the same as the py2 deployment

  4. Deploy py3 everywhere

If we need to land and test an emergency fix for Balrog after step 1), we can always revert those environments to py2 to verify before pushing to prod.

Assignee: nobody → oremj

(In reply to Chris AtLee [:catlee] from comment #0)

  1. Set up a canary py3 deployment in production. Watch that it behaves the same as the py2 deployment

Sounds great to me. Just to clarify, this means the public side only ?

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #1)

(In reply to Chris AtLee [:catlee] from comment #0)

  1. Set up a canary py3 deployment in production. Watch that it behaves the same as the py2 deployment

Sounds great to me. Just to clarify, this means the public side only ?

I suppose? How do we verify the admin side is working properly?

An option is staging releases (including on try), which will send data to staging balrog.

v2.57 is the currently deployed version, but there isn't a -py3 image for that version. Would it be possible to create one or shall we just wait for the next version?

Flags: needinfo?(catlee)

Would be easiest to just do a new version/prod push. It's been a while since I did one so I'll look into that.

In the meantime, dev is switched over to py3 now ? And stage will be next time I tag a release ? Since we haven't done point 2 (sanity checking) it would be good to hold there.

Flags: needinfo?(catlee)

Yep, that sounds good to me.

Let me know when this goes out so I can verify everything is set up correctly.

The 2.58 release process leading up to bug 1532910 has generated a py3 image with the v2.58-py3 tag.

What's the current status here?

I'm not aware of any issues with step 2 - staging releases against py3 admin seem fine although we don't exercise it much.

Ben merged a py36 to py37 change today, with the intention of having a single migration instead of two step, but it needed to be backed out again. Any idea if that's a small or large fix Ben ?

Flags: needinfo?(bhearsum)

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #10)

I'm not aware of any issues with step 2 - staging releases against py3 admin
seem fine although we don't exercise it much.

Ben merged a py36 to py37 change today, with the intention of having a
single migration instead of two step, but it needed to be backed out again.
Any idea if that's a small or large fix Ben ?

I'm not sure yet. If we can't find a Python 3.7 image with an old enough version of node, probably a large one. If it turns out that way, I don't think we should block on it.

Flags: needinfo?(bhearsum)

What version of node does balrog currently use, and what version are you given the option of installing?

Python 3.7 stuck the second time it landed, and auth0 is in prod, bravo! Anything else you'd like to wait on before py3 Ben ?

AIUI step 3 is to flush out any performance issues or bugs with high request volumes.

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #13)

Python 3.7 stuck the second time it landed, and auth0 is in prod, bravo!
Anything else you'd like to wait on before py3 Ben ?

AIUI step 3 is to flush out any performance issues or bugs with high request
volumes.

Sorry, missed this comment. We ended up doing MySQL 5.7 before Python 3. I'd like to actively work towards getting this done now, though. I talked with oremj on IRC yesterday, and we said we'd try it out on the web cluster on Monday. That will help flush out any performance issues, as you say, as well as help to find any potential unicode issues there.

I suspect we could do admin any time, but maybe it's best to wait until web is fully upgraded to avoid running a split stack.

Jeremy, do you have a preferred time to do this on Monday? Anytime before 4pm Eastern should be good for me.

Flags: needinfo?(oremj)

We tested this out earlier today.

Flags: needinfo?(oremj)
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.