1020491 - KumaScript: create dedicated cluster

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: KumaScript, enhancement)

Description

[Luke Crouch [:groovecoder]]

What problems would this solve?
===============================
KumaScript executes on the web-heads alongside django, making it impossible to isolate or scale independently of the web-heads.

Who would use this?
===================
webops; MDN devs; MDN users

What would users see?
=====================
webops - separate, isolated cluster of KumaScript nodes

MDN devs - new hosts in KUMASCRIPT_URL_TEMPLATE in settings_local.py

MDN users - no change

What would users do? What would happen as a result?
===================================================
webops & MDN devs - perform operations on kumascript nodes isolated from MDN web-heads

MDN users - no change

Is there anything else we should know?
======================================

Comment 1

[Jannis Leidel [:jezdez]]

Hi :cturra, I'm trying to figure out how complicated splitting off kumascript from the webheads would be. Kumascript is in general very CPU intensive because it deals with executing JS serverside and parses lots of wiki page markup via XML parsers. That is orthogonal to the I/O heavy load we have on the Django/Python app and seems like a bad match for putting it on one machine.

If we want to be able to better service MDN while increasing the number of contributors (who automatically increase the load on kumascript) we need to figure out a way to separate kuma and kumascript similar to how we've separated kuma and celery. I would assume that we could scale down the webheads in exchange.

I know that :lorchard planned to tackle this problem at the time when kumascript was put online, but never got to it. As his successor so to speak I'd love to restart this conversation again.

Comment 2

[Chris Turra [:cturra]]

i can't speak to the effort without digging into the details more. why don't you and i schedule some time to discuss these details together?

Comment 3

[Jannis Leidel [:jezdez]]

:cturra, I'm back again, so do you happen to know when a good time for a meeting would be? I'm in UTC+2 which means it'd be best for me to meet in your morning some time. Let me know :)

Comment 4

[Jannis Leidel [:jezdez]]

cturra is not with Moz anymore. Adding :cyliang to the mix.

Comment 5

[C. Liang [:cyliang]]

It looks like MDN currently is using nodejs 0.10.26.  This version is no longer available in the repo we're using; it looks like only 0.10.28 is available.

Since I'd rather not mix and match versions, would it be alright to upgrade nodejs everywhere?  Of course, I'd like to start by upgrading it in stage and making sure it doesn't upset anything.  If so, is there a good / bad time to do this?

Comment 6

[Luke Crouch [:groovecoder]]

We may need the stage server soon for a couple other big merges & pushes:

* django update
* django-pipeline

I would put this below those other projects, so maybe in 2/3 weeks?

Comment 7

[C. Liang [:cyliang]]

Sounds fine.  I'll pencil in a reminder to ask again around May 22nd.

Comment 8

[Jannis Leidel [:jezdez]]

Eh, what the hell, kanban?

Comment 9

[John Whitlock [:jwhitlock]]

The diagram in the repo [1] is missing some KumaScript -> Kuma interaction.  Step 4, "Replace macros", also includes calling APIs provided by Kuma, to get data about page hierarchy, etc.

A dedicated KumaScript node should include:

* a node server running KumaScript
* a dedicated Python server running Kuma with a read-only database connection

This is almost what our current web nodes look like, except that the KS server would talk to the local Kuma server, instead of calling out to the developer.mozilla.org load balancer.

There is code work required to ensure that the APIs work in read-only mode. I think this bug should become a MDN bug only, and I can open an ops bug to deploy once proven.

:fox2mike, can you remove the webops kanban linkage for this ticket?

[1] https://github.com/mozilla/kumascript

Comment 10

[Shyam Mani [:fox2mike]]

Done.

Comment 11

[John Whitlock [:jwhitlock]]

We're close on this one. As part of the Dockerization, we added an API server, so that KumaScript calls that instead of the web server:

web → kumascript → api

When we migrate to AWS, we'll have a scalable kumascript deployment, backed by a scalable api deployment.

SCL3 won't get this change.

Comment 12

[John Whitlock [:jwhitlock]]

The dedicated clusters have shipped. We now use:

* a deployment of Kuma web workers to handle user traffic
* a deployment of KumaScript renderers using node.js
* a deployment of Kuma web workers to handle KS's API requests
* a deployment of Kuma celery workers for backend processing

Each deployment is scalable, and is independently tracked in New Relic:

* kuma-web-prod-portland - just user-facing requests - https://rpm.newrelic.com/accounts/1299394/applications/34459612
* kuma-backend-prod-portland - both celery and API - https://rpm.newrelic.com/accounts/1299394/applications/34601198
* kuma-prod-portland - combines kuma-web- and kuma-backend - https://rpm.newrelic.com/accounts/1299394/applications/34419452
* kumascript-prod-portland - https://rpm.newrelic.com/accounts/1299394/applications/