KumaScript: create dedicated cluster

RESOLVED FIXED

Status

developer.mozilla.org
KumaScript
--
enhancement
RESOLVED FIXED
4 years ago
4 months ago

People

(Reporter: groovecoder, Unassigned)

Tracking

Details

(Whiteboard: [specification][type:feature][dev papercut])

(Reporter)

Description

4 years ago
What problems would this solve?
===============================
KumaScript executes on the web-heads alongside django, making it impossible to isolate or scale independently of the web-heads.

Who would use this?
===================
webops; MDN devs; MDN users

What would users see?
=====================
webops - separate, isolated cluster of KumaScript nodes

MDN devs - new hosts in KUMASCRIPT_URL_TEMPLATE in settings_local.py

MDN users - no change

What would users do? What would happen as a result?
===================================================
webops & MDN devs - perform operations on kumascript nodes isolated from MDN web-heads

MDN users - no change

Is there anything else we should know?
======================================
(Reporter)

Updated

4 years ago
Assignee: nobody → jezdez

Updated

4 years ago
Severity: normal → enhancement
Component: General → KumaScript
Whiteboard: [specification][type:feature] → [specification][type:feature][dev papercut]
Hi :cturra, I'm trying to figure out how complicated splitting off kumascript from the webheads would be. Kumascript is in general very CPU intensive because it deals with executing JS serverside and parses lots of wiki page markup via XML parsers. That is orthogonal to the I/O heavy load we have on the Django/Python app and seems like a bad match for putting it on one machine.

If we want to be able to better service MDN while increasing the number of contributors (who automatically increase the load on kumascript) we need to figure out a way to separate kuma and kumascript similar to how we've separated kuma and celery. I would assume that we could scale down the webheads in exchange.

I know that :lorchard planned to tackle this problem at the time when kumascript was put online, but never got to it. As his successor so to speak I'd love to restart this conversation again.
Flags: needinfo?(cturra)
i can't speak to the effort without digging into the details more. why don't you and i schedule some time to discuss these details together?
Flags: needinfo?(cturra)
:cturra, I'm back again, so do you happen to know when a good time for a meeting would be? I'm in UTC+2 which means it'd be best for me to meet in your morning some time. Let me know :)
Flags: needinfo?(cturra)
cturra is not with Moz anymore. Adding :cyliang to the mix.
Flags: needinfo?(cturra) → needinfo?(cliang)

Updated

3 years ago
Flags: needinfo?(cliang)

Updated

3 years ago
Whiteboard: [specification][type:feature][dev papercut] → [kanban:https://webops.kanbanize.com/ctrl_board/2/977] [specification][type:feature][dev papercut]

Updated

3 years ago
Depends on: 1160532

Updated

3 years ago
Depends on: 1160641

Updated

3 years ago
Depends on: 1160646

Comment 5

3 years ago
It looks like MDN currently is using nodejs 0.10.26.  This version is no longer available in the repo we're using; it looks like only 0.10.28 is available.

Since I'd rather not mix and match versions, would it be alright to upgrade nodejs everywhere?  Of course, I'd like to start by upgrading it in stage and making sure it doesn't upset anything.  If so, is there a good / bad time to do this?
Flags: needinfo?(jezdez)
(Reporter)

Comment 6

3 years ago
We may need the stage server soon for a couple other big merges & pushes:

* django update
* django-pipeline

I would put this below those other projects, so maybe in 2/3 weeks?

Comment 7

3 years ago
Sounds fine.  I'll pencil in a reminder to ask again around May 22nd.
Flags: needinfo?(jezdez)
Assignee: jezdez → nobody

Updated

2 years ago
Assignee: nobody → jezdez
Assignee: jezdez → nobody

Updated

2 years ago
Assignee: nobody → jezdez
Eh, what the hell, kanban?
Assignee: jezdez → nobody

Updated

2 years ago
Assignee: nobody → jezdez

Updated

2 years ago
Assignee: jezdez → nobody
The diagram in the repo [1] is missing some KumaScript -> Kuma interaction.  Step 4, "Replace macros", also includes calling APIs provided by Kuma, to get data about page hierarchy, etc.

A dedicated KumaScript node should include:

* a node server running KumaScript
* a dedicated Python server running Kuma with a read-only database connection

This is almost what our current web nodes look like, except that the KS server would talk to the local Kuma server, instead of calling out to the developer.mozilla.org load balancer.

There is code work required to ensure that the APIs work in read-only mode. I think this bug should become a MDN bug only, and I can open an ops bug to deploy once proven.

:fox2mike, can you remove the webops kanban linkage for this ticket?

[1] https://github.com/mozilla/kumascript
Flags: needinfo?(smani)

Updated

2 years ago
Flags: needinfo?(smani)
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/977] [specification][type:feature][dev papercut] → [specification][type:feature][dev papercut]
Done.
We're close on this one. As part of the Dockerization, we added an API server, so that KumaScript calls that instead of the web server:

web → kumascript → api

When we migrate to AWS, we'll have a scalable kumascript deployment, backed by a scalable api deployment.

SCL3 won't get this change.
Depends on: 1110799
The dedicated clusters have shipped. We now use:

* a deployment of Kuma web workers to handle user traffic
* a deployment of KumaScript renderers using node.js
* a deployment of Kuma web workers to handle KS's API requests
* a deployment of Kuma celery workers for backend processing

Each deployment is scalable, and is independently tracked in New Relic:

* kuma-web-prod-portland - just user-facing requests - https://rpm.newrelic.com/accounts/1299394/applications/34459612
* kuma-backend-prod-portland - both celery and API - https://rpm.newrelic.com/accounts/1299394/applications/34601198
* kuma-prod-portland - combines kuma-web- and kuma-backend - https://rpm.newrelic.com/accounts/1299394/applications/34419452
* kumascript-prod-portland - https://rpm.newrelic.com/accounts/1299394/applications/
Status: NEW → RESOLVED
Last Resolved: 4 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.