What problems would this solve? =============================== KumaScript executes on the web-heads alongside django, making it impossible to isolate or scale independently of the web-heads. Who would use this? =================== webops; MDN devs; MDN users What would users see? ===================== webops - separate, isolated cluster of KumaScript nodes MDN devs - new hosts in KUMASCRIPT_URL_TEMPLATE in settings_local.py MDN users - no change What would users do? What would happen as a result? =================================================== webops & MDN devs - perform operations on kumascript nodes isolated from MDN web-heads MDN users - no change Is there anything else we should know? ======================================
Severity: normal → enhancement
Component: General → KumaScript
Whiteboard: [specification][type:feature] → [specification][type:feature][dev papercut]
Hi :cturra, I'm trying to figure out how complicated splitting off kumascript from the webheads would be. Kumascript is in general very CPU intensive because it deals with executing JS serverside and parses lots of wiki page markup via XML parsers. That is orthogonal to the I/O heavy load we have on the Django/Python app and seems like a bad match for putting it on one machine. If we want to be able to better service MDN while increasing the number of contributors (who automatically increase the load on kumascript) we need to figure out a way to separate kuma and kumascript similar to how we've separated kuma and celery. I would assume that we could scale down the webheads in exchange. I know that :lorchard planned to tackle this problem at the time when kumascript was put online, but never got to it. As his successor so to speak I'd love to restart this conversation again.
i can't speak to the effort without digging into the details more. why don't you and i schedule some time to discuss these details together?
:cturra, I'm back again, so do you happen to know when a good time for a meeting would be? I'm in UTC+2 which means it'd be best for me to meet in your morning some time. Let me know :)
cturra is not with Moz anymore. Adding :cyliang to the mix.
Flags: needinfo?(cturra) → needinfo?(cliang)
Whiteboard: [specification][type:feature][dev papercut] → [kanban:https://webops.kanbanize.com/ctrl_board/2/977] [specification][type:feature][dev papercut]
It looks like MDN currently is using nodejs 0.10.26. This version is no longer available in the repo we're using; it looks like only 0.10.28 is available. Since I'd rather not mix and match versions, would it be alright to upgrade nodejs everywhere? Of course, I'd like to start by upgrading it in stage and making sure it doesn't upset anything. If so, is there a good / bad time to do this?
We may need the stage server soon for a couple other big merges & pushes: * django update * django-pipeline I would put this below those other projects, so maybe in 2/3 weeks?
Sounds fine. I'll pencil in a reminder to ask again around May 22nd.
Eh, what the hell, kanban?
The diagram in the repo  is missing some KumaScript -> Kuma interaction. Step 4, "Replace macros", also includes calling APIs provided by Kuma, to get data about page hierarchy, etc. A dedicated KumaScript node should include: * a node server running KumaScript * a dedicated Python server running Kuma with a read-only database connection This is almost what our current web nodes look like, except that the KS server would talk to the local Kuma server, instead of calling out to the developer.mozilla.org load balancer. There is code work required to ensure that the APIs work in read-only mode. I think this bug should become a MDN bug only, and I can open an ops bug to deploy once proven. :fox2mike, can you remove the webops kanban linkage for this ticket?  https://github.com/mozilla/kumascript
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/977] [specification][type:feature][dev papercut] → [specification][type:feature][dev papercut]
We're close on this one. As part of the Dockerization, we added an API server, so that KumaScript calls that instead of the web server: web → kumascript → api When we migrate to AWS, we'll have a scalable kumascript deployment, backed by a scalable api deployment. SCL3 won't get this change.
Depends on: 1110799
The dedicated clusters have shipped. We now use: * a deployment of Kuma web workers to handle user traffic * a deployment of KumaScript renderers using node.js * a deployment of Kuma web workers to handle KS's API requests * a deployment of Kuma celery workers for backend processing Each deployment is scalable, and is independently tracked in New Relic: * kuma-web-prod-portland - just user-facing requests - https://rpm.newrelic.com/accounts/1299394/applications/34459612 * kuma-backend-prod-portland - both celery and API - https://rpm.newrelic.com/accounts/1299394/applications/34601198 * kuma-prod-portland - combines kuma-web- and kuma-backend - https://rpm.newrelic.com/accounts/1299394/applications/34419452 * kumascript-prod-portland - https://rpm.newrelic.com/accounts/1299394/applications/
Status: NEW → RESOLVED
Last Resolved: 4 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.