1235409 - Track KumaScript macro execution time

Status: RESOLVED WONTFIX

(developer.mozilla.org Graveyard :: KumaScript, enhancement)

Description

[John Whitlock [:jwhitlock]]

What problem would this feature solve?
======================================
There is no data-driven way to analyze the performance of KumaScript macros, and thus no way to focus developer and writer efforts to improve performance.

Who has this problem?
=====================
Core contributors to MDN

How do you know that the users identified above have this problem?
==================================================================
The editing experience on MDN is worse on pages with long-running kumascript macros. If KS executes quickly, a user sees the page changes immediately. If not, then the editor sees a "page is being updated" message, and has to wait for the cache timeout to see their page.

Many pages on MDN have out-of-date content, because they contain the content at the time the page was rendered, but the data has changed or the KS macros have been updated.  Because KS execution time is slow, it is infeasible to periodically refresh content (it is estimated that a full refresh of MDN would be measured in days).

It is expected that, like most performance problems, 20% of the macros are causing 80% of the performance problems, so a small amount of effort would result in outsized rewards.

Despite the end-user benefits of improving KS macro execution, it is not done, because there is no data outside of anecdotes on how long it takes to execute particular KS macros.

How are the users identified above solving this problem now?
============================================================
Guessing at the KS macros that are causing performance issues. Waiting for the site to have performance problems that do appear in New Relic or tracebacks, and then going on hunting expeditions.

Do you have any suggestions for solving the problem? Please explain in detail.
==============================================================================
NewRelic can be used to instrument background tasks, which may be a good match for macro execution:

https://docs.newrelic.com/docs/agents/nodejs-agent/supported-features/nodejs-custom-instrumentation

The macro execution is centralized in macro-workers.js:

https://github.com/mozilla/kumascript/blob/master/lib/kumascript/macros-worker.js

Is there anything else we should know?
======================================
There are existing hooks for using statsd to record information such as counting errors and overall execution time.  The statsd server was decommissioned over a year ago. The developer consensus is that statsd was useful during migration to kuma / kumascript, and should not be reinstated during this maintenance phase.

Comment 1

[John Whitlock [:jwhitlock]]

On IRC it was pointed out that "developer consensus" was the wrong phrase here.

While at Mozlando, I sought out people that were around when KumaScript was being developed, mainly :orchard, and :groovecoder, to ask about statsd usage in the KumaScript code, why it was added, and what it was used for, to confirm my understanding from the code. The reaction from those I talked to was that the statsd measurement code was dead code that could be removed, and wouldn't be worth resurrecting without modifications in what was being measured.

I also talked to :fscholz, who is doing the work to clean up and remove macros in Q1 2016, about what kind of measurements would be useful. We sketched out a few measurement points that would require a little code and may quickly surface the most intensive macros, using New Relic for pretty graphs and potentially for transaction traces for diving deeper. This is where my suggestion comes from.

So, there's not a consensus from the dev team that statsd is a bad idea, or even a bad idea for this particular measurement. New Relic is the hammer that is currently closest at hand, and v2 or v3 might use statsd or another system for long-term monitoring.

Comment 2

[John Whitlock [:jwhitlock]]

No work has been done to implement this.

KumaScript was re-implemented, and looks more like modern JavaScript code. It was also simplified to always use POST, so there is even less information about what URLs are slow.

New Relic still supports customization hooks, which may make it easy to track this information:

https://docs.newrelic.com/docs/agents/nodejs-agent/supported-features/nodejs-custom-instrumentation

Rendering has moved out of workers and into a async promise-based system:

https://github.com/mdn/kumascript/blob/master/src/render.js

We're now using Datadog in production, which can be used as a statsd service.

Comment 3

[BMO Automation]

MDN Web Docs' bug reporting has now moved to GitHub. From now on, please file content bugs at https://github.com/mdn/sprints/issues/ and platform bugs at https://github.com/mdn/kuma/issues/.