866459 - KumaScript randomly failing

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: KumaScript, defect)

Description

[Jeremie Patonnier :Jeremie]

What did you do?
================
As I'm editing pages, when I save my change, sometimes, some kumascript template fails to be rendered

What happened?
==============
Some templates are just not rendered (nothing is appearing where it should). Sometimes it's a full error message that is displayed. In both case, a hard refresh seams to make it right.

What should have happened?
==========================
The template should always be rendered when a page is updated.

Is there anything else we should know?
======================================

Comment 1

[Les Orchard [:lorchard]]

Can you specify some pages where this is happening? I can't reproduce this issue from random places I've looked.

Comment 2

[Jeremie Patonnier :Jeremie]

I work daily on the WebAPI documentation, mostly all of those subpages have make it one or twice in the past 3 days:

https://developer.mozilla.org/en-US/docs/WebAPI

This is really a random stuff. Try editing a page, make some changes (if possible on a template) and sometimes some kuma templates fail to render (it can be really subtle because sometimes it's just something missing in a page that you won't notice). If you're lucky you'll get a giant template execution error but it happens less often.

It happens to me 4 or 5 times a days for 3 days.

Comment 3

[Jean-Yves Perrier [:teoli]]

I confirm this happens very often on diverse pages since about 4-5 days (at least).

One of the page that have the problem often is: https://developer.mozilla.org/en-US/docs/Specification_List

Comment 4

[Eric Shepherd [:sheppy]]

Yes, I'm seeing this semi-frequently over the last few days. Where pages won't pick up template changes for 5-10 minutes or the output of a template is missing in the middle of the page.

Comment 5

[Les Orchard [:lorchard]]

(In reply to Eric Shepherd [:sheppy] from comment #4)
> Yes, I'm seeing this semi-frequently over the last few days. Where pages
> won't pick up template changes for 5-10 minutes or the output of a template
> is missing in the middle of the page.

These are separate problems, caching vs failure - bug 866458 covers the caching. Please don't close one or the other as a dupe.

Trying to work out the reason for failures, and have been watching the servers for a day or so now. It appears that random kumascript processes are dying on the server with this message in the log:

    terminate called after throwing an instance of 'std::bad_alloc'
      what():  std::bad_alloc

The problem is, I can't reproduce the error in dev or prod, so I haven't yet worked out the cause. Retrying a page that seems to have resulted in the error doesn't cause it to happen again, so it would seem like this is some cumulative issue between document renderings.

Comment 6

[Les Orchard [:lorchard]]

And, FWIW, I can't make this error show up at *all* in my dev VM. I even tried hammering the server with some intense load testing, and never saw a hiccup. So far, my only leads come from watching what the service does in production :/

Comment 7

[Les Orchard [:lorchard]]

Filed an IT bug to investigate possible resource limits on the kumascript service that might be causing this issue

Comment 8

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/c2e887f5c46f3c7a247751d664f95dd8afbc08f6
bug 866459: Tweak kumascript retries and max jobs per process

https://github.com/mozilla/kuma/commit/9a50e218bd60279b9ef4e744ab5e5f0e7eb6a694
Merge pull request #1053 from lmorchard/866459-tweak-retries-and-max-jobs

bug 866459: Tweak kumascript retries and max jobs per process

Comment 9

[Les Orchard [:lorchard]]

The change in comment 8 has been pushed, which I hope helps the random failures.

It makes the service retry failed macros 10 times instead of 3. That should greatly increase the success of macros.

It also reduces the jobs executed by any worker process from 64 to 8. That should cut down on what I think is cumulative resource consumption which causes transient worker terminations.

I'm periodically watching the logs for signs of problems while working on other things, but let me know if this situation seems to have gotten worse or better.

Comment 10

[Les Orchard [:lorchard]]

I haven't seen trouble from this in a few days, and haven't gotten additional reports. Not saying it's totally fixed, but deferring this to the back-burner until I have more actionable information

Comment 11

[Jean-Yves Perrier [:teoli]]

I still have this problem every day, several time a day. Perhaps 10-15 times today, as much yesterday.

One interesting facts: when some templates have not been taken into account, it is all template of the same kind. But not always the same, and reload cure it most of the time.

Comment 12

[Les Orchard [:lorchard]]

(In reply to Jean-Yves Perrier [:teoli] from comment #11)
> I still have this problem every day, several time a day. Perhaps 10-15 times
> today, as much yesterday.
> 
> One interesting facts: when some templates have not been taken into account,
> it is all template of the same kind. But not always the same, and reload
> cure it most of the time.

Okay, so can you then please update this bug with failures you're seeing? Maybe some URLs, templates, and screenshots?

Comment 13

[Jean-Yves Perrier [:teoli]]

Attachment: Symptom 3: some templates lead to errors.

This is the less annoying case because we can't miss it.

Comment 14

[Jean-Yves Perrier [:teoli]]

Ok, I got 5 problems this morning. I can put them in three different kind of symptoms.

Symptom 1: (1 time) — maybe unrelated
I clicked "save" and the page is successfully saved but the old page is shown, without a warning saying that it is in the process of being updated (I got this warning a couple of time this morning).

Shift-reload made the correct modified page to appear.

Symptom 2:
I clicked "save" and got no error message, the edition are saved, but the displayed page has not resolved all the templates. There are blanks where the template content should be displayed. Note that not all templates are displayed blank on a page, but all templates of the same kind are.

Shift-reload fix the problem

Symptom 3:
I clicked save and got a list of error messages, 404-like about non-existing templates. (And these templates are very old and do exist).

Shift-reload fix the problem.

Comment 15

[Jean-Yves Perrier [:teoli]]

Attachment: Symptom 2: some template are ignored and replaced by ""

Comment 16

[Jean-Yves Perrier [:teoli]]

Attachment: Symptom 2b: the same page after a shift-reload

Comment 17

[Jean-Yves Perrier [:teoli]]

I don't have a screenshot for Symptom 1.

Also note this is happening on all kind of pages. Today, I noticed it in pages like:
https://developer.mozilla.org/en-US/docs/DOM/PannerNode
or 
https://developer.mozilla.org/en-US/docs/DOM/Mozilla_Event_Reference/ended_%28Web_Audio%29

I will continue to watch this problem.

Comment 18

[Jean-Yves Perrier [:teoli]]

I got the Symptom 3 again this morning (2 times), on these page:
https://developer.mozilla.org/en-US/docs/DOM/HTMLAnchorElement
https://developer.mozilla.org/en-US/docs/DOM/HTMLImageElement
And Symptom 2 there https://developer.mozilla.org/en-US/docs/DOM/HTMLIFrameElement
(on a few Templates, not all)

Comment 19

[Jean-Yves Perrier [:teoli]]

Got the symptom 2 and 3, 3-4 times this morning already (and it is 8.39am only), similar pages than the other days.

Btw, do we regenerate the pages when moving them? Because it would mean that this bug may happen on pages where we don't notice.

Comment 20

[Jean-Yves Perrier [:teoli]]

Got this problem regularly the last 5 days, but strangely, not today. Will report if it reappears.

Comment 21

[Jean-Yves Perrier [:teoli]]

And it happened right after :-(

Comment 22

[Eric Shepherd [:sheppy]]

This seems much better now.

Comment 23

[Jeremie Patonnier :Jeremie]

Yes it's better but I'm still facing those issue on a regular basis (but, yes, less often).

Comment 25

[Jean-Yves Perrier [:teoli]]

It looks to me that I didn't have any yesterday and this morning. Which is quite unusual (but really nice!)?

Maybe something was changed (I recall hearing about a memcache pref or something like that changed).

I will report again if it happens again.

Jeremie, Sheppy could you also report if you see it again? Just want to be sure that's just me being lucky for the last 36hours.

Comment 26

[Jean-Yves Perrier [:teoli]]

This seems to be fixed. Will reopen if it happens again.

Comment 27

[Jean-Yves Perrier [:teoli]]

Unfortunately this still happen; less often but it still happen: I got it twice this morning. It means we still have pages with missing content created several times a day.

Comment 28

[Jean-Yves Perrier [:teoli]]

So after a few days, I can say it still happens a lot. I get 20 times a day errors when saving: either scripts not found, either scripts ignored and blank inserted in pages.

Comment 30

[Jean-Yves Perrier [:teoli]]

I don't know if it is related to this bug or not, but I often get ISE today. That was very rare before this week-end.

I'm not able to tell if I get these instead of one of the two symptoms covered here, but I'm tempted to say yes.

Note that I get the ISE only when editing and that the commit of the edit (sometimes with the errors here) always happens successfully.

Finally a couple of time, I didn't even get the ISE as a result page, only a wholly blank page (but the commit of the edits happens successfully too).

Comment 31

[Jean-Yves Perrier [:teoli]]

Today, the MDN is unusable. On the about 30 commits I did this morning not a single worked normally: ISE, blank page, template not displayed, errors. Each time I get one of these.

I'm wondering if we should disable edition while you guys investigate.

Comment 32

[Les Orchard [:lorchard]]

(In reply to Jean-Yves Perrier [:teoli] from comment #30)
> I don't know if it is related to this bug or not, but I often get ISE today.
> That was very rare before this week-end.
> 
> I'm not able to tell if I get these instead of one of the two symptoms
> covered here, but I'm tempted to say yes.

Could you please file a new bug for this Internal Server Error? Or even file an overall tracker for these issues if they're related?

We're trying to track down multiple issues now, and we don't want to see them mashed together into unfocused bugs. More bugs are better, even if you think they might be related. We can merge them or close them all together later. Otherwise, if they're separate, we can better work on them in parallel

Comment 33

[John Karahalis [:openjck]]

Jean-Yves, can you please post the bug numbers and close the NEEDINFO after they have been opened?

Comment 34

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/a0ebc8a66f7269c3a007c1c1cf8629af0de9d9fd
bug 866459: Update kumascript to retry template loading on error

https://github.com/mozilla/kuma/commit/9febcb3ef4036b12ff7cb99cc76555789e539a4d
Merge pull request #1255 from lmorchard/866459-kumascript-loader-retries

bug 866459: Update kumascript to retry template loading on error

Comment 35

[Jean-Yves Perrier [:teoli]]

So to help make this problem clearer, I'm filling 4 bugs with the 4 different errors. I will add as many information as possible to them, and feel free to ask for more!

I will mark them as blocking this bug.

Comment 36

[Les Orchard [:lorchard]]

As noted in Comment 34, a change to KumaScript was pushed out that causes it to automatically retry attempts to load templates & modules. That should hopefully improve reliability vs transient Kuma issues.

That said, is this bug still happening often? And I mean, this specific problem: KumaScript macros sometimes fail at first and then succeed after a shift-refresh.

Any issues with editing or saving that don't directly involve KumaScript errors (eg. bug 902177) are out of scope for this bug.

Comment 37

[Jean-Yves Perrier [:teoli]]

(In reply to Les Orchard [:lorchard] from comment #36)
> That said, is this bug still happening often?
Impossible to know for the moment, as we always get a white page now. It looks that the pages are correct, but impossible to know for sure in the current state.

Comment 38

[Les Orchard [:lorchard]]

(In reply to Jean-Yves Perrier [:teoli] from comment #37)
> (In reply to Les Orchard [:lorchard] from comment #36)
> > That said, is this bug still happening often?
> Impossible to know for the moment, as we always get a white page now. It
> looks that the pages are correct, but impossible to know for sure in the
> current state.

Are you talking about bug 902177? If that's still happening to you, head over there & reopen with a comment about which page. I haven't seen a blank page on save today.

Comment 39

[Jean-Yves Perrier [:teoli]]

For the moment, I didn't get not valid Kumascript errors this morning. I will report when I get some.

Comment 40

[raymond [:retornam] (inactive)]

Could not find steps to reproduce this from my testing on stage mainly, again production DB content  dumps to stage will make testing a lot more easier and quicker

Comment 41

[Jean-Yves Perrier [:teoli]]

Since the fix(es) a week ago (or two? Time flies), I didn't saw any occurrence of this bug in production.

Comment 42

[Luke Crouch [:groovecoder]]

Closing. We can re-open if we need to.