Created attachment 585892 [details]
I was reading the follow page on MDN:
In the "text formatting" section this exception pops:
reference to undefined name 'syntax' Exception of type 'MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException' was thrown.
*** Bug 715331 has been marked as a duplicate of this bug. ***
*** Bug 715414 has been marked as a duplicate of this bug. ***
Sheppy, getting the syntax errors again. kicking MindTouch doesn't fix?
Yeah, the settings for the DHTML, syntax highlighter, and custom RSS extensions all vanished again. No idea why that happens (MindTouch can't figure it out either). I've restored the settings and we need the wiki restarted again to pick them up. With luck they won't get deleted again.
Also, there's a connectivity issue with a database on at least one host. I filed bug 715479 on that.
*** Bug 715544 has been marked as a duplicate of this bug. ***
Update: We have email and support tickets with MindTouch requesting additional information, as there's an error in the deki API log that is clearly related to this problem. It looks like, perhaps, there's a privilege issue between the settings store and the Deki API that's trying to access it, but we don't know for sure.
Occasionally getting this error:
Site settings could not be loaded
We were unable to locate the API to request site settings. Please see below for debugging information. If this is a new install, try refreshing - the API is simply taking its time loading up!
HTTP Response Status Code: 0
couldn't connect to host
API is the main offender for sure. Also getting django errors trying to fetch empty api-powered pages:
*** Bug 715691 has been marked as a duplicate of this bug. ***
Need to put the temporary banner back up.
MindTouch thinks they've figured this out. Basically, when a host starts up and goes to load extensions, it reads that extension's configuration from the database, then (for some reason I don't understand) deletes that row from the database. Then the extension re-writes the configuration back into the database.
This is fine for a single host, but when multiple hosts are starting up at once, this can result in one host reading and deleting, then other hosts finding no setting in the interim period, and as a result, eventually you wind up losing settings.
MindTouch has a patch in testing on their trunk now, and will back port to the release we're running on once they've finished testing. "Early next week," they say.
In the meantime, we'll just need to be patient -- or switch back to just one host for the interim.
The problem just started happening again, and we're in the middle of a doc sprint. Going down to one host might fix the problem, but would hamper performance at a very inconvenient time.
@Gen: yes, when this problem happens, it shows up on any page that has code examples on it.
With Sheppy's lead, we kicked MDN. Let us know if this crops up again, he has documented the recovery process in the wiki.
/me let you know that it happened again.
FWIW, instructions have been added to the Intranet on how to attempt to correct this when it crops up; jms, teoli, and I all have the permissions needed to try to fix it. But realistically, it's going to just keep happening until we get the patch. I'm waiting to hear back from MindTouch on a couple of emails I sent out this morning looking for a status report.
Email received from MindTouch: They apologize profusely for the aggravation and hope to have something for us later this afternoon.
*** Bug 720482 has been marked as a duplicate of this bug. ***
*** Bug 720813 has been marked as a duplicate of this bug. ***
(In reply to Stefan Plewako from comment #23)
Thanks, but we don't need any more reports of pages that demonstrate this problem. That part of the issue is well understood (that is, it appears on any page that uses code syntax highlighting).
MindTouch has installed a cronjob on the system which detects when this has happened and restores the lost configuration data, then restarts the extensions. This is currently being tested with the ajaxrss, syntax highlighter, and dhtml extensions. Tomorrow the rest of the extensions (activitystream, webcache, etc) will be added.
The long-term fix is the patch to fix the underlying problem; I don't know when we'll get that, but it's in work at MindTouch.
The rest of the extensions we use have been added to the auto-repair script. We're still waiting on the patch, which may take a few weeks unfortunately.
Alice: It looks like the error you were seeing has been fixed.
(In reply to John Karahalis [:openjck] from comment #28)
> Alice: It looks like the error you were seeing has been fixed.
Yeah, these will continue to happen periodically, then the repair script will come along and fix it, so it shouldn't happen for more than a few minutes at a time. That said, I will keep an eye on this! Thanks!
Looks like something's wrong, as this extension doesn't appear to be restarting. Does the auto-repair bot not know to restart the mediawiki extension?
(In reply to Alice0775 White from comment #33)
> Err :(
Any errors you saw on those pages don't seem to be happening right now. Also, the valueOf page is working again, so it finally did successfully get that extension restarted.
Strange. I am still seeing errors on the pages listed in comment 30 and comment 33.
So here's what I believe is happening: these extension crashes that are causing this error are happening extremely frequently. The site is recovering after a few minutes, but the recovery period is pretty short before the crash happens again. So we're seeing these errors a lot of the time, but not all of the time.
There's nothing we can do about it, so we might as well stop worrying about it. The only thing that will resolve this is getting off MindTouch.
Should I stop reporting the error here?
Alice: Yes. These errors will disappear when we release a major update to the MDN very soon. You do not need to report them for now.
This is fixed by virtue of having completed the migration of MDN to the new Kuma wiki system. Closing!
*** Bug 778564 has been marked as a duplicate of this bug. ***