Last Comment Bug 715341 - MDN wiki: Extension settings being deleted at startup, resulting in inline errors while browsing the site
: MDN wiki: Extension settings being deleted at startup, resulting in inline er...
Product: Mozilla Developer Network
Classification: Other
Component: General (show other bugs)
: unspecified
: x86 Mac OS X
-- normal with 4 votes (vote)
: ---
Assigned To: Jake Maul [:jakem]
: 715331 715414 715544 715691 720482 720813 778564 (view as bug list)
Depends on:
Blocks: 720162 720392
  Show dependency treegraph
Reported: 2012-01-04 14:27 PST by Hubert Figuiere [:hub]
Modified: 2013-11-11 18:34 PST (History)
34 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---

screenshot (428.94 KB, image/png)
2012-01-04 14:27 PST, Hubert Figuiere [:hub]
no flags Details

Description User image Hubert Figuiere [:hub] 2012-01-04 14:27:44 PST
Created attachment 585892 [details]

I was reading the follow page on MDN:

In the "text formatting" section this exception pops:

reference to undefined name 'syntax' Exception of type 'MindTouch.Deki.Script.Runtime.DekiScriptUndefinedNameException' was thrown.

see screenshot
Comment 1 User image Luke Crouch [:groovecoder] 2012-01-04 19:50:55 PST
*** Bug 715331 has been marked as a duplicate of this bug. ***
Comment 2 User image Luke Crouch [:groovecoder] 2012-01-04 20:11:11 PST
*** Bug 715414 has been marked as a duplicate of this bug. ***
Comment 3 User image Luke Crouch [:groovecoder] 2012-01-04 20:12:22 PST
Sheppy, getting the syntax errors again. kicking MindTouch doesn't fix?
Comment 4 User image Alice0775 White 2012-01-05 03:36:53 PST
Comment 5 User image Eric Shepherd [:sheppy] 2012-01-05 06:03:30 PST
Yeah, the settings for the DHTML, syntax highlighter, and custom RSS extensions all vanished again. No idea why that happens (MindTouch can't figure it out either). I've restored the settings and we need the wiki restarted again to pick them up. With luck they won't get deleted again.

Also, there's a connectivity issue with a database on at least one host. I filed bug 715479 on that.
Comment 6 User image Nickolay_Ponomarev 2012-01-05 09:52:47 PST
*** Bug 715544 has been marked as a duplicate of this bug. ***
Comment 7 User image Eric Shepherd [:sheppy] 2012-01-05 12:37:32 PST
Update: We have email and support tickets with MindTouch requesting additional information, as there's an error in the deki API log that is clearly related to this problem. It looks like, perhaps, there's a privilege issue between the settings store and the Deki API that's trying to access it, but we don't know for sure.
Comment 8 User image Janet Swisher 2012-01-05 12:54:15 PST
Occasionally getting this error:

Site settings could not be loaded

We were unable to locate the API to request site settings. Please see below for debugging information. If this is a new install, try refreshing - the API is simply taking its time loading up!

HTTP Response Status Code: 0

couldn't connect to host
Comment 9 User image Luke Crouch [:groovecoder] 2012-01-05 12:58:07 PST
API is the main offender for sure. Also getting django errors trying to fetch empty api-powered pages:
Comment 10 User image Luke Crouch [:groovecoder] 2012-01-05 15:59:41 PST
*** Bug 715691 has been marked as a duplicate of this bug. ***
Comment 11 User image Janet Swisher 2012-01-09 23:59:04 PST
Happening again.
Comment 12 User image Janet Swisher 2012-01-09 23:59:32 PST
Need to put the temporary banner back up.
Comment 13 User image Eric Shepherd [:sheppy] 2012-01-13 12:56:15 PST
MindTouch thinks they've figured this out. Basically, when a host starts up and goes to load extensions, it reads that extension's configuration from the database, then (for some reason I don't understand) deletes that row from the database. Then the extension re-writes the configuration back into the database.

This is fine for a single host, but when multiple hosts are starting up at once, this can result in one host reading and deleting, then other hosts finding no setting in the interim period, and as a result, eventually you wind up losing settings.

MindTouch has a patch in testing on their trunk now, and will back port to the release we're running on once they've finished testing. "Early next week," they say.

In the meantime, we'll just need to be patient -- or switch back to just one host for the interim.
Comment 14 User image Janet Swisher 2012-01-20 10:58:06 PST
The problem just started happening again, and we're in the middle of a doc sprint. Going down to one host might fix the problem, but would hamper performance at a very inconvenient time.
Comment 15 User image Gen Kanai [:gen] 2012-01-20 15:25:30 PST
Also happening here
Comment 16 User image Janet Swisher 2012-01-20 15:27:00 PST
@Gen: yes, when this problem happens, it shows up on any page that has code examples on it.
Comment 17 User image Michael Burns [:mburns] 2012-01-20 16:10:50 PST
With Sheppy's lead, we kicked MDN. Let us know if this crops up again, he has documented the recovery process in the wiki.
Comment 18 User image Andreas Wagner [:TheOne] 2012-01-23 07:57:05 PST
/me let you know that it happened again.
Comment 19 User image Eric Shepherd [:sheppy] 2012-01-23 12:20:23 PST
FWIW, instructions have been added to the Intranet on how to attempt to correct this when it crops up; jms, teoli, and I all have the permissions needed to try to fix it. But realistically, it's going to just keep happening until we get the patch. I'm waiting to hear back from MindTouch on a couple of emails I sent out this morning looking for a status report.
Comment 20 User image Eric Shepherd [:sheppy] 2012-01-23 13:01:12 PST
Email received from MindTouch: They apologize profusely for the aggravation and hope to have something for us later this afternoon.
Comment 21 User image Luke Crouch [:groovecoder] 2012-01-23 14:50:42 PST
*** Bug 720482 has been marked as a duplicate of this bug. ***
Comment 22 User image Luke Crouch [:groovecoder] 2012-01-24 15:09:19 PST
*** Bug 720813 has been marked as a duplicate of this bug. ***
Comment 24 User image Janet Swisher 2012-01-26 09:42:06 PST
(In reply to Stefan Plewako from comment #23)

Thanks, but we don't need any more reports of pages that demonstrate this problem. That part of the issue is well understood (that is, it appears on any page that uses code syntax highlighting).
Comment 25 User image Eric Shepherd [:sheppy] 2012-01-30 10:00:14 PST
MindTouch has installed a cronjob on the system which detects when this has happened and restores the lost configuration data, then restarts the extensions. This is currently being tested with the ajaxrss, syntax highlighter, and dhtml extensions. Tomorrow the rest of the extensions (activitystream, webcache, etc) will be added.

The long-term fix is the patch to fix the underlying problem; I don't know when we'll get that, but it's in work at MindTouch.
Comment 26 User image Eric Shepherd [:sheppy] 2012-02-03 14:20:38 PST
The rest of the extensions we use have been added to the auto-repair script. We're still waiting on the patch, which may take a few weeks unfortunately.
Comment 27 User image Alice0775 White 2012-07-14 10:40:14 PDT
also error
Comment 28 User image John Karahalis [:openjck] 2012-07-16 12:35:55 PDT
Alice: It looks like the error you were seeing has been fixed.
Comment 29 User image Alice0775 White 2012-07-16 12:43:51 PDT
(In reply to John Karahalis [:openjck] from comment #28)
> Alice: It looks like the error you were seeing has been fixed.

Confirmed.Thanks :)
Comment 31 User image Eric Shepherd [:sheppy] 2012-07-21 16:33:28 PDT
Yeah, these will continue to happen periodically, then the repair script will come along and fix it, so it shouldn't happen for more than a few minutes at a time. That said, I will keep an eye on this! Thanks!
Comment 32 User image Eric Shepherd [:sheppy] 2012-07-21 19:37:10 PDT
Looks like something's wrong, as this extension doesn't appear to be restarting. Does the auto-repair bot not know to restart the mediawiki extension?
Comment 34 User image Eric Shepherd [:sheppy] 2012-07-22 05:45:09 PDT
(In reply to Alice0775 White from comment #33)
> Err :(

Any errors you saw on those pages don't seem to be happening right now. Also, the valueOf page is working again, so it finally did successfully get that extension restarted.
Comment 35 User image John Karahalis [:openjck] 2012-07-23 10:27:24 PDT
Strange. I am still seeing errors on the pages listed in comment 30 and comment 33.
Comment 37 User image Eric Shepherd [:sheppy] 2012-07-26 03:47:46 PDT
So here's what I believe is happening: these extension crashes that are causing this error are happening extremely frequently. The site is recovering after a few minutes, but the recovery period is pretty short before the crash happens again. So we're seeing these errors a lot of the time, but not all of the time.

There's nothing we can do about it, so we might as well stop worrying about it. The only thing that will resolve this is getting off MindTouch.
Comment 38 User image Alice0775 White 2012-07-26 04:02:45 PDT
Should I stop reporting the error here?
Comment 39 User image John Karahalis [:openjck] 2012-07-26 14:06:29 PDT
Alice: Yes. These errors will disappear when we release a major update to the MDN very soon. You do not need to report them for now.
Comment 40 User image Jake Maul [:jakem] 2012-08-03 12:46:27 PDT
This is fixed by virtue of having completed the migration of MDN to the new Kuma wiki system. Closing!
Comment 41 User image Kohei Yoshino [:kohei] 2013-11-09 23:33:36 PST
*** Bug 778564 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.