Last Comment Bug 471945 - Dekiwiki causes replication failures on the slave databases frequently
: Dekiwiki causes replication failures on the slave databases frequently
Status: RESOLVED FIXED
:
Product: Mozilla Developer Network
Classification: Other
Component: General (show other bugs)
: unspecified
: x86 Linux
: -- critical (vote)
: ---
Assigned To: Dave Miller [:justdave] (justdave@bugzilla.org)
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-01-02 21:49 PST by Dave Miller [:justdave] (justdave@bugzilla.org)
Modified: 2012-09-18 23:12 PDT (History)
5 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Dave Miller [:justdave] (justdave@bugzilla.org) 2009-01-02 21:49:58 PST
This has been complained about loudly in the past in email to Sheppy but I don't see a bug, and I don't see mention of this being fixed on the Deki page Sheppy just linked to in his blog for the upcoming Lyons release, so creating a bug now to make sure it doesn't get forgotten.

I get paged about 2 to 3 times a day about replication in the C01 database cluster failing and have to go kick it, and 99% of the time it's Dekiwiki that's caused it by trying to write to in-memory tables that were created before the last time the slave database was restarted.

Restarting the dekiwiki application on either of the webheads will usually fix this problem for several days, until the next time we trade slaves around (except for the server that runs the backups, since it gets shut down every night for the backups to run anyway - so I still have to go fix it all the time).

The backups are probably corrupted because of this, btw.  It's taking the results of what it puts in that temp table and using it for the links and broken_links tables (which aren't temporary tables).  So I'm pretty sure what's getting backed up from those two tables probably doesn't match what's in the master database.

The problem seems to be that the application creates this table in memory at startup if it doesn't exist.  All times in the future when it wants to use it, it issues a "truncate table `temp`" statement first before doing all the inserts, then selecting from it to insert into one of the above two tables.

What they should really do is drop the table as soon as the inserts into the other tables are done, and recreate it from scratch every time they need it.  This would *vastly* reduce the likelihood of the table being missing when it's wanted during a replication stream.
Comment 1 Dave Miller [:justdave] (justdave@bugzilla.org) 2009-01-03 04:22:29 PST
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'22744'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',38872), value from temp'
Comment 2 Dave Miller [:justdave] (justdave@bugzilla.org) 2009-01-03 15:53:54 PST
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'38883'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'35'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',38883), value from temp'
Comment 3 Dave Miller [:justdave] (justdave@bugzilla.org) 2009-01-03 20:13:15 PST
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'4180'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36033'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36027'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36034'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',36026), value from temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'4180'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36033'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36027'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36034'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',36026), value from temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'4180'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36033'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36027'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36026'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',36034), value from temp'
Comment 4 Dave Miller [:justdave] (justdave@bugzilla.org) 2009-01-05 06:36:05 PST
This has happened 10 or 15 times since I last posted here, figured you'd seen enough to give them an idea what was happening by now.  Here's a error list from the other table, since it wasn't mentioned yet in any of the above log snippets:

Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.severity'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.message'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.type'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.relatedException'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.relatedData'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.location'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO brokenlinks (bl_from, bl_to) SELECT  NAME_CONST('pageid',38945), value from temp'
Comment 5 Eric Shepherd [:sheppy] 2009-01-05 09:12:52 PST
Filed with MindTouch:

http://bugs.developer.mindtouch.com/view.php?id=5543
Comment 6 Dave Miller [:justdave] (justdave@bugzilla.org) 2009-01-09 10:15:31 PST
Mindtouch has a patch on that bug that they'd like us to test.
Comment 7 Jeremy Orem [:oremj] 2009-01-09 10:25:40 PST
The patch has been applied.
Comment 8 Dave Miller [:justdave] (justdave@bugzilla.org) 2009-01-09 10:31:47 PST
(In reply to comment #7)
> The patch has been applied.

In production or on a staging server or?
Comment 9 Jeremy Orem [:oremj] 2009-01-09 10:57:20 PST
Production.
Comment 10 Dave Miller [:justdave] (justdave@bugzilla.org) 2009-01-09 11:42:11 PST
Cool.  I've disabled the auto-repair script I had running, we'll see if I get paged at all in the next couple days.
Comment 11 Dave Miller [:justdave] (justdave@bugzilla.org) 2009-01-22 16:05:12 PST
The patch turns out to have been to one of the installation scripts, and not live code, and said script actually has to be run to take effect.  Scheduling a downtime window for this since it's essentially making schema changes.
Comment 12 Dave Miller [:justdave] (justdave@bugzilla.org) 2009-01-22 21:46:56 PST
OK, this update has now been applied to both the library.mozilla.org and developer.mozilla.org databases.

The auto-recovery script has been disabled, so if it still happens at all I should get paged again right away.
Comment 13 Dave Miller [:justdave] (justdave@bugzilla.org) 2009-02-20 11:43:18 PST
This was fixed a while back.

But see bug 478458 for a similar problem.

Note You need to log in before you can comment on or make changes to this bug.