Dekiwiki causes replication failures on the slave databases frequently

RESOLVED FIXED

Status

Mozilla Developer Network
General
--
critical
RESOLVED FIXED
8 years ago
5 years ago

People

(Reporter: justdave, Assigned: justdave)

Tracking

Details

This has been complained about loudly in the past in email to Sheppy but I don't see a bug, and I don't see mention of this being fixed on the Deki page Sheppy just linked to in his blog for the upcoming Lyons release, so creating a bug now to make sure it doesn't get forgotten.

I get paged about 2 to 3 times a day about replication in the C01 database cluster failing and have to go kick it, and 99% of the time it's Dekiwiki that's caused it by trying to write to in-memory tables that were created before the last time the slave database was restarted.

Restarting the dekiwiki application on either of the webheads will usually fix this problem for several days, until the next time we trade slaves around (except for the server that runs the backups, since it gets shut down every night for the backups to run anyway - so I still have to go fix it all the time).

The backups are probably corrupted because of this, btw.  It's taking the results of what it puts in that temp table and using it for the links and broken_links tables (which aren't temporary tables).  So I'm pretty sure what's getting backed up from those two tables probably doesn't match what's in the master database.

The problem seems to be that the application creates this table in memory at startup if it doesn't exist.  All times in the future when it wants to use it, it issues a "truncate table `temp`" statement first before doing all the inserts, then selecting from it to insert into one of the above two tables.

What they should really do is drop the table as soon as the inserts into the other tables are done, and recreate it from scratch every time they need it.  This would *vastly* reduce the likelihood of the table being missing when it's wanted during a replication stream.
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'22744'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',38872), value from temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'38883'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'35'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',38883), value from temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'4180'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36033'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36027'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36034'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',36026), value from temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'4180'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36033'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36027'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36034'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',36026), value from temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'4180'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36033'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36027'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36026'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',36034), value from temp'
This has happened 10 or 15 times since I last posted here, figured you'd seen enough to give them an idea what was happening by now.  Here's a error list from the other table, since it wasn't mentioned yet in any of the above log snippets:

Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.severity'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.message'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.type'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.relatedException'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.relatedData'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.location'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO brokenlinks (bl_from, bl_to) SELECT  NAME_CONST('pageid',38945), value from temp'
Filed with MindTouch:

http://bugs.developer.mindtouch.com/view.php?id=5543
Mindtouch has a patch on that bug that they'd like us to test.
Assignee: nobody → oremj

Comment 7

8 years ago
The patch has been applied.
(In reply to comment #7)
> The patch has been applied.

In production or on a staging server or?

Comment 9

8 years ago
Production.
Cool.  I've disabled the auto-repair script I had running, we'll see if I get paged at all in the next couple days.
Assignee: oremj → justdave
The patch turns out to have been to one of the installation scripts, and not live code, and said script actually has to be run to take effect.  Scheduling a downtime window for this since it's essentially making schema changes.
OK, this update has now been applied to both the library.mozilla.org and developer.mozilla.org databases.

The auto-recovery script has been disabled, so if it still happens at all I should get paged again right away.
This was fixed a while back.

But see bug 478458 for a similar problem.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Component: Deki Infrastructure → Other
Product: Mozilla Developer Network → Mozilla Developer Network
You need to log in before you can comment on or make changes to this bug.