471945 - Dekiwiki causes replication failures on the slave databases frequently

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: General, defect)

Description

[Dave Miller [:justdave]]

This has been complained about loudly in the past in email to Sheppy but I don't see a bug, and I don't see mention of this being fixed on the Deki page Sheppy just linked to in his blog for the upcoming Lyons release, so creating a bug now to make sure it doesn't get forgotten.

I get paged about 2 to 3 times a day about replication in the C01 database cluster failing and have to go kick it, and 99% of the time it's Dekiwiki that's caused it by trying to write to in-memory tables that were created before the last time the slave database was restarted.

Restarting the dekiwiki application on either of the webheads will usually fix this problem for several days, until the next time we trade slaves around (except for the server that runs the backups, since it gets shut down every night for the backups to run anyway - so I still have to go fix it all the time).

The backups are probably corrupted because of this, btw.  It's taking the results of what it puts in that temp table and using it for the links and broken_links tables (which aren't temporary tables).  So I'm pretty sure what's getting backed up from those two tables probably doesn't match what's in the master database.

The problem seems to be that the application creates this table in memory at startup if it doesn't exist.  All times in the future when it wants to use it, it issues a "truncate table `temp`" statement first before doing all the inserts, then selecting from it to insert into one of the above two tables.

What they should really do is drop the table as soon as the inserts into the other tables are done, and recreate it from scratch every time they need it.  This would *vastly* reduce the likelihood of the table being missing when it's wanted during a replication stream.

Comment 1

[Dave Miller [:justdave]]

Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'22744'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',38872), value from temp'

Comment 2

[Dave Miller [:justdave]]

Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'38883'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'35'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',38883), value from temp'

Comment 3

[Dave Miller [:justdave]]

Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'4180'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36033'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36027'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36034'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',36026), value from temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'4180'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36033'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36027'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36034'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',36026), value from temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'4180'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36033'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36027'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'36026'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO links (l_from, l_to) SELECT  NAME_CONST('pageid',36034), value from temp'

Comment 4

[Dave Miller [:justdave]]

This has happened 10 or 15 times since I last posted here, figured you'd seen enough to give them an idea what was happening by now.  Here's a error list from the other table, since it wasn't mentioned yet in any of the above log snippets:

Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'truncate table temp'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.severity'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.message'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.type'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.relatedException'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.relatedData'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'insert temp values( NAME_CONST('_chunk',_utf8'En/DOM/DOMError.location'))'
Error 'Table 'wiki_mdc_deki.temp' doesn't exist' on query. Default database: 'wiki_mdc_deki'. Query: 'INSERT IGNORE INTO brokenlinks (bl_from, bl_to) SELECT  NAME_CONST('pageid',38945), value from temp'

Comment 5

[Eric Shepherd [:sheppy]]

Filed with MindTouch:

http://bugs.developer.mindtouch.com/view.php?id=5543

Comment 6

[Dave Miller [:justdave]]

Mindtouch has a patch on that bug that they'd like us to test.

Comment 7

[Jeremy Orem [:oremj]]

The patch has been applied.

Comment 8

[Dave Miller [:justdave]]

(In reply to comment #7)
> The patch has been applied.

In production or on a staging server or?

Comment 9

[Jeremy Orem [:oremj]]

Production.

Comment 10

[Dave Miller [:justdave]]

Cool.  I've disabled the auto-repair script I had running, we'll see if I get paged at all in the next couple days.

Comment 11

[Dave Miller [:justdave]]

The patch turns out to have been to one of the installation scripts, and not live code, and said script actually has to be run to take effect.  Scheduling a downtime window for this since it's essentially making schema changes.

Comment 12

[Dave Miller [:justdave]]

OK, this update has now been applied to both the library.mozilla.org and developer.mozilla.org databases.

The auto-recovery script has been disabled, so if it still happens at all I should get paged again right away.

Comment 13

[Dave Miller [:justdave]]

This was fixed a while back.

But see bug 478458 for a similar problem.