565289 - SUMO down: SQL error with FORCE INDEX

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[matthew zeier [:mrz]]

Attachment: curl output

Comment 1

[matthew zeier [:mrz]]

Attachment: screenshot

Comment 2

[matthew zeier [:mrz]]

Looks like the code's going into tiki-setup_base.php and spewing that error.  No app level debugging that I can find.

Comment 3

[James Socol [:jsocol, :james]]

At webroot/lib/userslib.php:1739 a FORCE INDEX clause (see line 1715) is causing the query to fail. No idea why this just started, we've never seen it on stage, and it was fine for several hours before failing.

We've patched the query on production and will investigate this today.

Comment 4

[James Socol [:jsocol, :james]]

Looks like the problem is: FORCE INDEX (permName) requires a *key*, not a column, named `permName`. Line 137303 of webroot/db/sumo_clean_schema.sql creates a UNIQUE key named `permName` on the column `permName`.

Our database conversion script had to drop the UNIQUE key to do the charset conversion but did not re-add it.

It looks like this bug was hidden by the tiki file cache--though clearing it is part of the normal cache-clearing steps so I'm not 100% sure why we didn't start seeing it immediately.

I'm guessing we didn't see it on stage because we didn't do a full cache clear after doing the database conversion, so this query never ran.

Comment 5

[James Socol [:jsocol, :james]]

This is patched and working in production. Filed bug 565320 to fix it permanently. Also filed bug 565317 to help keep bugs like this from hiding on stage.

Comment 6

[James Socol [:jsocol, :james]]

For the record, I'm fairly certain this is the same problem that initially presented after the first push last Thursday.