381500 - Possible AMO mysql DB replication failure

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Michael Morgan [:morgamic]]

We've had scattered reports of failed updates.  These failed updates all have correct DB records and their public listings are all up to date.  Yet when we check the update script old data is being served back -- for multiple extensions and authors.

Could someone check the configuration for the read-only db in AMO production?  It is SHADOW_DB in the config.

Also, could you do a quick check on replication of the AMO db overall?  Something seems awry.

Thanks. :)

Comment 1

[Aravind Gottipati [:aravind]]

which update script are you talking about?

Slave status looks okay.

Comment 2

[Michael Morgan [:morgamic]]

Here is a query to use to check replication.  It's taken from one of the problem cases.  Can you see if you get different results between master/slave?

        SELECT
            addons.guid as guid,
            addons.id as id,
            addons.addontype_id as type,
            applications.guid as appguid,
            appmin.version as min,
            appmax.version as max,
            files.id as file_id,
            files.hash,
            files.filename,
            versions.version as version
        FROM
            versions 
        INNER JOIN addons ON versions.addon_id = addons.id AND addons.guid = '{d9284e50-81fc-11da-a72b-0800200c9a66}' AND addons.status = 4 AND addons.inactive=0
        INNER JOIN applications_versions ON applications_versions.version_id = versions.id 
        INNER JOIN applications ON applications_versions.application_id = applications.id  AND applications.guid = '{ec8030f7-c20a-464f-9b0e-13a3a9e97384}'       
        INNER JOIN appversions appmin ON appmin.id = applications_versions.min
        INNER JOIN appversions appmax ON appmax.id = applications_versions.max  
        INNER JOIN files ON files.version_id = versions.id AND (files.platform_id = 1 OR files.platform_id = '3' ) AND files.status = 4
        ORDER BY
            versions.id DESC
        LIMIT 1;

Thanks.

Comment 3

[Aravind Gottipati [:aravind]]

From master:
{d9284e50-81fc-11da-a72b-0800200c9a66}  1833    1       {ec8030f7-c20a-464f-9b0e-13a3a9e97384}  1.5     2.0.0.* 15834   sha1:dbbff2e1368d51c1f1945523a53578aabf4a459a   yoono-4.0.0.4529-fx.xpi  4.0.0.4529

From slave:
{d9284e50-81fc-11da-a72b-0800200c9a66}  1833    1       {ec8030f7-c20a-464f-9b0e-13a3a9e97384}  1.5     2.0.0.* 15834   sha1:dbbff2e1368d51c1f1945523a53578aabf4a459a   yoono-4.0.0.4529-fx.xpi  4.0.0.4529

Comment 4

[Aravind Gottipati [:aravind]]

sorry, those were both from the master.

Here is the output from the slave

{d9284e50-81fc-11da-a72b-0800200c9a66}  1833    1       {ec8030f7-c20a-464f-9b0e-13a3a9e97384}  1.5     2.0.0.* 7465    sha1:4d82c8308328cd73e7de4dba9730badf7a264728   yoono-3.1.1.2999-fx.xpi  3.1.1.2999

Comment 5

[Justin Scott [:fligtar]]

Looks like there is a problem then :(

Comment 6

[Aravind Gottipati [:aravind]]

    Fixed the problem that was causing slave replication to hang.

    max_allowed_packet=32M

    This was set to 32M on the master and 16M on the slave.

    This caused us to hit this mysql bug http://bugs.mysql.com/bug.php?id=26201

    Replication should catch up in about 30 minutes or so..


Thanks to oremj for pointing me in the right direction.