1060651 - datazilla2.db.scl3 The table 'objectstore' is full

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Matt Pressman [:mpressman]]

It looks like this has hit again.
Error 'The table 'objectstore' is full' on query. Default database: 'talos_objectstore_1'

Comment 1

[Matt Pressman [:mpressman]]

I'm not as familiar with this engine. I see where the command:
alter table objectstore MAX_DATA_LENGTH=137438969856;

was run, but it doesn't look to have taken as table status shows Max_data_length: 68719484928
show table status like 'objectstore'\G
*************************** 1. row ***************************
           Name: objectstore
         Engine: Aria
        Version: 10
     Row_format: Page
           Rows: 4336809
 Avg_row_length: 15845
    Data_length: 68719370240
Max_data_length: 68719484928
   Index_length: 237035520
      Data_free: 0
 Auto_increment: 6604343
    Create_time: 2012-12-18 15:29:49
    Update_time: 2014-08-29 12:06:06
     Check_time: 2014-04-02 15:56:50
      Collation: utf8_general_ci
       Checksum: NULL
 Create_options: avg_row_length=31456 page_checksum=1

I found this https://mariadb.atlassian.net/browse/MDEV-3815 where it says this is a bug and the workaround is to increase the max_rows to force 6 byte pointers

Comment 2

[Matt Pressman [:mpressman]]

I ran:
mysql> alter table objectstore MAX_ROWS=10000000;
Query OK, 4336809 rows affected (40 min 28.92 sec)
Records: 4336809  Duplicates: 0  Warnings: 0

This seems to have done the trick. I restarted replication and, although behind, it's now able to run:
mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: datazilla1.db.scl3.mozilla.com
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: datazilla1-bin.001768
          Read_Master_Log_Pos: 864345969
               Relay_Log_File: datazilla2-relay-bin.000942
                Relay_Log_Pos: 204197076
        Relay_Master_Log_File: datazilla1-bin.001768
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 204196787
              Relay_Log_Space: 864346483
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 229441
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 22070046
1 row in set (0.00 sec)

Comment 3

[Matt Pressman [:mpressman]]

Leaving this bug open in case I am completely wrong as to how I fixed this and it can be corrected since we are now limited to a fixed set of rows

Comment 4

[Sheeri Cabral [:sheeri]]

FWIW, at least in this case, Aria is the same as MyISAM. I ran into this issue last week too and increased it - it was found when the replication check was fixed. I did not make a bug because I assumed it was a one-time problem.

Datazilla is going away by the end of the year if all goes well, so even if we need to increase this parameter a few more times, I think that's the right thing to do.