b01 database master crashing frequently

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
--
major
RESOLVED FIXED
8 years ago
3 years ago

People

(Reporter: justdave, Assigned: justdave)

Tracking

Details

tm-b01-master01 has hard-crashed 3 times today, all three times with the exact same crash dump.  The query in the buffer at crash time seems to suggest that Deki is somehow triggering it, though this is obviously a bug in MySQL...

090815 22:17:43 - mysqld got signal 6;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=125829120
read_buffer_size=131072
max_used_connections=35
max_connections=1200
threads_connected=28
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 7649270 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x2aaab4011040
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x45546fb0, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
(nil)
Stack trace seems successful - bottom reached
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x259fb570 = SELECT IF(right( NAME_CONST('_value',_utf8'zh_tw/Core_JavaScript_1.5_教學/建立新的物件/Defining_Methods\0zh_tw/Core_JavaScript_1.5_教學/建立新的物件/Using_this_for_Object_References\0zh_tw/Core_JavaScript_1.5_教學/建立新的物件/Defining_Getters_and_Setters\0zh_tw/Core_JavaScript_1.5_教學/建立新的物件/Deleting_Properties\0zh_tw/Core_JavaScript_1.5_教學/預先定義的核心物件\0zh_tw/Core_JavaScript_1.5_教學/預先定義的核心物件/Array_Object\0zh_tw/Core_JavaScript_1.5_教學/預先定義的核心物件/Boolean_Object\0zh_tw/Core_JavaScript_1.5_教學/預先定義的核心物件/Date_Object\0zh_tw/Core_JavaScript_1.5_教學/預先定義的核心物件/Function_Object\0zh_tw/Core_JavaScript_1.5_教學/預先定義的核心物件/Math_Object\0zh_tw/Core_JavaScript_1.5_教學/預先定義的核心物件/Number_Object\0zh_tw/Core_JavaScript_1.5_教學/預先定義的核心物件/RegExp_Object\0zh_tw/Core_JavaScript_1.5_教學/預先定義的核心物件/String_Object\0zh
thd->thread_id=1107
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
Fatal signal 11 while backtracing
Assignee: server-ops → justdave
I've emailed the CEO of MindTouch along with my usual contact there, with a copy of that information. Have we done anything about talking to people that work on MySQL, if we're reasonably sure this is a bug in MySQL?
ooh, just realized (when I was going through closing old terminal windows) that we actually got a real backtrace on the third one...  It printed it to the console instead of to the log, and since I was still logged in, it was on my console window.

*** glibc detected *** /usr/libexec/mysqld: malloc(): memory corruption: 0x00000000259fd5f0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3d178724ac]
/lib64/libc.so.6(__libc_malloc+0x7a)[0x3d1787402a]
/usr/libexec/mysqld(my_malloc+0x33)[0x832f63]
/usr/libexec/mysqld(init_alloc_root+0x7e)[0x833b2e]
/usr/libexec/mysqld(_Z11open_tablesP3THDPP13st_table_listPjj+0x34)[0x59a244]
/usr/libexec/mysqld(_Z20open_and_lock_tablesP3THDP13st_table_list+0x59)[0x59a769]
/usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x429)[0x56fca9]
/usr/libexec/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x1c)[0x66e7dc]
/usr/libexec/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0xac)[0x66e95c]
/usr/libexec/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x21c)[0x6731fc]
/usr/libexec/mysqld(_ZN7sp_head7executeEP3THD+0x41e)[0x6720be]
/usr/libexec/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x35d)[0x67271d]
/usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x45bf)[0x573e3f]
/usr/libexec/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x1c)[0x66e7dc]
/usr/libexec/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0xac)[0x66e95c]
/usr/libexec/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x21c)[0x6731fc]
/usr/libexec/mysqld(_ZN7sp_head7executeEP3THD+0x41e)[0x6720be]
/usr/libexec/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x35d)[0x67271d]
/usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x45bf)[0x573e3f]
/usr/libexec/mysqld(_Z11mysql_parseP3THDPKcjPS2_+0x193)[0x574c83]
/usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x440)[0x5751a0]
/usr/libexec/mysqld(_Z10do_commandP3THD+0xb3)[0x576483]
/usr/libexec/mysqld(handle_one_connection+0xa32)[0x577002]
/lib64/libpthread.so.0[0x3d18c06367]
/lib64/libc.so.6(clone+0x6d)[0x3d178d309d]
======= Memory map: ========
00400000-00a7a000 r-xp 00000000 fd:00 20092840                           /usr/libexec/mysqld
00c7a000-00ce6000 rw-p 0067a000 fd:00 20092840                           /usr/libexec/mysqld
00ce6000-00cf7000 rw-p 00ce6000 00:00 0 
00ee5000-00f45000 rw-p 006e5000 fd:00 20092840                           /usr/libexec/mysqld
1856c000-26f15000 rw-p 1856c000 00:00 0                                  [heap]
400df000-400e0000 ---p 400df000 00:00 0 
400e0000-40ae0000 rw-p 400e0000 00:00 0 
40ae0000-40ae1000 ---p 40ae0000 00:00 0 
40ae1000-414e1000 rw-p 40ae1000 00:00 0 
414e1000-414e2000 ---p 414e1000 00:00 0 
414e2000-41522000 rw-p 414e2000 00:00 0 
41522000-41523000 ---p 41522000 00:00 0 
41523000-41563000 rw-p 41523000 00:00 0 
41563000-41564000 ---p 41563000 00:00 0 
41564000-415a4000 rw-p 41564000 00:00 0 
4183e000-4183f000 ---p 4183e000 00:00 0 
4183f000-4223f000 rw-p 4183f000 00:00 0 
4223f000-42240000 ---p 4223f000 00:00 0 
42240000-42c40000 rw-p 42240000 00:00 0 
42c40000-42c41000 ---p 42c40000 00:00 0 
42c41000-43641000 rw-p 42c41000 00:00 0 
43641000-43642000 ---p 43641000 00:00 0 
43642000-44042000 rw-p 43642000 00:00 0 
44042000-44043000 ---p 44042000 00:00 0 
44043000-44a43000 rw-p 44043000 00:00 0 
44a43000-44a44000 ---p 44a43000 00:00 0 
44a44000-45444000 rw-p 44a44000 00:00 0 
45444000-45445000 ---p 45444000 00:00 0 
45445000-45485000 rw-p 45445000 00:00 0 
45485000-45486000 ---p 45485000 00:00 0 
45486000-454c6000 rw-p 45486000 00:00 0 
454c6000-454c7000 ---p 454c6000 00:00 0 
454c7000-45507000 rw-p 454c7000 00:00 0 
45507000-45508000 ---p 45507000 00:00 0 
45508000-45548000 rw-p 45508000 00:00 0 
45548000-45549000 ---p 45548000 00:00 0 
45549000-45589000 rw-p 45549000 00:00 0 
45589000-4558a000 ---p 45589000 00:00 0 
4558a000-455ca000 rw-p 4558a000 00:00 0 
455ca000-455cb000 ---p 455ca000 00:00 0 
455cb000-4560b000 rw-p 455cb000 00:00 0 
4560b000-4560c000 ---p 4560b000 00:00 0 
4560c000-4564c000 rw-p 4560c000 00:00 0 
4564c000-4564d000 ---p 4564c000 00:00 0 
4564d000-4568d000 rw-p 4564d000 00:00 0 
4568d000-4568e000 ---p 4568d000 00:00 0 
4568e000-456ce000 rw-p 4568e000 00:00 0 
456ce000-456cf000 ---p 456ce000 00:00 0 
456cf000-4570f000 rw-p 456cf000 00:00 0 
4570f000-45710000 ---p 4570f000 00:00 0 
45710000-45750000 rw-p 45710000 00:00 0 
45750000-45751000 ---p 45750000 00:00 0 
45751000-45791000 rw-p 45751000 00:00 0 
45791000-45792000 ---p 45791000 00:00 0 
45792000-457d2000 rw-p 45792000 00:00 0 
457d2000-457d3000 ---p 457d2000 00:00 0 
457d3000-45813000 rw-p 457d3000 00:00 0 
45813000-45814000 ---p 45813000 00:00 0 
45814000-45854000 rw-p 45814000 00:00 0 
45854000-45855000 ---p 45854000 00:00 0 
45855000-45895000 rw-p 45855000 00:00 0 
45895000-45896000 ---p 45895000 00:00 0 
45896000-458d6000 rw-p 45896000 00:00 0 
458d6000-458d7000 ---p 458d6000 00:00 0 
458d7000-45917000 rw-p 458d7000 00:00 0 
45917000-45918000 ---p 45917000 00:00 0 
45918000-45958000 rw-p 45918000 00:00 0 
45958000-45959000 ---p 45958000 00:00 0 
45959000-45999000 rw-p 45959000 00:00 0 
45999000-4599a000 ---p 45999000 00:00 0 
4599a000-459da000 rw-p 4599a000 00:00 0 
459da000-459db000 ---p 459da000 00:00 0 
459db000-45a1b000 rw-p 459db000 00:00 0 
45a1b000-45a1c000 ---p 45a1b000 00:00 0 
45a1c000-45a5c000 rw-p 45a1c000 00:00 0 
45a5c000-45a5d000 ---p 45a5c000 00:00 0 
45a5d000-45a9d000 rw-p 45a5d000 00:00 0 
45a9d000-45a9e000 ---p 45a9d000 00:00 0 
45a9e000-45ade000 rw-p 45a9e000 00:00 0 
45ade000-45adf000 ---p 45ade000 00:00 0 
45adf000-45b1f000 rw-p 45adf000 00:00 0 
45b1f000-45b20000 ---p 45b1f000 00:00 0 
45b20000-45b60000 rw-p 45b20000 00:00 0 
45b60000-45b61000 ---p 45b60000 00:00 0 
45b61000-45ba1000 rw-p 45b61000 00:00 0 
45ba1000-45ba2000 ---p 45ba1000 00:00 0 
45ba2000-45be2000 rw-p 45ba2000 00:00 0 
45be2000-45be3000 ---p 45be2000 00:00 0 
45be3000-45c23000 rw-p 45be3000 00:00 0 
45c23000-45c24000 ---p 45c23000 00:00 0 
45c24000-45c64000 rw-p 45c24000 00:00 0 
45c64000-45c65000 ---p 45c64000 00:00 0 
45c65000-45ca5000 rw-p 45c65000 00:00 0 
45ca5000-45ca6000 ---p 45ca5000 00:00 0 
45ca6000-45ce6000 rw-p 45ca6000 00:00 0 
3d17400000-3d1741c000 r-xp 00000000 fd:00 1507330                        /lib64/ld-2.5.so
3d1761b000-3d1761c000 r--p 0001b000 fd:00 1507330                        /lib64/ld-2.5.so
3d1761c000-3d1761d000 rw-p 0001c000 fd:00 1507330                        /lib64/ld-2.5.so
3d17800000-3d1794c000 r-xp 00000000 fd:00 1507331                        /lib64/libc-2.5.so
3d1794c000-3d17b4c000 ---p 0014c000 fd:00 1507331                        /lib64/libc-2.5.so
3d17b4c000-3d17b50000 r--p 0014c000 fd:00 1507331                        /lib64/libc-2.5.so
3d17b50000-3d17b51000 rw-p 00150000 fd:00 1507331                        /lib64/libc-2.5.so
3d17b51000-3d17b56000 rw-p 3d17b51000 00:00 0 
3d17c00000-3d17c02000 r-xp 00000000 fd:00 1507338                        /lib64/libdl-2.5.so
3d17c02000-3d17e02000 ---p 00002000 fd:00 1507338                        /lib64/libdl-2.5.so
3d17e02000-3d17e03000 r--p 00002000 fd:00 1507338                        /lib64/libdl-2.5.so
3d17e03000-3d17e04000 rw-p 00003000 fd:00 1507338                        /lib64/libdl-2.5.so
3d18000000-3d18015000 r-xp 00000000 fd:00 1507344                        /lib64/libselinux.so.1
3d18015000-3d18215000 ---p 00015000 fd:00 1507344                        /lib64/libselinux.so.1
3d18215000-3d18217000 rw-p 00015000 fd:00 1507344                        /lib64/libselinux.so.1
3d18217000-3d18218000 rw-p 3d18217000 00:00 0 
3d18400000-3d1843b000 r-xp 00000000 fd:00 1507343                        /lib64/libsepol.so.1
3d1843b000-3d1863b000 ---p 0003b000 fd:00 1507343                        /lib64/libsepol.so.1
3d1863b000-3d1863c000 rw-p 0003b000 fd:00 1507343                        /lib64/libsepol.so.1
3d1863c000-3d18646000 rw-p 3d1863c000 00:00 0 
3d18800000-3d18882000 r-xp 00000000 fd:00 1507334                        /lib64/libm-2.5.so
3d18882000-3d18a81000 ---p 00082000 fd:00 1507334                        /lib64/libm-2.5.so
3d18a81000-3d18a82000 r--p 00081000 fd:00 1507334                        /lib64/libm-2.5.so
3d18a82000-3d18a83000 rw-p 00082000 fd:00 1507334                        /lib64/libm-2.5.so
3d18c00000-3d18c16000 r-xp 00000000 fd:00 1507341                        /lib64/libpthread-2.5.so
3d18c16000-3d18e15000 ---p 00016000 fd:00 1507341                        /lib64/libpthread-2.5.so
3d18e15000-3d18e16000 r--p 00015000 fd:00 1507341                        /lib64/libpthread-2.5.so
3d18e16000-3d18e17000 rw-p 00016000 fd:00 1507341                        /lib64/libpthread-2.5.so
3d18e17000-3d18e1b000 rw-p 3d18e17000 00:00 0 
3d19000000-3d19014000 r-xp 00000000 fd:00 20099417                       /usr/lib64/libz.so.1.2.3
3d19014000-3d19213000 ---p 00014000 fd:00 20099417                       /usr/lib64/libz.so.1.2.3
3d19213000-3d19214000 rw-p 00013000 fd:00 20099417                       /usr/lib64/libz.so.1.2.3
3d19400000-3d19415000 r-xp 00000000 fd:00 1507336                        /lib64/libnsl-2.5.so
3d19415000-3d19614000 ---p 00015000 fd:00 1507336                        /lib64/libnsl-2.5.so
3d19614000-3d19615000 r--p 00014000 fd:00 1507336                        /lib64/libnsl-2.5.so
3d19615000-3d19616000 rw-p 00015000 fd:00 1507336                        /lib64/libnsl-2.5.so
3d19616000-3d19618000 rw-p 3d19616000 00:00 0 
3d19800000-3d19809000 r-xp 00000000 fd:00 1507339                        /lib64/libcrypt-2.5.so
3d19809000-3d19a08000 ---p 00009000 fd:00 1507339                        /lib64/libcrypt-2.5.so
3d19a08000-3d19a09000 r--p 00008000 fd:00 1507339                        /lib64/libcrypt-2.5.so
3d19a09000-3d19a0a000 rw-p 00009000 fd:00 1507339                        /lib64/libcrypt-2.5.so
3d19a0a000-3d19a38000 rw-p 3d19a0a000 00:00 0 
3d19c00000-3d19c11000 r-xp 00000000 fd:00 1507337                        /lib64/libresolv-2.5.so
3d19c11000-3d19e11000 ---p 00011000 fd:00 1507337                        /lib64/libresolv-2.5.so
3d19e11000-3d19e12000 r--p 00011000 fd:00 1507337                        /lib64/libresolv-2.5.so
3d19e12000-3d19e13000 rw-p 00012000 fd:00 1507337                        /lib64/libresolv-2.5.so
3d19e13000-3d19e15000 rw-p 3d19e13000 00:00 0 
3d1a000000-3d1a002000 r-xp 00000000 fd:00 1507342                        /lib64/libkeyutils-1.2.so
3d1a002000-3d1a201000 ---p 00002000 fd:00 1507342                        /lib64/libkeyutils-1.2.so
3d1a201000-3d1a202000 rw-p 00001000 fd:00 1507342                        /lib64/libkeyutils-1.2.so
3d1a400000-3d1a402000 r-xp 00000000 fd:00 1507345                        /lib64/libcom_err.so.2.1
3d1a402000-3d1a601000 ---p 00002000 fd:00 1507345                        /lib64/libcom_err.so.2.1
3d1a601000-3d1a602000 rw-p 00001000 fd:00 1507345                        /lib64/libcom_err.so.2.1
3d1a800000-3d1a807000 r-xp 00000000 fd:00 1507352                        /lib64/librt-2.5.so
3d1a807000-3d1aa07000 ---p 00007000 fd:00 1507352                        /lib64/librt-2.5.so
3d1aa07000-3d1aa08000 r--p 00007000 fd:00 1507352                        /lib64/librt-2.5.so
3d1aa08000-3d1aa09000 rw-p 00008000 fd:00 1507352                        /lib64/librt-2.5.so
3d1ac00000-3d1ad2d000 r-xp 00000000 fd:00 1507346                        /lib64/libcrypto.so.0.9.8e
3d1ad2d000-3d1af2c000 ---p 0012d000 fd:00 1507346                        /lib64/libcrypto.so.0.9.8e
3d1af2c000-3d1af4d000 rw-p 0012c000 fd:00 1507346                        /lib64/libcrypto.so.0.9.8e
3d1af4d000-3d1af51000 rw-p 3d1af4d000 00:00 0 
3d1b000000-3d1b008000 r-xp 00000000 fd:00 20099413                       /usr/lib64/libkrb5support.so.0.1
3d1b008000-3d1b207000 ---p 00008000 fd:00 20099413                       /usr/lib64/libkrb5support.so.0.1
3d1b207000-3d1b208000 rw-p 00007000 fd:00 20099413                       /usr/lib64/libkrb5support.so.0.1
3d1b400000-3d1b424000 r-xp 00000000 fd:00 20099414                       /usr/lib64/libk5crypto.so.3.1
3d1b424000-3d1b623000 ---p 00024000 fd:00 20099414                       /usr/lib64/libk5crypto.so.3.1
3d1b623000-3d1b625000 rw-p 00023000 fd:00 20099414                       /usr/lib64/libk5crypto.so.3.1
3d1b800000-3d1b82c000 r-xp 00000000 fd:00 20099416                       /usr/lib64/libgssapi_krb5.so.2.2
3d1b82c000-3d1ba2c000 ---p 0002c000 fd:00 20099416                       /usr/lib64/libgssapi_krb5.so.2.2
3d1ba2c000-3d1ba2e000 rw-p 0002c000 fd:00 20099416                       /usr/lib64/libgssapi_krb5.so.2.2
3d1bc00000-3d1bc91000 r-xp 00000000 fd:00 20099415                       /usr/lib64/libkrb5.so.3.3
3d1bc91000-3d1be91000 ---p 00091000 fd:00 20099415                       /usr/lib64/libkrb5.so.3.3
3d1be91000-3d1be95000 rw-p 00091000 fd:00 20099415                       /usr/lib64/libkrb5.so.3.3
3d1c000000-3d1c00d000 r-xp 00000000 fd:00 1507335                        /lib64/libgcc_s-4.1.2-20080825.so.1
3d1c00d000-3d1c20d000 ---p 0000d000 fd:00 1507335                        /lib64/libgcc_s-4.1.2-20080825.so.1
3d1c20d000-3d1c20e000 rw-p 0000d000 fd:00 1507335                        /lib64/libgcc_s-4.1.2-20080825.so.1
3d1c400000-3d1c444000 r-xp 00000000 fd:00 1507347                        /lib64/libssl.so.0.9.8e
3d1c444000-3d1c644000 ---p 00044000 fd:00 1507347                        /lib64/libssl.so.0.9.8e
3d1c644000-3d1c64a000 rw-p 00044000 fd:00 1507347                        /lib64/libssl.so.0.9.8e
3d1c800000-3d1c8e6000 r-xp 00000000 fd:00 20092236                       /usr/lib64/libstdc++.so.6.0.8
3d1c8e6000-3d1cae5000 ---p 000e6000 fd:00 20092236                       /usr/lib64/libstdc++.so.6.0.8
3d1cae5000-3d1caeb000 r--p 000e5000 fd:00 20092236                       /usr/lib64/libstdc++.so.6.0.8
3d1caeb000-3d1caee000 rw-p 000eb000 fd:00 20092236                       /usr/lib64/libstdc++.so.6.0.8
3d1caee000-3d1cb00000 rw-p 3d1caee000 00:00 0 
2aaaaaaac000-2aaab3355000 rw-p 2aaaaaaac000 00:00 0 
2aaab335e000-2aaab3368000 r-xp 00000000 fd:00 21364790                   /lib64/libnss_files-2.5.so
2aaab3368000-2aaab3567000 ---p 0000a000 fd:00 21364790                   /lib64/libnss_files-2.5.so
2aaab3567000-2aaab3568000 r--p 00009000 fd:00 21364790                   /lib64/libnss_files-2.5.so
2aaab3568000-2aaab3569000 rw-p 0000a000 fd:00 21364790                   /lib64/libnss_files-2.5.so
2aaab4000000-2aaab46a7000 rw-p 2aaab4000000 00:00 0 
2aaab46a7000-2aaab8000000 ---p 2aaab46a7000 00:00 0 
2aaadb6dd000-2aaae01d9000 rw-p 2aaadb6dd000 00:00 0 
2b8b1d76f000-2b8b1d770000 rw-p 2b8b1d76f000 00:00 0 
2b8b1d779000-2b8b1d783000 rw-p 2b8b1d779000 00:00 0 
2b8b1d783000-2b8b1d7b8000 r--s 00000000 fd:00 23560397                   /var/db/nscd/passwd
2b8b1d7b8000-2b8b1d7ed000 r--s 00000000 fd:00 23560398                   /var/db/nscd/group
2b8b1d7ed000-2b8cc2222000 rw-p 2b8b1d7ed000 00:00 0 
7fff8a41c000-7fff8a431000 rw-p 7ffffffea000 00:00 0                      [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  [vdso]
I've been searching RedHat's Bugzilla and MySQL's bug database looking for similar crashes... so far no luck, but they have a lot of crash bugs :(

Comment 4

8 years ago
Is there a benefit to building our own version of mysql to rule out RHEL bugs?
I'll be out taking my daughter to the aquarium today, but have my cell with me; feel free to call, text, or email if anything moves on this and/or my input is needed!

Comment 6

8 years ago
<MaxM> looking at the stack trace of the server i narrowed it down to a sproc
<MaxM> not sure why it's puking on you guys now but mindtouch 9.08 has most of
the calls to that sproc obsoleted
<MaxM> the remaining calls to util_String_SplitToTable are with numeric ids as
input instead of page titles which seemed to have caused the issue with
extended chars
Does this crash look like something that could result in database corruption if it's allowed to continue?

Should we consider upgrading to the not-yet-released 9.08 MindTouch, which apparently doesn't do the query that's causing this crash? I've asked MindTouch for insight into when they expect to release, what the current stability situation is, etc.

Should we consider trying to find and delete the pages whose titles are triggering this crash?  That doesn't seem to me like the best course of action (in that we're left with this as something that will hit us again next time a similar page is added), but it's one we could consider.
It actually hasn't happened again since those three times, so maybe it was just a fluke/lucky?

If it starts happening again we should probably consider upgrading.  The query in progress at the point of the crash is a SELECT, so the likelihood of dataloss is slim.  Your cluster has the per-transaction flush-to-disk thing enabled, too, and innodb is generally good at recovering from a crash.
Apparently there's a preview release of MindTouch 9.08 we could test on, to see if the database crash still happens there.

OK, glad to know corruption isn't very likely. I won't worry about this quite so much then.
Depends on: 511852
This crash did happen again yesterday once.  Seems pretty infrequent still,
though.
Just had this crash again.
Is this something we can mitigate by upgrading mysql?
That's possible, but I highly suspect we'll have the new version of Deki faster than we can push a MySQL upgrade through testing.
Yeah the new Deki update is due around August 28th.
so > 5 days to build mysql and deploy on b01?
(In reply to comment #15)
> so > 5 days to build mysql and deploy on b01?

a) set up a pair of blades with RAID cache cards and 8 GB RAM to act as a staging cluster and deploy the new MySQL to those.
b) test all of the apps that currently have databases on b01 and make sure they all work on the new version of MySQL.
c) deploy new version to b01.

Step b) is the one that'll take a while.

On b01 we have:
- bonsai.mozilla.org
- bonsai-l10n.mozilla.org
- bonsai-www.mozilla.org
- bugzilla.org vendor database
- buildbot
- despot.mozilla.org
- graphs.mozilla.org
- litmus.mozilla.org
- svn.mozilla.org viewvc
- developer.mozilla.org
Deki upgrade was supposed to fix this, and if it didn't, the MySQL version upgrade we picked up from RHEL with the 5.4 update probably will.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.