786558 - Undefined behavior caused by out-of-range shift in unixShmSystemLock and sqlite3GenerateRowDelete

Status: RESOLVED INACTIVE

(Core :: SQLite and Embedded Database Bindings, defect)

Description

[Matthew Gregan [:kinetik]]

Building mozilla-central with Clang 3.2 on x86_64 with -fcatch-undefined-behavior results in a binary that crashes in unixShmSystemLock and sqlite3GenerateRowDelete due to out-of-range left shifts:

86986 SQLITE_PRIVATE void sqlite3GenerateRowDelete(
...
87026     for(iCol=0; iCol<pTab->nCol; iCol++){
87027       if( mask==0xffffffff || mask&(1<<iCol) ){

pTab->nCol is 33, loop terminates with iCol == 32.

28874 static int unixShmSystemLock(
...
28908   mask = (1<<(ofst+n)) - (1<<ofst);

And here, ofst is 128.

Comment 1

[Marco Bonardo [:mak]]

Should be reported upstream if it's still existing, part of this code can't be found anymore