514291 - sqlite3.c:46028: failed assertion `pMem1->enc==pMem2->enc

Status: RESOLVED FIXED

(Core :: SQLite and Embedded Database Bindings, defect)

Description

[Curtis Bartley [:cbartley]]

The query

  SELECT 1 from moz_places 
  WHERE levenshteinDistance(url, title) <= 10 
  AND url <> ''

causes

  sqlite3.c:46028: failed assertion `pMem1->enc==pMem2->enc

If you disable the guts of the levenshteinDistance function (return a dummy result right before the construction of "A" in levenshteinDistanceFunction()) then the even simpler query 

  SELECT 1 from moz_places 
  WHERE levenshteinDistance(url, '123') <= 10 
  AND url <> ''

will cause the assertion failure, but the first query will not!

So I don't think it's a problem with the levenshteinDistance function proper, but rather with the "glue" code, either ours or Sqlite's I'm not sure yet.  The fact that the first query succeeds when the main levenshteinDistance code runs but fails when it's disabled sounds an awful lot like a memory overwrite to me.

I hate memory overwrites.

Comment 1

[D. Richard Hipp]

What version of SQLite are we talking about.  What does "SELECT sqlite_version()" say?

Can the OP run this in a debugger and tell me what pMem1[0] and pMem2[0] hold?

The pMem->enc field records the text encoding of the value.  This will be one of UTF-8, UTF-16be, or UTF-16le.  But does FF ever use anything other than UTF-8?

Comment 2

[Shawn Wilsher :sdwilsh]

(In reply to comment #1)
> What version of SQLite are we talking about.  What does "SELECT
> sqlite_version()" say?
3.6.16

> The pMem->enc field records the text encoding of the value.  This will be one
> of UTF-8, UTF-16be, or UTF-16le.  But does FF ever use anything other than
> UTF-8?
As far as I know, we don't specify except for a few test functions in our test harness.  We should get the default (I think that's UTF-8).

Comment 3

[Curtis Bartley [:cbartley]]

(In reply to comment #1)
> What version of SQLite are we talking about.  What does "SELECT
> sqlite_version()" say?

"3.6.16"

I won't have time to investigate this further for the next couple of days.  However, if the bug is on our side, I strongly suspect it's in this function, most likely in the four statements starting with "int aLen =":

void levenshteinDistanceFunction(sqlite3_context *aCtx,
                                 int aArgc,
                                 sqlite3_value **aArgv)
{
  NS_ASSERTION(2 == aArgc, "Invalid number of arguments!");

  // If either argument is a SQL NULL, then return SQL NULL.
  if (::sqlite3_value_type(aArgv[0]) == SQLITE_NULL ||
      ::sqlite3_value_type(aArgv[1]) == SQLITE_NULL) {
    ::sqlite3_result_null(aCtx);
    return;
  }

  int aLen = ::sqlite3_value_bytes16(aArgv[0]) / sizeof(PRUnichar);
  const PRUnichar *a = 
      static_cast<const PRUnichar *>(::sqlite3_value_text16(aArgv[0]));

  int bLen = ::sqlite3_value_bytes16(aArgv[1]) / sizeof(PRUnichar);
  const PRUnichar *b = 
      static_cast<const PRUnichar *>(::sqlite3_value_text16(aArgv[1]));
  
  ;;;;;;
//  ::sqlite3_result_int(aCtx, aLen > bLen ? aLen - bLen : bLen - aLen);
//  return;
  ;;;;;;

  // Compute the Levenshtein Distance, and return the result (or error).
  int distance = -1;
  const nsDependentString A(a, aLen);
  const nsDependentString B(b, bLen);
  int status = levenshteinDistance(A, B, &distance);
  if (status == SQLITE_OK) {
    ::sqlite3_result_int(aCtx, distance);    
  }
  else if (status == SQLITE_NOMEM) {
    ::sqlite3_result_error_nomem(aCtx);
  }
  else {
    ::sqlite3_result_error(aCtx, "User function returned error code", -1);
  }
}

Comment 4

[D. Richard Hipp]

I suspected as much.  I was in the process or writing a test case along the same lines as the function in the previous comment as the previous comment arrived in my mailbox.

What SQLite is doing is evaluating the two arguments to the levenshteinDistance() functions and storing those values in "Mem" objects.  The text encoding is UTF-8.  Then down inside levenshteinDistance(), sqlite3_value_text16() is called which forces a conversion to UTF-16.  After the levenshteinDistance() function returns, SQLite attempts to reuse the same Mem object in order to evaluate the url<>'' term, and at that point sees that the encodings no longer match.

SQLite was assuming that the parameters to a function do not change - which is mostly true.  Their values don't change, but the text encoding might.  I'll have to make a few tweaks to SQLite to get it to relax its assumption and not reuse the parameters to functions so aggressively.

I'll enter a corresponding bug into the SQLite bug database as soon as I have a test case.  I'll follow-up here with a link to that bug report once I have one.

Comment 5

[D. Richard Hipp]

See the SQLite ticket here:  http://www.sqlite.org/src/info/2ea2425d34

Comment 6

[D. Richard Hipp]

Problem fixed by this SQLite check-in: http://www.sqlite.org/src/info/f22e388727f0

The fix will appear in the next SQLite release (currently scheduled for 2009-09-14).  Let me know if you need me to build an amalgamation for you out of our latest code.

A temporary work-around might be to put the url<>'' test in front, like this:

  SELECT 1 from moz_places 
  WHERE url<>''
  AND levenshteinDistance(url, title) <= 10

Comment 7

[Shawn Wilsher :sdwilsh]

(In reply to comment #6)
> The fix will appear in the next SQLite release (currently scheduled for
> 2009-09-14).  Let me know if you need me to build an amalgamation for you out
> of our latest code.
That would be SQLite 3.6.18, yes?

Comment 8

[Shawn Wilsher :sdwilsh]

Fixed by bug 524144.