1926321 - (SQLite3.47.0) Upgrade to SQLite 3.47.0

Status: NEW

(Core :: SQLite and Embedded Database Bindings, task)

Description

[Ryan VanderMeulen [:RyanVM]]

SQLite Release 3.47.0 On 2024-10-21

• Allow arbitrary expressions in the second argument to the RAISE function.
• If the RHS of the ->> operator is negative, then access array elements counting from the right.
• Fix a problem with rolling back hot journal files in the seldom-used unix-dotfile VFS.
• FTS5 tables can now be dropped even if they use a non-standard tokenizer that has not been registered.
• Fix the group_concat() aggregate function so that it returns an empty string, not a NULL, if it receives a single input value which is an empty string.
• Enhance the generate_series() table-valued function so that it is able to recognize and use constraints on its output value.
• Preupdate hooks now recognize when a column added by ALTER TABLE ADD COLUMN has a non-null default value.
• Performance optimizations:
  • Improved reuse of subqueries associated with the IN operator, especially when the IN operator has been duplicated due to predicate push-down.
  • Use a Bloom filter on subqueries on the right-hand side of the IN operator, in cases where that seems likely to improve performance.
  • Ensure that queries like "SELECT func(a) FROM tab GROUP BY 1" only invoke the func() function once per row.
  • No attempt is made to create automatic indexes on a column that is known to be non-selective because of its use in other indexes that have been analyzed.
  • Adjustments to the query planner so that it produces better plans for star queries with a large number of dimension tables.
  • Add the "order-by-subquery" optimization, that seeks to disable sort operations in outer queries if the desired order is obtained naturally due to ORDER BY clauses in subqueries.
  • The "indexed-subtype-expr" optimization strives to use expressions that are part of an index rather than recomputing the expression based on table values, as long as the query planner can prove that the subtype of the expression will never be used.
  • Miscellaneous coding tweaks for faster runtimes.
• Enhancements to SQLite-related command-line programs:
  • Add the experimental sqlite3_rsync program.
  • Add extension functions median(), percentile(), percentile_cont(), and percentile_disc() to the CLI.
  • Add the .www dot-command to the CLI.
  • The sqlite3_analyzer utility now provides a break-out of statistics for WITHOUT ROWID tables.
  • The sqldiff utility avoids creating an empty database if its second argument does not exist.
• Enhance the sqlite_dbpage table-valued function such that INSERT can be used to increase or decrease the size of the database file.
• SQLite no longer makes any use of the "long double" data type, as hardware support for long double is becoming less common and long double creates challenges for some compiler tool chains. Instead, SQLite uses Dekker's algorithm when extended precision is needed.
• The TCL Interface for SQLite supports TCL9. Everything probably still works for TCL 8.5 and later, though this is not guaranteed. Users are encouraged to upgrade to TCL9.
• JavaScript/WASM:
  • Fix a corruption-causing bug in the JavaScript "opfs" VFS.
  • Correct "mode=ro" handling for the "opfs" VFS.
  • Work around a couple of browser-specific OPFS quirks.
• FTS5 Changes:
  • Add the fts5_tokenizer_v2 API and the locale=1 option, for creating custom locale-aware tokenizers and fts5 tables that may take advantage of them.
  • Add the contentless_unindexed=1 option, for creating contentless fts5 tables that store the values of any UNINDEXED columns persistently in the database.
  • Allow an FTS5 table to be dropped even if it uses a custom tokenizer whose implementation is not available.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

New Linux-only build bustage in the FTS5 code:
https://treeherder.mozilla.org/logviewer?job_id=479397015&repo=try&lineNumber=8229

Looks like it's from this change:
https://hg.mozilla.org/try/diff/1d0f22fcda6600da977bd88d213fe092e6b0a378/third_party/sqlite3/ext/fts5.c#l1.323

Comment 2

[Marco Bonardo [:mak]]

We use the fts5.c amalgamation from preprocessed source, it may not include stdint.h? I'll check later.

Comment 3

[Marco Bonardo [:mak]]

This is the reported error fts5Int.h:65:11: error: unknown type name 'uintptr_t'; did you mean '__intptr_t'?

Adding to the preprocessed fts5 what is normally present in sqliteint.h:

#ifdef HAVE_STDINT_H
#include <stdint.h>
#endif
#ifdef HAVE_INTTYPES_H
#include <inttypes.h>
#endif

solves the problem, so I suspect some includes may be missing when building the preprocessed fts5.c, that we use to build FTS5 as a loadable extension.

Sorry to nag you Richard, is this something we should handle on our side, or can it be fixed upstream in a next release?
This is not urgent, we're ok awaiting for the next release, as we don't have requirements related to 3.47.0 as of today.

Comment 4

[D. Richard Hipp]

Accomplished with SQLite check-in 1bf8daeb24142044.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Attachment: Bug 1926321 - Upgrade to SQLite 3.47.0.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

The attached patch incorporates the change from comment 4 into our vendored copy of fts5.c and is green on Try.
https://treeherder.mozilla.org/jobs?repo=try&revision=f275b9b99d230d0afd773457028ca2b700cde33f

Comment 7

[Marco Bonardo [:mak]]

We'll await for 3.47.x, we can wait.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

drh, were you planning to backport https://sqlite.org/src/info/6e7d498cda1d8fa7 & https://sqlite.org/src/info/1bf8daeb24142044 to branch-3.47 so they're included in 3.47.x?

Comment 9

[D. Richard Hipp]

Both patches have been cherrypicked onto branch-3.47 now.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Thanks!