2043658 - [llama.cpp] Update third_party/llama.cpp

Status: RESOLVED FIXED

(Core :: Machine Learning: On Device, enhancement, P1)

Description

[Valentin Pollet]

The version of llama.cpp built and used by Firefox is quite a bit behind upstream (~3000 commits, 10 months).

Setting this as P2 because it seems like quite a lot has happened upstream. This appears to be good idea to update before committing to more work involving the llama.cpp backend.

Why it may be worth doing

This is a lot of changes to go through to determine how much of a priority this should be, so I asked Claude to analyze it to get a sense of what Firefox could gain from bumping.

Prompt

From gg-ml/llama.cpp@98e480a32ec77bc494df887da3f69e53373122a7:

Firefox uses llama.cpp to run inference on device. llama.cpp is built in tree, from a version about 3000 commits behind head here. I ran git log d32e03f4495d3efa1c5126f53b449f1d429c5664..HEAD --oneline to get a summary of the changes that happened since, see @llama-cpp.changelog. I'm interested in the benefits Firefox would get from bumping. Firefox currently only uses the CPU backend, see @firefox-build.mozconfig for shipped optimizations/configurations.

Claude's answer

Firefox llama.cpp Bump — CPU-Focused Benefits

Range analyzed: d32e03f4495d3efa1c5126f53b449f1d429c5664..98e480a32ec77bc494df887da3f69e53373122a7 (~3000 commits). Filtered for CPU-backend relevance on the targets shipped in firefox-build.mozconfig (x86_64-AVX2/BMI2/F16C/FMA/SSE4.2, aarch64, arm, x86, generic).

Top wins justifying the bump

1. ARM64 K-quant repack GEMM/GEMV kernels (#16739, #17494, #18096, #18860, #18888, #19108, #19356, #19360) — dotprod + i8mm variants for q4_K/q5_K/q6_K/q8_0. Material prompt-prefill throughput on aarch64.
2. CPU Flash Attention — chunking (#16829), tiled FA for prefill (#19012), FA split across KV for TG (#19209), FA GEMM microkernel (#19422), and f16 FA mask (#23764) → significant speed + memory wins on the embedding path.
3. AVX2 wins — optimize avx2 q6_k (#22345), LUT for e8→f32 scales on x86 (#19288), optimized x86 q1_0 dot (#21636). Direct hits on the x86_64-AVX2 path Firefox ships.
4. New embedding architectures — EmbeddingGemma 300m (#15798 + dense modules #16367), ModernBERT (#18330), Jina v3 (#13693), Jina v5 Nano/EuroBERT (#19826), Granite Embedding (#15641), LFM2-ColBert-350M (#18607). Highly relevant to embedding robustness work.
5. Tokenizer crash fixes — segfault on repetitive input (#17786), Qwen2 regex segfault on long input (#21257), UGM XCDA bounds (#17215). Production stability.

Comment 1

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Vendoring fixes r=jbowser

Updated moz.yaml with two extra include globs as upstream refactored llama-model.cpp into multiple smaller files in src/models.

Updated cmakeparser.py to handle globs in CMakeLists.txt (introduced upstream to compile models/*.cpp).

Fixed run_parser.py needed to discard files excluded by moz.yaml.

Updated moz.build to provide src/ as an include directory during build. This was needed to compile models/*.cpp which include llama-model.h without a relative path.

Updated moz.build to export ggml/include/gguf.h as upstream now includes it in llama.h.

Comment 2

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Run mach vendor r=jbowser

This is the result of running the command:

./mach vendor third_party/llama.cpp/moz.yaml -r 74ade52741203e5c8f81eaf06a96cb1cfe15f2a3 --patch-mode=none

Comment 3

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Rebase patch 1 (exceptions) r=jbowser

Beyond the usual line-number and index shifts, this follows two upstream
file renames: llama-kv-cache-unified.cpp -> llama-kv-cache.cpp and
llama-sampling.cpp -> llama-sampler.cpp.

Comment 4

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Rebase patch 2 (std::filesystem) r=jbowser

Upstream split dynamic-library loading into the new ggml-backend-dl.{cpp,h},
which use std::filesystem. The patch grew to #if 0 out the fs::path-based
dl_load_library there and add the <memory>/<type_traits> includes it
otherwise pulled in transitively.

Comment 5

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Rebase patch 3 (std::future) r=jbowser

Comment 6

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Rebase patch 4 (std::wstring) r=jbowser

Upstream restructured the non-unicode regex split path: it now builds the
wstring directly from the codepoints instead of via unicode_wstring_from_utf8,
so the abort()/commented-out replacement was reapplied against the new code.
The std::wstring unicode_regex_split_stl overload was removed upstream, so
that #if 0 hunk is dropped.

Comment 7

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Rebase patch 5 (std::terminate) r=jbowser

Comment 8

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Rebase patch 6 (std::random_device) r=jbowser

Comment 9

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Rebase patch 7 (fix chat_detect_template) r=jbowser

Comment 10

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Rebase patch 8 (load model from fd/buffer) r=jbowser

Comment 11

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Rebase patch 9 (no static ctors) r=jbowser

Comment 12

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Rebase patch 10 (RLIMIT_MEMLOCK solaris) r=jbowser

Upstream added HAIKU to the RLIMIT_MEMLOCK guard, so sun is now
appended alongside it and the comment updated to match. Also drops a spurious
empty-file hunk the previous version of the patch carried.

Comment 13

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Rebase patch 11 (missing includes) r=jbowser

Comment 14

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Rebase patch 12 (profiler hooks) r=jbowser

Upstream moved threadpool parameter initialization into the new ggml-c.c, so
a hunk was added to zero-initialize the thread_create/destroy callbacks in
ggml_threadpool_params_init there, alongside the existing ggml-cpu-c.c hooks.

Comment 15

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - New patch (missing includes) r=jbowser

Comment 16

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Use flash_attn_type r=jbowser

Comment 17

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - New patch (skip metadata kv dump) r=jbowser,padenot

The metadata kv dump in llama_model_loader was measured to be very expensive (~10 milliseconds on e.g. Mac M5) for certain models.
This was not an issue prior to this bump in version of llama.cpp because we had a custom llama_model_load_from_file_handle.
Upstream added support for it so we adopted it, but our custom was skipping it.
The patch skips the dump using a pre-existing trace gate.

Comment 18

[Valentin Pollet]

Attachment: Bug 2043658 - Update llama.cpp to 74ade527 - Improve performance of catch blocks redefined by moz-overrides.h r=jbowser,padenot

This makes the exceptions declared in the redefine of catch static, avoiding construction/destruction of 4 objects.

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/72284c652553
https://hg.mozilla.org/integration/autoland/rev/e21fd1c1613a
Update llama.cpp to 74ade527 - Vendoring fixes r=jbowser,padenot
https://github.com/mozilla-firefox/firefox/commit/7ddd98990262
https://hg.mozilla.org/integration/autoland/rev/6811d1384280
Update llama.cpp to 74ade527 - Run mach vendor r=padenot
https://github.com/mozilla-firefox/firefox/commit/636febdaa0ac
https://hg.mozilla.org/integration/autoland/rev/730be2c048c3
Update llama.cpp to 74ade527 - Rebase patch 1 (exceptions) r=jbowser,padenot
https://github.com/mozilla-firefox/firefox/commit/2e011cb19834
https://hg.mozilla.org/integration/autoland/rev/25df54f4eb09
Update llama.cpp to 74ade527 - Rebase patch 2 (std::filesystem) r=jbowser,padenot
https://github.com/mozilla-firefox/firefox/commit/f8561c02cc82
https://hg.mozilla.org/integration/autoland/rev/dd511ec11000
Update llama.cpp to 74ade527 - Rebase patch 3 (std::future) r=jbowser,padenot
https://github.com/mozilla-firefox/firefox/commit/fce945379447
https://hg.mozilla.org/integration/autoland/rev/41444f8593da
Update llama.cpp to 74ade527 - Rebase patch 4 (std::wstring) r=jbowser,padenot
https://github.com/mozilla-firefox/firefox/commit/50e041d0af67
https://hg.mozilla.org/integration/autoland/rev/a314f319cb2d
Update llama.cpp to 74ade527 - Rebase patch 5 (std::terminate) r=jbowser,padenot
https://github.com/mozilla-firefox/firefox/commit/018402128fca
https://hg.mozilla.org/integration/autoland/rev/91a40b65026c
Update llama.cpp to 74ade527 - Rebase patch 6 (std::random_device) r=jbowser,padenot
https://github.com/mozilla-firefox/firefox/commit/5fef67cd0e85
https://hg.mozilla.org/integration/autoland/rev/859880280884
Update llama.cpp to 74ade527 - Rebase patch 7 (fix chat_detect_template) r=jbowser,padenot
https://github.com/mozilla-firefox/firefox/commit/af1e14a956b9
https://hg.mozilla.org/integration/autoland/rev/be9857e3ee1e
Update llama.cpp to 74ade527 - Rebase patch 8 (load model from fd/buffer) r=jbowser,padenot
https://github.com/mozilla-firefox/firefox/commit/9ab9f898d091
https://hg.mozilla.org/integration/autoland/rev/777fb9845371
Update llama.cpp to 74ade527 - Rebase patch 9 (no static ctors) r=jbowser,padenot
https://github.com/mozilla-firefox/firefox/commit/222f2f604438
https://hg.mozilla.org/integration/autoland/rev/3bd01ffe47b0
Update llama.cpp to 74ade527 - Rebase patch 10 (RLIMIT_MEMLOCK solaris) r=jbowser,padenot
https://github.com/mozilla-firefox/firefox/commit/5a7095ca5fbf
https://hg.mozilla.org/integration/autoland/rev/596654441a06
Update llama.cpp to 74ade527 - Rebase patch 11 (missing includes) r=padenot
https://github.com/mozilla-firefox/firefox/commit/5c663af001e1
https://hg.mozilla.org/integration/autoland/rev/9f49431b36d0
Update llama.cpp to 74ade527 - Rebase patch 12 (profiler hooks) r=padenot
https://github.com/mozilla-firefox/firefox/commit/0c6da8411919
https://hg.mozilla.org/integration/autoland/rev/8fd1d6633ade
Update llama.cpp to 74ade527 - New patch (missing includes) r=padenot
https://github.com/mozilla-firefox/firefox/commit/0a075fdf201b
https://hg.mozilla.org/integration/autoland/rev/29b6e547608f
Update llama.cpp to 74ade527 - Use flash_attn_type r=jbowser,ai-platform-reviewers,padenot
https://github.com/mozilla-firefox/firefox/commit/d1a2a4f7a807
https://hg.mozilla.org/integration/autoland/rev/a9bb42b8f851
Update llama.cpp to 74ade527 - New patch (skip metadata kv dump) r=padenot
https://github.com/mozilla-firefox/firefox/commit/ccb308090393
https://hg.mozilla.org/integration/autoland/rev/bcf9b1be5846
Update llama.cpp to 74ade527 - Improve performance of catch blocks redefined by moz-overrides.h r=padenot

Comment 20

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/e21fd1c1613a
https://hg.mozilla.org/mozilla-central/rev/6811d1384280
https://hg.mozilla.org/mozilla-central/rev/730be2c048c3
https://hg.mozilla.org/mozilla-central/rev/25df54f4eb09
https://hg.mozilla.org/mozilla-central/rev/dd511ec11000
https://hg.mozilla.org/mozilla-central/rev/41444f8593da
https://hg.mozilla.org/mozilla-central/rev/a314f319cb2d
https://hg.mozilla.org/mozilla-central/rev/91a40b65026c
https://hg.mozilla.org/mozilla-central/rev/859880280884
https://hg.mozilla.org/mozilla-central/rev/be9857e3ee1e
https://hg.mozilla.org/mozilla-central/rev/777fb9845371
https://hg.mozilla.org/mozilla-central/rev/3bd01ffe47b0
https://hg.mozilla.org/mozilla-central/rev/596654441a06
https://hg.mozilla.org/mozilla-central/rev/9f49431b36d0
https://hg.mozilla.org/mozilla-central/rev/8fd1d6633ade
https://hg.mozilla.org/mozilla-central/rev/29b6e547608f
https://hg.mozilla.org/mozilla-central/rev/a9bb42b8f851
https://hg.mozilla.org/mozilla-central/rev/bcf9b1be5846