1971158 - Vendor llama.cpp

Status: RESOLVED FIXED

(Core :: Machine Learning: General, enhancement, P2)

Comment 1

[Paul Adenot (:padenot)]

Attachment: Bug 1971158 - Vendor llama.cpp. r?glandium,sergesanspaille,atossou,tarek

update

Comment 2

[Paul Adenot (:padenot)]

Attachment: WIP: Bug 1971158 - DO NOT LAND - workaround for lack of weak alias on macOS

Comment 3

[Paul Adenot (:padenot)]

Attachment: Bug 1971158 - Copy and extend cmakeparser.py from libaom. r?sergesanspaille

Comment 4

[Paul Adenot (:padenot)]

Attachment: WIP: Bug 1971158 - Add llama.cpp CMakeLists.txt of directories we care about

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:tarek, could you have a look please?

For more information, please visit BugBot documentation.

Comment 6

[Paul Adenot (:padenot)]

Attachment: WIP: Bug 1971158 - Allow compiling llama.cpp without dynamic_cast and exceptions with this one weird trick.

Comment 7

[Paul Adenot (:padenot)]

Attachment: Bug 1971158 - Do not run our Sprintf Literal Checker clang plugin on third party code. r?marco

Comment 8

[Paul Adenot (:padenot)]

Attachment: Bug 1971158 - Do not compile llama.cpp on Android. r?tarek,atossou

Comment 9

[Tarek Ziadé (:tarek)]

Attachment: Bug 1971158 - Copy and extend cmakeparser.py from libaom. r?sergesanspaille,padenot

Comment 10

[Tarek Ziadé (:tarek)]

Attachment: Bug 1971158 - ruff pass r?padenot

Comment 11

[Paul Adenot (:padenot)]

Attachment: Bug 1971158 - Don't crash in llama.cpp when an exception is caught for control flow and not error handling. r?atossou

Comment 12

[Paul Adenot (:padenot)]

Attachment: Bug 1971158 - Compile llama.cpp w/ AVX2 on x86_64, allow knowing if we can use it via js. r?atossou,tarek

Comment 13

[Paul Adenot (:padenot)]

Attachment: Bug 1971158 - Patch llama.cpp to allow loading a model from a fd or a buffer. r?atossou,tarek

Comment 14

[Tarek Ziadé (:tarek)]

Attachment: Bug 1971158 - skip telemetry and engine test in verify r?tarek,padenot

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/8de931ce8e71
https://hg.mozilla.org/integration/autoland/rev/b915f51df79a
Copy and extend cmakeparser.py from libaom. r=padenot
https://github.com/mozilla-firefox/firefox/commit/33a418810d57
https://hg.mozilla.org/integration/autoland/rev/232bc92d24e2
ruff pass r=padenot
https://github.com/mozilla-firefox/firefox/commit/f7ea56505bce
https://hg.mozilla.org/integration/autoland/rev/47e0ba3acc33
Vendor llama.cpp. r=glandium
https://github.com/mozilla-firefox/firefox/commit/ab3716798c05
https://hg.mozilla.org/integration/autoland/rev/142950f7cc02
Do not run our Sprintf Literal Checker clang plugin on third party code. r=marco
https://github.com/mozilla-firefox/firefox/commit/3d3edcf8dcd5
https://hg.mozilla.org/integration/autoland/rev/216f4b30b0c0
Do not compile llama.cpp on Android. r=tarek
https://github.com/mozilla-firefox/firefox/commit/f8ecc2296b3a
https://hg.mozilla.org/integration/autoland/rev/d725386721e4
Don't crash in llama.cpp when an exception is caught for control flow and not error handling. r=atossou
https://github.com/mozilla-firefox/firefox/commit/f07d9fb2f755
https://hg.mozilla.org/integration/autoland/rev/755c877677b5
Compile llama.cpp w/ AVX2 on x86_64, allow knowing if we can use it via js. r=atossou
https://github.com/mozilla-firefox/firefox/commit/e56902821337
https://hg.mozilla.org/integration/autoland/rev/295143572cc2
Patch llama.cpp to allow loading a model from a fd or a buffer. r=atossou
https://github.com/mozilla-firefox/firefox/commit/01f78c04208f
https://hg.mozilla.org/integration/autoland/rev/347c59ae0bce
1973711, 1976784, 1978748: apply code formatting via Lando

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e09dab177edb
https://hg.mozilla.org/integration/autoland/rev/4efe072f5aa1
Revert "Bug 1971158, 1973711, 1976784, 1978748: apply code formatting via Lando" for causing Hazard bustages.

Comment 17

[Atila Butkovits]

Backed out for causing Hazard bustages.

Backout link: https://hg-edge.mozilla.org/integration/autoland/rev/4efe072f5aa1c69de8158337d9cddbaf7ec3ae05

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=347c59ae0bce3cde618bb738c6e630896f22cf8a&selectedTaskRun=WahxcdQjRPGMO-v-YgrlcQ.0

Failure log:
Hazard: https://treeherder.mozilla.org/logviewer?job_id=523781509&repo=autoland&task=WahxcdQjRPGMO-v-YgrlcQ.0&lineNumber=153071
Tier2 Snap: https://treeherder.mozilla.org/logviewer?job_id=523755213&repo=autoland&task=DAmutYuXQhqT2vtrOnel0g.0

Comment 18

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/b915f51df79a
https://hg.mozilla.org/mozilla-central/rev/232bc92d24e2
https://hg.mozilla.org/mozilla-central/rev/47e0ba3acc33
https://hg.mozilla.org/mozilla-central/rev/142950f7cc02
https://hg.mozilla.org/mozilla-central/rev/216f4b30b0c0
https://hg.mozilla.org/mozilla-central/rev/d725386721e4
https://hg.mozilla.org/mozilla-central/rev/755c877677b5
https://hg.mozilla.org/mozilla-central/rev/295143572cc2
https://hg.mozilla.org/mozilla-central/rev/347c59ae0bce

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e2cbda2dd0f6
https://hg.mozilla.org/mozilla-central/rev/6a6ff28c88f1
Revert "Bug 1971158, 1973711, 1976784, 1978748: apply code formatting via Lando" for causing Hazard bustages.

Comment 20

[amarc]

Backout merged to central: https://hg.mozilla.org/mozilla-central/rev/4efe072f5aa1

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/493cbdd30d11
https://hg.mozilla.org/integration/autoland/rev/192e6270b310
Copy and extend cmakeparser.py from libaom. r=padenot
https://github.com/mozilla-firefox/firefox/commit/11b851428fe8
https://hg.mozilla.org/integration/autoland/rev/152466554261
ruff pass r=padenot
https://github.com/mozilla-firefox/firefox/commit/b2e33cddea41
https://hg.mozilla.org/integration/autoland/rev/590f70f1aad4
Vendor llama.cpp. r=glandium
https://github.com/mozilla-firefox/firefox/commit/a1c82cd1af54
https://hg.mozilla.org/integration/autoland/rev/b050e6d485b0
Do not run our Sprintf Literal Checker clang plugin on third party code. r=marco
https://github.com/mozilla-firefox/firefox/commit/a90d27b100bc
https://hg.mozilla.org/integration/autoland/rev/2ef9bf65ad05
Do not compile llama.cpp on Android. r=tarek
https://github.com/mozilla-firefox/firefox/commit/4b9d33a0f2ef
https://hg.mozilla.org/integration/autoland/rev/934bc454421f
Don't crash in llama.cpp when an exception is caught for control flow and not error handling. r=atossou
https://github.com/mozilla-firefox/firefox/commit/ea7b4d1d7985
https://hg.mozilla.org/integration/autoland/rev/f603525b0b45
Compile llama.cpp w/ AVX2 on x86_64, allow knowing if we can use it via js. r=atossou
https://github.com/mozilla-firefox/firefox/commit/7913af2d8931
https://hg.mozilla.org/integration/autoland/rev/2084dabd20b7
Patch llama.cpp to allow loading a model from a fd or a buffer. r=atossou
https://github.com/mozilla-firefox/firefox/commit/73c82626f4c6
https://hg.mozilla.org/integration/autoland/rev/275bf8f32ff3
skip telemetry and engine test in verify r=tarek
https://github.com/mozilla-firefox/firefox/commit/bd6cb02b2f62
https://hg.mozilla.org/integration/autoland/rev/54bb42a9fc65
1973711, 1976784, 1978748: apply code formatting via Lando

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/eaa07185f404
https://hg.mozilla.org/integration/autoland/rev/3bed0e0e4ac0
Remove file patch-fs.patch for causing lint failures.

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a0dd6a10ce67
https://hg.mozilla.org/integration/autoland/rev/ad97718e3c54
Revert "Bug 1971158, Bug 1973711, Bug 1976784, Bug 1978748: apply code formatting via Lando"

Comment 24

[Atila Butkovits]

Backed out for causing Hazard bustages.

Backout link: https://hg-edge.mozilla.org/integration/autoland/rev/ad97718e3c54cfa88c7e480fde786dc961594b79

Push where failures started: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&searchStr=linux%2Cdebug%2Chazard-linux64-haz%2Fdebug%2Ch&revision=4f62441a0fd8eeb63139d01174dc9435a77cccb9&selectedTaskRun=Jlsuh2WVTDmjkYKYpFL2Ww.1

Failure log: https://treeherder.mozilla.org/logviewer?job_id=524018554&repo=autoland&task=Jlsuh2WVTDmjkYKYpFL2Ww.1&lineNumber=153087

Comment 25

[Florin Bilt]

(In reply to amarc from comment #20)

Backout merged to central: https://hg.mozilla.org/mozilla-central/rev/4efe072f5aa1

Perfherder has detected a awsy performance change from push 4efe072f5aa1c69de8158337d9cddbaf7ec3ae05.

If you have any questions, please reach out to a performance sheriff. Alternatively, you can find help on Slack by joining #perf-help, and on Matrix you can find help by joining #perftest.

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
4%	Base Content Heap Unclassified	windows11-64-24h2-shippable	fission	2,126,249.33 -> 2,039,947.33
3%	Base Content Heap Unclassified	linux1804-64-shippable-qr	fission	2,576,719.67 -> 2,493,361.67

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a performance sheriff to do that for you.

You can run all of these tests on try with ./mach try perf --alert 46509

The following documentation link provides more information about this command.

Comment 26

[Alex Finder]

Hi Paul! There was a build_metrics regression detected on the push mentioned below, and this bug was referenced. Do you think this bug may have caused the performance regression ?

Perfherder has detected a build_metrics performance change from push 347c59ae0bce3cde618bb738c6e630896f22cf8a. As author of one of the patches included in that push, we need your help to address this regression.

Please acknowledge, and begin investigating this alert within 3 business days, or the patch(es) may be backed out in accordance with our regression policy. Our guide to handling regression bugs has information about how you can proceed with this investigation.

If you have any questions or need any help with the investigation, please reach out to a performance sheriff. Alternatively, you can find help on Slack by joining #perf-help, and on Matrix you can find help by joining #perftest.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
6%	compiler_metrics num_static_constructors	osx-cross		91.00 -> 96.50
4%	compiler_metrics num_static_constructors	linux64-shippable		102.00 -> 106.17

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a performance sheriff to do that for you.

You can run all of these tests on try with ./mach try perf --alert 46416

The following documentation link provides more information about this command.

Comment 27

[Alex Finder]

(In reply to Pulsebot from comment #21)

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/493cbdd30d11
https://hg.mozilla.org/integration/autoland/rev/192e6270b310
Copy and extend cmakeparser.py from libaom. r=padenot
https://github.com/mozilla-firefox/firefox/commit/11b851428fe8
https://hg.mozilla.org/integration/autoland/rev/152466554261
ruff pass r=padenot
https://github.com/mozilla-firefox/firefox/commit/b2e33cddea41
https://hg.mozilla.org/integration/autoland/rev/590f70f1aad4
Vendor llama.cpp. r=glandium
https://github.com/mozilla-firefox/firefox/commit/a1c82cd1af54
https://hg.mozilla.org/integration/autoland/rev/b050e6d485b0
Do not run our Sprintf Literal Checker clang plugin on third party code.
r=marco
https://github.com/mozilla-firefox/firefox/commit/a90d27b100bc
https://hg.mozilla.org/integration/autoland/rev/2ef9bf65ad05
Do not compile llama.cpp on Android. r=tarek
https://github.com/mozilla-firefox/firefox/commit/4b9d33a0f2ef
https://hg.mozilla.org/integration/autoland/rev/934bc454421f
Don't crash in llama.cpp when an exception is caught for control flow and
not error handling. r=atossou
https://github.com/mozilla-firefox/firefox/commit/ea7b4d1d7985
https://hg.mozilla.org/integration/autoland/rev/f603525b0b45
Compile llama.cpp w/ AVX2 on x86_64, allow knowing if we can use it via js.
r=atossou
https://github.com/mozilla-firefox/firefox/commit/7913af2d8931
https://hg.mozilla.org/integration/autoland/rev/2084dabd20b7
Patch llama.cpp to allow loading a model from a fd or a buffer. r=atossou
https://github.com/mozilla-firefox/firefox/commit/73c82626f4c6
https://hg.mozilla.org/integration/autoland/rev/275bf8f32ff3
skip telemetry and engine test in verify r=tarek
https://github.com/mozilla-firefox/firefox/commit/bd6cb02b2f62
https://hg.mozilla.org/integration/autoland/rev/54bb42a9fc65
1973711, 1976784, 1978748: apply code formatting via Lando

Just to clarify, the regression reported below has been backed out, no further action is needed here.

Perfherder has detected a build_metrics performance change from push 54bb42a9fc65b0aafa2a4d168f5c001112d237f3. As author of one of the patches included in that push, we need your help to address this regression.

If you have any questions or need any help with the investigation, please reach out to a performance sheriff. Alternatively, you can find help on Slack by joining #perf-help, and on Matrix you can find help by joining #perftest.

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
9%	compiler_metrics num_static_constructors	linux64-snap		98.00 -> 106.50
8%	compiler_metrics num_static_constructors	linux64-snap		102.33 -> 111.00
7%	compiler_metrics num_static_constructors	windows2012-64		94.00 -> 101.00
7%	compiler_metrics num_static_constructors	osx-cross		87.00 -> 93.00
7%	compiler_metrics num_static_constructors	osx-cross		91.00 -> 97.00
6%	compiler_metrics num_static_constructors	windows2012-32		108.00 -> 115.00
5%	compiler_metrics num_static_constructors	osx-cross-aarch64	aarch64	92.75 -> 97.50
5%	compiler_metrics num_static_constructors	osx-shippable	instrumented	92.75 -> 97.50
5%	compiler_metrics num_static_constructors	linux64		102.00 -> 107.00
5%	compiler_metrics num_static_constructors	windows2012-64		148.00 -> 155.00
5%	compiler_metrics num_static_constructors	linux64		106.00 -> 111.00
4%	compiler_metrics num_static_constructors	windows2012-32		159.00 -> 166.00
4%	compiler_metrics num_static_constructors	linux64-shippable		102.00 -> 106.17
4%	compiler_metrics num_static_constructors	linux64-shippable	instrumented	108.00 -> 112.17

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a performance sheriff to do that for you.

You can run all of these tests on try with ./mach try perf --alert 46450

The following documentation link provides more information about this command.

Comment 28

[Alex Finder]

(In reply to Pulsebot from comment #16)

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e09dab177edb
https://hg.mozilla.org/integration/autoland/rev/4efe072f5aa1
Revert "Bug 1971158, 1973711, 1976784, 1978748: apply code formatting via
Lando" for causing Hazard bustages.

Perfherder has detected a build_metrics performance change from push 4efe072f5aa1c69de8158337d9cddbaf7ec3ae05.

If you have any questions, please reach out to a performance sheriff. Alternatively, you can find help on Slack by joining #perf-help, and on Matrix you can find help by joining #perftest.

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
6%	compiler_metrics num_static_constructors	osx-cross		97.00 -> 91.00
4%	compiler_metrics num_static_constructors	linux64-shippable		106.17 -> 102.00
4%	compiler_metrics num_static_constructors	linux64-shippable	instrumented	112.17 -> 108.00

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a performance sheriff to do that for you.

You can run all of these tests on try with ./mach try perf --alert 46435

The following documentation link provides more information about this command.

Comment 29

[Paul Adenot (:padenot)]

Attachment: Bug 1971158 - Patch out static ctors in llama.cpp. r?atossou

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/9da101c85d38
https://hg.mozilla.org/integration/autoland/rev/9fc97ed64612
Copy and extend cmakeparser.py from libaom. r=padenot
https://github.com/mozilla-firefox/firefox/commit/8f9cd2762bce
https://hg.mozilla.org/integration/autoland/rev/b50b606e99d4
ruff pass r=padenot
https://github.com/mozilla-firefox/firefox/commit/4dd522c6522d
https://hg.mozilla.org/integration/autoland/rev/a6b5919847a1
Vendor llama.cpp. r=glandium
https://github.com/mozilla-firefox/firefox/commit/b589b4c9b4c9
https://hg.mozilla.org/integration/autoland/rev/545c3770cabf
Do not run our Sprintf Literal Checker clang plugin on third party code. r=marco
https://github.com/mozilla-firefox/firefox/commit/ceecd3983a0b
https://hg.mozilla.org/integration/autoland/rev/0b4dbcbb8909
Do not compile llama.cpp on Android. r=tarek
https://github.com/mozilla-firefox/firefox/commit/e2507dfa2a1f
https://hg.mozilla.org/integration/autoland/rev/46b935061409
Don't crash in llama.cpp when an exception is caught for control flow and not error handling. r=atossou
https://github.com/mozilla-firefox/firefox/commit/966604ad65dc
https://hg.mozilla.org/integration/autoland/rev/cd6daec6e4ae
Compile llama.cpp w/ AVX2 on x86_64, allow knowing if we can use it via js. r=atossou
https://github.com/mozilla-firefox/firefox/commit/ce2ba2f0e465
https://hg.mozilla.org/integration/autoland/rev/cbc06957aed3
Patch llama.cpp to allow loading a model from a fd or a buffer. r=atossou
https://github.com/mozilla-firefox/firefox/commit/77d40a6ccc20
https://hg.mozilla.org/integration/autoland/rev/010afe1fcabb
Patch out static ctors in llama.cpp. r=atossou
https://github.com/mozilla-firefox/firefox/commit/aec5499de539
https://hg.mozilla.org/integration/autoland/rev/b56c81cb2ba2
skip telemetry and engine test in verify r=tarek
https://github.com/mozilla-firefox/firefox/commit/3ea8ea16834c
https://hg.mozilla.org/integration/autoland/rev/4c630ff1b325
1973711, 1976784, 1978748: apply code formatting via Lando

Comment 31

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/9fc97ed64612
https://hg.mozilla.org/mozilla-central/rev/b50b606e99d4
https://hg.mozilla.org/mozilla-central/rev/a6b5919847a1
https://hg.mozilla.org/mozilla-central/rev/545c3770cabf
https://hg.mozilla.org/mozilla-central/rev/0b4dbcbb8909
https://hg.mozilla.org/mozilla-central/rev/46b935061409
https://hg.mozilla.org/mozilla-central/rev/cd6daec6e4ae
https://hg.mozilla.org/mozilla-central/rev/cbc06957aed3
https://hg.mozilla.org/mozilla-central/rev/010afe1fcabb
https://hg.mozilla.org/mozilla-central/rev/b56c81cb2ba2
https://hg.mozilla.org/mozilla-central/rev/4c630ff1b325

Comment 32

[Nathan Teodosio :nteodosio]

Hi, may I ask what was the motivation behind this? Given Llamafile description:

by combining llama.cpp with Cosmopolitan Libc into one framework that collapses all the complexity of LLMs down to a single-file executable (called a "llamafile")

Comment 33

[Tarek Ziadé (:tarek)]

(In reply to Nathan Teodosio :nteodosio from comment #32)

Hi, may I ask what was the motivation behind this? Given Llamafile description:

by combining llama.cpp with Cosmopolitan Libc into one framework that collapses all the complexity of LLMs down to a single-file executable (called a "llamafile")

It does not serve the same purpose. llamafile is a great tool to run an llm as an HTTP service on your device, but we need an inference engine inside Firefox to run inference tasks. It's not practical for us to have two applications running on the device, e.g. firefox and then llamafile. It adds complexity, security issues, installation issues, performance considerations etc.

Vendoring the llama lib directly inside Firefox is the right design to add local inference in the browser.