2049478 - [llama.cpp] llama_model_params::use_mmap is exposed but not honored properly

Status: RESOLVED FIXED

(Core :: Machine Learning: On Device, defect, P2)

Description

[Valentin Pollet]

The native llama.cpp AI backend exposes a lot of options from llama.cpp to be configured by LlamaCppPipeline (the JS consumer). One of those options is use_mmap, set on llama_model_params. This option is exposed through PipelineOptions and re-exposed by the LlamaCppPipeline.

One of the patches we apply on top of llama.cpp unconditionally disables the option.

This is a defect because the only real consumer of the native backend, Link Preview, requests this option. The option should either not be exposed to the pipeline configuration, or properly honored.

Comment 1

[Valentin Pollet]

Attachment: Bug 2049478 - Update llama.cpp to 74ade527 - Set Link Preview useMmap off to match old behavior r=jbowser,padenot

The patch introducing loading from file descriptor was rewritten to reuse code upstream. The new version honors the use_mmap when the old one did not. This makes sure we're not changing the behavior of Link Preview (only consumer) which used to request it while it was silently ignored. Old behavior: pass use_mmap=true, ignored -> no mmap. New behavior: pass use_mmap=false, honored -> no mmap.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a3a1ad8e7b5b
https://hg.mozilla.org/integration/autoland/rev/50bc7de5fb65
Update llama.cpp to 74ade527 - Set Link Preview useMmap off to match old behavior  r=padenot,Mardak

Comment 3

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/50bc7de5fb65