Closed
Bug 1955353
Opened 1 year ago
Closed 2 months ago
[linkPreview] Initialize model with optimistic upper bound on numContext
Categories
(Core :: Machine Learning: On Device, enhancement, P2)
Core
Machine Learning: On Device
Tracking
()
RESOLVED
WONTFIX
People
(Reporter: atossou, Assigned: atossou)
References
(Blocks 1 open bug)
Details
(Whiteboard: [genai])
Currently, numContext is set to precisely match the estimated required context length for a given input.
This approach forces us to restart the engine for each new input.
The goal is to use a larger but reasonable value, such as the next power of 2. This value should remain the same across different input texts while not exceeding the exact requirement by too much.
|
||
Updated•1 year ago
|
|
||
Comment 1•3 months ago
|
||
We should fix this for correct llama.cpp usage for chat text generation.
Blocks: llama-cpp
|
|
||
Updated•3 months ago
|
Component: Machine Learning: General → Machine Learning: On Device
|
|
||
Updated•3 months ago
|
Priority: -- → P2
|
||
Comment 2•3 months ago
|
||
|
|
||
Updated•2 months ago
|
Status: ASSIGNED → RESOLVED
Closed: 2 months ago
Resolution: --- → WONTFIX
You need to log in
before you can comment on or make changes to this bug.
Description
•