1987081 - are Firefox AI summaries vulnerable to prompt injection?

Status: RESOLVED FIXED

(Core :: Machine Learning: Frontend, task)

Description

[Frederik Braun [:freddy]]

Mastodon user @[e-mail] reports:

Firefox AI summaries for websites is just inlining the website contents with quotes, not even escaping the quotes inside, plenty of prompt injection vulns here..

I am going to give them the benefit of doubt. As the developer of a reverse engineering tool, I expect them to have a somewhat credible sense of threat modeling and risk here.

Given the vagueness of this, I am filing this as "sec-audit" and "task" rather than a defect.

Do we have a shared understanding of what we are protecting against and what not?

Comment 1

[Frederik Braun [:freddy]]

Simplistic prompt injections like in https://frederikbraun.de/prompt-injections-and-a-demo.html used to work against the orbit extension, but I couldn't reproduce this here.

I tried a more targeted approach to escape the "<tabTitle>...</tabtitle>" as as well as the "<selection>...</selection>" for the text content, which were both successful. But it would still try to summarize what was outside of those "elements".

Comment 2

[Simon Friedberger [:simonf]]

I ran some experiments and the <|im_start|>system tokens can also be injected so it does indeed look like there is some sanitization missing.

Ed, is this something you can look into? Or can you redirect to the right person?

Comment 3

[Ed Lee :Mardak]

what is this referring to?

the examples with <tabTitle> are probably chatbot, and these summaries are processed by 3rd party, so is the expectation that firefox helps the chatbot provider from getting tricked by the page content?

a separate feature link previews do run a llm model on device and "key points" outputs are rendered as text, but yes potentially displaying some other text instead of the actual key points

Comment 4

[Simon Friedberger [:simonf]]

I think I have to ask some follow-ups here:

1. Do we know if the 3rd party is doing any processing to prevent tricking?
2. Is the local model you mean wllama?

I expect the impact to be minor because our prompt is indeed fairly minimal for now: https://searchfox.org/firefox-main/source/browser/locales/en-US/browser/genai.ftl#29
I agree that with the current setup this would mostly be a vandalization/pranking issue where a website can show different content in our summary window.

Comment 5

[Daniel Veditz [:dveditz]]

(In reply to Ed Lee :Mardak from comment #3)

what is this referring to?

the examples with <tabTitle> are probably chatbot, and these summaries are processed by 3rd party[...] A separate feature link previews do run a llm model on device

This is not about the "link summaries" feature, this is about the promoted sidebar AI chat feature.

Comment 6

[Tarek Ziadé (:tarek)]

Ed, I guess you could sanitize what's sent to the 3rd party ?

Comment 7

[Ed Lee :Mardak]

as in remove fake </tabTitle> and </selection> hint tags if the page context tries to use it? we could do that from https://searchfox.org/firefox-main/rev/251eeab1c468eb1557269952c712580e4cc16e29/browser/components/genai/GenAI.sys.mjs#982

we could also force a limit on the length of %tabTitle|50% similar to how we already truncate %selection|…% https://searchfox.org/firefox-main/rev/251eeab1c468eb1557269952c712580e4cc16e29/browser/components/genai/GenAI.sys.mjs#946-949

Comment 8

[Yubin Jamora]

Attachment: Bug 1987081 - Remove fake </tabTitle> and </selection> hint tags and sanitize prompts r?mardak

Comment 9

[Simon Friedberger [:simonf]]

Sanitization is good but does not solve prompt injection. Another thing to do here is to not use a logged-in session, such that no user data can be leaked and no actions can be performed on behalf of the user.

That means the summaries will still be vulnerable but at least the impact is reduced.

Comment 10

[Yubin Jamora]

Thank you. The sanitization helps with the structural injection part but i understand that doesn't really resolve prompt injection itself.
I have updated the patch to use a system principal so it doesn't run under a logged-in session. Please let me know what you think!!

Comment 11

[artur]

I agree that with the current setup this would mostly be a vandalization/pranking issue where a website can show different content in our summary window.

I think more that only vandalization/pranking can happen.

After a bit of playing around I've managed to make the LLMs:

• output funny things
• access external sites
• persist data into the users personal memory

Have a look: https://individual-it.net/llm-instructions/

I guess one now could insert data into the personal memory to trick the LLM to:

• answer political questions with a particular bias
• advertise for a product

Maybe it would be also possible to read the personal memory and by accessing external sites send it to the attacker.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/224bec7f9887
https://hg.mozilla.org/integration/autoland/rev/f4a225c064ee
Remove fake </tabTitle> and </selection> hint tags and sanitize prompts r=Mardak,ai-frontend-reviewers,Gijs

Comment 13

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/f4a225c064ee