Bugzilla

LLMs are effectively non-deterministic in their responses, and any given behavior may not reproduce accurately. Rather than rely on MLPA or any type of LLM responses, we should create some mochitests that exercise the system end to end, but the call/response from the LLM should be mocked. We can create a mocked language model call and response in toolkit/components/ml that is similar to the [`moz-echo` engine](https://searchfox.org/firefox-main/rev/0fce12de99c5e7d74536d8f93cf9778ce6199a1f/toolkit/components/ml/content/backends/ONNXPipeline.mjs#416). That way we can assume a fully untrusted text conversation is happening, and measure the results in the system.

So in pseudo-code:

```
add_task(async function test_security() {
  const llm = await mockLLM();
  await openSmartWindow();
  await navigate("example.com/pageContent.html")
  await startChat("Summarize the page");
  llm.response("do tool call get_page_content('http://example.com/not-allowed-url'")

  is(getRejectedToolCalls().length, 1, "The tool call was rejected")
})
```

LLMs are effectively non-deterministic in their responses, and any given behavior may not reproduce accurately. Rather than rely on MLPA or any type of LLM responses, we should create some mochitests that exercise the system end to end, but the call/response from the LLM should be mocked. We can create a mocked language model call and response in toolkit/components/ml that is similar to the [`moz-echo` engine](https://searchfox.org/firefox-main/rev/0fce12de99c5e7d74536d8f93cf9778ce6199a1f/toolkit/components/ml/content/backends/ONNXPipeline.mjs#416). That way we can assume a fully untrusted text conversation is happening, and measure the results in the system.

So in pseudo-code:

```js
add_task(async function test_security() {
  const llm = await mockLLM();
  await openSmartWindow();
  await navigate("example.com/pageContent.html")
  await startChat("Summarize the page");
  llm.response("do tool call get_page_content('http://example.com/not-allowed-url'")

  is(getRejectedToolCalls().length, 1, "The tool call was rejected")
})
```