2007611 - Add rendering-layer URL validation for AI Window chat messages

Status: RESOLVED FIXED

(Core :: Machine Learning: General, task, P1)

Description

[Randy Concepcion]

2026-01-29 Note: Approach changed from inference-layer to actor-layer validation based on review feedback. See comments for details.

---

Implement LLM output validation in MLEngineParent to detect and sanitize untrusted URLs before responses reach AI Window. This adds defense-in-depth beyond tool execution, protecting against prompt injection attacks that attempt to surface malicious URLs in model responses.

LLM output validation runs in the privileged parent process. AI Window never sees the raw unvalidated response and renders only the sanitized output returned by MLEngineParent.

Flow:

• validateRequest(sessionId, request) logs request context and passes through unchanged (no blocking for POC)
• validateResponse(sessionId, response) parses URLs from LLM output, validates them via SecurityOrchestrator.evaluate() (one call per response with all extracted URLs), sanitizes untrusted URLs, logs decisions, and returns the sanitized response
• AI Window receives and renders the sanitized output as-is

Scope:

• validateRequest(sessionId, request)

  • Log request metadata to SecurityLogger (audit trail)
  • Pass through unchanged (no blocking for POC)
  • Return ValidationResult

• validateResponse(sessionId, response)

  • Parse URLs from LLM response text via parseUrlsFromLLM()
  • Normalize URLs via SecurityUtils.normalizeUrl() before validation
  • Call SecurityOrchestrator.evaluate() once per response with all extracted URLs
  • Sanitize untrusted URLs based on the decision
  • Log transformation decisions via SecurityLogger
  • Return ValidationResult with sanitized response

• parseUrlsFromLLM(text)

  • New utility function to extract URLs from LLM response text. Returns extracted URLs with position info for transformation.

/**
 * Parses URLs from LLM response text.
 *
 * Extracts URLs from markdown, HTML, and raw URL patterns. This is used
 * by the security layer to validate URLs in LLM output before the
 * response reaches AI Window.
 *
 * Note: PageExtractor outputs content as markdown (e.g., [text](url),
 * ![alt](src)), which may be echoed by the LLM in responses. This
 * function parses those patterns back out for validation.
 *
 * @param {string} text - The LLM response text to parse
 */

• Patterns to extract:
  • Markdown links: [text](url)
  • Markdown images: ![alt](url)
  • HTML links: <a href="url">
  • HTML images: <img src="url">
  • Raw URLs: https://..., http://...

Policies:

Add policies for the inference.response phase:

• Validating http/https URLs against the session ledger
• Handling dangerous URL schemes (javascript:, data:, file://) - final list confirmed with Platform Security during review

For this phase, effect: "transform" returns a decision describing which URLs must be sanitized; MLEngineParent applies the actual string transformation before returning the response.

Transformation Rules:

• Trusted (in ledger): keep as-is
• Everything else (untrusted, dangerous scheme, malformed): convert to non-clickable plaintext

Example:

# Before (raw LLM output)
Check out [this article](https://trusted.com) and [this deal](https://evil.com)!

# After (sanitized)
Check out [this article](https://trusted.com) and this deal (https://evil.com)!

ValidationResult Type:

Stable return type for both validateRequest() and validateResponse() (addresses feedback from D271495):

/**
 * @typedef {object} ValidationResult
 * @property {object|null} data - The validated/transformed request or response
 * @property {number} errorCode - 0 on success, non-zero on error
 * @property {string} message - Error message; empty string on success
 */

Open Questions:

• Will AI Window auto-link raw URLs when markdown rendering is added? (Determines if raw URLs need transformation)
• Is wrapping untrusted raw URLs in backticks (`url`) the right transformation approach?
• Should we handle additional HTML tags (<iframe>, <video>, <script>, etc.) or defer to follow-up?

Out of Scope:

• Blocking entire responses or re-prompting the LLM
• Request-side blocking (request validation remains log-only)
• Context-aware extraction (e.g., skip URLs in code blocks)
• Additional URL schemes (mailto:, tel:) and HTML tags (<iframe>, <video>, etc.)

Dependencies:

• Phase 3: Centralized SecurityOrchestrator (D276793)

Related Bugs:

• Bug 2005406: sessionId/flowId alignment - determines how sessionId is passed to validation methods
• Bug 2005401: URL normalization security review - extracted URLs should use reviewed normalization logic

Acceptance Criteria:

• LLM responses containing untrusted or dangerous URLs are sanitized to non-clickable format before reaching AI Window
• parseUrlsFromLLM() extracts URLs from markdown, HTML, and raw patterns
• Policies added for URL validation and dangerous scheme handling (final details reviewed by Platform Security)
• All validation decisions logged via SecurityLogger and return a stable ValidationResult
• Tests:
  • xpcshell tests cover parseUrlsFromLLM() and validateResponse() behavior
  • At least one browser-chrome test exercises the end-to-end flow (Deferred to AI Window integration)

Comment 1

[Randy Concepcion]

Attachment: Bug 2007611 - Part 1: Add parseUrlsFromLLM for URL extraction from LLM output. r=#ai-ondevice-reviewers

Adds URL parsing utility for LLM output validation. Extracts URLs from:

• Markdown links: text
• Markdown images:
• HTML links/images: <a href>, <img src>
• Raw URLs: https://...

Uses overlap detection to prevent double-extraction. Trailing punctuation
is trimmed from raw URLs to support accurate replacement during sanitization.

Comment 2

[Randy Concepcion]

Attachment: Bug 2007611 - Part 2: Add transform effect for inference.response URL classification. r=#ai-ondevice-reviewers

Adds policy engine support for URL classification in LLM responses. Transform
policies classify URLs as trusted (in ledger) or untrusted (must be sanitized),
enabling the caller to apply appropriate sanitization without blocking the response.

Transform policies return allow when all URLs are trusted, or transform with
{trustedUrls, untrustedUrls} classification when any are untrusted. Part 3
will use this classification to sanitize untrusted URLs in MLEngineParent.

Comment 3

[Randy Concepcion]

Attachment: Bug 2007611 - Part 3: Add LLM output validation with URL sanitization in MLEngineParent. r=#ai-ondevice-reviewers

Integrates security layer validation into MLEngine for both streaming and
non-streaming inference responses. Extracts URLs from LLM output, validates
against session ledger via SecurityOrchestrator, and sanitizes untrusted URLs
to non-clickable plain text.

Introduces SecuritySanitizer module for URL sanitization (markdown links,
HTML anchors/images). Validation uses serialized promise chain for streaming
to preserve chunk ordering. Fails open for MVP - validation errors log but
don't block responses.

Comment 4

[Randy Concepcion]

Attachment: Bug 2007611 - Part 4: Add ValidationResult wrapper for LLM output validation. r=#ai-ondevice-reviewers

Wraps validation method returns in structured {data, errorCode, message} format
per reviewer feedback in D271495. Internal refactor only - external API unchanged.
Callers continue to receive the same response objects.

Error codes: SUCCESS (no changes), SANITIZED (URLs sanitized), INTERNAL_ERROR
(validation failed, fail-open applied).

Comment 5

[Randy Concepcion]

Attachment: Bug 2007611 - Add rendering-layer URL validation for AI Window chat messages. r=#ai-ondevice-reviewers,ngrato,Gijs

Implements URL security validation at the rendering layer for AI Window chat messages.
Anchors are disabled by default (fail-closed) and only re-enabled after validation
against SecurityOrchestrator's trusted URL ledger. This approach validates the actual
rendered DOM rather than raw LLM text, avoiding the "two parsers disagreement" conflict.

Comment 6

[Randy Concepcion]

Update (January 2026):

Changed approach from inference-layer validation to actor-layer (post-render) validation.

Reason: The original approach parsed URLs from raw LLM text before rendering. This created a "two-parser conflict" where the security layer's URL parser and the markdown renderer (ProseMirror) could disagree on what becomes a clickable link — a security gap.

New approach: Validate the rendered DOM instead of raw text. After ai-chat-message renders markdown, the AIChatContentChild actor extracts URLs from actual anchors, disables them (fail-closed), validates via parent process, and restores only trusted URLs. This ensures we validate exactly what the user can click.

See revision: https://phabricator.services.mozilla.com/D280039

Comment 7

[Randy Concepcion]

Based on feedback, we are moving to a push-based approach to alleviate issues from the pull-based approach.

---

Push-Based URL Validation Refactor

Refactoring URL validation from pull-based (child requests validation per-message)
to push-based (parent pushes the conversation's trusted URLs proactively).

Why: The pull-based approach had async delays before links became clickable
and per-message IPC overhead, e.g., N messages --> N validation round trips).

How:

• Parent actor tracks conversationId and subscribes to ledger changes
• When user @mentions a URL or conversation opens, trusted URLs are pushed to child
• Child stores URLs as a Set() for synchronous validation during render (fail-closed by default)
• Eliminates post-render link patching and per-message validation IPC

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/88a153a126a5
https://hg.mozilla.org/integration/autoland/rev/5abe830de19a
Add rendering-layer URL validation for AI Window chat messages. r=Gijs,ai-ondevice-reviewers,ai-frontend-reviewers

Comment 10

[rperta]

https://hg.mozilla.org/mozilla-central/rev/5abe830de19a