2012135 - [wpt-sync] Sync PR 57312 - [ai LanguageModel] Make RegExp case insensitive in prompt assertions

Status: NEW

(Testing :: web-platform-tests, task, P4)

Description

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Sync web-platform-tests PR 57312 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/web-platform-tests/wpt/pull/57312
Details from upstream follow.

Thomas Steiner <[e-mail]> wrote:

[ai LanguageModel] Make RegExp case insensitive in prompt assertions

The model responds sometimes uppercase, sometimes lowercase

CC: @foolip

Comment 1

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Pushed to try (stability) https://treeherder.mozilla.org/#/jobs?repo=try&revision=94094a257d5f645dedd456c8de1cbe4eaf6c280d

Comment 2

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

CI Results

Ran 9 Firefox configurations based on mozilla-central, and Firefox, Chrome, and Safari on GitHub CI

Total 22 tests and 5 subtests

Status Summary

Firefox

OK : 1[GitHub] 21[Gecko-android-em-14-x86_64-debug-geckoview, Gecko-android-em-14-x86_64-lite-opt-geckoview, Gecko-android-em-14-x86_64-opt-geckoview, Gecko-linux2404-64-debug, Gecko-linux2404-64-opt, Gecko-windows11-32-24h2-debug, Gecko-windows11-32-24h2-opt, Gecko-windows11-64-24h2-debug, Gecko-windows11-64-24h2-opt]
FAIL : 6[GitHub] 74[Gecko-android-em-14-x86_64-debug-geckoview, Gecko-android-em-14-x86_64-lite-opt-geckoview, Gecko-android-em-14-x86_64-opt-geckoview, Gecko-linux2404-64-debug, Gecko-linux2404-64-opt, Gecko-windows11-32-24h2-debug, Gecko-windows11-32-24h2-opt, Gecko-windows11-64-24h2-debug, Gecko-windows11-64-24h2-opt]
TIMEOUT: 1
ERROR : 1
NOTRUN : 14

Chrome

TIMEOUT: 2
NOTRUN : 5

Safari

OK : 1
FAIL : 6

Links

Gecko CI (Treeherder)
GitHub PR Head
GitHub PR Base

Details

New Tests That Don't Pass

• /ai/language-model/language-model-abort.tentative.https.window.html [wpt.fyi]
  • Aborting LanguageModel.create().: FAIL
  • Aborting LanguageModel.clone().: FAIL
  • Aborting LanguageModel.prompt().: FAIL
  • Aborting LanguageModel.promptStreaming().: FAIL
• /ai/language-model/language-model-availability-available-multimodal.tentative.https.window.html [wpt.fyi]
  • LanguageModel.availability() is available with multimodal audio option: FAIL
  • LanguageModel.availability() is available with multimodal image option: FAIL
  • LanguageModel.availability() returns available with supported multimodal options: FAIL
  • LanguageModel.availability() returns unavailable with unsupported multimodal options: FAIL
• /ai/language-model/language-model-availability-available.tentative.https.window.html [wpt.fyi]
  • LanguageModel.availability() is available with no options: FAIL
  • LanguageModel.availability() returns available with supported options: FAIL
  • LanguageModel.availability() returns unavailable with unsupported options: FAIL
  • LanguageModel.availability() rejects with invalid options: FAIL
• /ai/language-model/language-model-availability.tentative.https.window.html [wpt.fyi]
  • LanguageModel.availability() is defined: FAIL
  • LanguageModel.availability() returns a valid value with no options: FAIL
  • LanguageModel.availability() rejects when given invalid language tags: FAIL
  • LanguageModel.availability() returns a valid value with plausible options: FAIL
• /ai/language-model/language-model-clone.tentative.https.window.html [wpt.fyi]
  • Language Model Clone: FAIL
• /ai/language-model/language-model-create-multimodal.tentative.https.window.html [wpt.fyi]
  • LanguageModel.create() succeeds with supported multimodal type and language options: FAIL
  • LanguageModel.create() succeeds with supported multimodal initialPrompts: FAIL
  • LanguageModel.create() fails with unsupported multimodal initialPrompts: FAIL
• /ai/language-model/language-model-create-user-activation.tentative.https.window.html [wpt.fyi]
  • Create requires sticky user activation when availability is "downloadable": FAIL
• /ai/language-model/language-model-create.tentative.https.window.html [wpt.fyi]: ERROR
  • Ensure sessions can be created: FAIL
  • LanguageModel.create() returns a valid object with default options: FAIL
  • LanguageModel.create() notifies its monitor on downloadprogress: FAIL
  • Progress events are not emitted after aborted.: TIMEOUT
  • Create with topK and temperature: NOTRUN
  • Create with only topK should fail: NOTRUN
  • Create with only temperature should fail: NOTRUN
  • Create with negative temperature should fail: NOTRUN
  • Create with zero topK should fail: NOTRUN
  • Create with negative topK should fail: NOTRUN
  • Create with fractional topK should be rounded down: NOTRUN
  • Create with initialPrompts: NOTRUN
  • Create with empty initialPrompts: NOTRUN
  • Create with initialPrompts without system role: NOTRUN
  • Create with system role not ordered first should fail: NOTRUN
  • Create multiple system role entries should fail: NOTRUN
  • LanguageModel.create() rejects when given invalid language tags: NOTRUN
  • LanguageModel.create() canonicalizes language tags: NOTRUN
• /ai/language-model/language-model-destroy.tentative.https.window.html [wpt.fyi]
  • Language Model Destroy: FAIL
• /ai/language-model/language-model-from-detached-iframe.tentative.https.window.html [wpt.fyi]
  • Language Model From Detached Iframe: FAIL
• /ai/language-model/language-model-iframe.tentative.https.html [wpt.fyi]
  • Throw a 'NotAllowedError' when creating LanguageModel within cross-origin iframe: FAIL
  • LanguageModel can be created within cross-origin iframe with permission policy: FAIL
  • LanguageModel can be used within same-origin iframe: FAIL
  • LanguageModel is unavailable within cross-origin iframe: FAIL
  • LanguageModel is available within cross-origin iframe with permission policy: FAIL
  • LanguageModel is available within same-origin iframe: FAIL
• /ai/language-model/language-model-measure-input-usage.tentative.https.window.html [wpt.fyi]
  • Language Model Measure Input Usage: FAIL
• /ai/language-model/language-model-params.tentative.https.window.html [wpt.fyi]
  • Language Model Params: FAIL
• /ai/language-model/language-model-prompt-context-destroyed.tentative.https.window.html [wpt.fyi]
  • Detaching iframe while running prompt() should not cause memory leak: FAIL
• /ai/language-model/language-model-prompt-gc.tentative.https.window.html [wpt.fyi]
  • Prompt API must continue even after GC has been performed.: FAIL
• /ai/language-model/language-model-prompt-monitor-callback-exception.tentative.https.window.html [wpt.fyi]
  • Language Model Prompt Monitor Callback Exception: FAIL
• /ai/language-model/language-model-prompt-multimodal.tentative.https.window.html [wpt.fyi]
  • Prompt with type:"text" and image content should reject: FAIL
  • Prompt with assistant role should reject with multimodal input: FAIL
  • Prompt image without image expectedInput: FAIL
  • Test Image initialPrompt: FAIL
  • Prompt with Blob image content: FAIL
  • Prompt with ImageBitmap image content: FAIL
  • Prompt with VideoFrame image content: FAIL
  • Prompt with OffscreenCanvas image content: FAIL
  • Prompt with ImageData image content: FAIL
  • Prompt with HTMLImageElement image content: FAIL
  • Prompt with HTMLCanvasElement image content: FAIL
  • Prompt with ArrayBuffer image content: FAIL
  • Prompt with ArrayBufferView image content: FAIL
  • Prompt with ArrayBufferView image content with an offset.: FAIL
  • Prompt with HTMLImageElement image content (with SVG): FAIL
  • Prompt with SVGImageElement image content: FAIL
  • Prompt with HTMLVideoElement image content: FAIL
  • Prompt audio without audio expectedInput: FAIL
  • Test Audio initialPrompt: FAIL
  • Prompt with Blob audio content: FAIL
  • Prompt audio with blob containing invalid audio data.: FAIL
  • Prompt with AudioBuffer: FAIL
  • Prompt with BufferSource - ArrayBuffer: FAIL
• /ai/language-model/language-model-prompt-streaming-gc.tentative.https.window.html [wpt.fyi]
  • Prompt Streaming API must continue even after GC has been performed.: FAIL
• /ai/language-model/language-model-prompt-streaming.tentative.https.window.html [wpt.fyi]
  • Language Model Prompt Streaming: FAIL
• /ai/language-model/language-model-prompt.tentative.https.window.html [wpt.fyi]
  • Simple LanguageModel.prompt() call: FAIL (Chrome: TIMEOUT, Safari: FAIL)
  • Check empty input: FAIL (Chrome: NOTRUN, Safari: FAIL)
  • Check Shorthand: FAIL (Chrome: NOTRUN, Safari: FAIL)
  • Test that initialPrompt counts towards session inputUsage: FAIL (Chrome: NOTRUN, Safari: FAIL)
  • The quotaoverflow event is fired when overall usage exceeds the quota: FAIL (Chrome: NOTRUN, Safari: FAIL)
  • Test that prompt input exeeding the total quota rejects: FAIL
  • Test that prompt input exceeding the total quota rejects: FAIL (Chrome: NOTRUN, Safari: FAIL)
• /ai/language-model/language-model-quota-exceeded.tentative.https.window.html [wpt.fyi]
  • QuotaExceededError is thrown when initial prompts are too large.: FAIL
• /ai/language-model/language-model-response-json-schema.tentative.https.window.html [wpt.fyi]
  • Prompt API should fail if an invalid response json schema is provided: FAIL
  • Prompt API should work when a valid response json schema is provided.: FAIL
  • Prompt API should work when a valid response json schema and model prefix is provided.: FAIL
  • Prompt API should omit response schema from input.: FAIL
  • Prompt API should work when a valid regex constraint is provided.: FAIL