2071250 - [wpt-sync] Sync PR 62616 - [AI WPTs] Anchor polarity in bounded JSON schema constraint tests

Status: RESOLVED FIXED

(Testing :: web-platform-tests, task, P4)

Description

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Sync web-platform-tests PR 62616 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/web-platform-tests/wpt/pull/62616
Details from upstream follow.

Isaac Ahouma <[e-mail]> wrote:

[AI WPTs] Anchor polarity in bounded JSON schema constraint tests

Clarify the rating scale in the prompt to '-1.0 (worst) to 1.0 (best)'
and '-10 (worst) to 10 (best)'. Without explicit polarity anchors,
conversational models default to a 1-5 star review scale (producing
values like 4.0 or 5.0) which violates the test assertions.

Bug: 547963762
Change-Id: Ia42150e591e04d3e8b914d0f608fb3ed32d70e58
Fixed: 547963762
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/8392837
Reviewed-by: Mike Wasserman \<[e-mail]>
Auto-Submit: Isaac Ahouma \<[e-mail]>
Commit-Queue: Mike Wasserman \<[e-mail]>
Cr-Commit-Position: refs/heads/main@{#1695794}

Comment 1

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Pushed to try (stability) https://treeherder.mozilla.org/#/jobs?repo=try&revision=57b8aace2f57b5747ecc86af720eeb07d7b3a0a7

Comment 2

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

CI Results

Ran 9 Firefox configurations based on mozilla-central, and Firefox, Chrome, and Safari on GitHub CI

Total 15 tests and 1 subtests

Status Summary

Firefox

OK : 2[GitHub] 15[Gecko-android-em-14-x86_64-debug-geckoview, Gecko-android-em-14-x86_64-lite-opt-geckoview, Gecko-android-em-14-x86_64-opt-geckoview, Gecko-linux2404-64-debug, Gecko-linux2404-64-opt, Gecko-windows11-32-25h2-debug, Gecko-windows11-32-25h2-opt, Gecko-windows11-64-25h2-debug, Gecko-windows11-64-25h2-opt]
FAIL : 2[GitHub] 15[Gecko-android-em-14-x86_64-debug-geckoview, Gecko-android-em-14-x86_64-lite-opt-geckoview, Gecko-android-em-14-x86_64-opt-geckoview, Gecko-linux2404-64-debug, Gecko-linux2404-64-opt, Gecko-windows11-32-25h2-debug, Gecko-windows11-32-25h2-opt, Gecko-windows11-64-25h2-debug, Gecko-windows11-64-25h2-opt]

Chrome

TIMEOUT: 4

Safari

OK : 2
FAIL : 2

Links

Gecko CI (Treeherder)
GitHub PR Head
GitHub PR Base

Details

New Tests That Don't Pass

• /ai/language-model/response-constraint/json-schema/array.tentative.https.window.html [wpt.fyi]
  • Prompt should work with an array json schema constraint.: FAIL
• /ai/language-model/response-constraint/json-schema/boolean.tentative.https.window.html [wpt.fyi]
  • Prompt should work with a boolean json schema constraint.: FAIL
• /ai/language-model/response-constraint/json-schema/circular-references-rejection.tentative.https.window.html [wpt.fyi]
  • Prompt should reject response schemas with circular references: FAIL
• /ai/language-model/response-constraint/json-schema/integer-bounded.tentative.https.window.html [wpt.fyi]
  • Prompt should work with a bounded integer json schema constraint.: FAIL (Chrome: TIMEOUT, Safari: FAIL)
• /ai/language-model/response-constraint/json-schema/integer.tentative.https.window.html [wpt.fyi]
  • Prompt should work with an integer json schema constraint.: FAIL
• /ai/language-model/response-constraint/json-schema/invalid-type-rejection.tentative.https.window.html [wpt.fyi]
  • Prompt should reject response schemas with invalid types: FAIL
• /ai/language-model/response-constraint/json-schema/null.tentative.https.window.html [wpt.fyi]
  • Prompt should work with a null json schema constraint.: FAIL
• /ai/language-model/response-constraint/json-schema/number-bounded.tentative.https.window.html [wpt.fyi]
  • Prompt should work with a bounded number json schema constraint.: FAIL (Chrome: TIMEOUT, Safari: FAIL)
• /ai/language-model/response-constraint/json-schema/number.tentative.https.window.html [wpt.fyi]
  • Prompt should work with a number json schema constraint.: FAIL
• /ai/language-model/response-constraint/json-schema/object.tentative.https.window.html [wpt.fyi]
  • Prompt should work with an object json schema constraint.: FAIL
• /ai/language-model/response-constraint/json-schema/prefix-bad.tentative.https.window.html [wpt.fyi]
  • Prompt should reject if the prefix deviates from the json schema constraint.: FAIL
• /ai/language-model/response-constraint/json-schema/prefix-good.tentative.https.window.html [wpt.fyi]
  • Prompt should work when a valid response json schema and matching prefix is provided.: FAIL
• /ai/language-model/response-constraint/json-schema/response-schema-omitted-from-input.tentative.https.window.html [wpt.fyi]
  • Prompt should omit response schema from input.: FAIL
• /ai/language-model/response-constraint/json-schema/string.tentative.https.window.html [wpt.fyi]
  • Prompt should work with a string json schema constraint.: FAIL
• /ai/language-model/response-constraint/json-schema/valid-schema-success.tentative.https.window.html [wpt.fyi]
  • Prompt should work when a valid response json schema is provided.: FAIL

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/775dfc7556ec
https://hg.mozilla.org/integration/autoland/rev/aeeda486f19a
[wpt PR 62616] - [AI WPTs] Anchor polarity in bounded JSON schema constraint tests, a=testonly

Comment 4

[Goloman Adrian[:agoloman]]

https://hg.mozilla.org/mozilla-central/rev/aeeda486f19a