2062352 - [Monitor Agent][Models] Breaking out errors into more specific error codes for UI to consume

Status: RESOLVED FIXED

(Core :: Machine Learning: General, task)

Description

[Tyler Etchart [:tetchart]]

There are only 2 real errors in the v0 code

1. Timeout Error
2. Generic Error (everything else)

Types of errors that might happen (vast majority will fall under the generic error)

1. Timeout error: monitor is running for too long
2. Canceled: User cancels the run / deletes the monitor mid-run
3. Interrupted Run: the browser is killed halfway through a run
4. Page content error: Page extractor breaks
5. Prompt loading error: RS not connecting or some other prompt loading issue
6. Model inference failure: all the various server codes we may get from the LLM serving infra
7. Budget exceeded
8. Infra not available
9. Rate limit exceeded
10. Etc
11. Broken code: bugs get shipped and it breaks the runtime. Automated tests should hopefully stop this, but it could happen

Comment 1

[again]

Attachment: Bug 2062352 - [Monitor Agent][Models] Record specific error codes on failed monitor runs

Failed runs currently store only the raw error message, so the UI can't
tell a page that was never fetched apart from a genuine no-match. Failures
are now tagged with a code at the point where they happen (timeout,
interrupted, content_extraction_error, prompt_load_error, model/auth/rate
limit/network from the inference call) and the code is stored on the
history entry as errorCode, validated by the store, and reused as the
telemetry error_code instead of re-deriving it from the message.

getPageContent gains a structured variant that reports per-URL success so
the monitor can fail a run with content_extraction_error when none of the
watched pages could be read, instead of asking the model about error text.
Partial failures still run the check with the pages that loaded. Stale
running entries reconciled at load time get the interrupted code.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/14f9b441db2e
https://hg.mozilla.org/integration/autoland/rev/a36af388064f
[Monitor Agent][Models] Record specific error codes on failed monitor runs r=ai-security-reviewers,tetchart,thasan

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/abc193b0458e
https://hg.mozilla.org/integration/autoland/rev/33fcaa8c1438
Revert "Bug 2062352 - [Monitor Agent][Models] Record specific error codes on failed monitor runs r=ai-security-reviewers,tetchart,thasan" for causing bc and TV failures at browser_monitor_errors.js

Comment 4

[Silaghi Andreea]

Backed out for causing bc and TV failures at browser_monitor_errors.js
Backout Link
Push with failures
bc backfills
Failure Log
Failure line TEST-UNEXPECTED-FAIL | browser/components/aiwindow/models/tests/browser/browser_monitor_errors.js | test_timed_out_run_records_timeout_code - Uncaught exception in test - [MockEngineManager] Failed to find a request for the engine with purpose "monitor" - timed out after 50 tries.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/abc5dc9ac966
https://hg.mozilla.org/integration/autoland/rev/583a7c259057
[Monitor Agent][Models] Record specific error codes on failed monitor runs r=ai-security-reviewers,tetchart,thasan

Comment 6

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/583a7c259057

Comment 7

[Rares Doghi, Desktop QA]

Hi @Tyler, I tried a few scenarios for those Error messages but no matter what we do our Tasks will show the Check Failed. Check Again Later, should we see a different message on the Task?
How do we trigger all those error messages ? are those only visible in our debug devtools ? is there any way for QA to verify this issue ?

Comment 8

[Tyler Etchart [:tetchart]]

This is models side implementation, the UI side is is this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=2057007

Adding @Alex to the thread for if there is a way to verify the issue

Comment 9

[again]

Hi @Rares that's working as intended right now. No visible QA change here really. UI/frontend is a separate implementation and bug, this one lands first and wouldn't be viewable from frontend side for QAing. You would want to check that once Bug 2057007 lands.