1821194 - when wptserve cannot find files, do not print out Exception/Error, so treeherder doesn't parse the errors

Status: RESOLVED FIXED

(Testing :: General, task)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Currently treeherder parses >100K error lines/day on average. In reducing this, not only do we reduce the load on treeherder, but we also make it easier to read error summaries within treeherder.

here is an example:
https://treeherder.mozilla.org/logviewer?job_id=408328068&repo=try&lineNumber=52198:

[task 2023-03-08T22:10:27.237Z] 22:10:27  WARNING -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptserve/wptserve/handlers.py", line 322, in __call__
[task 2023-03-08T22:10:27.237Z] 22:10:27  WARNING -     self._load_file(request, response, func)
[task 2023-03-08T22:10:27.237Z] 22:10:27  WARNING -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptserve/wptserve/handlers.py", line 308, in _load_file
[task 2023-03-08T22:10:27.237Z] 22:10:27  WARNING -     return func(request, response, environ, path)
[task 2023-03-08T22:10:27.237Z] 22:10:27  WARNING -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptserve/wptserve/handlers.py", line 317, in func
[task 2023-03-08T22:10:27.237Z] 22:10:27  WARNING -     handler(request, response)
[task 2023-03-08T22:10:27.237Z] 22:10:27  WARNING -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptserve/wptserve/handlers.py", line 366, in __call__
[task 2023-03-08T22:10:27.237Z] 22:10:27  WARNING -     raise HTTPException(500, message=msg)
[task 2023-03-08T22:10:27.237Z] 22:10:27  WARNING - wptserve.utils.HTTPException: 500
[task 2023-03-08T22:10:27.237Z] 22:10:27  WARNING - 
[task 2023-03-08T22:10:27.238Z] 22:10:27  WARNING - wptserve Exception loading http://web-platform.test:8000/resource-timing/resources/fake_responses.py?url=importer_async.js: 500
[task 2023-03-08T22:10:27.239Z] 22:10:27     INFO - wptserve 500
[task 2023-03-08T22:10:27.275Z] 22:10:27     INFO - {'actions': [{'type': 'none', 'actions': [{'type': 'pause', 'duration': 16}, {'type': 'pause', 'duration': 16}], 'id': '12'}, {'type': 'key', 'actions': [{'type': 'keyDown', 'value': '\ue004'}, {'type': 'keyUp', 'value': '\ue004'}], 'id': '13'}]}
[task 2023-03-08T22:10:27.312Z] 22:10:27     INFO - .

There are two places to edit here:

• httpexception - should be change to HTTP 500 in handler: %s...
• check for if the file exists in handler.py. If it doesn't we are expecting to return javascript, so return b''

Ideally this will be solve to reduce these perma "failures" that are not affecting the results of the tests.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1821194 - reduce perma errors from wpt tasks. r=jgraham!

Comment 3

[Intermittent Failures Robot]

1 failures in 4161 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821194&startday=2023-03-20&endday=2023-03-26&tree=all

Comment 4

[James Graham [:jgraham]]

Attachment: Bug 1821194 - Allow suppressing tracebacks for handler errors,

When a wptserve HTTP handler throws an exception, wptserve by default
logs the traceback to stdout. This is a bit of a pain Gecko's CI
because we spot the traceback and add it to the log summary, which
causes people to believe that the errors they're seeing are important
to test failures, when usually they're coming from unrelated tests.

However suppressing these errors entirely is unhelpful when trying to
disgnose the server error itself.

To solve this we add a command line flag to wptrunner:
--suppress-handler-traceback that will be set in Gecko CI but not
elsewhere. This causes a brief error message to be logged, but not the
full traceback.

On the wptserve side, this is represented as a config option
logging.suppress_handler_traceback. For neatness, the existing
log_level option was moved under a logging sub-object.

In addition, for the default case we clean up the traceback a bit, so
that we get the traceback generated from the user code and omit the
traceback resulting from it being re-thrown as a HTTPException.

Comment 5

[James Graham [:jgraham]]

Attachment: Bug 1821194 - Allow suppressing tracebacks for handler errors,

When a wptserve HTTP handler throws an exception, wptserve by default
logs the traceback to stdout. This is a bit of a pain Gecko's CI
because we spot the traceback and add it to the log summary, which
causes people to believe that the errors they're seeing are important
to test failures, when usually they're coming from unrelated tests.

However suppressing these errors entirely is unhelpful when trying to
disgnose the server error itself.

To solve this we add a command line flag to wptrunner:
--suppress-handler-traceback that will be set in Gecko CI but not
elsewhere. This causes a brief error message to be logged, but not the
full traceback.

On the wptserve side, this is represented as a config option
logging.suppress_handler_traceback. For neatness, the existing
log_level option was moved under a logging sub-object.

In addition, for the default case we clean up the traceback a bit, so
that we get the traceback generated from the user code and omit the
traceback resulting from it being re-thrown as a HTTPException.

Comment 6

[James Graham [:jgraham]]

Attachment: Bug 1821194 - Allow suppressing tracebacks for handler errors,

When a wptserve HTTP handler throws an exception, wptserve by default
logs the traceback to stdout. This is a bit of a pain Gecko's CI
because we spot the traceback and add it to the log summary, which
causes people to believe that the errors they're seeing are important
to test failures, when usually they're coming from unrelated tests.

However suppressing these errors entirely is unhelpful when trying to
disgnose the server error itself.

To solve this we add a command line flag to wptrunner:
--suppress-handler-traceback that will be set in Gecko CI but not
elsewhere. This causes a brief error message to be logged, but not the
full traceback.

On the wptserve side, this is represented as a config option
logging.suppress_handler_traceback. For neatness, the existing
log_level option was moved under a logging sub-object.

In addition, for the default case we clean up the traceback a bit, so
that we get the traceback generated from the user code and omit the
traceback resulting from it being re-thrown as a HTTPException.

Comment 7

[James Graham [:jgraham]]

Attachment: Bug 1821194 - Improve error reporting from stash,

Comment 8

[James Graham [:jgraham]]

Attachment: Bug 1821194 - Fix pipe name in anonymous-iframe test,

Comment 9

[James Graham [:jgraham]]

Attachment: Bug 1821194 - Allow setting cookie value in CSP support file,

Some tests were assuming the value parameter could be used to set a stash key.

Comment 10

[James Graham [:jgraham]]

Attachment: Bug 1821194 - Log a warning if the client closed the connection during h2 setup

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

This good-first-bug hasn't had any activity for 2 months, it is automatically unassigned.
For more information, please visit BugBot documentation.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

:jgraham, there are 4 patches on this bug, 1 has landed and the other 3 haven't; is there another blocker? should those be abandoned?

Comment 13

[James Graham [:jgraham]]

They were all landed upstream; I've closed the phab revisions.

Comment 14

[Mathew Hodson]

It looks like this landed in bug 1830544 with https://hg.mozilla.org/mozilla-central/rev/1cb1d30fc14e

Comment 15

[Dianna Smith [:diannaS]]

Closing and marking as fixed per previous comments

Comment 16

[James Graham [:jgraham]]

Attachment: Bug 1821194 - Enable suppressing tracebacks for handler errors,

This enables the reduced traceback configuration in our CI, so that we
don't get errors that will be picked up by the log parser.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/01da2a1d9417
Enable suppressing tracebacks for handler errors, r=jmaher

Comment 18

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/01da2a1d9417