Closed Bug 2015448 Opened 9 months ago Closed 9 months ago

Marionette tests don't emit group_result, group_start, or group_end in structured logging

Categories

(Testing :: Marionette Client and Harness, enhancement, P3)

enhancement

Tracking

(firefox149 fixed)

RESOLVED FIXED
149 Branch
Tracking Status
firefox149 --- fixed

People

(Reporter: florian, Assigned: florian)

Details

(Whiteboard: [webdriver:m19][webdriver:external])

Attachments

(1 external file)

Marionette test jobs are missing group_result entries in their marionette_errorsummary.log files and group_start/group_end messages in their structured log output. This means manifest-level timing and status data isn't available for marionette tests, and resource usage profiling can't show markers for manifests. Additionally, fetch-manifest-data.js (being added in bug 2015029) doesn't even try to process marionette jobs.

There are three issues:

  1. Missing group_result entries in errorsummary: The marionette runner's StructuredTestResult (inherited from moztest) emits test events with IDs in the format "filepath TestClass.test_method" (via test.id()), but the error summary formatter's test_to_group mapping built during suite_start only has plain filepath keys. Since _clean_test_name only strips "(finished)" suffixes, the lookup always fails, group is always None, and self.groups is never populated from running test events. At suite_end, no group_result lines are emitted.

  2. Missing group_start/group_end messages in structured logging: The marionette runner never calls logger.group_start()/logger.group_end() around manifest execution, unlike mochitest and xpcshell. This prevents resource usage profiling from attributing consumption to specific manifests.

  3. fetch-manifest-data.js skips marionette jobs: getArtifactPrefix() doesn't recognize marionette job names, so it returns null and marionette jobs are silently skipped during runtime data collection.

Pass the test group (manifest name) through StructuredTestResult so that
test_start/test_end log events include the group field. Without this, the
errorsummary formatter cannot match test events to their groups because
marionette test IDs include "TestClass.test_method" after the filepath,
which doesn't match the filepath-only keys in test_to_group.

Also emit group_start/group_end around each manifest's tests in
run_test_set, and add "marionette" to the prefix list in
fetch-manifest-data.js so runtime collection includes marionette jobs.

Priority: -- → P3
Whiteboard: [webdriver:backlog]
Status: ASSIGNED → RESOLVED
Closed: 9 months ago
Resolution: --- → FIXED
Target Milestone: --- → 149 Branch
Whiteboard: [webdriver:backlog] → [webdriver:m19][webdriver:external]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: