No way to create a brasstacks report for a subset of a test run

RESOLVED WONTFIX

Status

Mozilla QA Graveyard
Mozmill Result Dashboard
RESOLVED WONTFIX
7 years ago
6 years ago

People

(Reporter: geo, Unassigned)

Tracking

Details

(Whiteboard: [testday-20110701])

(mozmill-tip)moco:functional geo$ mozmill -t testPopups/ --report="http://mozmill-archive.brasstacks.mozilla.com/db/" -b ~/auts/Nightly.app/
...
INFO Passed: 6
INFO Failed: 0
INFO Skipped: 0
Report document created at 'http://mozmill-archive.brasstacks.mozilla.com/db/3e673a01042add46ed7599eaba6a428e'

Navigated to mozmill-archive.brasstacks.mozilla.com and there's nothing under functional matching that view.

Was also seen by Aleksej executing the whole functional directory.

Since mozmill-the-CLI app supports reporting directly, the views should handle mozmill doing direct reports. Plus we need a way to run just a subset of tests and get a report displayed for debugging purposes, so it can't be dependent on the master scripts. 

Am I just not looking in the right place?

Comment 1

7 years ago
Compare these testrun_l10n reports:
mozmill -t: http://mozmill-crowd.brasstacks.mozilla.com/db/3e673a01042add46ed7599eaba6a518d
mozmill-env: http://mozmill-crowd.brasstacks.mozilla.com/db/3e673a01042add46ed7599eaba6a571a

The relevant difference seems to be the report_type in the beginning:
 -t: "report_type":"firefox-l10n"
env: "report_type":"mozmill-test"

I guess the site uses it as a guide on where to show the results: it shows data from reports with “firefox-l10n” under “L10n Tests”, but there is no view for data from reports with “mozmill-test”.
OS: Mac OS X → All
Hardware: x86 → All
whimboo explained the report process is dependent on our wrapper scripts, and feels the right answer is to improve them to handle subsets.

I have mixed feelings about the fact that we've made the reporting mechanism incompatible with vanilla Mozmill, but sounds like it's unavoidable for the moment. I'd like to explore being able to handle vanilla reports when we rewrite the dashboard.

In the meantime, I changed the summary to more generally describe the problem, and added bug 637227 as a dependency. 

I don't think it's a dupe, though; this is a process problem to be solved, rather than the specific implementation change to make to solve it.
Depends on: 637277
Summary: Results reported via mozmill --report="xxx" don't show up in the Brasstacks views → No way to create a brasstacks report for a subset of a test run
Erg, bug 637277 is the dependency. Got it right and the comment wrong.
Also we do not have atomic test names at the moment. With Mozmill you can run tests from within any folder. The parent folder name is not included in the report, so we can't assign those results to any other result with the expected atomic path.

I haven't remembered that fact when we talked about half an hour ago. Given all those reasons and our former decision to take that way, I will close it as WONTFIX.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → WONTFIX
(Assignee)

Updated

6 years ago
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.