668872 - No way to create a brasstacks report for a subset of a test run

Status: RESOLVED WONTFIX

(Mozilla QA Graveyard :: Mozmill Result Dashboard, defect)

Description

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

(mozmill-tip)moco:functional geo$ mozmill -t testPopups/ --report="http://mozmill-archive.brasstacks.mozilla.com/db/" -b ~/auts/Nightly.app/
...
INFO Passed: 6
INFO Failed: 0
INFO Skipped: 0
Report document created at 'http://mozmill-archive.brasstacks.mozilla.com/db/3e673a01042add46ed7599eaba6a428e'

Navigated to mozmill-archive.brasstacks.mozilla.com and there's nothing under functional matching that view.

Was also seen by Aleksej executing the whole functional directory.

Since mozmill-the-CLI app supports reporting directly, the views should handle mozmill doing direct reports. Plus we need a way to run just a subset of tests and get a report displayed for debugging purposes, so it can't be dependent on the master scripts. 

Am I just not looking in the right place?

Comment 1

[[:Aleksej]]

Compare these testrun_l10n reports:
mozmill -t: http://mozmill-crowd.brasstacks.mozilla.com/db/3e673a01042add46ed7599eaba6a518d
mozmill-env: http://mozmill-crowd.brasstacks.mozilla.com/db/3e673a01042add46ed7599eaba6a571a

The relevant difference seems to be the report_type in the beginning:
 -t: "report_type":"firefox-l10n"
env: "report_type":"mozmill-test"

I guess the site uses it as a guide on where to show the results: it shows data from reports with “firefox-l10n” under “L10n Tests”, but there is no view for data from reports with “mozmill-test”.

Comment 2

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

whimboo explained the report process is dependent on our wrapper scripts, and feels the right answer is to improve them to handle subsets.

I have mixed feelings about the fact that we've made the reporting mechanism incompatible with vanilla Mozmill, but sounds like it's unavoidable for the moment. I'd like to explore being able to handle vanilla reports when we rewrite the dashboard.

In the meantime, I changed the summary to more generally describe the problem, and added bug 637227 as a dependency. 

I don't think it's a dupe, though; this is a process problem to be solved, rather than the specific implementation change to make to solve it.

Comment 3

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Erg, bug 637277 is the dependency. Got it right and the comment wrong.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Also we do not have atomic test names at the moment. With Mozmill you can run tests from within any folder. The parent folder name is not included in the report, so we can't assign those results to any other result with the expected atomic path.

I haven't remembered that fact when we talked about half an hour ago. Given all those reasons and our former decision to take that way, I will close it as WONTFIX.