586792 - Automation scripts should create canonical test paths before sending data to brasstacks

Status: RESOLVED WONTFIX

(Mozilla QA Graveyard :: Mozmill Automation, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Right now we are sending full (absolute) paths in our reports up to brasstacks. That makes it hard to identify the same test modules. That said we should post-process the report data we get from Mozmill.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: WIP v1

This WIP introduces canonical path to our test results. It modifies the test path accordingly to the given root path (-t option) and the chosen test-run. It will not modify the "root_path" of the report.

Here some paths and the final results:

General tests
=============
c:\\test\\firefox\\nox\\mozmill-tests\\firefox\\testLayout\\testNavigateFTP.js
=> firefox/testLayout/testNavigateFTP.js

/test/nox/mozmill-tests/firefox/restartTests/testDefaultBookmarks/test1.js
=> firefox/restartTests/testDefaultBookmarks/test1.js

Addons tests
============
/test/firefox/nox/mozmill-tests/addons/[e-mail]/tests/testStart.js
=> addons/[e-mail]/tests/testStart.js

Update tests
============
/test/mozmill-tests/firefox/softwareUpdate/testDirectUpdate/test1.js
=> softwareUpdate/testDirectUpdate/test1.js

This patch has to work with Mozmill 1.4.1 and Mozmill 1.4.2. We have to keep the backward compatibility for now.

Geo, it would be great if you could test it with 1.4.1 at least on one platform and run tests with 1.4.2 on all platforms (win, linux, and mac). You can write the report to a file for Mozmill 1.4.2 (--report=file://filename) or send to brasstacks for Mozmill 1.4.1 (--report=http://brasstacks.mozilla.com/couchdb/mozmill). Sending to brasstacks for Mozmill 1.4.2 doesn't work at the moment due to bug 582979.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Geo, can you please make sure to check if this is what we want to have for our dashboard? I would like to push it as soon as possible. Thanks.

Comment 3

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Still in progress testing, but summarizing the IRC conversation we just had:

I raised a concern that putting this functionality in at the execution script level tightly couples our process to these execution scripts.  Alternate execution methods (for example, mozmill --report) won't properly massage the paths.

My suggestions was that if the process needs to be path-dependent, we handle this on the couchapp side as a data massaging task there instead.  That way, any reporter would be treated the same way.

Per Henrik, this code will basically work on the couchapp side too, so I'm continuing to test it out.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Another good point is that we are more flexible on the couchapp side. The only requirement we will have when following that way is, that we need an update design doc which handles those transformations on the server side.

Comment 5

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Paths look fine based on runs on Linux, Windows, and Mac.  I'd go with this code, but like I said, probably at the back end.

Ubuntu
--
http://brasstacks.mozilla.com/couchdb/mozmill-test/d37f8f06049accd405a3f2b55d8d003b
http://brasstacks.mozilla.com/couchdb/mozmill-test/d37f8f06049accd405a3f2b55d8d0573

WinXP
--
http://brasstacks.mozilla.com/couchdb/mozmill-test/d37f8f06049accd405a3f2b55d8ce571
http://brasstacks.mozilla.com/couchdb/mozmill-test/d37f8f06049accd405a3f2b55d8cf549

MacOS
--
http://brasstacks.mozilla.com/couchdb/mozmill-test/d37f8f06049accd405a3f2b55d839ff6
http://brasstacks.mozilla.com/couchdb/mozmill-test/d37f8f06049accd405a3f2b55d83b8e1

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks Geo. At the moment I'm tempted to move this code into the automation script itself until we have a working solution in the couchapp. That will make sure that all reports send via the script will have the correct path. Once bug 592652 has been fixed we can remove this code from the automation script.

Landed as:
http://hg.mozilla.org/qa/mozmill-automation/rev/c9d625e6961c

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 465839 [details] [diff] [review]
WIP v1

Ah, forgot the review. Andrew can you please check? I will still leave it in our repository because we need it starting from today with the upgrade to Mozmill 1.5.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Well, after talking with jchris from couchio and checking our current code of the dashboard we already convert the path on the couchdb side. Another result from the talk with jchris was that we shouldn't worry about the absolute paths when uploading the results. We can simply strip of unwanted parts of the path while creating the view or using the show function. That way we are more flexible with changes in the future. Patch upcoming...

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: backout patch

This will backout most of the latest changes except the little report_type fix.

Comment 10

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Comment on attachment 471978 [details] [diff] [review]
backout patch

Compared this patch to previous, questioned hskupin on any differences.  Looks great!

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Landed as with the removal of "import re":
http://hg.mozilla.org/qa/mozmill-automation/rev/21fccffb5256

Closing as wontfix in favor of the couchdb relaxation.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Move of Mozmill related project bugs to newly created components. You can
filter out those emails by using "Mozmill-Tests-to-MozillaQA" as criteria.