625818 - Check Panorama mochitest test suite coverage

Status: RESOLVED FIXED

(Firefox Graveyard :: Panorama, defect, P3)

Description

[Michael Yoshitaka Erlewine [:mitcho]]

Previously Mardak noted that they did some instrumenting on Weave at some point to see exactly what lines of code (or maybe just functions) were hit during the test suite, in order to see what code paths are never touched. In light of my experience with bug 625443, I would like to do this with Panorama.

Mardak, could you point us to general resources on how best to do this?

Comment 1

[Ed Lee :Mardak]

philiKON wrote the tool to us the js debugger to track which lines of code were executed as part of running tests.

Comment 2

[Ian Gilman [:iangilman]]

This is a fine thing to do after b11 ships.

Comment 3

[Michael Yoshitaka Erlewine [:mitcho]]

Attachment: Mochitest js coverage patch

Panorama test coverage report is here, based on the code in this patch:

http://mitcho.com/code/panorama-coverage/

Note that, with the jsd interruptHook on, the test suite runs ridiculously slowly... on my machine, the mochitest tabview suite took three hours or so.

The good news: we have about 86% code coverage. Some things which we should write tests for and/or remove:
- gPrivateBrowsing getter
- iQClass_position
- Storage.wipe
- GroupItem_arrange_evaluateDropSpace
- setResizable with no value arg
- GroupItems_saveAll
- GroupItems_getBoundingBox
- GroupItems_removeAll
- GroupItems_getNextGroupItemTab case where there's no activeGroupItem
- GroupItems_moveTabToGroupItem with a groupItemId arg
- Item.dropOptions.out, TabItems.dropOptions.drop, TabItems.dropOptions.over, TabItems.dropOptions.out
- TabItem_forceCanvasSize, TabItem_unforceCanvasSize
- TabPriorityQueue_pop, TabPriorityQueue_peek
- Drag_drag where parent is expanded
- Create an orphan tab on double click (is this really not tested?)
- UI_isIdle
- UI_onTabSelect for apptab
- GroupItem's handleKeyDown handling for return and escape keys and UI__setTabViewFrameKeyHandlers's getClosestTabBy and various keypresses
- UI__saveAll
- TabHandlers.onOther
- Utils.assertThrow

I'm going to make a follow-up blog post on the methodology.

Comment 4

[Michael Yoshitaka Erlewine [:mitcho]]

For reference, much of the code included in the patch above is based on PhiliKON's work on bug 569228.

Comment 5

[Michael Yoshitaka Erlewine [:mitcho]]

And now, a blog post on the method so hopefully others can use this tool in the future:

http://mitcho.com/blog/projects/checking-mochitest-test-coverage/

I now deem this bug resolved! :)

Comment 6

[Cedric Vivier [:cedricv]]

Attachment: Add coverage reporting capability to mochitest. b=625818

Comment 7

[Cedric Vivier [:cedricv]]

Attachment: Additional coverage report tools. b=625818

Comment 8

[Cedric Vivier [:cedricv]]

Comment on attachment 521029 [details] [diff] [review]
Mochitest js coverage patch

New patches above obsolete this patch.
They are updated to apply against recent revisions, plus the following changes:

- reporting tools moved to testing/tools rather than tools/
- more 'noise' lines are ignored as they should
- improved output (now looks like http://neonux.com/StyleEditor/coverage/)
- optional $COVERAGE_OUTPUT (file to store coverage stats if not default $OBJDIR/coverage.json)
- optional $COVERAGE_PROJECT and $COVERAGE_VERSION to give more information on the report

Comment 9

[Cedric Vivier [:cedricv]]

Attachment: Additional coverage report tools. b=625818