587115 - add script to publish code coverage report to csv

Status: RESOLVED FIXED

(Tamarin Graveyard :: Build Config, defect)

Description

[Dan Schaffer]

should add a script to the deep builders for code coverage to generate reports

we should focus on the uncovered functions by module
bullseye can output basic data with covfn --csv, we can process it in python to massage into a csv summary format a wiki chart and table can consume

Comment 1

[Dan Schaffer]

Attachment: python script to parse .cov into csv reports

Comment 2

[Dan Schaffer]

currently the module is just the directory.  we ignore pcre and zlib since they are 3rd party apis and we do not care about coverage of the api.  modules are currently: mmgc, vmpi, core, eval, extensions, nanojit, shell, vprof, platform.  we may want  to consolidate some smaller modules for example consolidate (vmpi, platform, vprof, and extensions)

Comment 3

[Brent Baker]

Attachment: combine platform coverage data

After processing coverage file locally, attempt to download all known platform coverage data. If downloading all files succeeds, then this is the last builder generating data, and we can combine all data into a unified coverage report.

I am opening up a new bug (#589168) to get rid of these duplicated scripts.

Comment 4

[Dan Schaffer]

Comment on attachment 467762 [details] [diff] [review]
combine platform coverage data

patch looks good.  little thing, comment typo: search for avaible
thinking of failure points, realistically one builder will always be slower but it may be possible theoretically for two builders to both find all 3 .cov files.  I think this situation is not very likely and if 2 still run it should still work.

Comment 5

[Brent Baker]

Comment on attachment 467762 [details] [diff] [review]
combine platform coverage data

Patch pushed as 5104:4c87f9f1ef87

Comment 6

[Brent Baker]

Attachment: post coverage data to server

- remove the old *.cov files from the latest directory
- post each platform cov file, name the file based on platform and TR rev
- post combined coverage file, name file based on TR rev

Comment 7

[Brent Baker]

Attachment: v2. post coverage data to server

fixes issue with combined coverage data file have a different relative path to the source from individual files. The generated coverage file needs to have the same relative depth to the source code.

Comment 8

[Brent Baker]

(In reply to comment #7)
> Created attachment 469095 [details] [diff] [review]
> v2. post coverage data to server
> 
There are 2 typos in the patch that I have fixed locally:
1) environment.sh -> scp_coverage should have ${scp_host} not ${scp_hos}
2) linux and windows coverage-process had a mistake in the echoed URL, dir should have been 'coverage' not 'coverge'

Comment 9

[Dan Schaffer]

Attachment: patch adding util-parsecodecoverage.py

Comment 10

[Dan Schaffer]

issues:
- the patch needs to be called from the coverage-process.sh scripts (lin,mac,win)
- also I kept the check for 'repo' to strip out from the csv since on 5011 and 5010 I saw the 'repo' in the path when running covfn

Comment 11

[Brent Baker]

Comment on attachment 469178 [details] [diff] [review]
patch adding util-parsecodecoverage.py

r+ with some minor requests

- include description for the '--incsvfile' option in the header with all of the other options
- should make a HUGE note that the cwd should be the root of the source tree in order for the 
- if the datadir does not exist it should be created, currently script will produce an error

Comment 12

[Dan Schaffer]

Attachment: v2 patch adding util-parsecodecoverage and hook into coverage-process.sh

minor updates to util-parsecodecoverage.py and hook into coverage-process.sh scripts to run parsecodecoverage script and upload results to server so wiki auto updates. 
- duplicated same code to (windows|mac|linux)-deep/scripts/coverage-process.sh at end of files
- tested locally ok

Comment 13

[Dan Schaffer]

Attachment: parse-codecoverage.py minor improvements

minor improvements to util-parse-codecoverage.py.  in codecoverage-summary.csv add the fn and bc overall coverage percent.  limit the number of historical builds to save (set to 20).

Comment 14

[Brent Baker]

Comment on attachment 469545 [details] [diff] [review]
parse-codecoverage.py minor improvements

just remove type from line 320 (duplicated token=line.split())

Comment 15

[Dan Schaffer]

pushed patches in:
5117:7a9bbfa644e1
5121:629c2b540011

remaining work for this bug is to generate and publish the html detail report

Comment 16

[Trevor Baker]

Marking fixed.